-----============= acceptance-small: conf-sanity ============----- Sun Mar 10 15:45:41 EDT 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg107-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg107-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg107-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg107-server oleg107-server: oleg107-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing load_modules_local oleg107-server: Loading modules from /home/green/git/lustre-release/lustre oleg107-server: detected 4 online CPUs by sysfs oleg107-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg107-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg107-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg107-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg107-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 15:47:19 (1710100039) start mds service on oleg107-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg107-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg107-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg107-server PASS 21a (31s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 15:47:52 (1710100072) start ost1 service on oleg107-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg107-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg107-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg107-server stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server PASS 21b (216s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 15:51:30 (1710100290) start mds service on oleg107-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg107-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Client mount with ost in logs, but none running start ost1 service on oleg107-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server mount lustre on /mnt/lustre..... Starting client: oleg107-client.virtnet: -o user_xattr,flock oleg107-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg107-client.virtnet /mnt/lustre (opts:-f) PASS Client mount with a running ost start ost1 service on oleg107-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg107-client.virtnet: -o user_xattr,flock oleg107-server@tcp:/lustre /mnt/lustre oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec oleg107-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012eb16800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012eb16800.ost_server_uuid in FULL state after 0 sec setup single mount lustre success PASS umount lustre on /mnt/lustre..... Stopping client oleg107-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg107-server unloading modules on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg107-client: oleg107-client: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-client: ssh exited with exit code 2 PASS 22 (77s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg107-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg107-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg107-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2 oleg107-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg107-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 46a: handle ost additional - wide striped file ========================================================== 15:52:49 (1710100369) Testing with 2 OSTs Stopping clients: oleg107-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg107-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2 oleg107-server: oleg107-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing load_modules_local oleg107-server: Loading modules from /home/green/git/lustre-release/lustre oleg107-server: detected 4 online CPUs by sysfs oleg107-server: Force libcfs to create 2 CPU partitions oleg107-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg107-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg107-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg107-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg107-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg107-server start mds service on oleg107-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg107-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg107-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg107-client.virtnet: -o user_xattr,flock oleg107-server@tcp:/lustre /mnt/lustre Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec oleg107-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800af165800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800af165800.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre2..... Starting client: oleg107-client.virtnet: -o user_xattr,flock oleg107-server@tcp:/lustre /mnt/lustre2 /mnt/lustre2 stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre2/widestripe lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 34 0x22 0x280000401 1 2 0x2 0x2c0000401 File: '/mnt/lustre/widestripe' Size: 3 Blocks: 2 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826934273 Links: 1 Access: (0674/-rw-rwxr--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-10 15:54:41.000000000 -0400 Modify: 2024-03-10 15:54:41.000000000 -0400 Change: 2024-03-10 15:54:42.000000000 -0400 Birth: - umount lustre on /mnt/lustre2..... Stopping client oleg107-client.virtnet /mnt/lustre2 (opts:) umount lustre on /mnt/lustre..... Stopping client oleg107-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost2 (opts:-f) on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg107-server stop ost1 service on oleg107-server stop mds service on oleg107-server stop mds service on oleg107-server unloading modules on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing unload_modules_local modules unloaded. oleg107-server: oleg107-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg107-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg107-client: oleg107-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg107-server: oleg107-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg107-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg107-client: oleg107-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg107-server: oleg107-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg107-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg107-client: oleg107-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg107-server: oleg107-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg107-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg107-client: oleg107-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg107-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg107-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2 oleg107-server: oleg107-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing load_modules_local oleg107-server: Loading modules from /home/green/git/lustre-release/lustre oleg107-server: detected 4 online CPUs by sysfs oleg107-server: Force libcfs to create 2 CPU partitions oleg107-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg107-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg107-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg107-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg107-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg107-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg107-server PASS 46a (213s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50b: lazystatfs all servers down ===== 15:56:23 (1710100583) start mds service on oleg107-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg107-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg107-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg107-client.virtnet: -o user_xattr,flock oleg107-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8801373ad000.lazystatfs=1 stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server oleg107-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8801373ad000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8801373ad000.ost_server_uuid in DISCONN state after 0 sec 0 UP mgc MGC192.168.201.107@tcp 21862d13-84ba-491a-b704-61edffe254e8 5 1 UP lov lustre-clilov-ffff8801373ad000 c967ba20-32d3-4eca-968b-6493a3a467c1 4 2 UP lmv lustre-clilmv-ffff8801373ad000 c967ba20-32d3-4eca-968b-6493a3a467c1 5 3 UP mdc lustre-MDT0000-mdc-ffff8801373ad000 c967ba20-32d3-4eca-968b-6493a3a467c1 5 4 UP mdc lustre-MDT0001-mdc-ffff8801373ad000 c967ba20-32d3-4eca-968b-6493a3a467c1 5 5 UP osc lustre-OST0000-osc-ffff8801373ad000 c967ba20-32d3-4eca-968b-6493a3a467c1 5 OSCs should all be DISCONN multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.7461 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3200 378368 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 383616 3072 378496 1% /mnt/lustre[MDT:1] filesystem_summary: 767232 6272 756864 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (1653) - No such process umount lustre on /mnt/lustre..... Stopping client oleg107-client.virtnet /mnt/lustre (opts:) stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg107-server PASS 50b (51s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50c: lazystatfs one server down ====== 15:57:17 (1710100637) start mds service on oleg107-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg107-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg107-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg107-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg107-client.virtnet: -o user_xattr,flock oleg107-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800aa960000.lazystatfs=1 stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server oleg107-server: oleg107-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.7461 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3328 378240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 383616 3072 378496 1% /mnt/lustre[MDT:1] lustre-OST0001_UUID 382976 4096 376832 2% /mnt/lustre[OST:1] filesystem_summary: 382976 4096 376832 2% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (4451) - No such process umount lustre on /mnt/lustre..... Stopping client oleg107-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg107-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg107-server oleg107-server: oleg107-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg107-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg107-client: oleg107-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg107-server: oleg107-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg107-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg107-client: oleg107-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg107-server: oleg107-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg107-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg107-client: oleg107-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg107-server: oleg107-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg107-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg107-client: oleg107-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg107-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg107-client.virtnet /mnt/lustre2 (opts:-f) oleg107-server: oleg107-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing load_modules_local oleg107-server: Loading modules from /home/green/git/lustre-release/lustre oleg107-server: detected 4 online CPUs by sysfs oleg107-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg107-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg107-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg107-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg107-server PASS 50c (112s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50e: normal statfs all servers down == 15:59:11 (1710100751) Stopping clients: oleg107-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg107-client.virtnet /mnt/lustre2 (opts:-f) oleg107-server: oleg107-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing load_modules_local oleg107-server: Loading modules from /home/green/git/lustre-release/lustre oleg107-server: detected 4 online CPUs by sysfs oleg107-server: Force libcfs to create 2 CPU partitions oleg107-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg107-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg107-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg107-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg107-server start mds service on oleg107-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg107-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg107-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server oleg107-server: oleg107-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 mount lustre on /mnt/lustre..... Starting client: oleg107-client.virtnet: -o user_xattr,flock oleg107-server@tcp:/lustre /mnt/lustre llite.lustre-ffff88012aa3f800.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.7461 start ost1 service on oleg107-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg107-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg107-server PASS 50e (136s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50f: normal statfs one server in down ========================================================== 16:01:31 (1710100891) start mds service on oleg107-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg107-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg107-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec start ost2 service on oleg107-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg107-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg107-server oleg107-server: oleg107-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg107-client.virtnet: -o user_xattr,flock oleg107-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800aaa6e000.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.7461 start ost2 service on oleg107-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0001 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg107-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg107-server umount lustre on /mnt/lustre..... Stopping client oleg107-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg107-server oleg107-server: oleg107-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg107-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg107-client: oleg107-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg107-server: oleg107-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg107-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg107-client: oleg107-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg107-server: oleg107-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg107-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg107-client: oleg107-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg107-server: oleg107-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg107-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg107-client: oleg107-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg107-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg107-client.virtnet /mnt/lustre2 (opts:-f) oleg107-server: oleg107-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing load_modules_local oleg107-server: Loading modules from /home/green/git/lustre-release/lustre oleg107-server: detected 4 online CPUs by sysfs oleg107-server: Force libcfs to create 2 CPU partitions oleg107-server: libkmod: kmod_module_get_refcnt: could not read integer from '/sys/module/acpi_cpufreq/refcnt': 'No such device' oleg107-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg107-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg107-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg107-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg107-server PASS 50f (219s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 69: replace an OST with the same index ========================================================== 16:05:13 (1710101113) start mds service on oleg107-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg107-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg107-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg107-client.virtnet: -o user_xattr,flock oleg107-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11840 inodes available. Want 99936. rc=0 - open/close 244 (time 1710101170.65 total 13.37 last 18.25) - open/close 1012 (time 1710101180.66 total 23.38 last 76.69) - open/close 1792 (time 1710101190.68 total 33.40 last 77.88) - open/close 2527 (time 1710101200.69 total 43.41 last 73.43) - open/close 3179 (time 1710101210.69 total 53.41 last 65.18) - open/close 3877 (time 1710101220.69 total 63.41 last 69.79) - open/close 4452 (time 1710101230.71 total 73.43 last 57.42) - open/close 5011 (time 1710101240.72 total 83.45 last 55.80) - open/close 5503 (time 1710101250.84 total 93.56 last 48.65) - open/close 6038 (time 1710101260.85 total 103.57 last 53.44) - open/close 6511 (time 1710101270.86 total 113.58 last 47.25) - open/close 6991 (time 1710101280.87 total 123.59 last 47.95) - open/close 7547 (time 1710101290.87 total 133.59 last 55.59) - open/close 8004 (time 1710101300.88 total 143.60 last 45.66) - open/close 8457 (time 1710101310.89 total 153.61 last 45.26) - open/close 8867 (time 1710101320.89 total 163.61 last 40.97) - open/close 9238 (time 1710101341.32 total 184.04 last 18.16) - open/close 9629 (time 1710101351.34 total 194.06 last 39.03) total: 10000 open/close in 203.34 seconds: 49.18 ops/second - unlinked 0 (time 1710101363 ; total 0 ; last 0) total: 10000 unlinks in 120 seconds: 83.333336 unlinks/second - open/close 298 (time 1710101501.09 total 14.75 last 20.20) - open/close 909 (time 1710101519.94 total 33.60 last 32.41) - open/close 1683 (time 1710101529.95 total 43.62 last 77.29) - open/close 2356 (time 1710101539.97 total 53.63 last 67.22) - open/close 2547 (time 1710101553.71 total 67.37 last 13.90) - open/close 3252 (time 1710101563.72 total 77.38 last 70.45) - open/close 3864 (time 1710101573.72 total 87.38 last 61.18) - open/close 4440 (time 1710101583.79 total 97.45 last 57.20) - open/close 4971 (time 1710101593.79 total 107.46 last 53.08) - open/close 5529 (time 1710101603.80 total 117.47 last 55.73) - open/close 5623 (time 1710101616.88 total 130.54 last 7.19) - open/close 6109 (time 1710101626.88 total 140.55 last 48.59) - open/close 6609 (time 1710101636.89 total 150.56 last 49.94) - open/close 7081 (time 1710101646.94 total 160.60 last 46.99) - open/close 7517 (time 1710101656.95 total 170.62 last 43.55) - open/close 7971 (time 1710101666.96 total 180.62 last 45.37) - open/close 8411 (time 1710101676.99 total 190.65 last 43.86) - open/close 8893 (time 1710101687.01 total 200.67 last 48.11) - open/close 9047 (time 1710101701.27 total 214.94 last 10.79) - open/close 9539 (time 1710101711.30 total 224.97 last 49.06) - open/close 9940 (time 1710101721.32 total 234.98 last 40.03) total: 10000 open/close in 236.96 seconds: 42.20 ops/second - unlinked 0 (time 1710101726 ; total 0 ; last 0) total: 10000 unlinks in 132 seconds: 75.757576 unlinks/second - open/close 882 (time 1710101871.15 total 10.01 last 88.10) - open/close 1565 (time 1710101881.16 total 20.02 last 68.23) - open/close 2335 (time 1710101891.17 total 30.03 last 76.91) - open/close 3056 (time 1710101901.18 total 40.04 last 72.03) - open/close 3690 (time 1710101911.19 total 50.05 last 63.36) - open/close 4267 (time 1710101921.23 total 60.09 last 57.46) - open/close 4868 (time 1710101931.26 total 70.12 last 59.91) - open/close 5303 (time 1710101950.15 total 89.01 last 23.03) - open/close 5874 (time 1710101960.15 total 99.02 last 57.08) - open/close 6386 (time 1710101970.16 total 109.03 last 51.13) - open/close 6850 (time 1710101980.16 total 119.03 last 46.40) - open/close 7374 (time 1710101990.17 total 129.04 last 52.36) - open/close 7880 (time 1710102000.18 total 139.05 last 50.54) - open/close 8304 (time 1710102010.26 total 149.12 last 42.09) - open/close 8749 (time 1710102020.27 total 159.14 last 44.43) - open/close 8843 (time 1710102033.22 total 172.08 last 7.26) - open/close 9320 (time 1710102043.23 total 182.09 last 47.66) - open/close 9756 (time 1710102053.24 total 192.10 last 43.55) total: 10000 open/close in 198.15 seconds: 50.47 ops/second - unlinked 0 (time 1710102061 ; total 0 ; last 0) total: 10000 unlinks in 122 seconds: 81.967216 unlinks/second - open/close 841 (time 1710102196.66 total 10.01 last 84.06) - open/close 1561 (time 1710102206.68 total 20.03 last 71.84) - open/close 2198 (time 1710102216.69 total 30.03 last 63.67) - open/close 2867 (time 1710102226.69 total 40.04 last 66.89) - open/close 3508 (time 1710102236.69 total 50.04 last 64.09) - open/close 4060 (time 1710102246.69 total 60.04 last 55.19) - open/close 4572 (time 1710102256.69 total 70.04 last 51.20) - open/close 5142 (time 1710102266.71 total 80.06 last 56.89) - open/close 5679 (time 1710102276.72 total 90.07 last 53.66) - open/close 6204 (time 1710102286.73 total 100.08 last 52.42) - open/close 6751 (time 1710102296.76 total 110.11 last 54.56) - open/close 7298 (time 1710102306.79 total 120.13 last 54.55) - open/close 7841 (time 1710102316.81 total 130.15 last 54.19) - open/close 8320 (time 1710102326.83 total 140.18 last 47.77) - open/close 8746 (time 1710102336.86 total 150.20 last 42.50) - open/close 9192 (time 1710102346.86 total 160.21 last 44.60) - open/close 9702 (time 1710102356.87 total 170.22 last 50.95) total: 10000 open/close in 176.20 seconds: 56.75 ops/second - unlinked 0 (time 1710102365 ; total 0 ; last 0) total: 10000 unlinks in 119 seconds: 84.033615 unlinks/second - open/close 817 (time 1710102497.00 total 10.00 last 81.68) - open/close 1442 (time 1710102516.30 total 29.30 last 32.38) - open/close 2102 (time 1710102526.30 total 39.31 last 65.98) - open/close 2214 (time 1710102538.78 total 51.79 last 8.97) - open/close 2835 (time 1710102548.80 total 61.80 last 62.00) - open/close 3457 (time 1710102558.81 total 71.82 last 62.10) - open/close 4092 (time 1710102568.81 total 81.82 last 63.49) - open/close 4691 (time 1710102578.82 total 91.83 last 59.86) - open/close 5278 (time 1710102588.84 total 101.84 last 58.61) - open/close 5783 (time 1710102598.84 total 111.84 last 50.48) - open/close 6312 (time 1710102608.85 total 121.85 last 52.85) - open/close 6774 (time 1710102618.86 total 131.86 last 46.16) - open/close 7208 (time 1710102628.87 total 141.88 last 43.35) - open/close 7709 (time 1710102638.88 total 151.89 last 50.03) - open/close 8162 (time 1710102648.89 total 161.90 last 45.27) - open/close 8593 (time 1710102658.90 total 171.91 last 43.04) - open/close 9015 (time 1710102668.91 total 181.91 last 42.17) - open/close 9397 (time 1710102678.92 total 191.92 last 38.17) - open/close 9824 (time 1710102688.94 total 201.94 last 42.62) total: 10000 open/close in 206.04 seconds: 48.53 ops/second - unlinked 0 (time 1710102696 ; total 1 ; last 1) total: 10000 unlinks in 120 seconds: 83.333336 unlinks/second - open/close 812 (time 1710102828.19 total 10.00 last 81.19) - open/close 1616 (time 1710102838.20 total 20.01 last 80.32) - open/close 2250 (time 1710102848.20 total 30.01 last 63.39) - open/close 2938 (time 1710102858.21 total 40.02 last 68.78) - open/close 3594 (time 1710102868.22 total 50.03 last 65.53) - open/close 4138 (time 1710102878.23 total 60.04 last 54.33) - open/close 4544 (time 1710102897.19 total 79.00 last 21.41) - open/close 5132 (time 1710102907.20 total 89.01 last 58.72) - open/close 5702 (time 1710102917.21 total 99.02 last 56.96) - open/close 6207 (time 1710102927.21 total 109.02 last 50.48) - open/close 6747 (time 1710102937.23 total 119.04 last 53.93) - open/close 7203 (time 1710102947.23 total 129.04 last 45.58) - open/close 7647 (time 1710102957.24 total 139.05 last 44.34) - open/close 8057 (time 1710102967.30 total 149.11 last 40.76) - open/close 8083 (time 1710102978.91 total 160.72 last 2.24) - open/close 8518 (time 1710102988.95 total 170.76 last 43.34) - open/close 8914 (time 1710102998.98 total 180.80 last 39.46) - open/close 9348 (time 1710103008.99 total 190.81 last 43.36) - open/close 9741 (time 1710103019.01 total 200.82 last 39.26) total: 10000 open/close in 206.52 seconds: 48.42 ops/second - unlinked 0 (time 1710103027 ; total 0 ; last 0) total: 10000 unlinks in 138 seconds: 72.463768 unlinks/second - open/close 936 (time 1710103177.80 total 10.00 last 93.59) - open/close 959 (time 1710103189.06 total 21.27 last 2.04) - open/close 1781 (time 1710103199.07 total 31.28 last 82.10) - open/close 2550 (time 1710103209.09 total 41.29 last 76.80) - open/close 2765 (time 1710103223.17 total 55.38 last 15.26) - open/close 3437 (time 1710103233.18 total 65.39 last 67.14) - open/close 4071 (time 1710103243.19 total 75.39 last 63.35) - open/close 4792 (time 1710103253.19 total 85.40 last 72.09) - open/close 5404 (time 1710103263.19 total 95.40 last 61.19) - open/close 6074 (time 1710103273.20 total 105.40 last 66.96) - open/close 6416 (time 1710103289.32 total 121.53 last 21.21) - open/close 6993 (time 1710103299.33 total 131.53 last 57.70) - open/close 7592 (time 1710103309.33 total 141.54 last 59.86) - open/close 8228 (time 1710103319.33 total 151.54 last 63.60) - open/close 8814 (time 1710103329.34 total 161.55 last 58.53) - open/close 9383 (time 1710103339.36 total 171.57 last 56.78) - open/close 9947 (time 1710103349.39 total 181.59 last 56.27) total: 10000 open/close in 182.45 seconds: 54.81 ops/second - unlinked 0 (time 1710103352 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 1086 (time 1710103463.93 total 10.00 last 108.59) - open/close 2089 (time 1710103473.94 total 20.01 last 100.24) - open/close 3050 (time 1710103483.94 total 30.01 last 96.08) - open/close 4028 (time 1710103493.94 total 40.01 last 97.80) - open/close 5037 (time 1710103503.94 total 50.01 last 100.86) - open/close 5979 (time 1710103513.95 total 60.02 last 94.14) - open/close 6935 (time 1710103523.96 total 70.03 last 95.51) - open/close 7887 (time 1710103533.96 total 80.03 last 95.16) - open/close 8803 (time 1710103543.97 total 90.04 last 91.53) - open/close 9673 (time 1710103553.97 total 100.04 last 86.99) total: 10000 open/close in 103.84 seconds: 96.31 ops/second - unlinked 0 (time 1710103559 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 2136 (time 1710103621.79 total 10.00 last 213.53) - open/close 3928 (time 1710103631.80 total 20.01 last 179.16) - open/close 5499 (time 1710103641.80 total 30.01 last 157.06) - open/close 6984 (time 1710103651.80 total 40.01 last 148.47) - open/close 8354 (time 1710103661.80 total 50.01 last 136.98) - open/close 9740 (time 1710103671.81 total 60.01 last 138.55) total: 10000 open/close in 62.05 seconds: 161.15 ops/second - unlinked 0 (time 1710103674 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 3716 (time 1710103719.75 total 10.00 last 371.48) - open/close 6399 (time 1710103729.75 total 20.00 last 268.26) - open/close 8829 (time 1710103739.75 total 30.01 last 242.91) total: 10000 open/close in 34.81 seconds: 287.26 ops/second - unlinked 0 (time 1710103745 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg107-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.107@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.107@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg107-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 14 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg107-client.virtnet: -o user_xattr,flock oleg107-server@tcp:/lustre /mnt/lustre On OST0, 17847 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg107-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg107-server unloading modules on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg107-client: oleg107-client: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-client: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2 PASS 69 (2705s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg107-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg107-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg107-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2 oleg107-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg107-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 93: register mulitple MDT at the same time ========================================================== 16:50:20 (1710103820) SKIP: conf-sanity test_93 needs >= 3 MDTs SKIP 93 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg107-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg107-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg107-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg107-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2 oleg107-server: oleg107-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing load_modules_local oleg107-server: Loading modules from /home/green/git/lustre-release/lustre oleg107-server: detected 4 online CPUs by sysfs oleg107-server: Force libcfs to create 2 CPU partitions oleg107-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg107-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3908 sec ========== 16:50:50 (1710103850) === conf-sanity: start cleanup 16:50:50 (1710103850) === === conf-sanity: finish cleanup 16:50:50 (1710103850) ===