-----============= acceptance-small: conf-sanity ============----- Sun Mar 10 15:47:38 EDT 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg117-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg117-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg117-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg117-server oleg117-server: oleg117-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing load_modules_local oleg117-server: Loading modules from /home/green/git/lustre-release/lustre oleg117-server: detected 4 online CPUs by sysfs oleg117-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg117-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg117-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg117-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 15:49:00 (1710100140) start mds service on oleg117-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg117-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg117-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server PASS 21a (33s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 15:49:35 (1710100175) start ost1 service on oleg117-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg117-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg117-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop mds service on oleg117-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server PASS 21b (231s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 15:53:29 (1710100409) start mds service on oleg117-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg117-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Client mount with ost in logs, but none running start ost1 service on oleg117-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server mount lustre on /mnt/lustre..... Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg117-client.virtnet /mnt/lustre (opts:-f) PASS Client mount with a running ost start ost1 service on oleg117-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 4 sec oleg117-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800aa0aa800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800aa0aa800.ost_server_uuid in FULL state after 0 sec setup single mount lustre success PASS umount lustre on /mnt/lustre..... Stopping client oleg117-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server unloading modules on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg117-client: oleg117-client: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-client: ssh exited with exit code 2 PASS 22 (82s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg117-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg117-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg117-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2 oleg117-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg117-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 46a: handle ost additional - wide striped file ========================================================== 15:54:54 (1710100494) Testing with 2 OSTs Stopping clients: oleg117-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg117-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2 oleg117-server: oleg117-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing load_modules_local oleg117-server: Loading modules from /home/green/git/lustre-release/lustre oleg117-server: detected 4 online CPUs by sysfs oleg117-server: Force libcfs to create 2 CPU partitions oleg117-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg117-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg117-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg117-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg117-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server start mds service on oleg117-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg117-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg117-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec mount lustre on /mnt/lustre..... Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec oleg117-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800aa0ac000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800aa0ac000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre2..... Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre2 /mnt/lustre2 stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre2/widestripe lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 34 0x22 0x280000401 1 2 0x2 0x2c0000401 File: '/mnt/lustre/widestripe' Size: 3 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826934273 Links: 1 Access: (0674/-rw-rwxr--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-10 15:56:29.000000000 -0400 Modify: 2024-03-10 15:56:29.000000000 -0400 Change: 2024-03-10 15:56:29.000000000 -0400 Birth: - umount lustre on /mnt/lustre2..... Stopping client oleg117-client.virtnet /mnt/lustre2 (opts:) umount lustre on /mnt/lustre..... Stopping client oleg117-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost2 (opts:-f) on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server stop ost1 service on oleg117-server stop mds service on oleg117-server stop mds service on oleg117-server unloading modules on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing unload_modules_local modules unloaded. oleg117-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg117-server: Is the ldiskfs module available? oleg117-server: oleg117-server: tunefs.lustre FATAL: failed to write local files oleg117-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg117-client: oleg117-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg117-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg117-server: Is the ldiskfs module available? oleg117-server: oleg117-server: tunefs.lustre FATAL: failed to write local files oleg117-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg117-client: oleg117-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.117@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.117@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg117-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg117-server: Is the ldiskfs module available? oleg117-server: oleg117-server: tunefs.lustre FATAL: failed to write local files oleg117-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg117-client: oleg117-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.117@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.117@tcp sys.timeout=20 oleg117-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg117-server: Is the ldiskfs module available? oleg117-server: oleg117-server: tunefs.lustre FATAL: failed to write local files oleg117-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg117-client: oleg117-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.117@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.117@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg117-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg117-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2 oleg117-server: oleg117-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing load_modules_local oleg117-server: Loading modules from /home/green/git/lustre-release/lustre oleg117-server: detected 4 online CPUs by sysfs oleg117-server: Force libcfs to create 2 CPU partitions oleg117-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg117-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg117-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg117-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg117-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server PASS 46a (207s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50b: lazystatfs all servers down ===== 15:58:22 (1710100702) start mds service on oleg117-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg117-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg117-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800b2a77000.lazystatfs=1 stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server oleg117-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8800b2a77000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b2a77000.ost_server_uuid in DISCONN state after 0 sec 0 UP mgc MGC192.168.201.117@tcp 59945edf-aacf-4828-b61f-e9866b64965f 5 1 UP lov lustre-clilov-ffff8800b2a77000 ba6170e9-4c58-4daf-9e47-d823e3629b7c 4 2 UP lmv lustre-clilmv-ffff8800b2a77000 ba6170e9-4c58-4daf-9e47-d823e3629b7c 5 3 UP mdc lustre-MDT0000-mdc-ffff8800b2a77000 ba6170e9-4c58-4daf-9e47-d823e3629b7c 5 4 UP mdc lustre-MDT0001-mdc-ffff8800b2a77000 ba6170e9-4c58-4daf-9e47-d823e3629b7c 5 5 UP osc lustre-OST0000-osc-ffff8800b2a77000 ba6170e9-4c58-4daf-9e47-d823e3629b7c 5 OSCs should all be DISCONN multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.7338 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 1632 84960 2% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 95248 1508 85084 2% /mnt/lustre[MDT:1] filesystem_summary: 190496 3140 170044 2% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (523) - No such process umount lustre on /mnt/lustre..... Stopping client oleg117-client.virtnet /mnt/lustre (opts:) stop mds service on oleg117-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server PASS 50b (47s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50c: lazystatfs one server down ====== 15:59:11 (1710100751) start mds service on oleg117-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg117-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg117-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg117-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800aa1aa800.lazystatfs=1 stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server oleg117-server: oleg117-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec oleg117-server: oleg117-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.7338 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 1668 84924 2% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 95248 1532 85060 2% /mnt/lustre[MDT:1] lustre-OST0001_UUID 142216 1524 126692 2% /mnt/lustre[OST:1] filesystem_summary: 142216 1524 126692 2% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (3351) - No such process umount lustre on /mnt/lustre..... Stopping client oleg117-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg117-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.117@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.117@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.117@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.117@tcp sys.timeout=20 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.117@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.117@tcp sys.timeout=20 PASS 50c (69s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50e: normal statfs all servers down == 16:00:22 (1710100822) Stopping clients: oleg117-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg117-client.virtnet /mnt/lustre2 (opts:-f) oleg117-server: oleg117-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing load_modules_local oleg117-server: Loading modules from /home/green/git/lustre-release/lustre oleg117-server: detected 4 online CPUs by sysfs oleg117-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg117-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg117-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg117-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server start mds service on oleg117-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg117-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg117-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server oleg117-server: oleg117-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec oleg117-server: oleg117-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 mount lustre on /mnt/lustre..... Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800b6232000.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.7338 start ost1 service on oleg117-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg117-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server PASS 50e (167s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50f: normal statfs one server in down ========================================================== 16:03:12 (1710100992) start mds service on oleg117-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg117-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg117-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec start ost2 service on oleg117-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg117-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg117-server oleg117-server: oleg117-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec oleg117-server: oleg117-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800ab0bc000.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.7338 start ost2 service on oleg117-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg117-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg117-server umount lustre on /mnt/lustre..... Stopping client oleg117-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.117@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.117@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.117@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.117@tcp sys.timeout=20 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.117@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.117@tcp sys.timeout=20 PASS 50f (147s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 69: replace an OST with the same index ========================================================== 16:05:44 (1710101144) start mds service on oleg117-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg117-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg117-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49440 inodes available. Want 99872. rc=0 - open/close 963 (time 1710101190.74 total 10.01 last 96.16) - open/close 1876 (time 1710101200.74 total 20.02 last 91.27) - open/close 2609 (time 1710101210.74 total 30.02 last 73.26) - open/close 3349 (time 1710101220.75 total 40.02 last 73.99) - open/close 4033 (time 1710101230.75 total 50.03 last 68.38) - open/close 4613 (time 1710101240.76 total 60.04 last 57.95) - open/close 5190 (time 1710101250.77 total 70.05 last 57.64) - open/close 5760 (time 1710101260.77 total 80.05 last 56.98) - open/close 6342 (time 1710101270.79 total 90.06 last 58.13) - open/close 6629 (time 1710101287.45 total 106.73 last 17.22) - open/close 7210 (time 1710101297.46 total 116.73 last 58.05) - open/close 7705 (time 1710101307.48 total 126.76 last 49.38) - open/close 8166 (time 1710101317.50 total 136.78 last 46.01) - open/close 8611 (time 1710101327.51 total 146.79 last 44.44) - open/close 9092 (time 1710101337.51 total 156.79 last 48.10) - open/close 9527 (time 1710101347.52 total 166.80 last 43.46) - open/close 9952 (time 1710101357.52 total 176.80 last 42.49) total: 10000 open/close in 177.90 seconds: 56.21 ops/second - unlinked 0 (time 1710101360 ; total 0 ; last 0) total: 10000 unlinks in 119 seconds: 84.033615 unlinks/second - open/close 405 (time 1710101497.50 total 15.08 last 26.86) - open/close 1191 (time 1710101507.51 total 25.09 last 78.53) - open/close 2005 (time 1710101517.52 total 35.10 last 81.32) - open/close 2767 (time 1710101527.53 total 45.10 last 76.15) - open/close 3252 (time 1710101546.02 total 63.60 last 26.22) - open/close 3883 (time 1710101556.02 total 73.60 last 63.09) - open/close 4503 (time 1710101566.03 total 83.60 last 61.98) - open/close 5125 (time 1710101576.03 total 93.61 last 62.15) - open/close 5645 (time 1710101586.05 total 103.63 last 51.90) - open/close 6167 (time 1710101596.07 total 113.65 last 52.12) - open/close 6683 (time 1710101606.09 total 123.67 last 51.50) - open/close 7222 (time 1710101616.10 total 133.68 last 53.83) - open/close 7697 (time 1710101626.10 total 143.68 last 47.49) - open/close 8207 (time 1710101636.11 total 153.69 last 50.94) - open/close 8657 (time 1710101646.12 total 163.70 last 44.97) - open/close 9119 (time 1710101656.14 total 173.72 last 46.10) - open/close 9598 (time 1710101666.15 total 183.72 last 47.88) total: 10000 open/close in 192.79 seconds: 51.87 ops/second - unlinked 0 (time 1710101677 ; total 0 ; last 0) total: 10000 unlinks in 133 seconds: 75.187973 unlinks/second - open/close 792 (time 1710101823.21 total 10.02 last 79.06) - open/close 1211 (time 1710101839.73 total 26.54 last 25.36) - open/close 2054 (time 1710101849.73 total 36.55 last 84.27) - open/close 2809 (time 1710101859.76 total 46.57 last 75.32) - open/close 3472 (time 1710101869.76 total 56.57 last 66.27) - open/close 3904 (time 1710101887.51 total 74.32 last 24.35) - open/close 4611 (time 1710101897.52 total 84.33 last 70.62) - open/close 5225 (time 1710101907.53 total 94.34 last 61.33) - open/close 5763 (time 1710101917.53 total 104.34 last 53.78) - open/close 6310 (time 1710101927.55 total 114.36 last 54.61) - open/close 6554 (time 1710101944.02 total 130.83 last 14.81) - open/close 7118 (time 1710101954.03 total 140.84 last 56.37) - open/close 7663 (time 1710101964.04 total 150.85 last 54.44) - open/close 8131 (time 1710101974.04 total 160.85 last 46.79) - open/close 8613 (time 1710101984.08 total 170.89 last 48.01) - open/close 9126 (time 1710101994.08 total 180.89 last 51.28) - open/close 9575 (time 1710102004.10 total 190.91 last 44.85) total: 10000 open/close in 200.55 seconds: 49.86 ops/second - unlinked 0 (time 1710102015 ; total 0 ; last 0) total: 10000 unlinks in 134 seconds: 74.626869 unlinks/second - open/close 918 (time 1710102161.78 total 10.02 last 91.65) - open/close 1707 (time 1710102171.79 total 20.02 last 78.87) - open/close 2427 (time 1710102181.80 total 30.03 last 71.92) - open/close 3135 (time 1710102191.80 total 40.04 last 70.77) - open/close 3822 (time 1710102201.81 total 50.04 last 68.64) - open/close 4427 (time 1710102211.83 total 60.06 last 60.42) - open/close 4970 (time 1710102221.84 total 70.07 last 54.25) - open/close 5574 (time 1710102231.85 total 80.08 last 60.31) - open/close 6065 (time 1710102241.86 total 90.09 last 49.04) - open/close 6542 (time 1710102251.87 total 100.10 last 47.66) - open/close 7036 (time 1710102261.88 total 110.11 last 49.37) - open/close 7543 (time 1710102271.89 total 120.12 last 50.65) - open/close 8098 (time 1710102281.90 total 130.14 last 55.40) - open/close 8592 (time 1710102291.91 total 140.14 last 49.39) - open/close 9029 (time 1710102301.91 total 150.15 last 43.67) - open/close 9538 (time 1710102311.93 total 160.16 last 50.84) - open/close 9980 (time 1710102321.93 total 170.16 last 44.17) total: 10000 open/close in 170.64 seconds: 58.60 ops/second - unlinked 0 (time 1710102325 ; total 0 ; last 0) total: 10000 unlinks in 121 seconds: 82.644630 unlinks/second - open/close 821 (time 1710102458.67 total 10.01 last 81.98) - open/close 1104 (time 1710102472.96 total 24.31 last 19.80) - open/close 1809 (time 1710102482.97 total 34.31 last 70.47) - open/close 2613 (time 1710102492.99 total 44.33 last 80.23) - open/close 3266 (time 1710102502.99 total 54.34 last 65.25) - open/close 3883 (time 1710102513.00 total 64.34 last 61.68) - open/close 4514 (time 1710102523.03 total 74.37 last 62.92) - open/close 5103 (time 1710102543.47 total 94.81 last 28.82) - open/close 5627 (time 1710102553.47 total 104.82 last 52.36) - open/close 6165 (time 1710102563.52 total 114.87 last 53.55) - open/close 6489 (time 1710102580.76 total 132.10 last 18.80) - open/close 6998 (time 1710102590.78 total 142.12 last 50.82) - open/close 7481 (time 1710102600.79 total 152.13 last 48.24) - open/close 8001 (time 1710102610.79 total 162.13 last 51.99) - open/close 8457 (time 1710102620.84 total 172.19 last 45.36) - open/close 8534 (time 1710102633.48 total 184.83 last 6.09) - open/close 9030 (time 1710102643.51 total 194.85 last 49.49) - open/close 9457 (time 1710102653.51 total 204.85 last 42.69) - open/close 9874 (time 1710102663.51 total 214.85 last 41.69) total: 10000 open/close in 217.89 seconds: 45.89 ops/second - unlinked 0 (time 1710102669 ; total 0 ; last 0) total: 10000 unlinks in 111 seconds: 90.090088 unlinks/second - open/close 868 (time 1710102793.07 total 10.01 last 86.75) - open/close 1750 (time 1710102803.07 total 20.01 last 88.17) - open/close 2481 (time 1710102813.07 total 30.01 last 73.09) - open/close 3202 (time 1710102823.10 total 40.03 last 71.94) - open/close 3816 (time 1710102833.11 total 50.04 last 61.33) - open/close 4435 (time 1710102843.11 total 60.05 last 61.86) - open/close 4996 (time 1710102853.18 total 70.12 last 55.72) - open/close 5556 (time 1710102863.20 total 80.14 last 55.89) - open/close 6110 (time 1710102873.22 total 90.15 last 55.31) - open/close 6602 (time 1710102883.24 total 100.18 last 49.10) - open/close 7122 (time 1710102893.24 total 110.18 last 51.98) - open/close 7584 (time 1710102903.26 total 120.19 last 46.14) - open/close 8077 (time 1710102913.27 total 130.21 last 49.23) - open/close 8562 (time 1710102923.32 total 140.25 last 48.29) - open/close 9041 (time 1710102933.40 total 150.34 last 47.51) - open/close 9476 (time 1710102943.40 total 160.34 last 43.49) - open/close 9938 (time 1710102953.40 total 170.34 last 46.20) total: 10000 open/close in 171.63 seconds: 58.27 ops/second - unlinked 0 (time 1710102957 ; total 0 ; last 0) total: 10000 unlinks in 120 seconds: 83.333336 unlinks/second - open/close 929 (time 1710103090.40 total 10.07 last 92.30) - open/close 1740 (time 1710103100.41 total 20.07 last 81.04) - open/close 2227 (time 1710103117.30 total 36.96 last 28.83) - open/close 2970 (time 1710103127.31 total 46.97 last 74.24) - open/close 3624 (time 1710103137.32 total 56.99 last 65.31) - open/close 3856 (time 1710103152.62 total 72.28 last 15.17) - open/close 4572 (time 1710103162.63 total 82.29 last 71.54) - open/close 5163 (time 1710103172.64 total 92.30 last 59.02) - open/close 5847 (time 1710103182.69 total 102.35 last 68.09) - open/close 6452 (time 1710103192.70 total 112.36 last 60.43) - open/close 7049 (time 1710103202.71 total 122.38 last 59.60) - open/close 7574 (time 1710103212.72 total 132.39 last 52.45) - open/close 8144 (time 1710103222.73 total 142.39 last 57.00) - open/close 8619 (time 1710103232.77 total 152.43 last 47.27) - open/close 9067 (time 1710103242.80 total 162.46 last 44.71) - open/close 9590 (time 1710103252.81 total 172.47 last 52.23) total: 10000 open/close in 181.40 seconds: 55.13 ops/second - unlinked 0 (time 1710103263 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 1093 (time 1710103365.34 total 10.00 last 109.29) - open/close 2106 (time 1710103375.35 total 20.01 last 101.17) - open/close 3022 (time 1710103385.36 total 30.02 last 91.54) - open/close 3988 (time 1710103395.37 total 40.03 last 96.52) - open/close 4834 (time 1710103405.37 total 50.03 last 84.58) - open/close 5620 (time 1710103415.38 total 60.04 last 78.53) - open/close 6429 (time 1710103425.39 total 70.05 last 80.83) - open/close 7195 (time 1710103435.40 total 80.06 last 76.53) - open/close 7894 (time 1710103445.40 total 90.06 last 69.88) - open/close 8595 (time 1710103455.41 total 100.07 last 70.06) - open/close 9269 (time 1710103465.41 total 110.07 last 67.35) - open/close 9925 (time 1710103475.42 total 120.08 last 65.57) total: 10000 open/close in 121.29 seconds: 82.45 ops/second - unlinked 0 (time 1710103478 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 1439 (time 1710103556.22 total 10.00 last 143.86) - open/close 2813 (time 1710103566.22 total 20.00 last 137.39) - open/close 4156 (time 1710103576.22 total 30.01 last 134.21) - open/close 5447 (time 1710103586.23 total 40.02 last 129.02) - open/close 6681 (time 1710103596.23 total 50.02 last 123.35) - open/close 7798 (time 1710103606.24 total 60.03 last 111.59) - open/close 8948 (time 1710103616.25 total 70.04 last 114.91) total: 10000 open/close in 79.97 seconds: 125.04 ops/second - unlinked 0 (time 1710103627 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 2934 (time 1710103678.75 total 10.00 last 293.30) - open/close 5418 (time 1710103688.76 total 20.01 last 248.34) - open/close 7502 (time 1710103698.76 total 30.01 last 208.37) - open/close 9296 (time 1710103708.76 total 40.01 last 179.31) total: 10000 open/close in 43.97 seconds: 227.43 ops/second - unlinked 0 (time 1710103713 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg117-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.117@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg117-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre On OST0, 15229 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg117-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server unloading modules on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg117-client: oleg117-client: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-client: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2 PASS 69 (2627s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg117-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg117-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg117-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2 oleg117-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg117-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 93: register mulitple MDT at the same time ========================================================== 16:49:33 (1710103773) SKIP: conf-sanity test_93 needs >= 3 MDTs SKIP 93 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg117-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg117-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg117-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg117-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2 oleg117-server: oleg117-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing load_modules_local oleg117-server: Loading modules from /home/green/git/lustre-release/lustre oleg117-server: detected 4 online CPUs by sysfs oleg117-server: Force libcfs to create 2 CPU partitions oleg117-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg117-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3735 sec ========== 16:49:53 (1710103793) === conf-sanity: start cleanup 16:49:54 (1710103794) === === conf-sanity: finish cleanup 16:49:54 (1710103794) ===