-----============= acceptance-small: conf-sanity ============----- Sun Mar 10 15:46:59 EDT 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg314-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg314-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg314-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg314-server oleg314-server: oleg314-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing load_modules_local oleg314-server: Loading modules from /home/green/git/lustre-release/lustre oleg314-server: detected 4 online CPUs by sysfs oleg314-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg314-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg314-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg314-server: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg314-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 15:48:01 (1710100081) start mds service on oleg314-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg314-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg314-server: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg314-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server PASS 21a (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 15:48:24 (1710100104) start ost1 service on oleg314-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg314-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg314-server: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg314-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop mds service on oleg314-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server PASS 21b (197s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 15:51:43 (1710100303) start mds service on oleg314-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Client mount with ost in logs, but none running start ost1 service on oleg314-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg314-server: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg314-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server mount lustre on /mnt/lustre..... Starting client: oleg314-client.virtnet: -o user_xattr,flock oleg314-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg314-client.virtnet /mnt/lustre (opts:-f) PASS Client mount with a running ost start ost1 service on oleg314-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg314-client.virtnet: -o user_xattr,flock oleg314-server@tcp:/lustre /mnt/lustre oleg314-server: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg314-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg314-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800ab3fb800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800ab3fb800.ost_server_uuid in FULL state after 0 sec setup single mount lustre success PASS umount lustre on /mnt/lustre..... Stopping client oleg314-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server unloading modules on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg314-client: oleg314-client: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-server: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-client: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-server: ssh exited with exit code 2 PASS 22 (52s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg314-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg314-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg314-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg314-client: oleg314-server: ssh exited with exit code 2 oleg314-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg314-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 46a: handle ost additional - wide striped file ========================================================== 15:52:37 (1710100357) Testing with 2 OSTs Stopping clients: oleg314-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg314-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg314-client: oleg314-server: ssh exited with exit code 2 oleg314-server: oleg314-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing load_modules_local oleg314-server: Loading modules from /home/green/git/lustre-release/lustre oleg314-server: detected 4 online CPUs by sysfs oleg314-server: Force libcfs to create 2 CPU partitions oleg314-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg314-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg314-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg314-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg314-server: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg314-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server start mds service on oleg314-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg314-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg314-server: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg314-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec mount lustre on /mnt/lustre..... Starting client: oleg314-client.virtnet: -o user_xattr,flock oleg314-server@tcp:/lustre /mnt/lustre Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg314-server: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg314-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg314-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012a4c6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a4c6800.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre2..... Starting client: oleg314-client.virtnet: -o user_xattr,flock oleg314-server@tcp:/lustre /mnt/lustre2 /mnt/lustre2 stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre2/widestripe lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 34 0x22 0x240000400 1 2 0x2 0x280000400 File: '/mnt/lustre/widestripe' Size: 3 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826934273 Links: 1 Access: (0674/-rw-rwxr--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-10 15:53:34.000000000 -0400 Modify: 2024-03-10 15:53:34.000000000 -0400 Change: 2024-03-10 15:53:34.000000000 -0400 Birth: - umount lustre on /mnt/lustre2..... Stopping client oleg314-client.virtnet /mnt/lustre2 (opts:) umount lustre on /mnt/lustre..... Stopping client oleg314-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost2 (opts:-f) on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server stop ost1 service on oleg314-server stop mds service on oleg314-server unloading modules on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing unload_modules_local modules unloaded. oleg314-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg314-server: Is the ldiskfs module available? oleg314-server: oleg314-server: tunefs.lustre FATAL: failed to write local files oleg314-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg314-client: oleg314-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg314-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg314-server: Is the ldiskfs module available? oleg314-server: oleg314-server: tunefs.lustre FATAL: failed to write local files oleg314-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg314-client: oleg314-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.114@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.114@tcp sys.timeout=20 oleg314-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg314-server: Is the ldiskfs module available? oleg314-server: oleg314-server: tunefs.lustre FATAL: failed to write local files oleg314-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg314-client: oleg314-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.114@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.114@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg314-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg314-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg314-client: oleg314-server: ssh exited with exit code 2 oleg314-server: oleg314-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing load_modules_local oleg314-server: Loading modules from /home/green/git/lustre-release/lustre oleg314-server: detected 4 online CPUs by sysfs oleg314-server: Force libcfs to create 2 CPU partitions oleg314-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg314-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg314-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg314-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg314-server: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg314-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server PASS 46a (115s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50b: lazystatfs all servers down ===== 15:54:34 (1710100474) start mds service on oleg314-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg314-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg314-client.virtnet: -o user_xattr,flock oleg314-server@tcp:/lustre /mnt/lustre llite.lustre-ffff88012a4c1000.lazystatfs=1 stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server oleg314-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff88012a4c1000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a4c1000.ost_server_uuid in DISCONN state after 0 sec 0 UP mgc MGC192.168.203.114@tcp a35b620f-1439-4138-aa65-329c5d1dfa68 5 1 UP lov lustre-clilov-ffff88012a4c1000 0c181aca-0368-4566-89eb-f4e94eefa85a 4 2 UP lmv lustre-clilmv-ffff88012a4c1000 0c181aca-0368-4566-89eb-f4e94eefa85a 5 3 UP mdc lustre-MDT0000-mdc-ffff88012a4c1000 0c181aca-0368-4566-89eb-f4e94eefa85a 5 4 UP osc lustre-OST0000-osc-ffff88012a4c1000 0c181aca-0368-4566-89eb-f4e94eefa85a 5 OSCs should all be DISCONN multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.6628 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 1616 84976 2% /mnt/lustre[MDT:0] filesystem_summary: 95248 1616 84976 2% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (26683) - No such process umount lustre on /mnt/lustre..... Stopping client oleg314-client.virtnet /mnt/lustre (opts:) stop mds service on oleg314-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server PASS 50b (41s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50c: lazystatfs one server down ====== 15:55:16 (1710100516) start mds service on oleg314-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg314-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg314-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg314-client.virtnet: -o user_xattr,flock oleg314-server@tcp:/lustre /mnt/lustre llite.lustre-ffff88012b4b7000.lazystatfs=1 stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server oleg314-server: oleg314-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg314-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.6628 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 1652 84940 2% /mnt/lustre[MDT:0] lustre-OST0001_UUID 142216 1388 126828 2% /mnt/lustre[OST:1] filesystem_summary: 142216 1388 126828 2% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (28920) - No such process umount lustre on /mnt/lustre..... Stopping client oleg314-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg314-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.114@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.114@tcp sys.timeout=20 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.114@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.114@tcp sys.timeout=20 PASS 50c (42s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50e: normal statfs all servers down == 15:56:00 (1710100560) Stopping clients: oleg314-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg314-client.virtnet /mnt/lustre2 (opts:-f) oleg314-server: oleg314-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing load_modules_local oleg314-server: Loading modules from /home/green/git/lustre-release/lustre oleg314-server: detected 4 online CPUs by sysfs oleg314-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg314-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg314-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg314-server: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg314-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server start mds service on oleg314-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg314-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg314-server: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg314-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server oleg314-server: oleg314-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg314-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg314-client.virtnet: -o user_xattr,flock oleg314-server@tcp:/lustre /mnt/lustre llite.lustre-ffff88012aa9c000.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.6628 start ost1 service on oleg314-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg314-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server PASS 50e (89s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:57:31 (1710100651) start mds service on oleg314-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg314-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg314-server: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg314-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec start ost2 service on oleg314-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg314-server: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg314-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec stop ost2 service on oleg314-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg314-server oleg314-server: oleg314-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg314-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg314-client.virtnet: -o user_xattr,flock oleg314-server@tcp:/lustre /mnt/lustre llite.lustre-ffff88012b4b2800.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.6628 start ost2 service on oleg314-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0001 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg314-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg314-server umount lustre on /mnt/lustre..... Stopping client oleg314-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.114@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.114@tcp sys.timeout=20 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.114@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.114@tcp sys.timeout=20 PASS 50f (91s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 69: replace an OST with the same index ========================================================== 15:59:03 (1710100743) start mds service on oleg314-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg314-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg314-client.virtnet: -o user_xattr,flock oleg314-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49602 inodes available. Want 99872. rc=0 - open/close 1831 (time 1710100767.60 total 10.01 last 182.97) - open/close 3466 (time 1710100777.61 total 20.01 last 163.44) - open/close 5086 (time 1710100787.61 total 30.02 last 161.92) - open/close 6956 (time 1710100797.61 total 40.02 last 186.97) - open/close 8767 (time 1710100807.62 total 50.02 last 181.05) total: 10000 open/close in 58.21 seconds: 171.80 ops/second - unlinked 0 (time 1710100816 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1887 (time 1710100873.02 total 10.00 last 188.69) - open/close 3708 (time 1710100883.02 total 20.01 last 182.01) - open/close 5275 (time 1710100893.03 total 30.01 last 156.67) - open/close 6688 (time 1710100903.03 total 40.01 last 141.26) - open/close 8273 (time 1710100913.03 total 50.01 last 158.49) - open/close 9631 (time 1710100923.03 total 60.02 last 135.73) total: 10000 open/close in 63.53 seconds: 157.41 ops/second - unlinked 0 (time 1710100927 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1433 (time 1710100987.07 total 10.00 last 143.25) - open/close 2676 (time 1710100997.07 total 20.01 last 124.27) - open/close 4052 (time 1710101007.08 total 30.01 last 137.53) - open/close 5174 (time 1710101017.08 total 40.02 last 112.14) - open/close 6214 (time 1710101027.09 total 50.02 last 103.94) - open/close 7205 (time 1710101037.10 total 60.04 last 98.97) - open/close 8203 (time 1710101047.11 total 70.05 last 99.70) - open/close 9063 (time 1710101057.11 total 80.05 last 85.99) - open/close 9913 (time 1710101067.12 total 90.05 last 84.93) total: 10000 open/close in 91.09 seconds: 109.79 ops/second - unlinked 0 (time 1710101069 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 1317 (time 1710101146.63 total 10.01 last 131.62) - open/close 2582 (time 1710101156.63 total 20.01 last 126.49) - open/close 3695 (time 1710101166.63 total 30.01 last 111.23) - open/close 4729 (time 1710101176.64 total 40.02 last 103.37) - open/close 5674 (time 1710101186.64 total 50.02 last 94.47) - open/close 6642 (time 1710101196.65 total 60.03 last 96.73) - open/close 7495 (time 1710101206.66 total 70.04 last 85.22) - open/close 8420 (time 1710101216.67 total 80.04 last 92.42) - open/close 9300 (time 1710101226.68 total 90.06 last 87.88) total: 10000 open/close in 98.40 seconds: 101.62 ops/second - unlinked 0 (time 1710101236 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1410 (time 1710101309.90 total 10.00 last 140.95) - open/close 2689 (time 1710101319.91 total 20.01 last 127.83) - open/close 3875 (time 1710101329.91 total 30.01 last 118.56) - open/close 4882 (time 1710101339.92 total 40.02 last 100.63) - open/close 5931 (time 1710101349.92 total 50.02 last 104.87) - open/close 6878 (time 1710101359.92 total 60.02 last 94.69) - open/close 7789 (time 1710101369.92 total 70.03 last 91.08) - open/close 8682 (time 1710101379.93 total 80.03 last 89.25) - open/close 9535 (time 1710101389.93 total 90.03 last 85.30) total: 10000 open/close in 96.06 seconds: 104.10 ops/second - unlinked 0 (time 1710101397 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1383 (time 1710101471.01 total 10.00 last 138.25) - open/close 2596 (time 1710101481.03 total 20.02 last 121.13) - open/close 3701 (time 1710101491.03 total 30.02 last 110.50) - open/close 4781 (time 1710101501.03 total 40.02 last 107.97) - open/close 5781 (time 1710101511.03 total 50.02 last 99.97) - open/close 6717 (time 1710101521.04 total 60.02 last 93.59) - open/close 7667 (time 1710101531.04 total 70.03 last 94.94) - open/close 8539 (time 1710101541.05 total 80.04 last 87.13) - open/close 9312 (time 1710101551.05 total 90.04 last 77.27) total: 10000 open/close in 98.85 seconds: 101.16 ops/second - unlinked 0 (time 1710101561 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1428 (time 1710101634.86 total 10.00 last 142.78) - open/close 2693 (time 1710101644.86 total 20.01 last 126.41) - open/close 3793 (time 1710101654.87 total 30.01 last 109.97) - open/close 4836 (time 1710101664.87 total 40.02 last 104.22) - open/close 5070 (time 1710101678.16 total 53.30 last 17.61) - open/close 6057 (time 1710101688.17 total 63.31 last 98.61) - open/close 7013 (time 1710101698.17 total 73.32 last 95.57) - open/close 7899 (time 1710101708.18 total 83.32 last 88.54) - open/close 8788 (time 1710101718.18 total 93.32 last 88.87) - open/close 9667 (time 1710101728.18 total 103.33 last 87.89) total: 10000 open/close in 107.27 seconds: 93.22 ops/second - unlinked 0 (time 1710101733 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 1436 (time 1710101805.30 total 10.00 last 143.55) - open/close 2757 (time 1710101815.30 total 20.01 last 132.00) - open/close 3934 (time 1710101825.31 total 30.01 last 117.67) - open/close 5021 (time 1710101835.31 total 40.02 last 108.64) - open/close 6033 (time 1710101845.32 total 50.02 last 101.14) - open/close 7003 (time 1710101855.32 total 60.03 last 96.98) - open/close 7933 (time 1710101865.33 total 70.03 last 92.93) - open/close 8782 (time 1710101875.33 total 80.03 last 84.90) - open/close 9566 (time 1710101885.33 total 90.04 last 78.39) total: 10000 open/close in 95.45 seconds: 104.77 ops/second - unlinked 0 (time 1710101892 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 1432 (time 1710101966.76 total 10.01 last 143.09) - open/close 2718 (time 1710101976.77 total 20.01 last 128.54) - open/close 3886 (time 1710101986.77 total 30.01 last 116.78) - open/close 5038 (time 1710101996.78 total 40.03 last 115.06) - open/close 6108 (time 1710102006.78 total 50.03 last 106.98) - open/close 7128 (time 1710102016.79 total 60.04 last 101.88) - open/close 8054 (time 1710102026.79 total 70.04 last 92.59) - open/close 8965 (time 1710102036.80 total 80.05 last 91.05) - open/close 9808 (time 1710102046.80 total 90.05 last 84.25) total: 10000 open/close in 92.24 seconds: 108.41 ops/second - unlinked 0 (time 1710102050 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 1438 (time 1710102125.97 total 10.00 last 143.76) - open/close 2719 (time 1710102135.97 total 20.00 last 128.09) - open/close 3937 (time 1710102145.97 total 30.00 last 121.79) - open/close 5088 (time 1710102155.98 total 40.02 last 114.96) - open/close 6212 (time 1710102165.99 total 50.02 last 112.32) - open/close 7294 (time 1710102175.99 total 60.03 last 108.16) - open/close 8337 (time 1710102186.00 total 70.03 last 104.26) - open/close 9345 (time 1710102196.00 total 80.04 last 100.75) total: 10000 open/close in 87.11 seconds: 114.80 ops/second - unlinked 0 (time 1710102204 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg314-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.114@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg314-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg314-server: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg314-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 19 sec mount lustre on /mnt/lustre..... Starting client: oleg314-client.virtnet: -o user_xattr,flock oleg314-server@tcp:/lustre /mnt/lustre On OST0, 20039 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg314-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server unloading modules on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg314-client: oleg314-client: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-server: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-client: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-server: ssh exited with exit code 2 PASS 69 (1573s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg314-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg314-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg314-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg314-client: oleg314-server: ssh exited with exit code 2 oleg314-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg314-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 93: register mulitple MDT at the same time ========================================================== 16:25:18 (1710102318) SKIP: conf-sanity test_93 needs >= 3 MDTs SKIP 93 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg314-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg314-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg314-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg314-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg314-client: oleg314-server: ssh exited with exit code 2 oleg314-server: oleg314-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing load_modules_local oleg314-server: Loading modules from /home/green/git/lustre-release/lustre oleg314-server: detected 4 online CPUs by sysfs oleg314-server: Force libcfs to create 2 CPU partitions oleg314-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg314-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 2322 sec ========== 16:25:42 (1710102342) === conf-sanity: start cleanup 16:25:43 (1710102343) === === conf-sanity: finish cleanup 16:25:43 (1710102343) ===