-----============= acceptance-small: conf-sanity ============----- Sun Mar 10 15:43:28 EDT 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg126-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg126-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg126-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg126-server oleg126-server: oleg126-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg126-server' oleg126-server: oleg126-server.virtnet: executing load_modules_local oleg126-server: Loading modules from /home/green/git/lustre-release/lustre oleg126-server: detected 4 online CPUs by sysfs oleg126-server: Force libcfs to create 2 CPU partitions oleg126-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg126-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg126-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg126-server: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg126-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server stop mds service on oleg126-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg126-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 15:44:32 (1710099872) start mds service on oleg126-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg126-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg126-server: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg126-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server stop mds service on oleg126-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg126-server PASS 21a (33s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 15:45:10 (1710099910) start ost1 service on oleg126-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg126-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg126-server: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg126-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop mds service on oleg126-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg126-server stop ost1 service on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server PASS 21b (201s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 15:48:32 (1710100112) start mds service on oleg126-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Client mount with ost in logs, but none running start ost1 service on oleg126-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg126-server: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg126-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec stop ost1 service on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server mount lustre on /mnt/lustre..... Starting client: oleg126-client.virtnet: -o user_xattr,flock oleg126-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg126-client.virtnet /mnt/lustre (opts:-f) PASS Client mount with a running ost start ost1 service on oleg126-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg126-client.virtnet: -o user_xattr,flock oleg126-server@tcp:/lustre /mnt/lustre oleg126-server: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg126-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg126-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800af033800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800af033800.ost_server_uuid in FULL state after 0 sec setup single mount lustre success PASS umount lustre on /mnt/lustre..... Stopping client oleg126-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server stop mds service on oleg126-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg126-server unloading modules on: 'oleg126-server' oleg126-server: oleg126-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg126-client: oleg126-client: ssh exited with exit code 2 pdsh@oleg126-client: oleg126-server: ssh exited with exit code 2 pdsh@oleg126-client: oleg126-server: ssh exited with exit code 2 pdsh@oleg126-client: oleg126-client: ssh exited with exit code 2 PASS 22 (52s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg126-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg126-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg126-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg126-client: oleg126-server: ssh exited with exit code 2 oleg126-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg126-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 46a: handle ost additional - wide striped file ========================================================== 15:49:26 (1710100166) Testing with 2 OSTs Stopping clients: oleg126-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg126-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg126-client: oleg126-server: ssh exited with exit code 2 oleg126-server: oleg126-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg126-server' oleg126-server: oleg126-server.virtnet: executing load_modules_local oleg126-server: Loading modules from /home/green/git/lustre-release/lustre oleg126-server: detected 4 online CPUs by sysfs oleg126-server: Force libcfs to create 2 CPU partitions oleg126-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg126-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg126-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg126-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg126-server: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg126-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server stop mds service on oleg126-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg126-server start mds service on oleg126-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg126-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg126-server: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg126-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec mount lustre on /mnt/lustre..... Starting client: oleg126-client.virtnet: -o user_xattr,flock oleg126-server@tcp:/lustre /mnt/lustre Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg126-server: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg126-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg126-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800af033000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800af033000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre2..... Starting client: oleg126-client.virtnet: -o user_xattr,flock oleg126-server@tcp:/lustre /mnt/lustre2 /mnt/lustre2 stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre2/widestripe lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2 0x2 0x280000400 0 34 0x22 0x240000400 File: '/mnt/lustre/widestripe' Size: 3 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826934273 Links: 1 Access: (0674/-rw-rwxr--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-10 15:50:29.000000000 -0400 Modify: 2024-03-10 15:50:29.000000000 -0400 Change: 2024-03-10 15:50:29.000000000 -0400 Birth: - umount lustre on /mnt/lustre2..... Stopping client oleg126-client.virtnet /mnt/lustre2 (opts:) umount lustre on /mnt/lustre..... Stopping client oleg126-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost2 (opts:-f) on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server stop mds service on oleg126-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg126-server stop ost1 service on oleg126-server stop mds service on oleg126-server unloading modules on: 'oleg126-server' oleg126-server: oleg126-server.virtnet: executing unload_modules_local modules unloaded. oleg126-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg126-server: Is the ldiskfs module available? oleg126-server: oleg126-server: tunefs.lustre FATAL: failed to write local files oleg126-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg126-client: oleg126-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg126-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg126-server: Is the ldiskfs module available? oleg126-server: oleg126-server: tunefs.lustre FATAL: failed to write local files oleg126-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg126-client: oleg126-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.126@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.126@tcp sys.timeout=20 oleg126-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg126-server: Is the ldiskfs module available? oleg126-server: oleg126-server: tunefs.lustre FATAL: failed to write local files oleg126-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg126-client: oleg126-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.126@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.126@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg126-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg126-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg126-client: oleg126-server: ssh exited with exit code 2 oleg126-server: oleg126-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg126-server' oleg126-server: oleg126-server.virtnet: executing load_modules_local oleg126-server: Loading modules from /home/green/git/lustre-release/lustre oleg126-server: detected 4 online CPUs by sysfs oleg126-server: Force libcfs to create 2 CPU partitions oleg126-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg126-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg126-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg126-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg126-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg126-server: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg126-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server stop mds service on oleg126-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg126-server PASS 46a (125s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50b: lazystatfs all servers down ===== 15:51:32 (1710100292) start mds service on oleg126-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg126-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg126-client.virtnet: -o user_xattr,flock oleg126-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800a9b86800.lazystatfs=1 stop ost1 service on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server oleg126-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8800a9b86800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a9b86800.ost_server_uuid in DISCONN state after 0 sec 0 UP mgc MGC192.168.201.126@tcp f23c2378-b0af-4232-90f8-5d80304b5ee4 5 1 UP lov lustre-clilov-ffff8800a9b86800 2c90f24c-277b-43eb-8d84-bcb675738cf3 4 2 UP lmv lustre-clilmv-ffff8800a9b86800 2c90f24c-277b-43eb-8d84-bcb675738cf3 5 3 UP mdc lustre-MDT0000-mdc-ffff8800a9b86800 2c90f24c-277b-43eb-8d84-bcb675738cf3 5 4 UP osc lustre-OST0000-osc-ffff8800a9b86800 2c90f24c-277b-43eb-8d84-bcb675738cf3 5 OSCs should all be DISCONN multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.6620 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 1616 84976 2% /mnt/lustre[MDT:0] filesystem_summary: 95248 1616 84976 2% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (26697) - No such process umount lustre on /mnt/lustre..... Stopping client oleg126-client.virtnet /mnt/lustre (opts:) stop mds service on oleg126-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg126-server PASS 50b (41s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50c: lazystatfs one server down ====== 15:52:15 (1710100335) start mds service on oleg126-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg126-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg126-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg126-client.virtnet: -o user_xattr,flock oleg126-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800b6f3a000.lazystatfs=1 stop ost1 service on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server oleg126-server: oleg126-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg126-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.6620 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 1652 84940 2% /mnt/lustre[MDT:0] lustre-OST0001_UUID 142216 1388 126828 2% /mnt/lustre[OST:1] filesystem_summary: 142216 1388 126828 2% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (28937) - No such process umount lustre on /mnt/lustre..... Stopping client oleg126-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg126-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg126-server stop mds service on oleg126-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg126-server checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.126@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.126@tcp sys.timeout=20 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.126@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.126@tcp sys.timeout=20 PASS 50c (52s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50e: normal statfs all servers down == 15:53:10 (1710100390) Stopping clients: oleg126-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg126-client.virtnet /mnt/lustre2 (opts:-f) oleg126-server: oleg126-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg126-server' oleg126-server: oleg126-server.virtnet: executing load_modules_local oleg126-server: Loading modules from /home/green/git/lustre-release/lustre oleg126-server: detected 4 online CPUs by sysfs oleg126-server: Force libcfs to create 2 CPU partitions oleg126-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg126-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg126-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg126-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg126-server: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg126-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server stop mds service on oleg126-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg126-server start mds service on oleg126-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg126-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg126-server: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg126-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec stop ost1 service on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server oleg126-server: oleg126-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg126-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg126-client.virtnet: -o user_xattr,flock oleg126-server@tcp:/lustre /mnt/lustre llite.lustre-ffff88012b273000.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.6620 start ost1 service on oleg126-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg126-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server stop mds service on oleg126-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg126-server PASS 50e (105s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:54:57 (1710100497) start mds service on oleg126-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg126-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg126-server: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg126-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec start ost2 service on oleg126-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg126-server: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg126-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg126-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg126-server oleg126-server: oleg126-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg126-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg126-client.virtnet: -o user_xattr,flock oleg126-server@tcp:/lustre /mnt/lustre llite.lustre-ffff880130fa3800.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.6620 start ost2 service on oleg126-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0001 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg126-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg126-server umount lustre on /mnt/lustre..... Stopping client oleg126-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server stop mds service on oleg126-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg126-server checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.126@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.126@tcp sys.timeout=20 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.126@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.126@tcp sys.timeout=20 PASS 50f (98s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 69: replace an OST with the same index ========================================================== 15:56:36 (1710100596) start mds service on oleg126-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg126-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg126-client.virtnet: -o user_xattr,flock oleg126-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49602 inodes available. Want 99872. rc=0 - open/close 1837 (time 1710100619.52 total 10.00 last 183.66) - open/close 2833 (time 1710100629.52 total 20.01 last 99.56) - open/close 3732 (time 1710100639.53 total 30.01 last 89.85) - open/close 4549 (time 1710100649.54 total 40.02 last 81.65) - open/close 5483 (time 1710100659.54 total 50.02 last 93.38) - open/close 6945 (time 1710100669.54 total 60.02 last 146.12) - open/close 8545 (time 1710100679.54 total 70.03 last 159.98) - open/close 9982 (time 1710100689.55 total 80.03 last 143.66) total: 10000 open/close in 80.18 seconds: 124.72 ops/second - unlinked 0 (time 1710100690 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 3101 (time 1710100734.43 total 10.00 last 310.03) - open/close 5642 (time 1710100744.44 total 20.00 last 254.06) - open/close 6728 (time 1710100754.45 total 30.01 last 108.50) - open/close 7671 (time 1710100764.45 total 40.01 last 94.29) - open/close 8653 (time 1710100774.45 total 50.02 last 98.13) total: 10000 open/close in 59.40 seconds: 168.36 ops/second - unlinked 0 (time 1710100784 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1313 (time 1710100843.91 total 10.01 last 131.23) - open/close 2420 (time 1710100853.91 total 20.01 last 110.69) - open/close 3440 (time 1710100863.93 total 30.02 last 101.89) - open/close 4219 (time 1710100873.93 total 40.02 last 77.89) - open/close 5040 (time 1710100883.93 total 50.02 last 82.09) - open/close 5881 (time 1710100893.93 total 60.02 last 84.09) - open/close 6619 (time 1710100903.93 total 70.02 last 73.78) - open/close 7291 (time 1710100913.93 total 80.02 last 67.20) - open/close 7988 (time 1710100923.93 total 90.02 last 69.69) - open/close 8621 (time 1710100933.94 total 100.03 last 63.25) - open/close 9273 (time 1710100943.95 total 110.04 last 65.14) - open/close 9981 (time 1710100953.96 total 120.05 last 70.74) total: 10000 open/close in 120.32 seconds: 83.11 ops/second - unlinked 0 (time 1710100956 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 1106 (time 1710101052.94 total 10.00 last 110.57) - open/close 2171 (time 1710101062.95 total 20.01 last 106.43) - open/close 3179 (time 1710101072.96 total 30.03 last 100.63) - open/close 4023 (time 1710101082.97 total 40.03 last 84.34) - open/close 4865 (time 1710101092.98 total 50.04 last 84.16) - open/close 5696 (time 1710101102.99 total 60.05 last 83.03) - open/close 6392 (time 1710101112.99 total 70.05 last 69.55) - open/close 7011 (time 1710101123.00 total 80.06 last 61.85) - open/close 7549 (time 1710101133.01 total 90.07 last 53.76) - open/close 8058 (time 1710101143.08 total 100.14 last 50.54) - open/close 8627 (time 1710101153.09 total 110.15 last 56.83) - open/close 9154 (time 1710101163.11 total 120.17 last 52.62) - open/close 9657 (time 1710101173.11 total 130.17 last 50.28) total: 10000 open/close in 137.84 seconds: 72.55 ops/second - unlinked 0 (time 1710101183 ; total 0 ; last 0) total: 10000 unlinks in 115 seconds: 86.956520 unlinks/second - open/close 808 (time 1710101310.60 total 10.00 last 80.76) - open/close 1622 (time 1710101320.60 total 20.01 last 81.39) - open/close 2347 (time 1710101330.61 total 30.02 last 72.38) - open/close 3091 (time 1710101340.63 total 40.04 last 74.29) - open/close 3716 (time 1710101350.64 total 50.05 last 62.43) - open/close 4360 (time 1710101360.65 total 60.06 last 64.33) - open/close 4940 (time 1710101370.66 total 70.07 last 57.94) - open/close 5524 (time 1710101380.68 total 80.09 last 58.28) - open/close 6123 (time 1710101390.70 total 90.11 last 59.79) - open/close 6663 (time 1710101400.70 total 100.11 last 53.98) - open/close 7169 (time 1710101410.71 total 110.12 last 50.55) - open/close 7654 (time 1710101420.72 total 120.13 last 48.44) - open/close 8175 (time 1710101430.73 total 130.14 last 52.06) - open/close 8656 (time 1710101440.76 total 140.17 last 47.96) - open/close 9200 (time 1710101450.77 total 150.18 last 54.34) - open/close 9334 (time 1710101464.61 total 164.02 last 9.68) - open/close 9756 (time 1710101474.62 total 174.03 last 42.14) total: 10000 open/close in 179.46 seconds: 55.72 ops/second - unlinked 0 (time 1710101482 ; total 0 ; last 0) total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second - open/close 893 (time 1710101598.10 total 10.00 last 89.28) - open/close 1701 (time 1710101608.14 total 20.04 last 80.48) - open/close 2463 (time 1710101618.14 total 30.04 last 76.19) - open/close 3147 (time 1710101628.15 total 40.05 last 68.35) - open/close 3835 (time 1710101638.16 total 50.06 last 68.71) - open/close 4466 (time 1710101648.16 total 60.07 last 63.08) - open/close 4989 (time 1710101658.18 total 70.09 last 52.20) - open/close 5603 (time 1710101668.19 total 80.09 last 61.39) - open/close 6196 (time 1710101678.20 total 90.11 last 59.20) - open/close 6818 (time 1710101688.22 total 100.12 last 62.11) - open/close 7351 (time 1710101698.22 total 110.12 last 53.29) - open/close 7910 (time 1710101708.22 total 120.13 last 55.87) - open/close 8404 (time 1710101718.23 total 130.13 last 49.36) - open/close 8853 (time 1710101728.24 total 140.14 last 44.88) - open/close 9277 (time 1710101738.25 total 150.15 last 42.35) - open/close 9715 (time 1710101748.26 total 160.16 last 43.76) total: 10000 open/close in 166.42 seconds: 60.09 ops/second - unlinked 0 (time 1710101756 ; total 0 ; last 0) total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second - open/close 959 (time 1710101875.85 total 10.01 last 95.81) - open/close 1773 (time 1710101885.86 total 20.02 last 81.34) - open/close 2609 (time 1710101895.87 total 30.03 last 83.51) - open/close 3394 (time 1710101905.88 total 40.04 last 78.43) - open/close 3824 (time 1710101923.33 total 57.48 last 24.65) - open/close 4469 (time 1710101933.33 total 67.49 last 64.47) - open/close 5090 (time 1710101943.34 total 77.50 last 62.05) - open/close 5538 (time 1710101962.79 total 96.95 last 23.03) - open/close 6098 (time 1710101972.81 total 106.96 last 55.94) - open/close 6125 (time 1710101984.57 total 118.73 last 2.29) - open/close 6679 (time 1710101994.58 total 128.74 last 55.34) - open/close 7205 (time 1710102004.60 total 138.75 last 52.53) - open/close 7718 (time 1710102014.60 total 148.75 last 51.29) - open/close 8185 (time 1710102024.60 total 158.76 last 46.68) - open/close 8650 (time 1710102034.61 total 168.76 last 46.47) - open/close 9196 (time 1710102044.62 total 178.77 last 54.55) - open/close 9632 (time 1710102054.63 total 188.79 last 43.55) total: 10000 open/close in 197.45 seconds: 50.65 ops/second - unlinked 0 (time 1710102065 ; total 0 ; last 0) total: 10000 unlinks in 119 seconds: 84.033615 unlinks/second - open/close 950 (time 1710102197.40 total 10.00 last 94.97) - open/close 1767 (time 1710102207.40 total 20.01 last 81.62) - open/close 2567 (time 1710102217.41 total 30.02 last 79.96) - open/close 3261 (time 1710102227.42 total 40.02 last 69.35) - open/close 3909 (time 1710102237.43 total 50.03 last 64.74) - open/close 4549 (time 1710102247.43 total 60.03 last 63.99) - open/close 5156 (time 1710102257.45 total 70.05 last 60.59) - open/close 5749 (time 1710102267.45 total 80.06 last 59.25) - open/close 6299 (time 1710102277.50 total 90.11 last 54.75) - open/close 6861 (time 1710102287.51 total 100.12 last 56.13) - open/close 7431 (time 1710102297.52 total 110.13 last 56.96) - open/close 7961 (time 1710102307.53 total 120.14 last 52.94) - open/close 8475 (time 1710102317.55 total 130.16 last 51.29) - open/close 8928 (time 1710102327.56 total 140.17 last 45.25) - open/close 9366 (time 1710102337.57 total 150.18 last 43.79) - open/close 9872 (time 1710102347.57 total 160.18 last 50.58) total: 10000 open/close in 162.78 seconds: 61.43 ops/second - unlinked 0 (time 1710102352 ; total 0 ; last 0) total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second - open/close 877 (time 1710102472.06 total 10.03 last 87.41) - open/close 1704 (time 1710102482.06 total 20.04 last 82.66) - open/close 2489 (time 1710102492.06 total 30.04 last 78.46) - open/close 3176 (time 1710102502.07 total 40.05 last 68.66) - open/close 3832 (time 1710102512.07 total 50.05 last 65.57) - open/close 4473 (time 1710102522.08 total 60.06 last 64.04) - open/close 5152 (time 1710102532.09 total 70.07 last 67.84) - open/close 5735 (time 1710102542.10 total 80.08 last 58.27) - open/close 6269 (time 1710102552.10 total 90.08 last 53.38) - open/close 6854 (time 1710102562.11 total 100.09 last 58.45) - open/close 7353 (time 1710102572.11 total 110.09 last 49.88) - open/close 7930 (time 1710102582.12 total 120.10 last 57.64) - open/close 8393 (time 1710102592.13 total 130.10 last 46.29) - open/close 8875 (time 1710102602.15 total 140.13 last 48.06) - open/close 9312 (time 1710102612.18 total 150.16 last 43.57) - open/close 9706 (time 1710102622.20 total 160.18 last 39.34) total: 10000 open/close in 166.87 seconds: 59.93 ops/second - unlinked 0 (time 1710102631 ; total 0 ; last 0) total: 10000 unlinks in 124 seconds: 80.645164 unlinks/second - open/close 868 (time 1710102767.61 total 10.00 last 86.78) - open/close 1654 (time 1710102777.62 total 20.01 last 78.52) - open/close 2433 (time 1710102787.67 total 30.06 last 77.52) - open/close 3250 (time 1710102797.68 total 40.07 last 81.64) - open/close 3916 (time 1710102807.70 total 50.09 last 66.48) - open/close 4533 (time 1710102817.71 total 60.10 last 61.64) - open/close 5056 (time 1710102837.16 total 79.55 last 26.89) - open/close 5613 (time 1710102847.16 total 89.55 last 55.67) - open/close 6174 (time 1710102857.18 total 99.57 last 55.99) - open/close 6759 (time 1710102867.19 total 109.57 last 58.47) - open/close 7270 (time 1710102877.20 total 119.59 last 51.03) - open/close 7736 (time 1710102887.21 total 129.60 last 46.56) - open/close 8224 (time 1710102897.21 total 139.60 last 48.76) - open/close 8751 (time 1710102907.23 total 149.62 last 52.63) - open/close 9289 (time 1710102917.24 total 159.63 last 53.73) - open/close 9748 (time 1710102927.24 total 169.63 last 45.89) total: 10000 open/close in 175.68 seconds: 56.92 ops/second - unlinked 0 (time 1710102935 ; total 0 ; last 0) total: 10000 unlinks in 114 seconds: 87.719299 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg126-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.126@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg126-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg126-server: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg126-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec mount lustre on /mnt/lustre..... Starting client: oleg126-client.virtnet: -o user_xattr,flock oleg126-server@tcp:/lustre /mnt/lustre On OST0, 20235 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg126-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server stop mds service on oleg126-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg126-server unloading modules on: 'oleg126-server' oleg126-server: oleg126-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg126-client: oleg126-server: ssh exited with exit code 2 pdsh@oleg126-client: oleg126-client: ssh exited with exit code 2 pdsh@oleg126-client: oleg126-client: ssh exited with exit code 2 pdsh@oleg126-client: oleg126-server: ssh exited with exit code 2 PASS 69 (2505s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg126-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg126-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg126-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg126-client: oleg126-server: ssh exited with exit code 2 oleg126-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg126-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 93: register mulitple MDT at the same time ========================================================== 16:38:25 (1710103105) SKIP: conf-sanity test_93 needs >= 3 MDTs SKIP 93 (3s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg126-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg126-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg126-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg126-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg126-client: oleg126-server: ssh exited with exit code 2 oleg126-server: oleg126-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg126-server' oleg126-server: oleg126-server.virtnet: executing load_modules_local oleg126-server: Loading modules from /home/green/git/lustre-release/lustre oleg126-server: detected 4 online CPUs by sysfs oleg126-server: Force libcfs to create 2 CPU partitions oleg126-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg126-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3341 sec ========== 16:39:09 (1710103149) === conf-sanity: start cleanup 16:39:11 (1710103151) === === conf-sanity: finish cleanup 16:39:11 (1710103151) ===