-----============= acceptance-small: conf-sanity ============----- Sun Mar 10 15:47:05 EDT 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg150-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg150-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg150-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg150-server oleg150-server: oleg150-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_refcnt: could not read integer from '/sys/module/pcc_cpufreq/refcnt': 'No such device' loading modules on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing load_modules_local oleg150-server: Loading modules from /home/green/git/lustre-release/lustre oleg150-server: detected 4 online CPUs by sysfs oleg150-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg150-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg150-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg150-server: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 15:48:06 (1710100086) start mds service on oleg150-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg150-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg150-server: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server PASS 21a (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 15:48:29 (1710100109) start ost1 service on oleg150-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg150-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg150-server: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop mds service on oleg150-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server PASS 21b (198s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 15:51:48 (1710100308) start mds service on oleg150-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Client mount with ost in logs, but none running start ost1 service on oleg150-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg150-server: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server mount lustre on /mnt/lustre..... Starting client: oleg150-client.virtnet: -o user_xattr,flock oleg150-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg150-client.virtnet /mnt/lustre (opts:-f) PASS Client mount with a running ost start ost1 service on oleg150-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg150-client.virtnet: -o user_xattr,flock oleg150-server@tcp:/lustre /mnt/lustre oleg150-server: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg150-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b6fd7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6fd7800.ost_server_uuid in FULL state after 0 sec setup single mount lustre success PASS umount lustre on /mnt/lustre..... Stopping client oleg150-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server unloading modules on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg150-client: oleg150-client: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-client: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2 PASS 22 (49s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg150-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg150-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg150-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2 oleg150-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg150-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 46a: handle ost additional - wide striped file ========================================================== 15:52:40 (1710100360) Testing with 2 OSTs Stopping clients: oleg150-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg150-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2 oleg150-server: oleg150-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing load_modules_local oleg150-server: Loading modules from /home/green/git/lustre-release/lustre oleg150-server: detected 4 online CPUs by sysfs oleg150-server: Force libcfs to create 2 CPU partitions oleg150-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg150-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg150-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg150-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg150-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg150-server: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server start mds service on oleg150-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg150-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg150-server: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg150-client.virtnet: -o user_xattr,flock oleg150-server@tcp:/lustre /mnt/lustre Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg150-server: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg150-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800aa941800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800aa941800.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre2..... Starting client: oleg150-client.virtnet: -o user_xattr,flock oleg150-server@tcp:/lustre /mnt/lustre2 /mnt/lustre2 stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre2/widestripe lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 34 0x22 0x240000400 1 2 0x2 0x280000400 File: '/mnt/lustre/widestripe' Size: 3 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826934273 Links: 1 Access: (0674/-rw-rwxr--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-10 15:54:04.000000000 -0400 Modify: 2024-03-10 15:54:04.000000000 -0400 Change: 2024-03-10 15:54:05.000000000 -0400 Birth: - umount lustre on /mnt/lustre2..... Stopping client oleg150-client.virtnet /mnt/lustre2 (opts:) umount lustre on /mnt/lustre..... Stopping client oleg150-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost2 (opts:-f) on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server stop ost1 service on oleg150-server stop mds service on oleg150-server unloading modules on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing unload_modules_local modules unloaded. oleg150-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg150-server: Is the ldiskfs module available? oleg150-server: oleg150-server: tunefs.lustre FATAL: failed to write local files oleg150-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg150-client: oleg150-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg150-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg150-server: Is the ldiskfs module available? oleg150-server: oleg150-server: tunefs.lustre FATAL: failed to write local files oleg150-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg150-client: oleg150-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.150@tcp sys.timeout=20 oleg150-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg150-server: Is the ldiskfs module available? oleg150-server: oleg150-server: tunefs.lustre FATAL: failed to write local files oleg150-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg150-client: oleg150-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.150@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg150-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg150-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2 oleg150-server: oleg150-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing load_modules_local oleg150-server: Loading modules from /home/green/git/lustre-release/lustre oleg150-server: detected 4 online CPUs by sysfs oleg150-server: Force libcfs to create 2 CPU partitions oleg150-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg150-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg150-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg150-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg150-server: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server PASS 46a (166s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50b: lazystatfs all servers down ===== 15:55:29 (1710100529) start mds service on oleg150-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg150-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg150-client.virtnet: -o user_xattr,flock oleg150-server@tcp:/lustre /mnt/lustre llite.lustre-ffff880136935000.lazystatfs=1 stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server oleg150-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff880136935000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff880136935000.ost_server_uuid in DISCONN state after 0 sec 0 UP mgc MGC192.168.201.150@tcp 039a5b36-8d8a-4d59-98e0-be73a1ad2c3e 5 1 UP lov lustre-clilov-ffff880136935000 8717b131-7b0f-4c03-8383-cf0a9c807b75 4 2 UP lmv lustre-clilmv-ffff880136935000 8717b131-7b0f-4c03-8383-cf0a9c807b75 5 3 UP mdc lustre-MDT0000-mdc-ffff880136935000 8717b131-7b0f-4c03-8383-cf0a9c807b75 5 4 UP osc lustre-OST0000-osc-ffff880136935000 8717b131-7b0f-4c03-8383-cf0a9c807b75 5 OSCs should all be DISCONN multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.6662 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 1616 84976 2% /mnt/lustre[MDT:0] filesystem_summary: 95248 1616 84976 2% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (26755) - No such process umount lustre on /mnt/lustre..... Stopping client oleg150-client.virtnet /mnt/lustre (opts:) stop mds service on oleg150-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server PASS 50b (44s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50c: lazystatfs one server down ====== 15:56:14 (1710100574) start mds service on oleg150-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg150-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg150-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg150-client.virtnet: -o user_xattr,flock oleg150-server@tcp:/lustre /mnt/lustre llite.lustre-ffff880130142000.lazystatfs=1 stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server oleg150-server: oleg150-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.6662 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 1652 84940 2% /mnt/lustre[MDT:0] lustre-OST0001_UUID 142216 1388 126828 2% /mnt/lustre[OST:1] filesystem_summary: 142216 1388 126828 2% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (28996) - No such process umount lustre on /mnt/lustre..... Stopping client oleg150-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg150-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.150@tcp sys.timeout=20 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.150@tcp sys.timeout=20 PASS 50c (48s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50e: normal statfs all servers down == 15:57:05 (1710100625) Stopping clients: oleg150-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg150-client.virtnet /mnt/lustre2 (opts:-f) oleg150-server: oleg150-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing load_modules_local oleg150-server: Loading modules from /home/green/git/lustre-release/lustre oleg150-server: detected 4 online CPUs by sysfs oleg150-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg150-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg150-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg150-server: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server start mds service on oleg150-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg150-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg150-server: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server oleg150-server: oleg150-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg150-client.virtnet: -o user_xattr,flock oleg150-server@tcp:/lustre /mnt/lustre llite.lustre-ffff880130140000.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.6662 start ost1 service on oleg150-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg150-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server PASS 50e (105s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:58:52 (1710100732) start mds service on oleg150-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg150-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg150-server: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec start ost2 service on oleg150-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg150-server: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg150-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg150-server oleg150-server: oleg150-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg150-client.virtnet: -o user_xattr,flock oleg150-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800ab23f000.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.6662 start ost2 service on oleg150-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0001 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg150-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg150-server umount lustre on /mnt/lustre..... Stopping client oleg150-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.150@tcp sys.timeout=20 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.150@tcp sys.timeout=20 PASS 50f (96s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 69: replace an OST with the same index ========================================================== 16:00:32 (1710100832) start mds service on oleg150-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg150-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg150-client.virtnet: -o user_xattr,flock oleg150-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49602 inodes available. Want 99872. rc=0 - open/close 1309 (time 1710100861.80 total 10.00 last 130.89) - open/close 2272 (time 1710100871.81 total 20.01 last 96.24) - open/close 3119 (time 1710100881.83 total 30.03 last 84.55) - open/close 4046 (time 1710100891.84 total 40.04 last 92.61) - open/close 4859 (time 1710100901.85 total 50.05 last 81.17) - open/close 5591 (time 1710100911.86 total 60.06 last 73.14) - open/close 6329 (time 1710100921.86 total 70.06 last 73.79) - open/close 7061 (time 1710100931.87 total 80.07 last 73.15) - open/close 7797 (time 1710100941.88 total 90.08 last 73.49) - open/close 8513 (time 1710100951.89 total 100.09 last 71.54) - open/close 9193 (time 1710100961.91 total 110.11 last 67.86) - open/close 9805 (time 1710100971.93 total 120.13 last 61.07) total: 10000 open/close in 123.18 seconds: 81.18 ops/second - unlinked 0 (time 1710100976 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 1176 (time 1710101073.49 total 10.01 last 117.53) - open/close 2118 (time 1710101083.49 total 20.01 last 94.19) - open/close 3068 (time 1710101093.49 total 30.01 last 94.97) - open/close 3984 (time 1710101103.50 total 40.02 last 91.55) - open/close 4802 (time 1710101113.50 total 50.02 last 81.78) - open/close 5535 (time 1710101123.51 total 60.03 last 73.25) - open/close 6105 (time 1710101133.51 total 70.03 last 56.98) - open/close 6682 (time 1710101143.52 total 80.04 last 57.66) - open/close 7292 (time 1710101153.53 total 90.05 last 60.93) - open/close 7889 (time 1710101163.54 total 100.06 last 59.63) - open/close 8448 (time 1710101173.56 total 110.08 last 55.78) - open/close 8992 (time 1710101183.59 total 120.11 last 54.26) - open/close 9544 (time 1710101193.60 total 130.11 last 55.16) total: 10000 open/close in 139.48 seconds: 71.69 ops/second - unlinked 0 (time 1710101205 ; total 0 ; last 0) total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second - open/close 828 (time 1710101325.68 total 10.05 last 82.39) - open/close 1665 (time 1710101335.69 total 20.06 last 83.65) - open/close 2421 (time 1710101345.69 total 30.06 last 75.56) - open/close 3096 (time 1710101355.71 total 40.08 last 67.38) - open/close 3776 (time 1710101365.72 total 50.09 last 67.95) - open/close 4407 (time 1710101375.72 total 60.09 last 63.08) - open/close 5044 (time 1710101385.72 total 70.09 last 63.67) - open/close 5658 (time 1710101395.72 total 80.09 last 61.40) - open/close 6222 (time 1710101405.73 total 90.10 last 56.38) - open/close 6800 (time 1710101415.73 total 100.10 last 57.80) - open/close 7321 (time 1710101425.74 total 110.11 last 52.04) - open/close 7868 (time 1710101435.76 total 120.13 last 54.61) - open/close 8385 (time 1710101445.77 total 130.14 last 51.61) - open/close 8872 (time 1710101455.78 total 140.15 last 48.66) - open/close 9307 (time 1710101465.79 total 150.15 last 43.48) - open/close 9720 (time 1710101475.81 total 160.18 last 41.21) total: 10000 open/close in 167.06 seconds: 59.86 ops/second - unlinked 0 (time 1710101485 ; total 0 ; last 0) total: 10000 unlinks in 115 seconds: 86.956520 unlinks/second - open/close 848 (time 1710101613.28 total 10.00 last 84.80) - open/close 1679 (time 1710101623.29 total 20.01 last 83.04) - open/close 2436 (time 1710101633.30 total 30.02 last 75.64) - open/close 3145 (time 1710101643.30 total 40.02 last 70.85) - open/close 3768 (time 1710101653.31 total 50.03 last 62.25) - open/close 4379 (time 1710101663.35 total 60.07 last 60.87) - open/close 4948 (time 1710101673.35 total 70.07 last 56.89) - open/close 5138 (time 1710101687.23 total 83.95 last 13.69) - open/close 5791 (time 1710101697.23 total 93.95 last 65.26) - open/close 6407 (time 1710101707.25 total 103.97 last 61.51) - open/close 6932 (time 1710101717.26 total 113.98 last 52.46) - open/close 7425 (time 1710101727.26 total 123.98 last 49.28) - open/close 7937 (time 1710101737.28 total 134.00 last 51.08) - open/close 8469 (time 1710101747.29 total 144.01 last 53.15) - open/close 8957 (time 1710101757.30 total 154.02 last 48.77) - open/close 9460 (time 1710101767.30 total 164.02 last 50.27) - open/close 9929 (time 1710101777.34 total 174.06 last 46.73) total: 10000 open/close in 175.39 seconds: 57.01 ops/second - unlinked 0 (time 1710101780 ; total 0 ; last 0) total: 10000 unlinks in 118 seconds: 84.745766 unlinks/second - open/close 852 (time 1710101921.39 total 20.72 last 41.12) - open/close 1668 (time 1710101931.39 total 30.72 last 81.57) - open/close 2410 (time 1710101941.41 total 40.74 last 74.09) - open/close 3170 (time 1710101951.42 total 50.75 last 75.93) - open/close 3670 (time 1710101969.22 total 68.55 last 28.08) - open/close 4347 (time 1710101979.23 total 78.56 last 67.66) - open/close 5019 (time 1710101989.24 total 88.57 last 67.15) - open/close 5687 (time 1710101999.24 total 98.57 last 66.76) - open/close 6276 (time 1710102009.25 total 108.59 last 58.83) - open/close 6821 (time 1710102019.26 total 118.59 last 54.47) - open/close 7311 (time 1710102029.26 total 128.59 last 48.99) - open/close 7856 (time 1710102039.27 total 138.60 last 54.47) - open/close 8477 (time 1710102049.27 total 148.60 last 62.09) - open/close 8983 (time 1710102059.27 total 158.61 last 50.57) - open/close 9535 (time 1710102069.29 total 168.62 last 55.13) total: 10000 open/close in 178.45 seconds: 56.04 ops/second - unlinked 0 (time 1710102081 ; total 0 ; last 0) total: 10000 unlinks in 128 seconds: 78.125000 unlinks/second - open/close 331 (time 1710102227.80 total 15.25 last 21.71) - open/close 1210 (time 1710102237.80 total 25.25 last 87.90) - open/close 1969 (time 1710102247.81 total 35.25 last 75.85) - open/close 2700 (time 1710102257.81 total 45.26 last 73.05) - open/close 3414 (time 1710102267.81 total 55.26 last 71.40) - open/close 3654 (time 1710102282.36 total 69.80 last 16.50) - open/close 4417 (time 1710102292.36 total 79.81 last 76.23) - open/close 5091 (time 1710102302.37 total 89.82 last 67.38) - open/close 5745 (time 1710102312.37 total 99.82 last 65.40) - open/close 6389 (time 1710102322.39 total 109.84 last 64.27) - open/close 6944 (time 1710102332.39 total 119.84 last 55.46) - open/close 7475 (time 1710102342.40 total 129.85 last 53.05) - open/close 8010 (time 1710102352.42 total 139.86 last 53.43) - open/close 8552 (time 1710102362.43 total 149.87 last 54.15) - open/close 9121 (time 1710102372.45 total 159.90 last 56.75) - open/close 9658 (time 1710102382.46 total 169.91 last 53.64) total: 10000 open/close in 177.84 seconds: 56.23 ops/second - unlinked 0 (time 1710102393 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 935 (time 1710102507.94 total 10.01 last 93.40) - open/close 1754 (time 1710102517.96 total 20.02 last 81.79) - open/close 2597 (time 1710102527.97 total 30.03 last 84.23) - open/close 3352 (time 1710102537.97 total 40.04 last 75.44) - open/close 4002 (time 1710102547.98 total 50.04 last 65.00) - open/close 4649 (time 1710102557.98 total 60.05 last 64.65) - open/close 5268 (time 1710102567.98 total 70.05 last 61.89) - open/close 5882 (time 1710102577.99 total 80.06 last 61.34) - open/close 6473 (time 1710102588.01 total 90.07 last 59.03) - open/close 7026 (time 1710102598.01 total 100.08 last 55.28) - open/close 7581 (time 1710102608.02 total 110.08 last 55.47) - open/close 8098 (time 1710102618.02 total 120.08 last 51.70) - open/close 8614 (time 1710102628.03 total 130.09 last 51.55) - open/close 9142 (time 1710102638.04 total 140.10 last 52.74) - open/close 9639 (time 1710102648.06 total 150.12 last 49.61) total: 10000 open/close in 158.29 seconds: 63.17 ops/second - unlinked 0 (time 1710102659 ; total 0 ; last 0) total: 10000 unlinks in 106 seconds: 94.339622 unlinks/second - open/close 557 (time 1710102785.61 total 17.42 last 31.98) - open/close 1436 (time 1710102795.62 total 27.42 last 87.85) - open/close 2211 (time 1710102805.63 total 37.44 last 77.38) - open/close 2587 (time 1710102821.56 total 53.37 last 23.60) - open/close 3315 (time 1710102831.58 total 63.39 last 72.68) - open/close 3956 (time 1710102841.59 total 73.40 last 64.04) - open/close 4573 (time 1710102851.60 total 83.40 last 61.67) - open/close 5166 (time 1710102861.60 total 93.40 last 59.28) - open/close 5758 (time 1710102871.61 total 103.41 last 59.15) - open/close 6303 (time 1710102881.61 total 113.42 last 54.47) - open/close 6840 (time 1710102891.62 total 123.42 last 53.68) - open/close 7352 (time 1710102901.62 total 133.43 last 51.17) - open/close 7852 (time 1710102911.64 total 143.45 last 49.91) - open/close 8399 (time 1710102921.65 total 153.45 last 54.67) - open/close 8894 (time 1710102931.65 total 163.45 last 49.48) - open/close 9392 (time 1710102941.65 total 173.46 last 49.80) - open/close 9867 (time 1710102951.67 total 183.47 last 47.43) total: 10000 open/close in 186.45 seconds: 53.63 ops/second - unlinked 0 (time 1710102957 ; total 0 ; last 0) total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second - open/close 977 (time 1710103074.70 total 10.01 last 97.63) - open/close 1778 (time 1710103084.70 total 20.01 last 80.10) - open/close 2625 (time 1710103094.71 total 30.02 last 84.63) - open/close 3398 (time 1710103104.71 total 40.02 last 77.29) - open/close 4135 (time 1710103114.74 total 50.04 last 73.51) - open/close 4791 (time 1710103124.74 total 60.05 last 65.59) - open/close 5424 (time 1710103134.74 total 70.05 last 63.27) - open/close 5984 (time 1710103144.76 total 80.07 last 55.91) - open/close 6564 (time 1710103154.76 total 90.07 last 57.98) - open/close 7138 (time 1710103164.76 total 100.07 last 57.39) - open/close 7645 (time 1710103174.77 total 110.08 last 50.68) - open/close 8277 (time 1710103184.78 total 120.09 last 63.11) - open/close 8845 (time 1710103194.78 total 130.09 last 56.80) - open/close 9033 (time 1710103209.58 total 144.88 last 12.71) - open/close 9125 (time 1710103222.23 total 157.54 last 7.27) - open/close 9595 (time 1710103232.24 total 167.55 last 46.98) total: 10000 open/close in 175.98 seconds: 56.82 ops/second - unlinked 0 (time 1710103242 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 1104 (time 1710103342.56 total 10.00 last 110.39) - open/close 2136 (time 1710103352.57 total 20.01 last 103.12) - open/close 3034 (time 1710103362.58 total 30.01 last 89.77) - open/close 3946 (time 1710103372.58 total 40.01 last 91.19) - open/close 4783 (time 1710103382.58 total 50.02 last 83.67) - open/close 5592 (time 1710103392.58 total 60.02 last 80.87) - open/close 6379 (time 1710103402.59 total 70.03 last 78.62) - open/close 7058 (time 1710103412.60 total 80.04 last 67.84) - open/close 7822 (time 1710103422.61 total 90.04 last 76.38) - open/close 8514 (time 1710103432.61 total 100.04 last 69.18) - open/close 9238 (time 1710103442.61 total 110.05 last 72.35) - open/close 9921 (time 1710103452.63 total 120.06 last 68.22) total: 10000 open/close in 121.27 seconds: 82.46 ops/second - unlinked 0 (time 1710103455 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg150-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.150@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg150-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg150-server: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 13 sec mount lustre on /mnt/lustre..... Starting client: oleg150-client.virtnet: -o user_xattr,flock oleg150-server@tcp:/lustre /mnt/lustre On OST0, 20235 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg150-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server unloading modules on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg150-client: oleg150-client: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-client: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2 PASS 69 (2739s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg150-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg150-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg150-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2 oleg150-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg150-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 93: register mulitple MDT at the same time ========================================================== 16:46:13 (1710103573) SKIP: conf-sanity test_93 needs >= 3 MDTs SKIP 93 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg150-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg150-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg150-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg150-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2 oleg150-server: oleg150-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing load_modules_local oleg150-server: Loading modules from /home/green/git/lustre-release/lustre oleg150-server: detected 4 online CPUs by sysfs oleg150-server: Force libcfs to create 2 CPU partitions oleg150-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg150-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3568 sec ========== 16:46:34 (1710103594) === conf-sanity: start cleanup 16:46:34 (1710103594) === === conf-sanity: finish cleanup 16:46:34 (1710103594) ===