-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:45:34 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg443-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg443-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg443-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg443-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg443-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg443-server oleg443-server: oleg443-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing load_modules_local oleg443-server: Loading modules from /home/green/git/lustre-release/lustre oleg443-server: detected 4 online CPUs by sysfs oleg443-server: Force libcfs to create 2 CPU partitions oleg443-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg443-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg443-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg443-server: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg443-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg443-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg443-server stop mds service on oleg443-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg443-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:46:49 (1709786809) start mds service on oleg443-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg443-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0000 oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg443-client.virtnet: -o user_xattr,flock oleg443-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h47m07s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800b58b2000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:47:51.000000000 -0500 Modify: 2024-03-06 23:47:51.000000000 -0500 Change: 2024-03-06 23:47:51.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 76 lines, 76 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg443-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg443-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg443-server stop mds service on oleg443-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg443-server unloading modules on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing unload_modules_local modules unloaded. oleg443-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg443-server: Is the ldiskfs module available? oleg443-server: oleg443-server: tunefs.lustre FATAL: failed to write local files oleg443-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg443-client: oleg443-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg443-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg443-server: Is the ldiskfs module available? oleg443-server: oleg443-server: tunefs.lustre FATAL: failed to write local files oleg443-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg443-client: oleg443-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.143@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.143@tcp sys.timeout=20 oleg443-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg443-server: Is the ldiskfs module available? oleg443-server: oleg443-server: tunefs.lustre FATAL: failed to write local files oleg443-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg443-client: oleg443-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.143@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.143@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg443-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg443-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2 oleg443-server: oleg443-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing load_modules_local oleg443-server: Loading modules from /home/green/git/lustre-release/lustre oleg443-server: detected 4 online CPUs by sysfs oleg443-server: Force libcfs to create 2 CPU partitions oleg443-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg443-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg443-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg443-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg443-server: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg443-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg443-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg443-server stop mds service on oleg443-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg443-server PASS 35b (157s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:49:30 (1709786970) start mds service on oleg443-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg443-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0000 oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg443-client.virtnet: -o user_xattr,flock oleg443-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg443-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg443-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg443-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg443-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg443-client.virtnet: -o user_xattr,flock oleg443-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg443-client.virtnet /mnt/lustre (opts:) stop mds service on oleg443-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg443-server 000000 2 000008 oleg443-server: 8+0 records in oleg443-server: 8+0 records out oleg443-server: 4096 bytes (4.1 kB) copied, 0.000365601 s, 11.2 MB/s start mds service on oleg443-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg443-client.virtnet: -o user_xattr,flock oleg443-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg443-client.virtnet /mnt/lustre (opts:) stop mds service on oleg443-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg443-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg443-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg443-server stop mds service on oleg443-server unloading modules on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg443-client: oleg443-client: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-client: ssh exited with exit code 2 PASS 38 (93s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg443-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg443-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg443-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2 oleg443-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg443-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:51:06 (1709787066) Stopping clients: oleg443-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg443-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2 oleg443-server: oleg443-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing load_modules_local oleg443-server: Loading modules from /home/green/git/lustre-release/lustre oleg443-server: detected 4 online CPUs by sysfs oleg443-server: Force libcfs to create 2 CPU partitions oleg443-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg443-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg443-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg443-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg443-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg443-client.virtnet: -o user_xattr,flock oleg443-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg443-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg443-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg443-server umount lustre on /mnt/lustre..... Stopping client oleg443-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg443-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg443-server stop mds service on oleg443-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg443-server unloading modules on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing unload_modules_local modules unloaded. oleg443-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg443-server: Is the ldiskfs module available? oleg443-server: oleg443-server: tunefs.lustre FATAL: failed to write local files oleg443-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg443-client: oleg443-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg443-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg443-server: Is the ldiskfs module available? oleg443-server: oleg443-server: tunefs.lustre FATAL: failed to write local files oleg443-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg443-client: oleg443-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.143@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.143@tcp sys.timeout=20 oleg443-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg443-server: Is the ldiskfs module available? oleg443-server: oleg443-server: tunefs.lustre FATAL: failed to write local files oleg443-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg443-client: oleg443-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.143@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.143@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg443-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg443-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2 oleg443-server: oleg443-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing load_modules_local oleg443-server: Loading modules from /home/green/git/lustre-release/lustre oleg443-server: detected 4 online CPUs by sysfs oleg443-server: Force libcfs to create 2 CPU partitions oleg443-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg443-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg443-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg443-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg443-server: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg443-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg443-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg443-server stop mds service on oleg443-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg443-server PASS 51 (196s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:54:26 (1709787266) start mds service on oleg443-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg443-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0000 oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg443-client.virtnet: -o user_xattr,flock oleg443-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 135 (time 1709787303.83 total 10.13 last 13.33) - open/close 307 (time 1709787314.43 total 20.73 last 16.21) - open/close 502 (time 1709787324.97 total 31.27 last 18.51) - open/close 704 (time 1709787334.99 total 41.29 last 20.16) - open/close 909 (time 1709787344.99 total 51.29 last 20.49) - open/close 1065 (time 1709787355.02 total 61.32 last 15.55) - open/close 1258 (time 1709787365.03 total 71.33 last 19.28) - open/close 1458 (time 1709787377.38 total 83.68 last 16.19) - open/close 1619 (time 1709787387.86 total 94.16 last 15.37) - open/close 1858 (time 1709787397.90 total 104.20 last 23.80) - open/close 2046 (time 1709787408.55 total 114.85 last 17.65) - open/close 2213 (time 1709787418.70 total 125.00 last 16.45) - open/close 2424 (time 1709787429.45 total 135.75 last 19.62) - open/close 2618 (time 1709787439.97 total 146.27 last 18.44) - open/close 2762 (time 1709787450.56 total 156.86 last 13.60) - open/close 2882 (time 1709787460.58 total 166.88 last 11.98) - open/close 3027 (time 1709787471.42 total 177.72 last 13.37) - open/close 3261 (time 1709787482.36 total 188.66 last 21.39) - open/close 3424 (time 1709787492.36 total 198.67 last 16.29) - open/close 3580 (time 1709787503.14 total 209.44 last 14.48) - open/close 3786 (time 1709787513.31 total 219.61 last 20.25) - open/close 3972 (time 1709787524.03 total 230.33 last 17.34) - open/close 4034 (time 1709787539.11 total 245.41 last 4.11) - open/close 4257 (time 1709787549.13 total 255.43 last 22.26) - open/close 4421 (time 1709787559.13 total 265.43 last 16.39) - open/close 4653 (time 1709787569.65 total 275.95 last 22.07) - open/close 4786 (time 1709787579.65 total 285.95 last 13.29) - open/close 4961 (time 1709787590.32 total 296.62 last 16.40) - open/close 5254 (time 1709787600.46 total 306.76 last 28.90) - open/close 5458 (time 1709787611.39 total 317.69 last 18.67) - open/close 5594 (time 1709787621.83 total 328.13 last 13.03) - open/close 5785 (time 1709787631.84 total 338.14 last 19.08) - open/close 6003 (time 1709787642.36 total 348.66 last 20.72) - open/close 6298 (time 1709787652.95 total 359.25 last 27.86) - open/close 6495 (time 1709787662.96 total 369.26 last 19.67) - open/close 6671 (time 1709787673.92 total 380.22 last 16.06) - open/close 6759 (time 1709787684.80 total 391.10 last 8.09) - open/close 6872 (time 1709787694.94 total 401.24 last 11.15) - open/close 7073 (time 1709787704.97 total 411.27 last 20.04) - open/close 7240 (time 1709787714.97 total 421.27 last 16.70) - open/close 7349 (time 1709787724.98 total 431.28 last 10.89) - open/close 7508 (time 1709787735.49 total 441.79 last 15.13) - open/close 7717 (time 1709787746.46 total 452.76 last 19.05) - open/close 7957 (time 1709787756.62 total 462.92 last 23.63) - open/close 8128 (time 1709787767.06 total 473.36 last 16.39) - open/close 8307 (time 1709787777.07 total 483.37 last 17.87) - open/close 8655 (time 1709787787.80 total 494.10 last 32.45) - open/close 8759 (time 1709787798.64 total 504.94 last 9.59) - open/close 8951 (time 1709787809.04 total 515.34 last 18.46) - open/close 9214 (time 1709787819.04 total 525.34 last 26.29) - open/close 9404 (time 1709787829.05 total 535.35 last 18.99) - open/close 10000 (time 1709787838.54 total 544.84 last 62.79) - open/close 10633 (time 1709787848.55 total 554.85 last 63.24) - open/close 11237 (time 1709787858.57 total 564.87 last 60.32) - open/close 11882 (time 1709787868.57 total 574.87 last 64.44) - open/close 12520 (time 1709787878.59 total 584.89 last 63.70) - open/close 13156 (time 1709787888.60 total 594.90 last 63.52) - open/close 13746 (time 1709787898.61 total 604.91 last 58.95) - open/close 14246 (time 1709787908.62 total 614.92 last 49.97) - open/close 14753 (time 1709787918.62 total 624.92 last 50.66) - open/close 15256 (time 1709787928.64 total 634.94 last 50.22) - open/close 15775 (time 1709787938.65 total 644.95 last 51.86) - open/close 16266 (time 1709787948.65 total 654.95 last 49.09) - open/close 16804 (time 1709787958.65 total 664.95 last 53.77) - open/close 17284 (time 1709787968.65 total 674.95 last 48.00) - open/close 17762 (time 1709787978.66 total 684.96 last 47.76) - open/close 18240 (time 1709787988.67 total 694.97 last 47.77) - open/close 18757 (time 1709787998.69 total 705.00 last 51.56) - open/close 19150 (time 1709788008.71 total 715.02 last 39.22) - open/close 19539 (time 1709788018.74 total 725.04 last 38.82) - open/close 19964 (time 1709788028.75 total 735.05 last 42.43) - open/close 20414 (time 1709788038.76 total 745.06 last 44.97) - open/close 20864 (time 1709788048.76 total 755.06 last 45.00) - open/close 21263 (time 1709788058.78 total 765.08 last 39.82) - open/close 21690 (time 1709788068.80 total 775.10 last 42.63) - open/close 22134 (time 1709788078.81 total 785.11 last 44.35) - open/close 22576 (time 1709788088.81 total 795.11 last 44.19) - open/close 22979 (time 1709788098.81 total 805.11 last 40.29) - open/close 23376 (time 1709788108.82 total 815.12 last 39.68) - open/close 23781 (time 1709788118.82 total 825.12 last 40.48) - open/close 24178 (time 1709788128.83 total 835.13 last 39.68) - open/close 24539 (time 1709788138.84 total 845.14 last 36.06) - open/close 24903 (time 1709788148.91 total 855.21 last 36.15) - open/close 25206 (time 1709788158.92 total 865.22 last 30.27) - open/close 25520 (time 1709788168.94 total 875.24 last 31.33) - open/close 25884 (time 1709788178.95 total 885.26 last 36.35) - open/close 26269 (time 1709788188.96 total 895.26 last 38.48) - open/close 26600 (time 1709788198.99 total 905.29 last 32.99) - open/close 26955 (time 1709788209.01 total 915.31 last 35.45) - open/close 27329 (time 1709788219.02 total 925.32 last 37.35) - open/close 27683 (time 1709788229.03 total 935.33 last 35.38) - open/close 28012 (time 1709788239.05 total 945.35 last 32.82) - open/close 28331 (time 1709788249.06 total 955.36 last 31.87) - open/close 28726 (time 1709788259.07 total 965.37 last 39.44) - open/close 29088 (time 1709788269.10 total 975.40 last 36.09) - open/close 29422 (time 1709788279.11 total 985.41 last 33.38) - open/close 29760 (time 1709788289.13 total 995.43 last 33.72) - open/close 30000 (time 1709788295.98 total 1002.28 last 35.06) - open/close 30340 (time 1709788306.01 total 1012.31 last 33.90) - open/close 30664 (time 1709788316.04 total 1022.34 last 32.30) - open/close 30996 (time 1709788326.05 total 1032.35 last 33.19) - open/close 31351 (time 1709788336.06 total 1042.36 last 35.44) - open/close 31673 (time 1709788346.08 total 1052.38 last 32.13) - open/close 31994 (time 1709788356.11 total 1062.41 last 32.01) - open/close 32321 (time 1709788366.13 total 1072.43 last 32.65) - open/close 32671 (time 1709788376.15 total 1082.45 last 34.91) - open/close 33002 (time 1709788386.16 total 1092.46 last 33.08) - open/close 33344 (time 1709788396.17 total 1102.47 last 34.15) - open/close 33669 (time 1709788406.18 total 1112.48 last 32.48) - open/close 33974 (time 1709788416.18 total 1122.48 last 30.50) - open/close 34252 (time 1709788426.21 total 1132.51 last 27.72) - open/close 34537 (time 1709788436.21 total 1142.51 last 28.49) - open/close 34819 (time 1709788446.24 total 1152.54 last 28.14) - open/close 35101 (time 1709788456.28 total 1162.58 last 28.08) - open/close 35381 (time 1709788466.28 total 1172.58 last 27.99) - open/close 35684 (time 1709788476.30 total 1182.60 last 30.26) - open/close 35946 (time 1709788486.30 total 1192.60 last 26.19) - open/close 36237 (time 1709788496.32 total 1202.63 last 29.03) - open/close 36530 (time 1709788506.33 total 1212.63 last 29.29) - open/close 36840 (time 1709788516.33 total 1222.63 last 30.99) - open/close 37100 (time 1709788526.34 total 1232.64 last 25.99) - open/close 37390 (time 1709788536.34 total 1242.64 last 28.98) - open/close 37697 (time 1709788546.35 total 1252.65 last 30.67) - open/close 37983 (time 1709788556.36 total 1262.66 last 28.57) - open/close 38261 (time 1709788566.39 total 1272.69 last 27.74) - open/close 38552 (time 1709788576.40 total 1282.70 last 29.07) - open/close 38823 (time 1709788586.41 total 1292.71 last 27.07) - open/close 39134 (time 1709788596.41 total 1302.71 last 31.10) - open/close 39437 (time 1709788606.45 total 1312.75 last 30.18) - open/close 39727 (time 1709788616.45 total 1322.76 last 28.98) - open/close 40000 (time 1709788625.75 total 1332.05 last 29.38) - open/close 40295 (time 1709788635.78 total 1342.08 last 29.42) - open/close 40589 (time 1709788645.80 total 1352.10 last 29.34) - open/close 40867 (time 1709788655.80 total 1362.10 last 27.78) - open/close 41149 (time 1709788665.81 total 1372.11 last 28.17) - open/close 41366 (time 1709788675.86 total 1382.16 last 21.60) - open/close 41566 (time 1709788685.86 total 1392.16 last 20.00) - open/close 41789 (time 1709788695.88 total 1402.18 last 22.26) - open/close 42039 (time 1709788705.89 total 1412.19 last 24.96) - open/close 42289 (time 1709788715.92 total 1422.22 last 24.94) - open/close 42539 (time 1709788725.94 total 1432.24 last 24.95) - open/close 42794 (time 1709788735.94 total 1442.24 last 25.49) - open/close 43028 (time 1709788745.99 total 1452.29 last 23.29) - open/close 43263 (time 1709788756.01 total 1462.31 last 23.45) - open/close 43537 (time 1709788766.01 total 1472.31 last 27.40) - open/close 43809 (time 1709788776.02 total 1482.32 last 27.18) - open/close 44040 (time 1709788786.06 total 1492.37 last 22.99) - open/close 44286 (time 1709788796.09 total 1502.39 last 24.53) - open/close 44529 (time 1709788806.12 total 1512.42 last 24.23) - open/close 44790 (time 1709788816.15 total 1522.45 last 26.03) - open/close 45042 (time 1709788826.18 total 1532.48 last 25.11) - open/close 45293 (time 1709788836.19 total 1542.49 last 25.08) - open/close 45526 (time 1709788846.23 total 1552.53 last 23.21) - open/close 45755 (time 1709788856.26 total 1562.56 last 22.83) - open/close 46024 (time 1709788866.27 total 1572.57 last 26.87) - open/close 46296 (time 1709788876.30 total 1582.60 last 27.12) - open/close 46549 (time 1709788886.33 total 1592.63 last 25.23) - open/close 46804 (time 1709788896.34 total 1602.64 last 25.48) - open/close 47051 (time 1709788906.34 total 1612.64 last 24.70) - open/close 47302 (time 1709788916.34 total 1622.64 last 25.09) - open/close 47530 (time 1709788926.35 total 1632.65 last 22.78) - open/close 47779 (time 1709788936.37 total 1642.67 last 24.85) - open/close 48020 (time 1709788946.37 total 1652.67 last 24.09) - open/close 48255 (time 1709788956.37 total 1662.67 last 23.49) - open/close 48492 (time 1709788966.41 total 1672.71 last 23.61) - open/close 48733 (time 1709788976.41 total 1682.71 last 24.10) - open/close 48955 (time 1709788986.43 total 1692.73 last 22.16) - open/close 49192 (time 1709788996.44 total 1702.74 last 23.69) - open/close 49426 (time 1709789006.44 total 1712.74 last 23.38) - open/close 49646 (time 1709789016.48 total 1722.78 last 21.93) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1725.09 seconds: 28.81 ops/second - unlinked 0 (time 1709789020 ; total 0 ; last 0) - unlinked 10000 (time 1709789506 ; total 486 ; last 486) - unlinked 20000 (time 1709789860 ; total 840 ; last 354) - unlinked 30000 (time 1709790035 ; total 1015 ; last 175) - unlinked 40000 (time 1709790121 ; total 1101 ; last 86) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1141 seconds: 43.556530 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg443-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg443-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg443-server stop mds service on oleg443-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg443-server unloading modules on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg443-client: oleg443-client: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-client: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2 PASS 101a (2910s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg443-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg443-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg443-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2 oleg443-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg443-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:42:58 (1709790178) start mds service on oleg443-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing load_modules_local oleg443-server: Loading modules from /home/green/git/lustre-release/lustre oleg443-server: detected 4 online CPUs by sysfs oleg443-server: Force libcfs to create 2 CPU partitions oleg443-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg443-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg443-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg443-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0000 oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg443-client.virtnet: -o user_xattr,flock oleg443-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg443-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg443-server start ost1 service on oleg443-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0000 oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 12 sec oleg443-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b6e18800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6e18800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg443-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg443-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg443-server stop mds service on oleg443-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg443-server unloading modules on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg443-client: oleg443-client: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-client: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2 PASS 101b (85s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg443-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg443-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg443-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg443-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2 oleg443-server: oleg443-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing load_modules_local oleg443-server: Loading modules from /home/green/git/lustre-release/lustre oleg443-server: detected 4 online CPUs by sysfs oleg443-server: Force libcfs to create 2 CPU partitions oleg443-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg443-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3545 sec ========== 00:44:40 (1709790280) === conf-sanity: start cleanup 00:44:40 (1709790280) === === conf-sanity: finish cleanup 00:44:40 (1709790280) ===