-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:40:15 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg101-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg101-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg101-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg101-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg101-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg101-server oleg101-server: oleg101-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing load_modules_local oleg101-server: Loading modules from /home/green/git/lustre-release/lustre oleg101-server: detected 4 online CPUs by sysfs oleg101-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg101-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg101-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg101-server: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg101-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg101-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg101-server stop mds service on oleg101-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg101-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:41:18 (1709786478) start mds service on oleg101-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg101-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-OST0000 oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg101-client.virtnet: -o user_xattr,flock oleg101-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h41m33s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff88012aadc000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:42:17.000000000 -0500 Modify: 2024-03-06 23:42:17.000000000 -0500 Change: 2024-03-06 23:42:17.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 72 lines, 72 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg101-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg101-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg101-server stop mds service on oleg101-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg101-server unloading modules on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing unload_modules_local modules unloaded. oleg101-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg101-server: Is the ldiskfs module available? oleg101-server: oleg101-server: tunefs.lustre FATAL: failed to write local files oleg101-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg101-client: oleg101-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg101-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg101-server: Is the ldiskfs module available? oleg101-server: oleg101-server: tunefs.lustre FATAL: failed to write local files oleg101-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg101-client: oleg101-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.101@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.101@tcp sys.timeout=20 oleg101-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg101-server: Is the ldiskfs module available? oleg101-server: oleg101-server: tunefs.lustre FATAL: failed to write local files oleg101-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg101-client: oleg101-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.101@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.101@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg101-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg101-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2 oleg101-server: oleg101-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing load_modules_local oleg101-server: Loading modules from /home/green/git/lustre-release/lustre oleg101-server: detected 4 online CPUs by sysfs oleg101-server: Force libcfs to create 2 CPU partitions oleg101-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg101-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg101-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg101-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg101-server: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg101-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg101-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg101-server stop mds service on oleg101-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg101-server PASS 35b (124s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:43:24 (1709786604) start mds service on oleg101-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg101-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-OST0000 oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg101-client.virtnet: -o user_xattr,flock oleg101-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg101-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg101-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg101-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg101-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg101-client.virtnet: -o user_xattr,flock oleg101-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg101-client.virtnet /mnt/lustre (opts:) stop mds service on oleg101-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg101-server 000000 2 000008 oleg101-server: 8+0 records in oleg101-server: 8+0 records out oleg101-server: 4096 bytes (4.1 kB) copied, 0.000541553 s, 7.6 MB/s start mds service on oleg101-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg101-client.virtnet: -o user_xattr,flock oleg101-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg101-client.virtnet /mnt/lustre (opts:) stop mds service on oleg101-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg101-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg101-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg101-server stop mds service on oleg101-server unloading modules on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-client: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-client: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2 PASS 38 (79s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg101-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg101-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg101-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2 oleg101-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg101-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:44:47 (1709786687) Stopping clients: oleg101-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg101-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2 oleg101-server: oleg101-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing load_modules_local oleg101-server: Loading modules from /home/green/git/lustre-release/lustre oleg101-server: detected 4 online CPUs by sysfs oleg101-server: Force libcfs to create 2 CPU partitions oleg101-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg101-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg101-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg101-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg101-client.virtnet: -o user_xattr,flock oleg101-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg101-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg101-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg101-server umount lustre on /mnt/lustre..... Stopping client oleg101-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg101-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg101-server stop mds service on oleg101-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg101-server unloading modules on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing unload_modules_local modules unloaded. oleg101-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg101-server: Is the ldiskfs module available? oleg101-server: oleg101-server: tunefs.lustre FATAL: failed to write local files oleg101-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg101-client: oleg101-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg101-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg101-server: Is the ldiskfs module available? oleg101-server: oleg101-server: tunefs.lustre FATAL: failed to write local files oleg101-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg101-client: oleg101-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.101@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.101@tcp sys.timeout=20 oleg101-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg101-server: Is the ldiskfs module available? oleg101-server: oleg101-server: tunefs.lustre FATAL: failed to write local files oleg101-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg101-client: oleg101-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.101@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.101@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg101-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg101-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2 oleg101-server: oleg101-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing load_modules_local oleg101-server: Loading modules from /home/green/git/lustre-release/lustre oleg101-server: detected 4 online CPUs by sysfs oleg101-server: Force libcfs to create 2 CPU partitions oleg101-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg101-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg101-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg101-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg101-server: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg101-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg101-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg101-server stop mds service on oleg101-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg101-server PASS 51 (171s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:47:41 (1709786861) start mds service on oleg101-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg101-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-OST0000 oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg101-client.virtnet: -o user_xattr,flock oleg101-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 143 (time 1709786894.61 total 10.02 last 14.27) - open/close 405 (time 1709786904.63 total 20.03 last 26.16) - open/close 652 (time 1709786914.64 total 30.05 last 24.67) - open/close 825 (time 1709786924.67 total 40.07 last 17.25) - open/close 1087 (time 1709786935.38 total 50.79 last 24.45) - open/close 1307 (time 1709786945.60 total 61.01 last 21.53) - open/close 1489 (time 1709786955.84 total 71.25 last 17.78) - open/close 1771 (time 1709786966.15 total 81.56 last 27.34) - open/close 2095 (time 1709786976.16 total 91.57 last 32.37) - open/close 2284 (time 1709786986.20 total 101.61 last 18.82) - open/close 2421 (time 1709786996.75 total 112.16 last 12.98) - open/close 2637 (time 1709787007.01 total 122.41 last 21.07) - open/close 2763 (time 1709787017.52 total 132.93 last 11.99) - open/close 3001 (time 1709787028.49 total 143.89 last 21.70) - open/close 3284 (time 1709787038.66 total 154.07 last 27.81) - open/close 3584 (time 1709787049.31 total 164.72 last 28.18) - open/close 3814 (time 1709787060.15 total 175.55 last 21.22) - open/close 3976 (time 1709787070.58 total 185.98 last 15.53) - open/close 4215 (time 1709787080.58 total 195.99 last 23.88) - open/close 4421 (time 1709787091.35 total 206.75 last 19.14) - open/close 4586 (time 1709787103.08 total 218.49 last 14.06) - open/close 4890 (time 1709787113.09 total 228.49 last 30.39) - open/close 4993 (time 1709787123.10 total 238.50 last 10.29) - open/close 5187 (time 1709787133.98 total 249.39 last 17.82) - open/close 5338 (time 1709787144.53 total 259.94 last 14.31) - open/close 5502 (time 1709787154.54 total 269.95 last 16.39) - open/close 5684 (time 1709787164.86 total 280.27 last 17.64) - open/close 5782 (time 1709787175.75 total 291.16 last 9.00) - open/close 5993 (time 1709787186.32 total 301.72 last 19.98) - open/close 6169 (time 1709787196.32 total 311.73 last 17.59) - open/close 6408 (time 1709787206.64 total 322.04 last 23.17) - open/close 6635 (time 1709787216.96 total 332.36 last 22.00) - open/close 6782 (time 1709787227.26 total 342.66 last 14.27) - open/close 6865 (time 1709787237.86 total 353.27 last 7.83) - open/close 7023 (time 1709787248.26 total 363.67 last 15.19) - open/close 7254 (time 1709787261.22 total 376.63 last 17.83) - open/close 7436 (time 1709787271.24 total 386.65 last 18.16) - open/close 7618 (time 1709787282.21 total 397.62 last 16.59) - open/close 7849 (time 1709787292.21 total 407.62 last 23.10) - open/close 7961 (time 1709787302.32 total 417.73 last 11.08) - open/close 8149 (time 1709787312.86 total 428.26 last 17.85) - open/close 8356 (time 1709787323.82 total 439.23 last 18.87) - open/close 8524 (time 1709787333.85 total 449.25 last 16.76) - open/close 8711 (time 1709787344.61 total 460.01 last 17.38) - open/close 8827 (time 1709787355.25 total 470.65 last 10.90) - open/close 9031 (time 1709787365.26 total 480.67 last 20.38) - open/close 9215 (time 1709787375.28 total 490.69 last 18.36) - open/close 9336 (time 1709787385.95 total 501.36 last 11.34) - open/close 9488 (time 1709787395.96 total 511.37 last 15.18) - open/close 9637 (time 1709787405.97 total 521.37 last 14.90) - open/close 9833 (time 1709787415.98 total 531.38 last 19.58) - open/close 10000 (time 1709787424.04 total 539.45 last 20.71) - open/close 10230 (time 1709787439.31 total 554.72 last 15.06) - open/close 10731 (time 1709787449.32 total 564.73 last 50.04) - open/close 11233 (time 1709787459.33 total 574.74 last 50.16) - open/close 11728 (time 1709787469.34 total 584.74 last 49.47) - open/close 12156 (time 1709787479.34 total 594.75 last 42.77) - open/close 12634 (time 1709787489.35 total 604.76 last 47.74) - open/close 13098 (time 1709787499.36 total 614.77 last 46.38) - open/close 13574 (time 1709787509.38 total 624.78 last 47.51) - open/close 13997 (time 1709787519.41 total 634.82 last 42.16) - open/close 14286 (time 1709787536.40 total 651.81 last 17.01) - open/close 14697 (time 1709787546.42 total 661.83 last 41.01) - open/close 15123 (time 1709787556.43 total 671.84 last 42.55) - open/close 15521 (time 1709787566.44 total 681.85 last 39.77) - open/close 15924 (time 1709787576.45 total 691.86 last 40.26) - open/close 16351 (time 1709787586.47 total 701.88 last 42.60) - open/close 16755 (time 1709787596.48 total 711.88 last 40.39) - open/close 17135 (time 1709787606.50 total 721.90 last 37.93) - open/close 17535 (time 1709787616.51 total 731.92 last 39.94) - open/close 17929 (time 1709787626.53 total 741.94 last 39.33) - open/close 18319 (time 1709787636.55 total 751.96 last 38.93) - open/close 18694 (time 1709787646.60 total 762.01 last 37.30) - open/close 19085 (time 1709787656.62 total 772.02 last 39.04) - open/close 19516 (time 1709787666.63 total 782.04 last 43.05) - open/close 19917 (time 1709787676.67 total 792.08 last 39.92) - open/close 20000 (time 1709787678.74 total 794.14 last 40.28) - open/close 20323 (time 1709787688.76 total 804.17 last 32.22) - open/close 20644 (time 1709787698.81 total 814.22 last 31.94) - open/close 21006 (time 1709787708.82 total 824.22 last 36.18) - open/close 21393 (time 1709787718.83 total 834.24 last 38.64) - open/close 21793 (time 1709787728.86 total 844.26 last 39.89) - open/close 22171 (time 1709787738.87 total 854.28 last 37.74) - open/close 22500 (time 1709787748.91 total 864.31 last 32.79) - open/close 22859 (time 1709787758.92 total 874.32 last 35.86) - open/close 23197 (time 1709787768.93 total 884.33 last 33.77) - open/close 23544 (time 1709787778.93 total 894.34 last 34.68) - open/close 23879 (time 1709787788.95 total 904.36 last 33.43) - open/close 24184 (time 1709787798.97 total 914.37 last 30.47) - open/close 24513 (time 1709787808.99 total 924.39 last 32.84) - open/close 24877 (time 1709787819.00 total 934.40 last 36.36) - open/close 25193 (time 1709787829.07 total 944.48 last 31.36) - open/close 25482 (time 1709787839.08 total 954.49 last 28.88) - open/close 25739 (time 1709787849.10 total 964.50 last 25.65) - open/close 26033 (time 1709787859.11 total 974.51 last 29.37) - open/close 26340 (time 1709787869.12 total 984.53 last 30.65) - open/close 26658 (time 1709787879.14 total 994.55 last 31.74) - open/close 26948 (time 1709787889.15 total 1004.56 last 28.98) - open/close 27250 (time 1709787899.17 total 1014.57 last 30.15) - open/close 27537 (time 1709787909.20 total 1024.60 last 28.61) - open/close 27814 (time 1709787919.24 total 1034.65 last 27.57) - open/close 28092 (time 1709787929.27 total 1044.68 last 27.72) - open/close 28374 (time 1709787939.30 total 1054.71 last 28.13) - open/close 28659 (time 1709787949.30 total 1064.71 last 28.48) - open/close 29004 (time 1709787959.34 total 1074.74 last 34.38) - open/close 29289 (time 1709787969.36 total 1084.77 last 28.43) - open/close 29598 (time 1709787979.38 total 1094.79 last 30.84) - open/close 29937 (time 1709787989.41 total 1104.81 last 33.82) - open/close 30262 (time 1709787999.43 total 1114.84 last 32.43) - open/close 30589 (time 1709788009.44 total 1124.85 last 32.65) - open/close 30892 (time 1709788019.47 total 1134.88 last 30.22) - open/close 31201 (time 1709788029.48 total 1144.89 last 30.86) - open/close 31517 (time 1709788039.50 total 1154.91 last 31.55) - open/close 31842 (time 1709788049.52 total 1164.93 last 32.41) - open/close 32153 (time 1709788059.54 total 1174.94 last 31.07) - open/close 32463 (time 1709788069.57 total 1184.98 last 30.89) - open/close 32780 (time 1709788079.61 total 1195.02 last 31.57) - open/close 33082 (time 1709788089.61 total 1205.02 last 30.20) - open/close 33393 (time 1709788099.64 total 1215.05 last 31.01) - open/close 33676 (time 1709788109.65 total 1225.06 last 28.27) - open/close 33970 (time 1709788119.68 total 1235.09 last 29.32) - open/close 34271 (time 1709788129.69 total 1245.09 last 30.08) - open/close 34566 (time 1709788139.71 total 1255.11 last 29.44) - open/close 34841 (time 1709788149.75 total 1265.15 last 27.39) - open/close 35107 (time 1709788159.75 total 1275.15 last 26.60) - open/close 35365 (time 1709788169.77 total 1285.17 last 25.75) - open/close 35649 (time 1709788179.78 total 1295.19 last 28.36) - open/close 35923 (time 1709788189.78 total 1305.19 last 27.39) - open/close 36187 (time 1709788199.78 total 1315.19 last 26.40) - open/close 36476 (time 1709788209.80 total 1325.20 last 28.86) - open/close 36766 (time 1709788219.81 total 1335.22 last 28.96) - open/close 37070 (time 1709788229.83 total 1345.23 last 30.36) - open/close 37370 (time 1709788239.84 total 1355.25 last 29.95) - open/close 37658 (time 1709788249.85 total 1365.26 last 28.77) - open/close 37932 (time 1709788259.87 total 1375.28 last 27.34) - open/close 38197 (time 1709788269.89 total 1385.30 last 26.44) - open/close 38461 (time 1709788279.91 total 1395.31 last 26.37) - open/close 38742 (time 1709788289.93 total 1405.34 last 28.02) - open/close 39022 (time 1709788299.95 total 1415.35 last 27.96) - open/close 39294 (time 1709788309.98 total 1425.39 last 27.11) - open/close 39577 (time 1709788320.02 total 1435.43 last 28.19) - open/close 39857 (time 1709788330.06 total 1445.46 last 27.90) - open/close 40000 (time 1709788335.13 total 1450.54 last 28.17) - open/close 40264 (time 1709788345.15 total 1460.56 last 26.36) - open/close 40535 (time 1709788355.17 total 1470.58 last 27.04) - open/close 40810 (time 1709788365.20 total 1480.60 last 27.43) - open/close 41078 (time 1709788375.20 total 1490.61 last 26.78) - open/close 41345 (time 1709788385.22 total 1500.63 last 26.65) - open/close 41606 (time 1709788395.24 total 1510.64 last 26.06) - open/close 41876 (time 1709788405.25 total 1520.66 last 26.96) - open/close 42144 (time 1709788415.26 total 1530.66 last 26.79) - open/close 42392 (time 1709788425.28 total 1540.69 last 24.73) - open/close 42650 (time 1709788435.30 total 1550.71 last 25.76) - open/close 42900 (time 1709788445.30 total 1560.71 last 24.99) - open/close 43160 (time 1709788455.32 total 1570.72 last 25.97) - open/close 43406 (time 1709788465.35 total 1580.76 last 24.51) - open/close 43642 (time 1709788475.38 total 1590.79 last 23.54) - open/close 43873 (time 1709788485.41 total 1600.82 last 23.03) - open/close 44123 (time 1709788495.44 total 1610.84 last 24.93) - open/close 44373 (time 1709788505.46 total 1620.86 last 24.96) - open/close 44629 (time 1709788515.46 total 1630.87 last 25.59) - open/close 44875 (time 1709788525.47 total 1640.88 last 24.57) - open/close 45121 (time 1709788535.47 total 1650.88 last 24.60) - open/close 45360 (time 1709788545.50 total 1660.90 last 23.85) - open/close 45599 (time 1709788555.51 total 1670.92 last 23.87) - open/close 45842 (time 1709788565.51 total 1680.92 last 24.29) - open/close 46074 (time 1709788575.52 total 1690.93 last 23.18) - open/close 46298 (time 1709788585.54 total 1700.95 last 22.36) - open/close 46539 (time 1709788595.57 total 1710.98 last 24.02) - open/close 46784 (time 1709788605.58 total 1720.98 last 24.49) - open/close 47025 (time 1709788615.60 total 1731.01 last 24.05) - open/close 47277 (time 1709788625.62 total 1741.03 last 25.14) - open/close 47524 (time 1709788635.66 total 1751.07 last 24.61) - open/close 47762 (time 1709788645.67 total 1761.07 last 23.78) - open/close 48001 (time 1709788655.70 total 1771.11 last 23.82) - open/close 48234 (time 1709788665.72 total 1781.13 last 23.25) - open/close 48461 (time 1709788675.73 total 1791.14 last 22.68) - open/close 48692 (time 1709788685.77 total 1801.17 last 23.02) - open/close 48912 (time 1709788695.77 total 1811.18 last 21.98) - open/close 49139 (time 1709788705.79 total 1821.19 last 22.67) - open/close 49365 (time 1709788715.79 total 1831.19 last 22.60) - open/close 49599 (time 1709788725.82 total 1841.23 last 23.32) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1845.53 seconds: 26.93 ops/second - unlinked 0 (time 1709788731 ; total 0 ; last 0) - unlinked 10000 (time 1709789248 ; total 517 ; last 517) - unlinked 20000 (time 1709789669 ; total 938 ; last 421) - unlinked 30000 (time 1709789971 ; total 1240 ; last 302) - unlinked 40000 (time 1709790131 ; total 1400 ; last 160) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1453 seconds: 34.203716 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg101-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg101-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg101-server stop mds service on oleg101-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg101-server unloading modules on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg101-client: oleg101-client: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-client: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2 PASS 101a (3344s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg101-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg101-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg101-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2 oleg101-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg101-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:43:27 (1709790207) start mds service on oleg101-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing load_modules_local oleg101-server: Loading modules from /home/green/git/lustre-release/lustre oleg101-server: detected 4 online CPUs by sysfs oleg101-server: Force libcfs to create 2 CPU partitions oleg101-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg101-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg101-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-OST0000 oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg101-client.virtnet: -o user_xattr,flock oleg101-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg101-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg101-server start ost1 service on oleg101-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-OST0000 oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec oleg101-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012d8f5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012d8f5800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg101-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg101-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg101-server stop mds service on oleg101-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg101-server unloading modules on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg101-client: oleg101-client: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-client: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2 PASS 101b (89s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg101-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg101-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg101-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg101-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2 oleg101-server: oleg101-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing load_modules_local oleg101-server: Loading modules from /home/green/git/lustre-release/lustre oleg101-server: detected 4 online CPUs by sysfs oleg101-server: Force libcfs to create 2 CPU partitions oleg101-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg101-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3901 sec ========== 00:45:16 (1709790316) === conf-sanity: start cleanup 00:45:16 (1709790316) === === conf-sanity: finish cleanup 00:45:16 (1709790316) ===