-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:45:29 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg451-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg451-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg451-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg451-server oleg451-server: oleg451-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: detected 4 online CPUs by sysfs oleg451-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg451-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg451-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg451-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:46:39 (1709786799) start mds service on oleg451-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg451-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock oleg451-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h46m56s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800aae32800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:47:43.000000000 -0500 Modify: 2024-03-06 23:47:43.000000000 -0500 Change: 2024-03-06 23:47:43.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 76 lines, 76 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server unloading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing unload_modules_local modules unloaded. oleg451-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg451-server: Is the ldiskfs module available? oleg451-server: oleg451-server: tunefs.lustre FATAL: failed to write local files oleg451-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg451-client: oleg451-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg451-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg451-server: Is the ldiskfs module available? oleg451-server: oleg451-server: tunefs.lustre FATAL: failed to write local files oleg451-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg451-client: oleg451-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 oleg451-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg451-server: Is the ldiskfs module available? oleg451-server: oleg451-server: tunefs.lustre FATAL: failed to write local files oleg451-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg451-client: oleg451-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg451-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg451-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 oleg451-server: oleg451-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: detected 4 online CPUs by sysfs oleg451-server: Force libcfs to create 2 CPU partitions oleg451-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg451-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg451-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg451-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg451-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg451-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg451-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server PASS 35b (155s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:49:18 (1709786958) start mds service on oleg451-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg451-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock oleg451-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg451-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock oleg451-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server 000000 2 000008 oleg451-server: 8+0 records in oleg451-server: 8+0 records out oleg451-server: 4096 bytes (4.1 kB) copied, 0.00110197 s, 3.7 MB/s start mds service on oleg451-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock oleg451-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server unloading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 PASS 38 (93s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg451-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg451-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg451-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 oleg451-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg451-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:50:54 (1709787054) Stopping clients: oleg451-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg451-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 oleg451-server: oleg451-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: detected 4 online CPUs by sysfs oleg451-server: Force libcfs to create 2 CPU partitions oleg451-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg451-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg451-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg451-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock oleg451-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg451-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec stop ost2 service on oleg451-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg451-server umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server unloading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing unload_modules_local modules unloaded. oleg451-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg451-server: Is the ldiskfs module available? oleg451-server: oleg451-server: tunefs.lustre FATAL: failed to write local files oleg451-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg451-client: oleg451-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg451-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg451-server: Is the ldiskfs module available? oleg451-server: oleg451-server: tunefs.lustre FATAL: failed to write local files oleg451-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg451-client: oleg451-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 oleg451-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg451-server: Is the ldiskfs module available? oleg451-server: oleg451-server: tunefs.lustre FATAL: failed to write local files oleg451-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg451-client: oleg451-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg451-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg451-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 oleg451-server: oleg451-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: detected 4 online CPUs by sysfs oleg451-server: Force libcfs to create 2 CPU partitions oleg451-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg451-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg451-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg451-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg451-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server PASS 51 (187s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:54:05 (1709787245) start mds service on oleg451-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg451-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock oleg451-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 171 (time 1709787279.78 total 10.14 last 16.86) - open/close 403 (time 1709787289.79 total 20.15 last 23.18) - open/close 548 (time 1709787300.80 total 31.16 last 13.18) - open/close 739 (time 1709787310.80 total 41.16 last 19.09) - open/close 954 (time 1709787321.48 total 51.84 last 20.13) - open/close 1206 (time 1709787331.51 total 61.88 last 25.12) - open/close 1399 (time 1709787342.38 total 72.74 last 17.77) - open/close 1531 (time 1709787353.24 total 83.60 last 12.16) - open/close 1668 (time 1709787363.39 total 93.75 last 13.49) - open/close 1910 (time 1709787373.40 total 103.76 last 24.19) - open/close 2080 (time 1709787384.30 total 114.66 last 15.60) - open/close 2305 (time 1709787394.54 total 124.90 last 21.96) - open/close 2497 (time 1709787404.73 total 135.09 last 18.84) - open/close 2688 (time 1709787415.04 total 145.40 last 18.52) - open/close 2911 (time 1709787425.05 total 155.41 last 22.28) - open/close 3081 (time 1709787435.54 total 165.90 last 16.21) - open/close 3254 (time 1709787445.60 total 175.97 last 17.18) - open/close 3461 (time 1709787456.46 total 186.82 last 19.07) - open/close 3655 (time 1709787467.02 total 197.38 last 18.37) - open/close 3852 (time 1709787477.47 total 207.83 last 18.84) - open/close 4078 (time 1709787487.48 total 217.85 last 22.57) - open/close 4246 (time 1709787498.00 total 228.36 last 15.97) - open/close 4408 (time 1709787508.47 total 238.83 last 15.48) - open/close 4663 (time 1709787519.41 total 249.77 last 23.31) - open/close 4899 (time 1709787529.42 total 259.78 last 23.57) - open/close 5050 (time 1709787540.35 total 270.71 last 13.82) - open/close 5276 (time 1709787550.40 total 280.76 last 22.48) - open/close 5445 (time 1709787561.28 total 291.64 last 15.53) - open/close 5622 (time 1709787571.30 total 301.66 last 17.66) - open/close 5723 (time 1709787582.24 total 312.60 last 9.23) - open/close 5986 (time 1709787592.25 total 322.61 last 26.28) - open/close 6319 (time 1709787602.26 total 332.62 last 33.26) - open/close 6524 (time 1709787612.44 total 342.80 last 20.14) - open/close 6704 (time 1709787622.46 total 352.82 last 17.96) - open/close 6855 (time 1709787633.11 total 363.47 last 14.17) - open/close 7113 (time 1709787643.48 total 373.84 last 24.89) - open/close 7330 (time 1709787654.01 total 384.37 last 20.60) - open/close 7565 (time 1709787664.73 total 395.09 last 21.93) - open/close 7787 (time 1709787675.28 total 405.64 last 21.03) - open/close 8057 (time 1709787685.30 total 415.66 last 26.97) - open/close 8339 (time 1709787695.91 total 426.28 last 26.55) - open/close 8523 (time 1709787706.41 total 436.77 last 17.53) - open/close 8694 (time 1709787716.42 total 446.78 last 17.08) - open/close 8937 (time 1709787727.26 total 457.62 last 22.42) - open/close 9176 (time 1709787737.26 total 467.62 last 23.89) - open/close 9365 (time 1709787747.26 total 477.63 last 18.90) - open/close 9545 (time 1709787757.77 total 488.13 last 17.13) - open/close 9696 (time 1709787767.81 total 498.17 last 15.05) - open/close 9781 (time 1709787778.16 total 508.52 last 8.21) - open/close 9892 (time 1709787793.95 total 524.31 last 7.03) - open/close 10000 (time 1709787803.55 total 533.91 last 11.25) - open/close 10402 (time 1709787813.56 total 543.92 last 40.15) - open/close 10930 (time 1709787823.57 total 553.94 last 52.73) - open/close 11422 (time 1709787833.58 total 563.94 last 49.16) - open/close 11968 (time 1709787843.59 total 573.95 last 54.57) - open/close 12495 (time 1709787853.59 total 583.95 last 52.68) - open/close 13086 (time 1709787863.60 total 593.96 last 59.04) - open/close 13617 (time 1709787873.62 total 603.98 last 53.02) - open/close 14175 (time 1709787883.63 total 613.99 last 55.74) - open/close 14680 (time 1709787893.64 total 624.00 last 50.43) - open/close 15172 (time 1709787903.65 total 634.02 last 49.15) - open/close 15646 (time 1709787913.68 total 644.04 last 47.30) - open/close 16123 (time 1709787923.69 total 654.05 last 47.65) - open/close 16602 (time 1709787933.70 total 664.06 last 47.82) - open/close 17054 (time 1709787943.72 total 674.08 last 45.12) - open/close 17498 (time 1709787953.73 total 684.09 last 44.38) - open/close 17940 (time 1709787963.73 total 694.09 last 44.18) - open/close 18361 (time 1709787973.74 total 704.10 last 42.06) - open/close 18779 (time 1709787983.75 total 714.11 last 41.75) - open/close 19219 (time 1709787993.76 total 724.12 last 43.95) - open/close 19646 (time 1709788003.77 total 734.13 last 42.67) - open/close 20000 (time 1709788012.69 total 743.06 last 39.66) - open/close 20375 (time 1709788022.71 total 753.07 last 37.45) - open/close 20764 (time 1709788032.72 total 763.08 last 38.85) - open/close 21141 (time 1709788042.74 total 773.10 last 37.61) - open/close 21516 (time 1709788052.75 total 783.11 last 37.47) - open/close 21900 (time 1709788062.76 total 793.12 last 38.38) - open/close 22294 (time 1709788072.77 total 803.13 last 39.35) - open/close 22677 (time 1709788082.79 total 813.15 last 38.23) - open/close 23080 (time 1709788092.79 total 823.15 last 40.29) - open/close 23453 (time 1709788102.79 total 833.15 last 37.28) - open/close 23827 (time 1709788112.80 total 843.16 last 37.38) - open/close 24222 (time 1709788122.81 total 853.17 last 39.47) - open/close 24576 (time 1709788132.84 total 863.20 last 35.28) - open/close 24913 (time 1709788142.87 total 873.23 last 33.60) - open/close 25234 (time 1709788152.88 total 883.24 last 32.08) - open/close 25536 (time 1709788162.90 total 893.26 last 30.13) - open/close 25852 (time 1709788172.92 total 903.28 last 31.53) - open/close 26214 (time 1709788182.93 total 913.29 last 36.17) - open/close 26570 (time 1709788192.94 total 923.30 last 35.57) - open/close 26844 (time 1709788202.95 total 933.31 last 27.37) - open/close 27164 (time 1709788212.95 total 943.31 last 32.00) - open/close 27476 (time 1709788222.98 total 953.34 last 31.11) - open/close 27787 (time 1709788233.00 total 963.36 last 31.04) - open/close 28067 (time 1709788243.02 total 973.38 last 27.95) - open/close 28372 (time 1709788253.05 total 983.41 last 30.41) - open/close 28733 (time 1709788263.07 total 993.43 last 36.02) - open/close 29046 (time 1709788273.10 total 1003.46 last 31.20) - open/close 29333 (time 1709788283.11 total 1013.47 last 28.67) - open/close 29660 (time 1709788293.12 total 1023.48 last 32.66) - open/close 29978 (time 1709788303.15 total 1033.51 last 31.73) - open/close 30270 (time 1709788313.16 total 1043.52 last 29.16) - open/close 30570 (time 1709788323.17 total 1053.54 last 29.96) - open/close 30885 (time 1709788333.20 total 1063.56 last 31.43) - open/close 31180 (time 1709788343.22 total 1073.58 last 29.43) - open/close 31458 (time 1709788353.25 total 1083.61 last 27.73) - open/close 31735 (time 1709788363.27 total 1093.63 last 27.64) - open/close 32018 (time 1709788373.27 total 1103.63 last 28.29) - open/close 32310 (time 1709788383.29 total 1113.65 last 29.14) - open/close 32607 (time 1709788393.33 total 1123.69 last 29.59) - open/close 32901 (time 1709788403.35 total 1133.71 last 29.32) - open/close 33191 (time 1709788413.39 total 1143.75 last 28.90) - open/close 33456 (time 1709788423.42 total 1153.78 last 26.41) - open/close 33730 (time 1709788433.46 total 1163.82 last 27.30) - open/close 33986 (time 1709788443.46 total 1173.82 last 25.60) - open/close 34242 (time 1709788453.48 total 1183.84 last 25.55) - open/close 34505 (time 1709788463.50 total 1193.86 last 26.25) - open/close 34786 (time 1709788473.51 total 1203.87 last 28.08) - open/close 35049 (time 1709788483.53 total 1213.89 last 26.24) - open/close 35316 (time 1709788493.55 total 1223.91 last 26.65) - open/close 35589 (time 1709788503.55 total 1233.92 last 27.28) - open/close 35876 (time 1709788513.58 total 1243.95 last 28.62) - open/close 36124 (time 1709788523.63 total 1253.99 last 24.69) - open/close 36362 (time 1709788533.64 total 1264.00 last 23.78) - open/close 36643 (time 1709788543.67 total 1274.03 last 28.02) - open/close 36926 (time 1709788553.69 total 1284.05 last 28.24) - open/close 37180 (time 1709788563.69 total 1294.05 last 25.39) - open/close 37450 (time 1709788573.70 total 1304.06 last 26.98) - open/close 37714 (time 1709788583.71 total 1314.07 last 26.38) - open/close 37994 (time 1709788593.72 total 1324.08 last 27.98) - open/close 38279 (time 1709788603.73 total 1334.09 last 28.46) - open/close 38547 (time 1709788613.76 total 1344.12 last 26.73) - open/close 38823 (time 1709788623.79 total 1354.15 last 27.50) - open/close 39106 (time 1709788633.79 total 1364.16 last 28.29) - open/close 39384 (time 1709788643.81 total 1374.18 last 27.74) - open/close 39650 (time 1709788653.85 total 1384.21 last 26.51) - open/close 39918 (time 1709788663.88 total 1394.24 last 26.72) - open/close 40000 (time 1709788667.13 total 1397.49 last 25.22) - open/close 40206 (time 1709788677.15 total 1407.51 last 20.56) - open/close 40413 (time 1709788687.19 total 1417.56 last 20.61) - open/close 40638 (time 1709788697.23 total 1427.60 last 22.41) - open/close 40890 (time 1709788707.25 total 1437.61 last 25.16) - open/close 41135 (time 1709788717.27 total 1447.64 last 24.44) - open/close 41376 (time 1709788727.31 total 1457.67 last 24.02) - open/close 41627 (time 1709788737.32 total 1467.68 last 25.07) - open/close 41865 (time 1709788747.36 total 1477.72 last 23.72) - open/close 42101 (time 1709788757.39 total 1487.76 last 23.51) - open/close 42362 (time 1709788767.40 total 1497.76 last 26.08) - open/close 42616 (time 1709788777.43 total 1507.79 last 25.33) - open/close 42848 (time 1709788787.47 total 1517.83 last 23.10) - open/close 43087 (time 1709788797.49 total 1527.85 last 23.86) - open/close 43321 (time 1709788807.50 total 1537.86 last 23.37) - open/close 43565 (time 1709788817.52 total 1547.88 last 24.36) - open/close 43801 (time 1709788827.52 total 1557.88 last 23.60) - open/close 44041 (time 1709788837.53 total 1567.89 last 23.98) - open/close 44262 (time 1709788847.53 total 1577.89 last 22.10) - open/close 44485 (time 1709788857.54 total 1587.90 last 22.28) - open/close 44732 (time 1709788867.54 total 1597.91 last 24.68) - open/close 44982 (time 1709788877.57 total 1607.93 last 24.93) - open/close 45217 (time 1709788887.61 total 1617.97 last 23.41) - open/close 45450 (time 1709788897.64 total 1628.00 last 23.23) - open/close 45686 (time 1709788907.66 total 1638.02 last 23.56) - open/close 45925 (time 1709788917.70 total 1648.07 last 23.78) - open/close 46152 (time 1709788927.74 total 1658.10 last 22.63) - open/close 46384 (time 1709788937.75 total 1668.11 last 23.17) - open/close 46616 (time 1709788947.77 total 1678.13 last 23.16) - open/close 46850 (time 1709788957.78 total 1688.14 last 23.37) - open/close 47074 (time 1709788967.80 total 1698.16 last 22.35) - open/close 47299 (time 1709788977.82 total 1708.18 last 22.46) - open/close 47513 (time 1709788987.83 total 1718.19 last 21.38) - open/close 47735 (time 1709788997.84 total 1728.20 last 22.18) - open/close 47954 (time 1709789007.88 total 1738.24 last 21.81) - open/close 48170 (time 1709789017.93 total 1748.29 last 21.50) - open/close 48391 (time 1709789027.95 total 1758.31 last 22.05) - open/close 48604 (time 1709789037.95 total 1768.31 last 21.30) - open/close 48833 (time 1709789048.00 total 1778.36 last 22.79) - open/close 49061 (time 1709789058.04 total 1788.41 last 22.71) - open/close 49289 (time 1709789068.06 total 1798.42 last 22.77) - open/close 49516 (time 1709789078.08 total 1808.44 last 22.64) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1816.43 seconds: 27.36 ops/second - unlinked 0 (time 1709789087 ; total 0 ; last 0) - unlinked 10000 (time 1709789616 ; total 529 ; last 529) - unlinked 20000 (time 1709790016 ; total 929 ; last 400) - unlinked 30000 (time 1709790241 ; total 1154 ; last 225) - unlinked 40000 (time 1709790315 ; total 1228 ; last 74) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1246 seconds: 39.886036 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server unloading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 PASS 101a (3098s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg451-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg451-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg451-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 oleg451-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg451-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:45:45 (1709790345) start mds service on oleg451-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: detected 4 online CPUs by sysfs oleg451-server: Force libcfs to create 2 CPU partitions oleg451-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg451-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg451-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock oleg451-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server start ost1 service on oleg451-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 10 sec oleg451-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800afc21800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800afc21800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server unloading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 PASS 101b (80s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg451-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg451-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg451-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg451-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 oleg451-server: oleg451-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: detected 4 online CPUs by sysfs oleg451-server: Force libcfs to create 2 CPU partitions oleg451-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg451-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3711 sec ========== 00:47:21 (1709790441) === conf-sanity: start cleanup 00:47:22 (1709790442) === === conf-sanity: finish cleanup 00:47:22 (1709790442) ===