-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:39:03 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg109-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg109-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg109-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg109-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg109-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg109-server oleg109-server: oleg109-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg109-server' oleg109-server: oleg109-server.virtnet: executing load_modules_local oleg109-server: Loading modules from /home/green/git/lustre-release/lustre oleg109-server: detected 4 online CPUs by sysfs oleg109-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg109-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg109-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg109-server: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg109-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg109-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg109-server stop mds service on oleg109-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg109-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:40:06 (1709786406) start mds service on oleg109-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg109-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0000 oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg109-client.virtnet: -o user_xattr,flock oleg109-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h40m22s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800ab722000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:41:05.000000000 -0500 Modify: 2024-03-06 23:41:05.000000000 -0500 Change: 2024-03-06 23:41:05.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 56 lines, 56 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg109-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg109-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg109-server stop mds service on oleg109-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg109-server unloading modules on: 'oleg109-server' oleg109-server: oleg109-server.virtnet: executing unload_modules_local modules unloaded. oleg109-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg109-server: Is the ldiskfs module available? oleg109-server: oleg109-server: tunefs.lustre FATAL: failed to write local files oleg109-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg109-client: oleg109-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg109-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg109-server: Is the ldiskfs module available? oleg109-server: oleg109-server: tunefs.lustre FATAL: failed to write local files oleg109-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg109-client: oleg109-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.109@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.109@tcp sys.timeout=20 oleg109-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg109-server: Is the ldiskfs module available? oleg109-server: oleg109-server: tunefs.lustre FATAL: failed to write local files oleg109-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg109-client: oleg109-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.109@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.109@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg109-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg109-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg109-client: oleg109-server: ssh exited with exit code 2 oleg109-server: oleg109-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg109-server' oleg109-server: oleg109-server.virtnet: executing load_modules_local oleg109-server: Loading modules from /home/green/git/lustre-release/lustre oleg109-server: detected 4 online CPUs by sysfs oleg109-server: Force libcfs to create 2 CPU partitions oleg109-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg109-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg109-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg109-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg109-server: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg109-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg109-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg109-server stop mds service on oleg109-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg109-server PASS 35b (116s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:42:04 (1709786524) start mds service on oleg109-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg109-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0000 oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg109-client.virtnet: -o user_xattr,flock oleg109-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg109-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg109-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg109-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg109-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg109-client.virtnet: -o user_xattr,flock oleg109-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg109-client.virtnet /mnt/lustre (opts:) stop mds service on oleg109-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg109-server 000000 2 000008 oleg109-server: 8+0 records in oleg109-server: 8+0 records out oleg109-server: 4096 bytes (4.1 kB) copied, 0.000387513 s, 10.6 MB/s start mds service on oleg109-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg109-client.virtnet: -o user_xattr,flock oleg109-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg109-client.virtnet /mnt/lustre (opts:) stop mds service on oleg109-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg109-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg109-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg109-server stop mds service on oleg109-server unloading modules on: 'oleg109-server' oleg109-server: oleg109-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg109-client: oleg109-client: ssh exited with exit code 2 pdsh@oleg109-client: oleg109-server: ssh exited with exit code 2 pdsh@oleg109-client: oleg109-client: ssh exited with exit code 2 pdsh@oleg109-client: oleg109-server: ssh exited with exit code 2 PASS 38 (71s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg109-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg109-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg109-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg109-client: oleg109-server: ssh exited with exit code 2 oleg109-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg109-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:43:17 (1709786597) Stopping clients: oleg109-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg109-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg109-client: oleg109-server: ssh exited with exit code 2 oleg109-server: oleg109-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg109-server' oleg109-server: oleg109-server.virtnet: executing load_modules_local oleg109-server: Loading modules from /home/green/git/lustre-release/lustre oleg109-server: detected 4 online CPUs by sysfs oleg109-server: Force libcfs to create 2 CPU partitions oleg109-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg109-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg109-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg109-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg109-client.virtnet: -o user_xattr,flock oleg109-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg109-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg109-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg109-server umount lustre on /mnt/lustre..... Stopping client oleg109-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg109-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg109-server stop mds service on oleg109-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg109-server unloading modules on: 'oleg109-server' oleg109-server: oleg109-server.virtnet: executing unload_modules_local modules unloaded. oleg109-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg109-server: Is the ldiskfs module available? oleg109-server: oleg109-server: tunefs.lustre FATAL: failed to write local files oleg109-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg109-client: oleg109-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg109-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg109-server: Is the ldiskfs module available? oleg109-server: oleg109-server: tunefs.lustre FATAL: failed to write local files oleg109-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg109-client: oleg109-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.109@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.109@tcp sys.timeout=20 oleg109-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg109-server: Is the ldiskfs module available? oleg109-server: oleg109-server: tunefs.lustre FATAL: failed to write local files oleg109-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg109-client: oleg109-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.109@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.109@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg109-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg109-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg109-client: oleg109-server: ssh exited with exit code 2 oleg109-server: oleg109-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg109-server' oleg109-server: oleg109-server.virtnet: executing load_modules_local oleg109-server: Loading modules from /home/green/git/lustre-release/lustre oleg109-server: detected 4 online CPUs by sysfs oleg109-server: Force libcfs to create 2 CPU partitions oleg109-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg109-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg109-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg109-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg109-server: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg109-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg109-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg109-server stop mds service on oleg109-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg109-server PASS 51 (151s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:45:52 (1709786752) start mds service on oleg109-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg109-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0000 oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg109-client.virtnet: -o user_xattr,flock oleg109-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 174 (time 1709786783.90 total 10.66 last 16.32) - open/close 505 (time 1709786794.44 total 21.20 last 31.41) - open/close 607 (time 1709786804.63 total 31.39 last 10.00) - open/close 869 (time 1709786815.22 total 41.98 last 24.75) - open/close 1248 (time 1709786826.01 total 52.77 last 35.13) - open/close 1512 (time 1709786836.02 total 62.78 last 26.37) - open/close 1744 (time 1709786846.98 total 73.74 last 21.17) - open/close 1875 (time 1709786857.01 total 83.77 last 13.06) - open/close 2048 (time 1709786867.83 total 94.59 last 15.99) - open/close 2262 (time 1709786877.84 total 104.60 last 21.38) - open/close 2459 (time 1709786887.86 total 114.62 last 19.65) - open/close 2691 (time 1709786898.74 total 125.49 last 21.34) - open/close 2885 (time 1709786908.74 total 135.50 last 19.40) - open/close 3157 (time 1709786918.75 total 145.51 last 27.17) - open/close 3347 (time 1709786929.19 total 155.95 last 18.19) - open/close 3572 (time 1709786939.20 total 165.95 last 22.49) - open/close 3780 (time 1709786949.35 total 176.11 last 20.48) - open/close 4044 (time 1709786960.32 total 187.07 last 24.08) - open/close 4324 (time 1709786970.32 total 197.07 last 28.00) - open/close 4514 (time 1709786980.62 total 207.38 last 18.44) - open/close 4761 (time 1709786991.31 total 218.07 last 23.10) - open/close 4985 (time 1709787001.32 total 228.08 last 22.38) - open/close 5217 (time 1709787011.68 total 238.44 last 22.40) - open/close 5410 (time 1709787021.86 total 248.61 last 18.96) - open/close 5542 (time 1709787031.88 total 258.64 last 13.17) - open/close 5854 (time 1709787041.88 total 268.64 last 31.18) - open/close 6075 (time 1709787052.62 total 279.38 last 20.58) - open/close 6182 (time 1709787062.64 total 289.40 last 10.68) - open/close 6380 (time 1709787072.64 total 299.40 last 19.79) - open/close 6551 (time 1709787082.74 total 309.49 last 16.95) - open/close 6808 (time 1709787093.30 total 320.06 last 24.32) - open/close 6953 (time 1709787104.16 total 330.92 last 13.36) - open/close 7164 (time 1709787114.17 total 340.93 last 21.08) - open/close 7307 (time 1709787124.18 total 350.94 last 14.28) - open/close 7414 (time 1709787135.43 total 362.19 last 9.51) - open/close 7582 (time 1709787146.14 total 372.90 last 15.68) - open/close 7798 (time 1709787156.72 total 383.48 last 20.43) - open/close 7993 (time 1709787167.63 total 394.39 last 17.87) - open/close 8227 (time 1709787178.29 total 405.04 last 21.96) - open/close 8406 (time 1709787188.56 total 415.31 last 17.43) - open/close 8560 (time 1709787198.84 total 425.60 last 14.97) - open/close 8690 (time 1709787209.29 total 436.04 last 12.45) - open/close 8912 (time 1709787219.29 total 446.05 last 22.20) - open/close 9074 (time 1709787229.57 total 456.33 last 15.75) - open/close 9348 (time 1709787239.82 total 466.58 last 26.72) - open/close 9551 (time 1709787250.50 total 477.26 last 19.02) - open/close 9656 (time 1709787261.15 total 487.91 last 9.86) - open/close 9890 (time 1709787271.99 total 498.75 last 21.58) - open/close 10000 (time 1709787276.61 total 503.36 last 23.84) - open/close 10315 (time 1709787287.65 total 514.41 last 28.52) - open/close 10435 (time 1709787298.22 total 524.98 last 11.35) - open/close 10625 (time 1709787309.11 total 535.86 last 17.46) - open/close 10976 (time 1709787319.11 total 545.87 last 35.09) - open/close 11438 (time 1709787329.11 total 555.87 last 46.18) - open/close 11720 (time 1709787345.70 total 572.46 last 17.00) - open/close 12141 (time 1709787355.70 total 582.46 last 42.10) - open/close 12550 (time 1709787365.71 total 592.46 last 40.87) - open/close 12972 (time 1709787375.73 total 602.49 last 42.09) - open/close 13358 (time 1709787385.77 total 612.53 last 38.45) - open/close 13359 (time 1709787396.84 total 623.60 last 0.09) - open/close 13738 (time 1709787406.85 total 633.60 last 37.88) - open/close 14099 (time 1709787416.85 total 643.61 last 36.08) - open/close 14489 (time 1709787426.86 total 653.62 last 38.96) - open/close 14672 (time 1709787443.71 total 670.46 last 10.86) - open/close 15005 (time 1709787453.72 total 680.47 last 33.27) - open/close 15332 (time 1709787463.73 total 690.49 last 32.65) - open/close 15622 (time 1709787473.73 total 700.49 last 28.99) - open/close 15920 (time 1709787483.77 total 710.53 last 29.68) - open/close 16238 (time 1709787493.79 total 720.55 last 31.76) - open/close 16543 (time 1709787503.79 total 730.55 last 30.48) - open/close 16856 (time 1709787513.80 total 740.56 last 31.29) - open/close 17182 (time 1709787523.80 total 750.56 last 32.58) - open/close 17505 (time 1709787533.81 total 760.56 last 32.30) - open/close 17804 (time 1709787543.81 total 770.57 last 29.89) - open/close 18099 (time 1709787553.82 total 780.58 last 29.46) - open/close 18395 (time 1709787563.83 total 790.59 last 29.57) - open/close 18679 (time 1709787573.84 total 800.60 last 28.39) - open/close 18985 (time 1709787583.89 total 810.65 last 30.44) - open/close 19274 (time 1709787593.91 total 820.67 last 28.83) - open/close 19570 (time 1709787603.93 total 830.69 last 29.54) - open/close 19917 (time 1709787613.95 total 840.71 last 34.64) - open/close 20000 (time 1709787616.81 total 843.57 last 29.01) - open/close 20298 (time 1709787626.82 total 853.58 last 29.79) - open/close 20600 (time 1709787636.83 total 863.59 last 30.16) - open/close 20905 (time 1709787646.85 total 873.61 last 30.44) - open/close 21196 (time 1709787656.88 total 883.63 last 29.02) - open/close 21496 (time 1709787666.92 total 893.68 last 29.86) - open/close 21782 (time 1709787676.93 total 903.69 last 28.58) - open/close 22061 (time 1709787686.96 total 913.72 last 27.82) - open/close 22332 (time 1709787696.97 total 923.73 last 27.07) - open/close 22616 (time 1709787707.01 total 933.76 last 28.31) - open/close 22921 (time 1709787717.03 total 943.79 last 30.42) - open/close 23230 (time 1709787727.05 total 953.81 last 30.86) - open/close 23544 (time 1709787737.06 total 963.81 last 31.37) - open/close 23617 (time 1709787751.12 total 977.88 last 5.19) - open/close 23881 (time 1709787761.17 total 987.92 last 26.28) - open/close 24161 (time 1709787771.19 total 997.95 last 27.94) - open/close 24420 (time 1709787781.21 total 1007.97 last 25.83) - open/close 24676 (time 1709787791.22 total 1017.97 last 25.60) - open/close 24927 (time 1709787801.23 total 1027.99 last 25.06) - open/close 25222 (time 1709787811.26 total 1038.02 last 29.43) - open/close 25524 (time 1709787821.29 total 1048.05 last 30.10) - open/close 25775 (time 1709787831.32 total 1058.08 last 25.03) - open/close 26009 (time 1709787841.35 total 1068.11 last 23.33) - open/close 26234 (time 1709787851.37 total 1078.13 last 22.46) - open/close 26472 (time 1709787861.38 total 1088.14 last 23.77) - open/close 26725 (time 1709787871.39 total 1098.15 last 25.28) - open/close 26981 (time 1709787881.43 total 1108.19 last 25.49) - open/close 26999 (time 1709787893.49 total 1120.25 last 1.49) - open/close 27263 (time 1709787903.51 total 1130.27 last 26.36) - open/close 27505 (time 1709787913.55 total 1140.31 last 24.10) - open/close 27730 (time 1709787923.56 total 1150.32 last 22.47) - open/close 27958 (time 1709787933.59 total 1160.35 last 22.73) - open/close 28191 (time 1709787943.66 total 1170.41 last 23.15) - open/close 28442 (time 1709787953.66 total 1180.42 last 25.09) - open/close 28702 (time 1709787963.67 total 1190.43 last 25.97) - open/close 28932 (time 1709787973.71 total 1200.47 last 22.91) - open/close 29216 (time 1709787983.71 total 1210.47 last 28.40) - open/close 29516 (time 1709787993.72 total 1220.48 last 29.96) - open/close 29797 (time 1709788003.74 total 1230.50 last 28.05) - open/close 30000 (time 1709788010.96 total 1237.72 last 28.14) - open/close 30248 (time 1709788020.99 total 1247.74 last 24.73) - open/close 30514 (time 1709788030.99 total 1257.74 last 26.60) - open/close 30789 (time 1709788041.00 total 1267.76 last 27.46) - open/close 31069 (time 1709788051.01 total 1277.77 last 27.99) - open/close 31340 (time 1709788061.03 total 1287.79 last 27.04) - open/close 31606 (time 1709788071.07 total 1297.83 last 26.49) - open/close 31895 (time 1709788081.11 total 1307.87 last 28.79) - open/close 32177 (time 1709788091.11 total 1317.87 last 28.20) - open/close 32448 (time 1709788101.15 total 1327.91 last 27.00) - open/close 32716 (time 1709788111.15 total 1337.91 last 26.78) - open/close 32986 (time 1709788121.16 total 1347.92 last 26.98) - open/close 33248 (time 1709788131.19 total 1357.95 last 26.14) - open/close 33520 (time 1709788141.22 total 1367.98 last 27.11) - open/close 33777 (time 1709788151.24 total 1377.99 last 25.66) - open/close 34023 (time 1709788161.24 total 1388.00 last 24.58) - open/close 34264 (time 1709788171.25 total 1398.01 last 24.07) - open/close 34508 (time 1709788181.30 total 1408.06 last 24.29) - open/close 34750 (time 1709788191.34 total 1418.09 last 24.12) - open/close 34998 (time 1709788201.34 total 1428.10 last 24.78) - open/close 35275 (time 1709788211.36 total 1438.12 last 27.66) - open/close 35535 (time 1709788221.37 total 1448.13 last 25.96) - open/close 35814 (time 1709788231.40 total 1458.16 last 27.81) - open/close 36070 (time 1709788241.42 total 1468.18 last 25.56) - open/close 36327 (time 1709788251.43 total 1478.19 last 25.68) - open/close 36551 (time 1709788261.43 total 1488.19 last 22.39) - open/close 36785 (time 1709788271.46 total 1498.22 last 23.33) - open/close 37028 (time 1709788281.49 total 1508.25 last 24.22) - open/close 37280 (time 1709788291.50 total 1518.26 last 25.17) - open/close 37528 (time 1709788301.55 total 1528.31 last 24.69) - open/close 37772 (time 1709788311.59 total 1538.35 last 24.31) - open/close 38013 (time 1709788321.59 total 1548.35 last 24.10) - open/close 38259 (time 1709788331.63 total 1558.39 last 24.50) - open/close 38499 (time 1709788341.68 total 1568.44 last 23.88) - open/close 38736 (time 1709788351.72 total 1578.47 last 23.62) - open/close 38972 (time 1709788361.74 total 1588.50 last 23.53) - open/close 39206 (time 1709788371.74 total 1598.50 last 23.40) - open/close 39443 (time 1709788381.76 total 1608.52 last 23.67) - open/close 39674 (time 1709788391.78 total 1618.54 last 23.06) - open/close 39904 (time 1709788401.80 total 1628.56 last 22.95) - open/close 40000 (time 1709788405.78 total 1632.54 last 24.11) - open/close 40234 (time 1709788415.80 total 1642.56 last 23.35) - open/close 40459 (time 1709788425.81 total 1652.56 last 22.49) - open/close 40693 (time 1709788435.83 total 1662.58 last 23.35) - open/close 40910 (time 1709788445.84 total 1672.60 last 21.67) - open/close 41143 (time 1709788455.86 total 1682.62 last 23.25) - open/close 41347 (time 1709788465.87 total 1692.63 last 20.38) - open/close 41560 (time 1709788475.91 total 1702.67 last 21.22) - open/close 41771 (time 1709788485.92 total 1712.68 last 21.07) - open/close 41999 (time 1709788495.95 total 1722.71 last 22.75) - open/close 42226 (time 1709788505.96 total 1732.72 last 22.66) - open/close 42468 (time 1709788516.00 total 1742.76 last 24.10) - open/close 42699 (time 1709788526.04 total 1752.80 last 23.02) - open/close 42936 (time 1709788536.04 total 1762.80 last 23.70) - open/close 43165 (time 1709788546.06 total 1772.82 last 22.84) - open/close 43393 (time 1709788556.08 total 1782.83 last 22.77) - open/close 43630 (time 1709788566.12 total 1792.88 last 23.60) - open/close 43855 (time 1709788576.16 total 1802.91 last 22.41) - open/close 44061 (time 1709788586.17 total 1812.93 last 20.56) - open/close 44280 (time 1709788596.20 total 1822.95 last 21.85) - open/close 44501 (time 1709788606.23 total 1832.98 last 22.03) - open/close 44735 (time 1709788616.26 total 1843.02 last 23.33) - open/close 44977 (time 1709788626.28 total 1853.04 last 24.14) - open/close 45216 (time 1709788636.31 total 1863.07 last 23.84) - open/close 45433 (time 1709788646.34 total 1873.10 last 21.63) - open/close 45652 (time 1709788656.34 total 1883.10 last 21.89) - open/close 45861 (time 1709788666.39 total 1893.15 last 20.81) - open/close 46064 (time 1709788676.40 total 1903.16 last 20.28) - open/close 46265 (time 1709788686.42 total 1913.17 last 20.06) - open/close 46476 (time 1709788696.42 total 1923.18 last 21.08) - open/close 46688 (time 1709788706.46 total 1933.22 last 21.13) - open/close 46904 (time 1709788716.49 total 1943.25 last 21.54) - open/close 47116 (time 1709788726.53 total 1953.28 last 21.12) - open/close 47325 (time 1709788736.54 total 1963.30 last 20.86) - open/close 47516 (time 1709788746.57 total 1973.33 last 19.04) - open/close 47703 (time 1709788756.60 total 1983.36 last 18.64) - open/close 47890 (time 1709788766.64 total 1993.40 last 18.63) - open/close 48085 (time 1709788776.65 total 2003.40 last 19.49) - open/close 48283 (time 1709788786.66 total 2013.42 last 19.78) - open/close 48495 (time 1709788796.66 total 2023.42 last 21.19) - open/close 48691 (time 1709788806.69 total 2033.45 last 19.54) - open/close 48905 (time 1709788816.73 total 2043.49 last 21.32) - open/close 49098 (time 1709788826.76 total 2053.52 last 19.23) - open/close 49293 (time 1709788836.77 total 2063.53 last 19.49) - open/close 49482 (time 1709788846.80 total 2073.55 last 18.85) - open/close 49671 (time 1709788856.84 total 2083.60 last 18.81) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 2085.02 seconds: 23.84 ops/second