-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:39:28 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg112-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg112-server oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg112-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:40:30 (1709786430) start mds service on oleg112-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock oleg112-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h40m45s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800aab98000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:41:28.000000000 -0500 Modify: 2024-03-06 23:41:28.000000000 -0500 Change: 2024-03-06 23:41:28.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 54 lines, 54 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server unloading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local modules unloaded. oleg112-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg112-server: Is the ldiskfs module available? oleg112-server: oleg112-server: tunefs.lustre FATAL: failed to write local files oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg112-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg112-server: Is the ldiskfs module available? oleg112-server: oleg112-server: tunefs.lustre FATAL: failed to write local files oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.112@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.112@tcp sys.timeout=20 oleg112-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg112-server: Is the ldiskfs module available? oleg112-server: oleg112-server: tunefs.lustre FATAL: failed to write local files oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.112@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.112@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg112-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server PASS 35b (123s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:42:36 (1709786556) start mds service on oleg112-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock oleg112-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg112-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock oleg112-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server 000000 2 000008 oleg112-server: 8+0 records in oleg112-server: 8+0 records out oleg112-server: 4096 bytes (4.1 kB) copied, 0.00039306 s, 10.4 MB/s start mds service on oleg112-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock oleg112-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server unloading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 PASS 38 (72s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg112-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg112-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg112-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 oleg112-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg112-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:43:51 (1709786631) Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg112-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock oleg112-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg112-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg112-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg112-server umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server unloading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local modules unloaded. oleg112-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg112-server: Is the ldiskfs module available? oleg112-server: oleg112-server: tunefs.lustre FATAL: failed to write local files oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg112-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg112-server: Is the ldiskfs module available? oleg112-server: oleg112-server: tunefs.lustre FATAL: failed to write local files oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.112@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.112@tcp sys.timeout=20 oleg112-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg112-server: Is the ldiskfs module available? oleg112-server: oleg112-server: tunefs.lustre FATAL: failed to write local files oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.112@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.112@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg112-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server PASS 51 (173s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:46:48 (1709786808) start mds service on oleg112-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock oleg112-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 204 (time 1709786839.85 total 10.54 last 19.36) - open/close 355 (time 1709786850.16 total 20.85 last 14.65) - open/close 436 (time 1709786861.43 total 32.12 last 7.19) - open/close 649 (time 1709786871.47 total 42.15 last 21.22) - open/close 849 (time 1709786882.26 total 52.94 last 18.54) - open/close 1024 (time 1709786892.52 total 63.20 last 17.05) - open/close 1190 (time 1709786902.83 total 73.52 last 16.09) - open/close 1606 (time 1709786912.84 total 83.52 last 41.58) - open/close 1836 (time 1709786923.74 total 94.42 last 21.10) - open/close 2060 (time 1709786934.73 total 105.42 last 20.38) - open/close 2290 (time 1709786945.15 total 115.83 last 22.08) - open/close 2445 (time 1709786955.26 total 125.94 last 15.33) - open/close 2598 (time 1709786965.26 total 135.95 last 15.29) - open/close 2779 (time 1709786975.82 total 146.50 last 17.15) - open/close 3018 (time 1709786985.87 total 156.55 last 23.77) - open/close 3273 (time 1709786996.78 total 167.47 last 23.37) - open/close 3467 (time 1709787008.23 total 178.91 last 16.95) - open/close 3673 (time 1709787018.24 total 188.92 last 20.58) - open/close 3933 (time 1709787028.25 total 198.93 last 25.98) - open/close 4248 (time 1709787038.54 total 209.22 last 30.61) - open/close 4364 (time 1709787049.29 total 219.97 last 10.79) - open/close 4640 (time 1709787059.32 total 230.00 last 27.51) - open/close 4808 (time 1709787069.45 total 240.14 last 16.58) - open/close 4969 (time 1709787080.25 total 250.93 last 14.92) - open/close 5210 (time 1709787090.29 total 260.97 last 24.01) - open/close 5375 (time 1709787100.52 total 271.21 last 16.12) - open/close 5620 (time 1709787110.53 total 281.21 last 24.49) - open/close 5834 (time 1709787120.54 total 291.22 last 21.37) - open/close 5962 (time 1709787130.85 total 301.53 last 12.42) - open/close 6106 (time 1709787140.85 total 311.54 last 14.39) - open/close 6247 (time 1709787151.56 total 322.25 last 13.16) - open/close 6386 (time 1709787161.57 total 332.26 last 13.89) - open/close 6552 (time 1709787172.09 total 342.77 last 15.78) - open/close 6770 (time 1709787182.73 total 353.42 last 20.48) - open/close 6955 (time 1709787193.47 total 364.15 last 17.24) - open/close 7205 (time 1709787203.48 total 374.16 last 24.97) - open/close 7394 (time 1709787214.09 total 384.78 last 17.80) - open/close 7520 (time 1709787224.11 total 394.79 last 12.59) - open/close 7726 (time 1709787234.74 total 405.42 last 19.37) - open/close 7850 (time 1709787245.55 total 416.23 last 11.47) - open/close 8075 (time 1709787255.74 total 426.42 last 22.09) - open/close 8251 (time 1709787266.25 total 436.93 last 16.75) - open/close 8453 (time 1709787276.70 total 447.39 last 19.32) - open/close 8509 (time 1709787289.96 total 460.65 last 4.22) - open/close 8631 (time 1709787300.72 total 471.41 last 11.34) - open/close 8759 (time 1709787310.78 total 481.46 last 12.73) - open/close 8936 (time 1709787320.88 total 491.57 last 17.51) - open/close 9137 (time 1709787331.44 total 502.12 last 19.05) - open/close 9317 (time 1709787341.48 total 512.16 last 17.93) - open/close 9474 (time 1709787352.39 total 523.08 last 14.38) - open/close 9587 (time 1709787362.42 total 533.10 last 11.27) - open/close 9797 (time 1709787372.42 total 543.10 last 21.00) - open/close 10000 (time 1709787379.49 total 550.17 last 28.70) - open/close 10442 (time 1709787389.50 total 560.18 last 44.17) - open/close 10880 (time 1709787399.51 total 570.20 last 43.73) - open/close 11294 (time 1709787409.53 total 580.21 last 41.33) - open/close 11734 (time 1709787419.57 total 590.26 last 43.81) - open/close 12150 (time 1709787429.59 total 600.28 last 41.52) - open/close 12547 (time 1709787439.60 total 610.29 last 39.66) - open/close 12887 (time 1709787449.62 total 620.30 last 33.96) - open/close 13236 (time 1709787459.62 total 630.30 last 34.90) - open/close 13584 (time 1709787469.63 total 640.31 last 34.77) - open/close 13930 (time 1709787479.65 total 650.33 last 34.53) - open/close 14331 (time 1709787489.66 total 660.34 last 40.04) - open/close 14720 (time 1709787499.66 total 670.35 last 38.89) - open/close 15101 (time 1709787509.67 total 680.35 last 38.08) - open/close 15461 (time 1709787519.68 total 690.36 last 35.97) - open/close 15874 (time 1709787529.69 total 700.37 last 41.24) - open/close 16240 (time 1709787539.71 total 710.39 last 36.55) - open/close 16525 (time 1709787559.65 total 730.34 last 14.29) - open/close 16826 (time 1709787569.66 total 740.35 last 30.07) - open/close 17154 (time 1709787579.68 total 750.36 last 32.76) - open/close 17490 (time 1709787589.70 total 760.38 last 33.52) - open/close 17783 (time 1709787599.73 total 770.41 last 29.21) - open/close 18105 (time 1709787609.75 total 780.43 last 32.14) - open/close 18455 (time 1709787619.79 total 790.47 last 34.87) - open/close 18769 (time 1709787629.81 total 800.49 last 31.33) - open/close 19108 (time 1709787639.84 total 810.52 last 33.80) - open/close 19417 (time 1709787649.84 total 820.52 last 30.89) - open/close 19725 (time 1709787659.87 total 830.55 last 30.71) - open/close 20000 (time 1709787668.56 total 839.25 last 31.63) - open/close 20301 (time 1709787678.58 total 849.27 last 30.04) - open/close 20582 (time 1709787688.59 total 859.28 last 28.08) - open/close 20892 (time 1709787698.60 total 869.28 last 30.98) - open/close 21189 (time 1709787708.61 total 879.29 last 29.68) - open/close 21512 (time 1709787718.61 total 889.29 last 32.30) - open/close 21847 (time 1709787728.63 total 899.32 last 33.40) - open/close 22164 (time 1709787738.66 total 909.34 last 31.63) - open/close 22418 (time 1709787748.68 total 919.37 last 25.33) - open/close 22697 (time 1709787758.70 total 929.38 last 27.86) - open/close 22974 (time 1709787768.70 total 939.38 last 27.69) - open/close 23270 (time 1709787778.72 total 949.40 last 29.56) - open/close 23538 (time 1709787788.74 total 959.43 last 26.73) - open/close 23824 (time 1709787798.74 total 969.43 last 28.60) - open/close 24118 (time 1709787808.76 total 979.44 last 29.35) - open/close 24431 (time 1709787818.77 total 989.45 last 31.27) - open/close 24707 (time 1709787828.78 total 999.46 last 27.58) - open/close 24953 (time 1709787838.79 total 1009.48 last 24.56) - open/close 25203 (time 1709787848.84 total 1019.52 last 24.89) - open/close 25470 (time 1709787858.89 total 1029.57 last 26.57) - open/close 25728 (time 1709787868.91 total 1039.59 last 25.75) - open/close 25998 (time 1709787878.93 total 1049.62 last 26.93) - open/close 26267 (time 1709787888.96 total 1059.65 last 26.82) - open/close 26526 (time 1709787898.98 total 1069.67 last 25.84) - open/close 26674 (time 1709787915.69 total 1086.38 last 8.86) - open/close 26888 (time 1709787925.71 total 1096.39 last 21.37) - open/close 27126 (time 1709787935.71 total 1106.39 last 23.79) - open/close 27365 (time 1709787945.73 total 1116.42 last 23.85) - open/close 27668 (time 1709787955.75 total 1126.44 last 30.24) - open/close 27942 (time 1709787965.76 total 1136.45 last 27.37) - open/close 28178 (time 1709787975.77 total 1146.45 last 23.58) - open/close 28462 (time 1709787985.78 total 1156.46 last 28.37) - open/close 28749 (time 1709787995.82 total 1166.50 last 28.60) - open/close 29021 (time 1709788005.84 total 1176.52 last 27.13) - open/close 29274 (time 1709788015.87 total 1186.55 last 25.22) - open/close 29529 (time 1709788025.90 total 1196.58 last 25.43) - open/close 29833 (time 1709788035.93 total 1206.61 last 30.31) - open/close 30000 (time 1709788041.98 total 1212.66 last 27.62) - open/close 30271 (time 1709788051.99 total 1222.67 last 27.07) - open/close 30541 (time 1709788062.00 total 1232.69 last 26.96) - open/close 30820 (time 1709788072.02 total 1242.70 last 27.85) - open/close 31108 (time 1709788082.03 total 1252.71 last 28.79) - open/close 31373 (time 1709788092.04 total 1262.72 last 26.46) - open/close 31630 (time 1709788102.08 total 1272.76 last 25.60) - open/close 31874 (time 1709788112.09 total 1282.77 last 24.38) - open/close 32110 (time 1709788122.12 total 1292.80 last 23.52) - open/close 32357 (time 1709788132.13 total 1302.81 last 24.68) - open/close 32613 (time 1709788142.14 total 1312.82 last 25.57) - open/close 32861 (time 1709788152.14 total 1322.83 last 24.79) - open/close 33095 (time 1709788162.15 total 1332.83 last 23.38) - open/close 33328 (time 1709788172.18 total 1342.86 last 23.24) - open/close 33565 (time 1709788182.19 total 1352.87 last 23.67) - open/close 33801 (time 1709788192.21 total 1362.89 last 23.55) - open/close 34050 (time 1709788202.22 total 1372.90 last 24.88) - open/close 34328 (time 1709788212.22 total 1382.91 last 27.78) - open/close 34584 (time 1709788222.23 total 1392.91 last 25.59) - open/close 34874 (time 1709788232.25 total 1402.93 last 28.94) - open/close 35136 (time 1709788242.25 total 1412.94 last 26.18) - open/close 35383 (time 1709788252.28 total 1422.96 last 24.64) - open/close 35606 (time 1709788262.29 total 1432.97 last 22.28) - open/close 35832 (time 1709788272.30 total 1442.98 last 22.58) - open/close 36079 (time 1709788282.31 total 1452.99 last 24.67) - open/close 36339 (time 1709788292.32 total 1463.00 last 25.97) - open/close 36596 (time 1709788302.33 total 1473.01 last 25.67) - open/close 36836 (time 1709788312.34 total 1483.02 last 23.98) - open/close 37084 (time 1709788322.37 total 1493.06 last 24.72) - open/close 37338 (time 1709788332.39 total 1503.07 last 25.35) - open/close 37577 (time 1709788342.43 total 1513.12 last 23.80) - open/close 37822 (time 1709788352.43 total 1523.12 last 24.50) - open/close 38060 (time 1709788362.44 total 1533.12 last 23.79) - open/close 38304 (time 1709788372.47 total 1543.15 last 24.32) - open/close 38559 (time 1709788382.49 total 1553.17 last 25.45) - open/close 38805 (time 1709788392.50 total 1563.19 last 24.57) - open/close 39042 (time 1709788402.52 total 1573.20 last 23.67) - open/close 39299 (time 1709788412.55 total 1583.24 last 25.60) - open/close 39530 (time 1709788422.57 total 1593.25 last 23.08) - open/close 39749 (time 1709788432.60 total 1603.28 last 21.83) - open/close 39977 (time 1709788442.63 total 1613.31 last 22.74) - open/close 40205 (time 1709788452.66 total 1623.34 last 22.73) - open/close 40415 (time 1709788462.69 total 1633.37 last 20.93) - open/close 40622 (time 1709788472.71 total 1643.40 last 20.65) - open/close 40838 (time 1709788482.77 total 1653.45 last 21.49) - open/close 41055 (time 1709788492.77 total 1663.46 last 21.68) - open/close 41283 (time 1709788502.79 total 1673.48 last 22.76) - open/close 41524 (time 1709788512.80 total 1683.49 last 24.08) - open/close 41757 (time 1709788522.84 total 1693.53 last 23.21) - open/close 41990 (time 1709788532.89 total 1703.58 last 23.18) - open/close 42222 (time 1709788542.92 total 1713.61 last 23.13) - open/close 42449 (time 1709788552.95 total 1723.64 last 22.63) - open/close 42680 (time 1709788562.98 total 1733.66 last 23.05) - open/close 42912 (time 1709788573.00 total 1743.68 last 23.15) - open/close 43107 (time 1709788583.02 total 1753.70 last 19.45) - open/close 43323 (time 1709788593.05 total 1763.73 last 21.54) - open/close 43548 (time 1709788603.09 total 1773.78 last 22.40) - open/close 43780 (time 1709788613.13 total 1783.82 last 23.11) - open/close 44025 (time 1709788623.17 total 1793.85 last 24.42) - open/close 44267 (time 1709788633.17 total 1803.86 last 24.18) - open/close 44493 (time 1709788643.19 total 1813.88 last 22.55) - open/close 44715 (time 1709788653.23 total 1823.91 last 22.12) - open/close 44936 (time 1709788663.26 total 1833.94 last 22.02) - open/close 45148 (time 1709788673.28 total 1843.96 last 21.16) - open/close 45348 (time 1709788683.30 total 1853.98 last 19.97) - open/close 45547 (time 1709788693.32 total 1864.01 last 19.84) - open/close 45750 (time 1709788703.35 total 1874.03 last 20.25) - open/close 45970 (time 1709788713.39 total 1884.07 last 21.91) - open/close 46195 (time 1709788723.41 total 1894.10 last 22.44) - open/close 46413 (time 1709788733.44 total 1904.12 last 21.75) - open/close 46618 (time 1709788743.46 total 1914.15 last 20.45) - open/close 46797 (time 1709788753.50 total 1924.18 last 17.83) - open/close 46984 (time 1709788763.55 total 1934.23 last 18.61) - open/close 47161 (time 1709788773.60 total 1944.28 last 17.60) - open/close 47340 (time 1709788783.62 total 1954.31 last 17.86) - open/close 47539 (time 1709788793.64 total 1964.33 last 19.86) - open/close 47745 (time 1709788803.67 total 1974.35 last 20.54) - open/close 47960 (time 1709788813.68 total 1984.37 last 21.47) - open/close 48169 (time 1709788823.69 total 1994.37 last 20.89) - open/close 48383 (time 1709788833.72 total 2004.40 last 21.35) - open/close 48594 (time 1709788843.72 total 2014.40 last 21.09) - open/close 48805 (time 1709788853.73 total 2024.41 last 21.08) - open/close 49004 (time 1709788863.74 total 2034.42 last 19.89) - open/close 49221 (time 1709788873.78 total 2044.46 last 21.62) - open/close 49431 (time 1709788883.80 total 2054.48 last 20.96) - open/close 49641 (time 1709788893.83 total 2064.51 last 20.93) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 2067.04 seconds: 24.04 ops/second - unlinked 0 (time 1709788897 ; total 0 ; last 0) - unlinked 10000 (time 1709789454 ; total 557 ; last 557) - unlinked 20000 (time 1709789916 ; total 1019 ; last 462) - unlinked 30000 (time 1709790234 ; total 1337 ; last 318) - unlinked 40000 (time 1709790368 ; total 1471 ; last 134) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1515 seconds: 32.803959 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:)