-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:37:22 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg145-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg145-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg145-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg145-server oleg145-server: oleg145-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing load_modules_local oleg145-server: Loading modules from /home/green/git/lustre-release/lustre oleg145-server: detected 4 online CPUs by sysfs oleg145-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg145-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg145-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg145-server: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg145-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server stop mds service on oleg145-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg145-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:38:23 (1709786303) start mds service on oleg145-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg145-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg145-client.virtnet: -o user_xattr,flock oleg145-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h38m39s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff88012a650800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:39:22.000000000 -0500 Modify: 2024-03-06 23:39:22.000000000 -0500 Change: 2024-03-06 23:39:22.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 72 lines, 72 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg145-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server stop mds service on oleg145-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg145-server unloading modules on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing unload_modules_local modules unloaded. oleg145-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg145-server: Is the ldiskfs module available? oleg145-server: oleg145-server: tunefs.lustre FATAL: failed to write local files oleg145-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg145-client: oleg145-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg145-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg145-server: Is the ldiskfs module available? oleg145-server: oleg145-server: tunefs.lustre FATAL: failed to write local files oleg145-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg145-client: oleg145-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.145@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.145@tcp sys.timeout=20 oleg145-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg145-server: Is the ldiskfs module available? oleg145-server: oleg145-server: tunefs.lustre FATAL: failed to write local files oleg145-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg145-client: oleg145-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.145@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.145@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg145-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg145-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2 oleg145-server: oleg145-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing load_modules_local oleg145-server: Loading modules from /home/green/git/lustre-release/lustre oleg145-server: detected 4 online CPUs by sysfs oleg145-server: Force libcfs to create 2 CPU partitions oleg145-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg145-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg145-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg145-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg145-server: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg145-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server stop mds service on oleg145-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg145-server PASS 35b (113s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:40:18 (1709786418) start mds service on oleg145-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg145-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg145-client.virtnet: -o user_xattr,flock oleg145-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg145-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg145-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg145-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg145-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg145-client.virtnet: -o user_xattr,flock oleg145-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg145-client.virtnet /mnt/lustre (opts:) stop mds service on oleg145-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg145-server 000000 2 000008 oleg145-server: 8+0 records in oleg145-server: 8+0 records out oleg145-server: 4096 bytes (4.1 kB) copied, 0.000361183 s, 11.3 MB/s start mds service on oleg145-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg145-client.virtnet: -o user_xattr,flock oleg145-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg145-client.virtnet /mnt/lustre (opts:) stop mds service on oleg145-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg145-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server stop mds service on oleg145-server unloading modules on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg145-client: oleg145-client: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-client: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2 PASS 38 (59s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg145-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg145-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg145-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2 oleg145-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg145-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:41:19 (1709786479) Stopping clients: oleg145-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg145-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2 oleg145-server: oleg145-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing load_modules_local oleg145-server: Loading modules from /home/green/git/lustre-release/lustre oleg145-server: detected 4 online CPUs by sysfs oleg145-server: Force libcfs to create 2 CPU partitions oleg145-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg145-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg145-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg145-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg145-client.virtnet: -o user_xattr,flock oleg145-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg145-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec stop ost2 service on oleg145-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg145-server umount lustre on /mnt/lustre..... Stopping client oleg145-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server stop mds service on oleg145-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg145-server unloading modules on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing unload_modules_local modules unloaded. oleg145-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg145-server: Is the ldiskfs module available? oleg145-server: oleg145-server: tunefs.lustre FATAL: failed to write local files oleg145-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg145-client: oleg145-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg145-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg145-server: Is the ldiskfs module available? oleg145-server: oleg145-server: tunefs.lustre FATAL: failed to write local files oleg145-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg145-client: oleg145-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.145@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.145@tcp sys.timeout=20 oleg145-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg145-server: Is the ldiskfs module available? oleg145-server: oleg145-server: tunefs.lustre FATAL: failed to write local files oleg145-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg145-client: oleg145-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.145@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.145@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg145-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg145-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2 oleg145-server: oleg145-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing load_modules_local oleg145-server: Loading modules from /home/green/git/lustre-release/lustre oleg145-server: detected 4 online CPUs by sysfs oleg145-server: Force libcfs to create 2 CPU partitions oleg145-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg145-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg145-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg145-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg145-server: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg145-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server stop mds service on oleg145-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg145-server PASS 51 (140s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:43:42 (1709786622) start mds service on oleg145-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg145-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg145-client.virtnet: -o user_xattr,flock oleg145-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 167 (time 1709786653.99 total 10.11 last 16.53) - open/close 548 (time 1709786664.73 total 20.85 last 35.46) - open/close 857 (time 1709786674.91 total 31.03 last 30.36) - open/close 1124 (time 1709786685.56 total 41.68 last 25.06) - open/close 1339 (time 1709786695.62 total 51.73 last 21.39) - open/close 1519 (time 1709786706.02 total 62.14 last 17.30) - open/close 1794 (time 1709786717.23 total 73.34 last 24.54) - open/close 2049 (time 1709786728.02 total 84.13 last 23.63) - open/close 2301 (time 1709786738.03 total 94.14 last 25.18) - open/close 2611 (time 1709786748.85 total 104.97 last 28.63) - open/close 2820 (time 1709786759.36 total 115.48 last 19.90) - open/close 3008 (time 1709786769.80 total 125.92 last 18.01) - open/close 3164 (time 1709786780.72 total 136.84 last 14.28) - open/close 3367 (time 1709786790.86 total 146.98 last 20.02) - open/close 3674 (time 1709786800.88 total 157.00 last 30.63) - open/close 3933 (time 1709786810.89 total 167.01 last 25.88) - open/close 4173 (time 1709786821.45 total 177.56 last 22.74) - open/close 4379 (time 1709786832.12 total 188.24 last 19.29) - open/close 4605 (time 1709786842.41 total 198.53 last 21.96) - open/close 4780 (time 1709786852.42 total 208.54 last 17.49) - open/close 4956 (time 1709786862.43 total 218.55 last 17.58) - open/close 5232 (time 1709786872.44 total 228.56 last 27.56) - open/close 5459 (time 1709786882.72 total 238.84 last 22.08) - open/close 5589 (time 1709786893.44 total 249.56 last 12.13) - open/close 5766 (time 1709786904.38 total 260.50 last 16.18) - open/close 5957 (time 1709786914.64 total 270.76 last 18.61) - open/close 6218 (time 1709786925.62 total 281.73 last 23.79) - open/close 6363 (time 1709786935.62 total 291.74 last 14.50) - open/close 6768 (time 1709786946.42 total 302.54 last 37.50) - open/close 6967 (time 1709786958.52 total 314.64 last 16.44) - open/close 7210 (time 1709786968.65 total 324.77 last 23.98) - open/close 7412 (time 1709786978.66 total 334.77 last 20.19) - open/close 7658 (time 1709786989.43 total 345.55 last 22.83) - open/close 7800 (time 1709786999.44 total 355.56 last 14.19) - open/close 7975 (time 1709787009.76 total 365.88 last 16.95) - open/close 8087 (time 1709787019.78 total 375.90 last 11.18) - open/close 8277 (time 1709787029.78 total 385.90 last 18.99) - open/close 8476 (time 1709787039.89 total 396.00 last 19.70) - open/close 8673 (time 1709787050.13 total 406.25 last 19.23) - open/close 8918 (time 1709787061.02 total 417.14 last 22.50) - open/close 9044 (time 1709787071.52 total 427.64 last 12.00) - open/close 9199 (time 1709787082.94 total 439.05 last 13.58) - open/close 9334 (time 1709787093.48 total 449.59 last 12.81) - open/close 9491 (time 1709787103.50 total 459.62 last 15.67) - open/close 9661 (time 1709787113.52 total 469.64 last 16.96) - open/close 10000 (time 1709787119.27 total 475.39 last 58.97) - open/close 10601 (time 1709787129.27 total 485.39 last 60.09) - open/close 11144 (time 1709787139.28 total 495.39 last 54.27) - open/close 11679 (time 1709787149.28 total 505.40 last 53.49) - open/close 12195 (time 1709787159.29 total 515.40 last 51.56) - open/close 12705 (time 1709787169.31 total 525.43 last 50.89) - open/close 13217 (time 1709787179.32 total 535.44 last 51.12) - open/close 13752 (time 1709787189.34 total 545.46 last 53.39) - open/close 14180 (time 1709787199.37 total 555.48 last 42.71) - open/close 14611 (time 1709787209.38 total 565.49 last 43.06) - open/close 15046 (time 1709787219.42 total 575.53 last 43.32) - open/close 15459 (time 1709787229.44 total 585.56 last 41.19) - open/close 15881 (time 1709787239.44 total 595.56 last 42.20) - open/close 16329 (time 1709787249.45 total 605.56 last 44.78) - open/close 16797 (time 1709787259.45 total 615.57 last 46.79) - open/close 17265 (time 1709787269.46 total 625.58 last 46.73) - open/close 17723 (time 1709787279.47 total 635.59 last 45.77) - open/close 18135 (time 1709787289.48 total 645.60 last 41.15) - open/close 18577 (time 1709787299.49 total 655.61 last 44.18) - open/close 18954 (time 1709787309.51 total 665.62 last 37.64) - open/close 19341 (time 1709787319.52 total 675.63 last 38.65) - open/close 19736 (time 1709787329.54 total 685.65 last 39.43) - open/close 20000 (time 1709787336.07 total 692.19 last 40.37) - open/close 20336 (time 1709787346.09 total 702.21 last 33.54) - open/close 20697 (time 1709787356.11 total 712.22 last 36.05) - open/close 21026 (time 1709787366.13 total 722.25 last 32.82) - open/close 21339 (time 1709787376.15 total 732.27 last 31.24) - open/close 21675 (time 1709787386.16 total 742.28 last 33.58) - open/close 21985 (time 1709787396.16 total 752.28 last 30.99) - open/close 22309 (time 1709787406.19 total 762.31 last 32.31) - open/close 22658 (time 1709787416.22 total 772.34 last 34.79) - open/close 23006 (time 1709787426.22 total 782.34 last 34.79) - open/close 23302 (time 1709787436.25 total 792.37 last 29.52) - open/close 23566 (time 1709787446.25 total 802.37 last 26.39) - open/close 23857 (time 1709787456.26 total 812.38 last 29.07) - open/close 24157 (time 1709787466.28 total 822.40 last 29.94) - open/close 24405 (time 1709787476.31 total 832.43 last 24.74) - open/close 24676 (time 1709787486.35 total 842.46 last 27.00) - open/close 24965 (time 1709787496.37 total 852.49 last 28.83) - open/close 25238 (time 1709787506.38 total 862.49 last 27.28) - open/close 25500 (time 1709787516.38 total 872.50 last 26.18) - open/close 25805 (time 1709787526.40 total 882.52 last 30.45) - open/close 26113 (time 1709787536.43 total 892.55 last 30.70) - open/close 26376 (time 1709787546.45 total 902.57 last 26.24) - open/close 26666 (time 1709787556.46 total 912.57 last 28.99) - open/close 26909 (time 1709787566.47 total 922.58 last 24.28) - open/close 27144 (time 1709787576.49 total 932.60 last 23.45) - open/close 27419 (time 1709787586.51 total 942.62 last 27.45) - open/close 27692 (time 1709787596.52 total 952.63 last 27.27) - open/close 27952 (time 1709787606.53 total 962.65 last 25.95) - open/close 28237 (time 1709787616.58 total 972.70 last 28.36) - open/close 28520 (time 1709787626.58 total 982.70 last 28.30) - open/close 28781 (time 1709787636.59 total 992.71 last 26.07) - open/close 29057 (time 1709787646.62 total 1002.74 last 27.52) - open/close 29337 (time 1709787656.64 total 1012.75 last 27.97) - open/close 29661 (time 1709787666.66 total 1022.78 last 32.32) - open/close 29944 (time 1709787676.68 total 1032.79 last 28.25) - open/close 30216 (time 1709787686.70 total 1042.82 last 27.12) - open/close 30455 (time 1709787696.72 total 1052.83 last 23.87) - open/close 30742 (time 1709787706.73 total 1062.85 last 28.66) - open/close 31018 (time 1709787716.76 total 1072.87 last 27.53) - open/close 31309 (time 1709787726.78 total 1082.89 last 29.05) - open/close 31604 (time 1709787736.79 total 1092.91 last 29.44) - open/close 31838 (time 1709787756.87 total 1112.99 last 11.66) - open/close 32105 (time 1709787766.88 total 1122.99 last 26.68) - open/close 32216 (time 1709787781.96 total 1138.08 last 7.36) - open/close 32466 (time 1709787791.98 total 1148.10 last 24.95) - open/close 32713 (time 1709787801.99 total 1158.10 last 24.68) - open/close 32999 (time 1709787812.01 total 1168.12 last 28.55) - open/close 33267 (time 1709787822.02 total 1178.14 last 26.76) - open/close 33510 (time 1709787832.03 total 1188.15 last 24.26) - open/close 33745 (time 1709787842.06 total 1198.18 last 23.45) - open/close 33985 (time 1709787852.10 total 1208.22 last 23.89) - open/close 34238 (time 1709787862.11 total 1218.23 last 25.28) - open/close 34496 (time 1709787872.15 total 1228.27 last 25.70) - open/close 34748 (time 1709787882.16 total 1238.28 last 25.17) - open/close 34988 (time 1709787892.17 total 1248.29 last 23.98) - open/close 35237 (time 1709787902.20 total 1258.32 last 24.82) - open/close 35479 (time 1709787912.22 total 1268.34 last 24.16) - open/close 35705 (time 1709787922.26 total 1278.37 last 22.52) - open/close 35936 (time 1709787932.27 total 1288.39 last 23.08) - open/close 36160 (time 1709787942.30 total 1298.41 last 22.34) - open/close 36400 (time 1709787952.33 total 1308.44 last 23.93) - open/close 36670 (time 1709787962.35 total 1318.47 last 26.94) - open/close 36915 (time 1709787972.39 total 1328.51 last 24.39) - open/close 37181 (time 1709787982.40 total 1338.51 last 26.59) - open/close 37466 (time 1709787992.41 total 1348.52 last 28.47) - open/close 37755 (time 1709788002.41 total 1358.53 last 28.89) - open/close 38032 (time 1709788012.44 total 1368.56 last 27.61) - open/close 38293 (time 1709788022.47 total 1378.59 last 26.03) - open/close 38585 (time 1709788032.50 total 1388.62 last 29.12) - open/close 38854 (time 1709788042.52 total 1398.64 last 26.84) - open/close 39134 (time 1709788052.52 total 1408.64 last 28.00) - open/close 39404 (time 1709788062.54 total 1418.66 last 26.96) - open/close 39675 (time 1709788072.54 total 1428.66 last 27.09) - open/close 39943 (time 1709788082.58 total 1438.69 last 26.71) - open/close 40000 (time 1709788084.74 total 1440.86 last 26.30) - open/close 40267 (time 1709788094.79 total 1450.90 last 26.59) - open/close 40531 (time 1709788104.83 total 1460.94 last 26.30) - open/close 40782 (time 1709788114.86 total 1470.97 last 25.02) - open/close 40943 (time 1709788132.06 total 1488.17 last 9.36) - open/close 41197 (time 1709788142.13 total 1498.25 last 25.21) - open/close 41434 (time 1709788152.13 total 1508.25 last 23.70) - open/close 41674 (time 1709788162.17 total 1518.28 last 23.92) - open/close 41896 (time 1709788172.18 total 1528.30 last 22.17) - open/close 42136 (time 1709788182.22 total 1538.34 last 23.90) - open/close 42369 (time 1709788192.27 total 1548.39 last 23.18) - open/close 42617 (time 1709788202.28 total 1558.39 last 24.80) - open/close 42873 (time 1709788212.28 total 1568.40 last 25.59) - open/close 43126 (time 1709788222.32 total 1578.43 last 25.20) - open/close 43402 (time 1709788232.34 total 1588.45 last 27.55) - open/close 43663 (time 1709788242.34 total 1598.46 last 26.09) - open/close 43909 (time 1709788252.37 total 1608.49 last 24.52) - open/close 44156 (time 1709788262.39 total 1618.51 last 24.65) - open/close 44394 (time 1709788272.41 total 1628.53 last 23.76) - open/close 44639 (time 1709788282.45 total 1638.56 last 24.41) - open/close 44891 (time 1709788292.48 total 1648.60 last 25.11) - open/close 45137 (time 1709788302.49 total 1658.61 last 24.58) - open/close 45382 (time 1709788312.49 total 1668.61 last 24.50) - open/close 45634 (time 1709788322.52 total 1678.64 last 25.13) - open/close 45880 (time 1709788332.54 total 1688.65 last 24.56) - open/close 46126 (time 1709788342.57 total 1698.68 last 24.53) - open/close 46377 (time 1709788352.59 total 1708.70 last 25.04) - open/close 46616 (time 1709788362.63 total 1718.75 last 23.80) - open/close 46864 (time 1709788372.66 total 1728.78 last 24.71) - open/close 47108 (time 1709788382.68 total 1738.80 last 24.36) - open/close 47345 (time 1709788392.71 total 1748.83 last 23.63) - open/close 47586 (time 1709788402.75 total 1758.86 last 24.02) - open/close 47833 (time 1709788412.78 total 1768.89 last 24.63) - open/close 48064 (time 1709788422.78 total 1778.90 last 23.09) - open/close 48296 (time 1709788432.82 total 1788.94 last 23.11) - open/close 48520 (time 1709788442.84 total 1798.95 last 22.36) - open/close 48751 (time 1709788452.86 total 1808.98 last 23.05) - open/close 48978 (time 1709788462.87 total 1818.99 last 22.68) - open/close 49188 (time 1709788472.89 total 1829.01 last 20.96) - open/close 49408 (time 1709788482.89 total 1839.01 last 22.00) - open/close 49634 (time 1709788492.93 total 1849.05 last 22.51) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1851.81 seconds: 26.84 ops/second - unlinked 0 (time 1709788497 ; total 0 ; last 0) - unlinked 10000 (time 1709788984 ; total 487 ; last 487) - unlinked 20000 (time 1709789416 ; total 919 ; last 432) - unlinked 30000 (time 1709789732 ; total 1235 ; last 316) - unlinked 40000 (time 1709789915 ; total 1418 ; last 183) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1480 seconds: 33.579731 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg145-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server stop mds service on oleg145-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg145-server unloading modules on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg145-client: oleg145-client: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-client: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2 PASS 101a (3371s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg145-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg145-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg145-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2 oleg145-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg145-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:39:56 (1709789996) start mds service on oleg145-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing load_modules_local oleg145-server: Loading modules from /home/green/git/lustre-release/lustre oleg145-server: detected 4 online CPUs by sysfs oleg145-server: Force libcfs to create 2 CPU partitions oleg145-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg145-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg145-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg145-client.virtnet: -o user_xattr,flock oleg145-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server start ost1 service on oleg145-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec oleg145-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012aaf9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012aaf9000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg145-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server stop mds service on oleg145-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg145-server unloading modules on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg145-client: oleg145-client: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-client: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2 PASS 101b (90s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg145-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg145-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg145-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg145-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2 oleg145-server: oleg145-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing load_modules_local oleg145-server: Loading modules from /home/green/git/lustre-release/lustre oleg145-server: detected 4 online CPUs by sysfs oleg145-server: Force libcfs to create 2 CPU partitions oleg145-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg145-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3865 sec ========== 00:41:47 (1709790107) === conf-sanity: start cleanup 00:41:48 (1709790108) === === conf-sanity: finish cleanup 00:41:48 (1709790108) ===