-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:22:43 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg231-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg231-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg231-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg231-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg231-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg231-server oleg231-server: oleg231-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing load_modules_local oleg231-server: Loading modules from /home/green/git/lustre-release/lustre oleg231-server: detected 4 online CPUs by sysfs oleg231-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg231-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg231-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg231-server: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg231-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg231-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg231-server stop mds service on oleg231-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg231-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:23:44 (1709785424) start mds service on oleg231-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg231-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0000 oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg231-client.virtnet: -o user_xattr,flock oleg231-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h23m58s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800aa374800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:24:41.000000000 -0500 Modify: 2024-03-06 23:24:41.000000000 -0500 Change: 2024-03-06 23:24:41.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 54 lines, 54 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg231-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg231-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg231-server stop mds service on oleg231-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg231-server unloading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing unload_modules_local modules unloaded. oleg231-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg231-server: Is the ldiskfs module available? oleg231-server: oleg231-server: tunefs.lustre FATAL: failed to write local files oleg231-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg231-client: oleg231-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg231-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg231-server: Is the ldiskfs module available? oleg231-server: oleg231-server: tunefs.lustre FATAL: failed to write local files oleg231-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg231-client: oleg231-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.131@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.131@tcp sys.timeout=20 oleg231-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg231-server: Is the ldiskfs module available? oleg231-server: oleg231-server: tunefs.lustre FATAL: failed to write local files oleg231-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg231-client: oleg231-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.131@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.131@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg231-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg231-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 oleg231-server: oleg231-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing load_modules_local oleg231-server: Loading modules from /home/green/git/lustre-release/lustre oleg231-server: detected 4 online CPUs by sysfs oleg231-server: Force libcfs to create 2 CPU partitions oleg231-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg231-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg231-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg231-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg231-server: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg231-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg231-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg231-server stop mds service on oleg231-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg231-server PASS 35b (111s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:25:36 (1709785536) start mds service on oleg231-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg231-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0000 oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg231-client.virtnet: -o user_xattr,flock oleg231-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg231-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg231-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg231-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg231-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg231-client.virtnet: -o user_xattr,flock oleg231-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg231-client.virtnet /mnt/lustre (opts:) stop mds service on oleg231-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg231-server 000000 2 000008 oleg231-server: 8+0 records in oleg231-server: 8+0 records out oleg231-server: 4096 bytes (4.1 kB) copied, 0.000335112 s, 12.2 MB/s start mds service on oleg231-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg231-client.virtnet: -o user_xattr,flock oleg231-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg231-client.virtnet /mnt/lustre (opts:) stop mds service on oleg231-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg231-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg231-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg231-server stop mds service on oleg231-server unloading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg231-client: oleg231-client: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-client: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 PASS 38 (58s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg231-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg231-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg231-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 oleg231-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg231-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:26:35 (1709785595) Stopping clients: oleg231-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg231-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 oleg231-server: oleg231-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing load_modules_local oleg231-server: Loading modules from /home/green/git/lustre-release/lustre oleg231-server: detected 4 online CPUs by sysfs oleg231-server: Force libcfs to create 2 CPU partitions oleg231-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg231-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg231-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg231-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg231-client.virtnet: -o user_xattr,flock oleg231-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg231-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec stop ost2 service on oleg231-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg231-server umount lustre on /mnt/lustre..... Stopping client oleg231-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg231-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg231-server stop mds service on oleg231-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg231-server unloading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing unload_modules_local modules unloaded. oleg231-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg231-server: Is the ldiskfs module available? oleg231-server: oleg231-server: tunefs.lustre FATAL: failed to write local files oleg231-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg231-client: oleg231-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg231-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg231-server: Is the ldiskfs module available? oleg231-server: oleg231-server: tunefs.lustre FATAL: failed to write local files oleg231-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg231-client: oleg231-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.131@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.131@tcp sys.timeout=20 oleg231-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg231-server: Is the ldiskfs module available? oleg231-server: oleg231-server: tunefs.lustre FATAL: failed to write local files oleg231-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg231-client: oleg231-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.131@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.131@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg231-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg231-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 oleg231-server: oleg231-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing load_modules_local oleg231-server: Loading modules from /home/green/git/lustre-release/lustre oleg231-server: detected 4 online CPUs by sysfs oleg231-server: Force libcfs to create 2 CPU partitions oleg231-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg231-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg231-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg231-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg231-server: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg231-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg231-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg231-server stop mds service on oleg231-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg231-server PASS 51 (123s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:28:40 (1709785720) start mds service on oleg231-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg231-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0000 oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg231-client.virtnet: -o user_xattr,flock oleg231-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 461 (time 1709785745.93 total 10.00 last 46.09) - open/close 1269 (time 1709785756.64 total 20.71 last 75.48) - open/close 1813 (time 1709785766.99 total 31.06 last 52.56) - open/close 2488 (time 1709785777.55 total 41.62 last 63.93) - open/close 3094 (time 1709785788.18 total 52.25 last 56.99) - open/close 3684 (time 1709785798.79 total 62.86 last 55.59) - open/close 4297 (time 1709785809.58 total 73.65 last 56.83) - open/close 4866 (time 1709785820.46 total 84.52 last 52.32) - open/close 5388 (time 1709785831.17 total 95.24 last 48.71) - open/close 5708 (time 1709785841.20 total 105.27 last 31.89) - open/close 6019 (time 1709785851.32 total 115.39 last 30.76) - open/close 6464 (time 1709785861.98 total 126.05 last 41.72) - open/close 6972 (time 1709785872.95 total 137.02 last 46.30) - open/close 7425 (time 1709785883.76 total 147.83 last 41.91) - open/close 7724 (time 1709785895.97 total 160.04 last 24.49) - open/close 8204 (time 1709785906.09 total 170.16 last 47.41) - open/close 8561 (time 1709785917.71 total 181.77 last 30.75) - open/close 8932 (time 1709785927.71 total 191.78 last 37.08) - open/close 9135 (time 1709785937.73 total 201.80 last 20.25) - open/close 9396 (time 1709785948.18 total 212.25 last 24.99) - open/close 9805 (time 1709785958.18 total 222.25 last 40.87) - open/close 10000 (time 1709785962.20 total 226.26 last 48.62) - open/close 10354 (time 1709785973.04 total 237.11 last 32.66) - open/close 11398 (time 1709785983.04 total 247.11 last 104.37) - open/close 13014 (time 1709785993.04 total 257.11 last 161.52) - open/close 14585 (time 1709786003.04 total 267.11 last 157.09) - open/close 16033 (time 1709786013.05 total 277.12 last 144.78) - open/close 17341 (time 1709786023.05 total 287.12 last 130.77) - open/close 18512 (time 1709786033.06 total 297.13 last 117.00) - open/close 19497 (time 1709786043.06 total 307.13 last 98.49) - open/close 20000 (time 1709786047.22 total 311.29 last 120.84) - open/close 21097 (time 1709786057.23 total 321.30 last 109.60) - open/close 22014 (time 1709786067.23 total 331.30 last 91.70) - open/close 22859 (time 1709786077.24 total 341.30 last 84.45) - open/close 23742 (time 1709786087.24 total 351.31 last 88.27) - open/close 24624 (time 1709786097.25 total 361.32 last 88.11) - open/close 25444 (time 1709786107.25 total 371.32 last 82.00) - open/close 26310 (time 1709786117.26 total 381.33 last 86.52) - open/close 27139 (time 1709786127.26 total 391.33 last 82.87) - open/close 27905 (time 1709786137.26 total 401.33 last 76.59) - open/close 28675 (time 1709786147.27 total 411.33 last 76.98) - open/close 29438 (time 1709786157.27 total 421.34 last 76.28) - open/close 30000 (time 1709786164.72 total 428.79 last 75.41) - open/close 30731 (time 1709786174.73 total 438.80 last 73.06) - open/close 31428 (time 1709786184.73 total 448.80 last 69.67) - open/close 32082 (time 1709786194.74 total 458.81 last 65.30) - open/close 32775 (time 1709786204.75 total 468.82 last 69.26) - open/close 33392 (time 1709786214.76 total 478.83 last 61.65) - open/close 34009 (time 1709786224.76 total 488.83 last 61.69) - open/close 34609 (time 1709786234.77 total 498.84 last 59.93) - open/close 35150 (time 1709786244.77 total 508.84 last 54.09) - open/close 35692 (time 1709786254.79 total 518.86 last 54.09) - open/close 36235 (time 1709786264.79 total 528.86 last 54.29) - open/close 36840 (time 1709786274.81 total 538.88 last 60.39) - open/close 37372 (time 1709786284.82 total 548.89 last 53.16) - open/close 37750 (time 1709786294.83 total 558.90 last 37.76) - open/close 38174 (time 1709786304.83 total 568.90 last 42.39) - open/close 38595 (time 1709786314.85 total 578.92 last 42.03) - open/close 39040 (time 1709786324.86 total 588.93 last 44.46) - open/close 39434 (time 1709786334.87 total 598.94 last 39.35) - open/close 39848 (time 1709786344.89 total 608.96 last 41.33) - open/close 40000 (time 1709786348.12 total 612.19 last 47.06) - open/close 40392 (time 1709786358.14 total 622.21 last 39.12) - open/close 40823 (time 1709786368.16 total 632.23 last 43.04) - open/close 41259 (time 1709786378.17 total 642.24 last 43.53) - open/close 41630 (time 1709786388.17 total 652.24 last 37.10) - open/close 41952 (time 1709786398.18 total 662.25 last 32.17) - open/close 42405 (time 1709786408.20 total 672.27 last 45.23) - open/close 42868 (time 1709786418.21 total 682.28 last 46.25) - open/close 43297 (time 1709786428.23 total 692.29 last 42.82) - open/close 43736 (time 1709786438.24 total 702.31 last 43.82) - open/close 44096 (time 1709786448.25 total 712.32 last 35.96) - open/close 44536 (time 1709786458.26 total 722.33 last 43.97) - open/close 45004 (time 1709786468.27 total 732.34 last 46.76) - open/close 45469 (time 1709786478.28 total 742.35 last 46.45) - open/close 45901 (time 1709786488.29 total 752.36 last 43.16) - open/close 46308 (time 1709786498.29 total 762.36 last 40.69) - open/close 46705 (time 1709786508.31 total 772.38 last 39.61) - open/close 47068 (time 1709786518.32 total 782.39 last 36.28) - open/close 47440 (time 1709786528.33 total 792.40 last 37.15) - open/close 47846 (time 1709786538.35 total 802.42 last 40.55) - open/close 48192 (time 1709786548.37 total 812.43 last 34.53) - open/close 48476 (time 1709786558.39 total 822.46 last 28.33) - open/close 48841 (time 1709786568.40 total 832.47 last 36.47) - open/close 49174 (time 1709786578.40 total 842.47 last 33.29) - open/close 49479 (time 1709786588.41 total 852.47 last 30.49) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 860.75 seconds: 57.74 ops/second - unlinked 0 (time 1709786598 ; total 0 ; last 0) - unlinked 10000 (time 1709787031 ; total 433 ; last 433) - unlinked 20000 (time 1709787458 ; total 860 ; last 427) - unlinked 30000 (time 1709787780 ; total 1182 ; last 322) - unlinked 40000 (time 1709787958 ; total 1360 ; last 178) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1419 seconds: 35.023254 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg231-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg231-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg231-server stop mds service on oleg231-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg231-server unloading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg231-client: oleg231-client: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-client: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 PASS 101a (2308s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg231-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg231-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg231-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 oleg231-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg231-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:07:10 (1709788030) start mds service on oleg231-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing load_modules_local oleg231-server: Loading modules from /home/green/git/lustre-release/lustre oleg231-server: detected 4 online CPUs by sysfs oleg231-server: Force libcfs to create 2 CPU partitions oleg231-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg231-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg231-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0000 oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg231-client.virtnet: -o user_xattr,flock oleg231-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg231-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg231-server start ost1 service on oleg231-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0000 oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec oleg231-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012a5bd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a5bd800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg231-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg231-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg231-server stop mds service on oleg231-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg231-server unloading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg231-client: oleg231-client: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-client: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 PASS 101b (86s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg231-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg231-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg231-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg231-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 oleg231-server: oleg231-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing load_modules_local oleg231-server: Loading modules from /home/green/git/lustre-release/lustre oleg231-server: detected 4 online CPUs by sysfs oleg231-server: Force libcfs to create 2 CPU partitions oleg231-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg231-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 2773 sec ========== 00:08:56 (1709788136) === conf-sanity: start cleanup 00:08:56 (1709788136) === === conf-sanity: finish cleanup 00:08:56 (1709788136) ===