-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:40:40 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg217-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg217-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg217-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg217-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg217-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg217-server oleg217-server: oleg217-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: detected 4 online CPUs by sysfs oleg217-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg217-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg217-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg217-server: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg217-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg217-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg217-server stop mds service on oleg217-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg217-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:41:44 (1709786504) start mds service on oleg217-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg217-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0000 oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg217-client.virtnet: -o user_xattr,flock oleg217-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h42m01s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800b5811000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:42:45.000000000 -0500 Modify: 2024-03-06 23:42:45.000000000 -0500 Change: 2024-03-06 23:42:45.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 76 lines, 76 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg217-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg217-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg217-server stop mds service on oleg217-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg217-server unloading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing unload_modules_local modules unloaded. oleg217-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg217-server: Is the ldiskfs module available? oleg217-server: oleg217-server: tunefs.lustre FATAL: failed to write local files oleg217-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg217-client: oleg217-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg217-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg217-server: Is the ldiskfs module available? oleg217-server: oleg217-server: tunefs.lustre FATAL: failed to write local files oleg217-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg217-client: oleg217-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.117@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.117@tcp sys.timeout=20 oleg217-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg217-server: Is the ldiskfs module available? oleg217-server: oleg217-server: tunefs.lustre FATAL: failed to write local files oleg217-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg217-client: oleg217-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.117@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.117@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg217-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg217-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 oleg217-server: oleg217-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: detected 4 online CPUs by sysfs oleg217-server: Force libcfs to create 2 CPU partitions oleg217-server: libkmod: kmod_module_get_refcnt: could not read integer from '/sys/module/pcc_cpufreq/refcnt': 'No such device' oleg217-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg217-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg217-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg217-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg217-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg217-server: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg217-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg217-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg217-server stop mds service on oleg217-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg217-server PASS 35b (133s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:44:00 (1709786640) start mds service on oleg217-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg217-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0000 oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg217-client.virtnet: -o user_xattr,flock oleg217-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg217-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg217-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg217-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg217-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg217-client.virtnet: -o user_xattr,flock oleg217-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg217-client.virtnet /mnt/lustre (opts:) stop mds service on oleg217-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg217-server 000000 2 000008 oleg217-server: 8+0 records in oleg217-server: 8+0 records out oleg217-server: 4096 bytes (4.1 kB) copied, 0.000676054 s, 6.1 MB/s start mds service on oleg217-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg217-client.virtnet: -o user_xattr,flock oleg217-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg217-client.virtnet /mnt/lustre (opts:) stop mds service on oleg217-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg217-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg217-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg217-server stop mds service on oleg217-server unloading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg217-client: oleg217-client: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-client: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 PASS 38 (83s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg217-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg217-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg217-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 oleg217-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg217-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:45:26 (1709786726) Stopping clients: oleg217-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg217-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 oleg217-server: oleg217-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: detected 4 online CPUs by sysfs oleg217-server: Force libcfs to create 2 CPU partitions oleg217-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg217-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg217-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg217-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg217-client.virtnet: -o user_xattr,flock oleg217-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg217-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec stop ost2 service on oleg217-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg217-server umount lustre on /mnt/lustre..... Stopping client oleg217-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg217-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg217-server stop mds service on oleg217-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg217-server unloading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing unload_modules_local modules unloaded. oleg217-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg217-server: Is the ldiskfs module available? oleg217-server: oleg217-server: tunefs.lustre FATAL: failed to write local files oleg217-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg217-client: oleg217-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg217-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg217-server: Is the ldiskfs module available? oleg217-server: oleg217-server: tunefs.lustre FATAL: failed to write local files oleg217-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg217-client: oleg217-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.117@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.117@tcp sys.timeout=20 oleg217-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg217-server: Is the ldiskfs module available? oleg217-server: oleg217-server: tunefs.lustre FATAL: failed to write local files oleg217-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg217-client: oleg217-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.117@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.117@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg217-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg217-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 oleg217-server: oleg217-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: detected 4 online CPUs by sysfs oleg217-server: Force libcfs to create 2 CPU partitions oleg217-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg217-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg217-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg217-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg217-server: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg217-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg217-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg217-server stop mds service on oleg217-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg217-server PASS 51 (181s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:48:30 (1709786910) start mds service on oleg217-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg217-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0000 oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg217-client.virtnet: -o user_xattr,flock oleg217-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 160 (time 1709786943.38 total 10.98 last 14.57) - open/close 372 (time 1709786953.84 total 21.43 last 20.27) - open/close 601 (time 1709786964.81 total 32.40 last 20.88) - open/close 831 (time 1709786974.81 total 42.41 last 22.99) - open/close 1078 (time 1709786984.81 total 52.41 last 24.69) - open/close 1211 (time 1709786997.62 total 65.22 last 10.39) - open/close 1390 (time 1709787007.63 total 75.23 last 17.88) - open/close 1541 (time 1709787017.63 total 85.23 last 15.10) - open/close 1694 (time 1709787027.69 total 95.28 last 15.22) - open/close 1910 (time 1709787037.69 total 105.29 last 21.59) - open/close 2139 (time 1709787047.72 total 115.31 last 22.84) - open/close 2313 (time 1709787057.80 total 125.40 last 17.26) - open/close 2455 (time 1709787068.32 total 135.91 last 13.50) - open/close 2703 (time 1709787079.32 total 146.91 last 22.55) - open/close 2902 (time 1709787089.32 total 156.92 last 19.89) - open/close 3160 (time 1709787100.09 total 167.68 last 23.96) - open/close 3398 (time 1709787110.09 total 177.69 last 23.78) - open/close 3536 (time 1709787120.38 total 187.97 last 13.42) - open/close 3701 (time 1709787131.26 total 198.86 last 15.16) - open/close 3783 (time 1709787143.96 total 211.56 last 6.46) - open/close 4063 (time 1709787154.90 total 222.50 last 25.59) - open/close 4277 (time 1709787165.64 total 233.24 last 19.92) - open/close 4476 (time 1709787175.81 total 243.40 last 19.58) - open/close 4602 (time 1709787185.93 total 253.53 last 12.44) - open/close 4780 (time 1709787196.47 total 264.06 last 16.90) - open/close 4974 (time 1709787207.06 total 274.66 last 18.31) - open/close 5161 (time 1709787217.16 total 284.75 last 18.52) - open/close 5403 (time 1709787227.32 total 294.91 last 23.82) - open/close 5663 (time 1709787237.33 total 304.93 last 25.97) - open/close 5851 (time 1709787248.35 total 315.95 last 17.06) - open/close 6019 (time 1709787258.35 total 325.95 last 16.80) - open/close 6222 (time 1709787268.69 total 336.29 last 19.64) - open/close 6434 (time 1709787279.29 total 346.89 last 20.00) - open/close 6572 (time 1709787290.23 total 357.83 last 12.61) - open/close 6737 (time 1709787300.80 total 368.40 last 15.62) - open/close 6959 (time 1709787310.84 total 378.44 last 22.11) - open/close 7226 (time 1709787320.85 total 388.45 last 26.68) - open/close 7329 (time 1709787331.01 total 398.61 last 10.13) - open/close 7523 (time 1709787341.02 total 408.62 last 19.39) - open/close 7675 (time 1709787351.61 total 419.20 last 14.36) - open/close 7809 (time 1709787362.27 total 429.87 last 12.57) - open/close 7977 (time 1709787372.27 total 439.87 last 16.79) - open/close 8155 (time 1709787383.10 total 450.70 last 16.45) - open/close 8370 (time 1709787393.11 total 460.70 last 21.48) - open/close 8504 (time 1709787403.11 total 470.71 last 13.39) - open/close 8633 (time 1709787415.77 total 483.36 last 10.19) - open/close 8747 (time 1709787425.85 total 493.45 last 11.31) - open/close 8851 (time 1709787435.85 total 503.45 last 10.40) - open/close 9025 (time 1709787446.69 total 514.28 last 16.06) - open/close 9202 (time 1709787456.69 total 524.28 last 17.70) - open/close 9349 (time 1709787466.69 total 534.29 last 14.69) - open/close 9490 (time 1709787477.00 total 544.60 last 13.68) - open/close 9710 (time 1709787487.01 total 554.60 last 21.98) - open/close 9835 (time 1709787497.29 total 564.89 last 12.15) - open/close 10000 (time 1709787505.18 total 572.78 last 20.91) - open/close 10301 (time 1709787515.31 total 582.91 last 29.73) - open/close 10768 (time 1709787525.32 total 592.91 last 46.66) - open/close 11184 (time 1709787535.34 total 602.94 last 41.50) - open/close 11638 (time 1709787545.38 total 612.97 last 45.24) - open/close 12015 (time 1709787555.41 total 623.00 last 37.59) - open/close 12146 (time 1709787570.01 total 637.61 last 8.97) - open/close 12596 (time 1709787580.02 total 647.62 last 44.95) - open/close 13012 (time 1709787590.06 total 657.66 last 41.44) - open/close 13457 (time 1709787600.08 total 667.68 last 44.41) - open/close 13872 (time 1709787610.09 total 677.69 last 41.46) - open/close 14249 (time 1709787620.10 total 687.69 last 37.67) - open/close 14621 (time 1709787630.12 total 697.71 last 37.14) - open/close 15039 (time 1709787640.12 total 707.72 last 41.78) - open/close 15429 (time 1709787650.14 total 717.74 last 38.91) - open/close 15784 (time 1709787660.14 total 727.74 last 35.50) - open/close 16159 (time 1709787670.15 total 737.74 last 37.49) - open/close 16501 (time 1709787680.18 total 747.77 last 34.10) - open/close 16875 (time 1709787690.20 total 757.80 last 37.32) - open/close 17236 (time 1709787700.20 total 767.80 last 36.09) - open/close 17594 (time 1709787710.21 total 777.81 last 35.78) - open/close 17920 (time 1709787720.21 total 787.81 last 32.58) - open/close 18248 (time 1709787730.23 total 797.83 last 32.75) - open/close 18591 (time 1709787740.25 total 807.85 last 34.23) - open/close 18885 (time 1709787750.30 total 817.90 last 29.26) - open/close 19213 (time 1709787760.30 total 827.90 last 32.79) - open/close 19547 (time 1709787770.36 total 837.96 last 33.21) - open/close 19884 (time 1709787780.37 total 847.97 last 33.66) - open/close 20000 (time 1709787784.25 total 851.84 last 29.94) - open/close 20339 (time 1709787794.26 total 861.85 last 33.87) - open/close 20456 (time 1709787808.52 total 876.12 last 8.20) - open/close 20768 (time 1709787818.56 total 886.16 last 31.07) - open/close 21110 (time 1709787828.57 total 896.17 last 34.16) - open/close 21429 (time 1709787838.60 total 906.20 last 31.81) - open/close 21724 (time 1709787848.63 total 916.23 last 29.40) - open/close 21980 (time 1709787858.69 total 926.28 last 25.46) - open/close 22261 (time 1709787868.72 total 936.31 last 28.02) - open/close 22400 (time 1709787884.43 total 952.02 last 8.85) - open/close 22680 (time 1709787894.49 total 962.08 last 27.83) - open/close 22935 (time 1709787904.50 total 972.09 last 25.48) - open/close 23210 (time 1709787914.52 total 982.12 last 27.43) - open/close 23503 (time 1709787924.56 total 992.16 last 29.18) - open/close 23792 (time 1709787934.58 total 1002.18 last 28.84) - open/close 24081 (time 1709787944.60 total 1012.20 last 28.83) - open/close 24427 (time 1709787954.61 total 1022.21 last 34.58) - open/close 24761 (time 1709787964.63 total 1032.22 last 33.34) - open/close 25086 (time 1709787974.65 total 1042.25 last 32.42) - open/close 25391 (time 1709787984.66 total 1052.26 last 30.47) - open/close 25730 (time 1709787994.69 total 1062.29 last 33.80) - open/close 26119 (time 1709788004.69 total 1072.29 last 38.89) - open/close 26505 (time 1709788014.69 total 1082.29 last 38.60) - open/close 26923 (time 1709788024.71 total 1092.30 last 41.75) - open/close 27338 (time 1709788034.71 total 1102.31 last 41.46) - open/close 27773 (time 1709788044.73 total 1112.33 last 43.43) - open/close 28189 (time 1709788054.74 total 1122.34 last 41.56) - open/close 28581 (time 1709788064.74 total 1132.34 last 39.20) - open/close 28952 (time 1709788074.76 total 1142.35 last 37.04) - open/close 29360 (time 1709788084.77 total 1152.36 last 40.76) - open/close 29747 (time 1709788094.78 total 1162.38 last 38.65) - open/close 30000 (time 1709788101.67 total 1169.27 last 36.71) - open/close 30376 (time 1709788111.68 total 1179.28 last 37.57) - open/close 30751 (time 1709788121.68 total 1189.28 last 37.49) - open/close 31115 (time 1709788131.69 total 1199.29 last 36.37) - open/close 31487 (time 1709788141.70 total 1209.29 last 37.18) - open/close 31863 (time 1709788151.70 total 1219.30 last 37.57) - open/close 32239 (time 1709788161.72 total 1229.31 last 37.55) - open/close 32621 (time 1709788171.74 total 1239.33 last 38.13) - open/close 32993 (time 1709788181.74 total 1249.33 last 37.19) - open/close 33360 (time 1709788191.74 total 1259.34 last 36.70) - open/close 33725 (time 1709788201.75 total 1269.34 last 36.48) - open/close 34088 (time 1709788211.76 total 1279.36 last 36.24) - open/close 34459 (time 1709788221.77 total 1289.36 last 37.08) - open/close 34820 (time 1709788231.78 total 1299.38 last 36.04) - open/close 35168 (time 1709788241.81 total 1309.40 last 34.72) - open/close 35525 (time 1709788251.82 total 1319.42 last 35.64) - open/close 35871 (time 1709788261.83 total 1329.42 last 34.58) - open/close 36203 (time 1709788271.84 total 1339.44 last 33.16) - open/close 36528 (time 1709788281.85 total 1349.44 last 32.48) - open/close 36835 (time 1709788291.87 total 1359.47 last 30.62) - open/close 37154 (time 1709788301.90 total 1369.50 last 31.81) - open/close 37474 (time 1709788311.91 total 1379.51 last 31.97) - open/close 37787 (time 1709788321.94 total 1389.53 last 31.22) - open/close 38093 (time 1709788331.95 total 1399.54 last 30.57) - open/close 38397 (time 1709788341.98 total 1409.57 last 30.31) - open/close 38700 (time 1709788351.98 total 1419.57 last 30.30) - open/close 38997 (time 1709788361.98 total 1429.58 last 29.69) - open/close 39294 (time 1709788371.99 total 1439.59 last 29.65) - open/close 39586 (time 1709788382.01 total 1449.61 last 29.16) - open/close 39858 (time 1709788392.03 total 1459.62 last 27.15) - open/close 40000 (time 1709788396.83 total 1464.42 last 29.59) - open/close 40307 (time 1709788406.85 total 1474.44 last 30.63) - open/close 40606 (time 1709788416.87 total 1484.46 last 29.84) - open/close 40908 (time 1709788426.88 total 1494.47 last 30.17) - open/close 41202 (time 1709788436.91 total 1504.50 last 29.31) - open/close 41497 (time 1709788446.91 total 1514.51 last 29.49) - open/close 41799 (time 1709788456.94 total 1524.54 last 30.11) - open/close 42085 (time 1709788466.95 total 1534.54 last 28.59) - open/close 42382 (time 1709788476.95 total 1544.55 last 29.69) - open/close 42689 (time 1709788486.95 total 1554.55 last 30.69) - open/close 42989 (time 1709788496.96 total 1564.55 last 30.00) - open/close 43286 (time 1709788506.98 total 1574.57 last 29.63) - open/close 43594 (time 1709788516.99 total 1584.59 last 30.76) - open/close 43892 (time 1709788527.02 total 1594.61 last 29.72) - open/close 44185 (time 1709788537.02 total 1604.62 last 29.29) - open/close 44474 (time 1709788547.03 total 1614.63 last 28.87) - open/close 44764 (time 1709788557.05 total 1624.65 last 28.93) - open/close 45057 (time 1709788567.08 total 1634.67 last 29.23) - open/close 45351 (time 1709788577.09 total 1644.69 last 29.35) - open/close 45642 (time 1709788587.11 total 1654.71 last 29.05) - open/close 45943 (time 1709788597.14 total 1664.74 last 30.02) - open/close 46231 (time 1709788607.17 total 1674.76 last 28.72) - open/close 46516 (time 1709788617.19 total 1684.79 last 28.43) - open/close 46802 (time 1709788627.20 total 1694.80 last 28.58) - open/close 47075 (time 1709788637.21 total 1704.80 last 27.29) - open/close 47347 (time 1709788647.23 total 1714.83 last 27.13) - open/close 47622 (time 1709788657.25 total 1724.84 last 27.46) - open/close 47899 (time 1709788667.26 total 1734.86 last 27.66) - open/close 48173 (time 1709788677.28 total 1744.87 last 27.35) - open/close 48443 (time 1709788687.31 total 1754.91 last 26.91) - open/close 48716 (time 1709788697.32 total 1764.91 last 27.29) - open/close 48985 (time 1709788707.32 total 1774.92 last 26.88) - open/close 49252 (time 1709788717.33 total 1784.93 last 26.68) - open/close 49526 (time 1709788727.34 total 1794.93 last 27.38) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1801.35 seconds: 27.59 ops/second - unlinked 0 (time 1709788734 ; total 0 ; last 0) - unlinked 10000 (time 1709789185 ; total 451 ; last 451) - unlinked 20000 (time 1709789571 ; total 837 ; last 386) - unlinked 30000 (time 1709789846 ; total 1112 ; last 275) - unlinked 40000 (time 1709789980 ; total 1246 ; last 134) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1288 seconds: 38.585403 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg217-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg217-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg217-server stop mds service on oleg217-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg217-server unloading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg217-client: oleg217-client: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-client: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 PASS 101a (3126s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg217-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg217-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg217-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 oleg217-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg217-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:40:37 (1709790037) start mds service on oleg217-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: detected 4 online CPUs by sysfs oleg217-server: Force libcfs to create 2 CPU partitions oleg217-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg217-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg217-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0000 oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg217-client.virtnet: -o user_xattr,flock oleg217-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg217-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg217-server start ost1 service on oleg217-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0000 oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 11 sec oleg217-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b593d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b593d800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg217-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg217-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg217-server stop mds service on oleg217-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg217-server unloading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg217-client: oleg217-client: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-client: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 PASS 101b (85s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg217-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg217-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg217-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg217-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 oleg217-server: oleg217-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: detected 4 online CPUs by sysfs oleg217-server: Force libcfs to create 2 CPU partitions oleg217-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg217-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3698 sec ========== 00:42:18 (1709790138) === conf-sanity: start cleanup 00:42:19 (1709790139) === === conf-sanity: finish cleanup 00:42:19 (1709790139) ===