-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:38:16 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg351-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg351-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg351-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg351-server oleg351-server: oleg351-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: detected 4 online CPUs by sysfs oleg351-server: Force libcfs to create 2 CPU partitions oleg351-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg351-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg351-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg351-server: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg351-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:39:29 (1709786369) start mds service on oleg351-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg351-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg351-client.virtnet: -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h39m44s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800ab66c800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:40:28.000000000 -0500 Modify: 2024-03-06 23:40:28.000000000 -0500 Change: 2024-03-06 23:40:28.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 72 lines, 72 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg351-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server unloading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing unload_modules_local modules unloaded. oleg351-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg351-server: Is the ldiskfs module available? oleg351-server: oleg351-server: tunefs.lustre FATAL: failed to write local files oleg351-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg351-client: oleg351-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg351-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg351-server: Is the ldiskfs module available? oleg351-server: oleg351-server: tunefs.lustre FATAL: failed to write local files oleg351-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg351-client: oleg351-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.151@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.151@tcp sys.timeout=20 oleg351-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg351-server: Is the ldiskfs module available? oleg351-server: oleg351-server: tunefs.lustre FATAL: failed to write local files oleg351-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg351-client: oleg351-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.151@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.151@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg351-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg351-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 oleg351-server: oleg351-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: detected 4 online CPUs by sysfs oleg351-server: Force libcfs to create 2 CPU partitions oleg351-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg351-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg351-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg351-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg351-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg351-server: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg351-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server PASS 35b (115s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:41:25 (1709786485) start mds service on oleg351-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg351-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg351-client.virtnet: -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg351-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg351-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg351-client.virtnet: -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg351-client.virtnet /mnt/lustre (opts:) stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server 000000 2 000008 oleg351-server: 8+0 records in oleg351-server: 8+0 records out oleg351-server: 4096 bytes (4.1 kB) copied, 0.000392808 s, 10.4 MB/s start mds service on oleg351-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg351-client.virtnet: -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg351-client.virtnet /mnt/lustre (opts:) stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server stop mds service on oleg351-server unloading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 PASS 38 (72s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg351-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg351-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg351-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 oleg351-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg351-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:42:40 (1709786560) Stopping clients: oleg351-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg351-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 oleg351-server: oleg351-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: detected 4 online CPUs by sysfs oleg351-server: Force libcfs to create 2 CPU partitions oleg351-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg351-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg351-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg351-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg351-client.virtnet: -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg351-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg351-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg351-server umount lustre on /mnt/lustre..... Stopping client oleg351-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server unloading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing unload_modules_local modules unloaded. oleg351-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg351-server: Is the ldiskfs module available? oleg351-server: oleg351-server: tunefs.lustre FATAL: failed to write local files oleg351-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg351-client: oleg351-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg351-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg351-server: Is the ldiskfs module available? oleg351-server: oleg351-server: tunefs.lustre FATAL: failed to write local files oleg351-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg351-client: oleg351-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.151@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.151@tcp sys.timeout=20 oleg351-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg351-server: Is the ldiskfs module available? oleg351-server: oleg351-server: tunefs.lustre FATAL: failed to write local files oleg351-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg351-client: oleg351-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.151@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.151@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg351-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg351-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 oleg351-server: oleg351-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: detected 4 online CPUs by sysfs oleg351-server: Force libcfs to create 2 CPU partitions oleg351-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg351-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg351-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg351-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg351-server: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg351-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server PASS 51 (169s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:45:32 (1709786732) start mds service on oleg351-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg351-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg351-client.virtnet: -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 310 (time 1709786759.67 total 10.94 last 28.35) - open/close 596 (time 1709786769.98 total 21.24 last 27.75) - open/close 913 (time 1709786780.68 total 31.95 last 29.61) - open/close 1221 (time 1709786790.87 total 42.13 last 30.24) - open/close 1460 (time 1709786801.77 total 53.03 last 21.93) - open/close 1693 (time 1709786812.21 total 63.47 last 22.32) - open/close 1968 (time 1709786823.02 total 74.29 last 25.42) - open/close 2306 (time 1709786833.94 total 85.20 last 30.96) - open/close 2576 (time 1709786843.95 total 95.21 last 26.99) - open/close 3029 (time 1709786853.95 total 105.21 last 45.28) - open/close 3284 (time 1709786864.91 total 116.17 last 23.27) - open/close 3617 (time 1709786875.48 total 126.74 last 31.50) - open/close 3849 (time 1709786886.29 total 137.55 last 21.46) - open/close 4116 (time 1709786896.34 total 147.61 last 26.55) - open/close 4348 (time 1709786906.52 total 157.78 last 22.80) - open/close 4512 (time 1709786917.44 total 168.71 last 15.02) - open/close 4676 (time 1709786927.70 total 178.96 last 16.00) - open/close 4787 (time 1709786937.70 total 188.96 last 11.09) - open/close 4921 (time 1709786948.11 total 199.37 last 12.87) - open/close 5101 (time 1709786958.12 total 209.38 last 17.99) - open/close 5349 (time 1709786968.12 total 219.38 last 24.79) - open/close 5581 (time 1709786978.15 total 229.41 last 23.13) - open/close 5767 (time 1709786988.65 total 239.91 last 17.72) - open/close 5936 (time 1709786998.67 total 249.94 last 16.85) - open/close 6051 (time 1709787009.45 total 260.71 last 10.67) - open/close 6310 (time 1709787019.93 total 271.19 last 24.72) - open/close 6381 (time 1709787030.08 total 281.34 last 6.99) - open/close 6504 (time 1709787040.36 total 291.63 last 11.96) - open/close 6753 (time 1709787051.13 total 302.40 last 23.12) - open/close 6994 (time 1709787061.14 total 312.41 last 24.08) - open/close 7225 (time 1709787071.94 total 323.20 last 21.40) - open/close 7381 (time 1709787090.13 total 341.39 last 8.58) - open/close 7621 (time 1709787100.14 total 351.40 last 23.98) - open/close 7686 (time 1709787110.58 total 361.84 last 6.22) - open/close 7899 (time 1709787120.58 total 371.84 last 21.30) - open/close 8122 (time 1709787131.24 total 382.50 last 20.92) - open/close 8328 (time 1709787141.75 total 393.01 last 19.60) - open/close 8485 (time 1709787154.36 total 405.62 last 12.45) - open/close 8639 (time 1709787165.22 total 416.49 last 14.18) - open/close 8810 (time 1709787175.51 total 426.77 last 16.62) - open/close 9007 (time 1709787185.52 total 436.78 last 19.69) - open/close 9145 (time 1709787195.58 total 446.84 last 13.72) - open/close 9541 (time 1709787205.59 total 456.86 last 39.53) - open/close 10000 (time 1709787214.07 total 465.33 last 54.17) - open/close 10502 (time 1709787224.09 total 475.35 last 50.11) - open/close 11091 (time 1709787234.09 total 485.36 last 58.85) - open/close 11631 (time 1709787244.12 total 495.38 last 53.85) - open/close 12193 (time 1709787254.15 total 505.41 last 56.04) - open/close 12293 (time 1709787267.11 total 518.38 last 7.71) - open/close 12531 (time 1709787283.15 total 534.41 last 14.84) - open/close 13066 (time 1709787293.15 total 544.42 last 53.46) - open/close 13553 (time 1709787303.17 total 554.43 last 48.63) - open/close 14064 (time 1709787313.17 total 564.43 last 51.09) - open/close 14551 (time 1709787323.19 total 574.45 last 48.63) - open/close 15038 (time 1709787333.23 total 584.49 last 48.49) - open/close 15540 (time 1709787343.23 total 594.49 last 50.19) - open/close 16009 (time 1709787353.27 total 604.53 last 46.73) - open/close 16488 (time 1709787363.27 total 614.53 last 47.88) - open/close 16894 (time 1709787373.29 total 624.55 last 40.53) - open/close 17281 (time 1709787383.30 total 634.56 last 38.66) - open/close 17715 (time 1709787393.30 total 644.57 last 43.37) - open/close 18091 (time 1709787403.31 total 654.57 last 37.57) - open/close 18466 (time 1709787413.34 total 664.60 last 37.40) - open/close 18882 (time 1709787423.34 total 674.60 last 41.59) - open/close 19270 (time 1709787433.34 total 684.61 last 38.79) - open/close 19669 (time 1709787443.35 total 694.61 last 39.88) - open/close 20000 (time 1709787452.05 total 703.31 last 38.06) - open/close 20361 (time 1709787462.06 total 713.32 last 36.05) - open/close 20608 (time 1709787472.07 total 723.33 last 24.69) - open/close 20944 (time 1709787482.08 total 733.35 last 33.54) - open/close 21296 (time 1709787492.12 total 743.38 last 35.08) - open/close 21560 (time 1709787502.14 total 753.40 last 26.34) - open/close 21893 (time 1709787512.16 total 763.42 last 33.25) - open/close 22213 (time 1709787522.16 total 773.43 last 31.98) - open/close 22524 (time 1709787532.18 total 783.44 last 31.04) - open/close 22838 (time 1709787542.20 total 793.47 last 31.33) - open/close 23171 (time 1709787552.24 total 803.50 last 33.19) - open/close 23467 (time 1709787562.26 total 813.53 last 29.52) - open/close 23737 (time 1709787572.32 total 823.58 last 26.85) - open/close 24007 (time 1709787582.34 total 833.61 last 26.94) - open/close 24311 (time 1709787592.38 total 843.64 last 30.29) - open/close 24555 (time 1709787602.45 total 853.71 last 24.24) - open/close 24838 (time 1709787612.51 total 863.77 last 28.14) - open/close 25108 (time 1709787622.51 total 873.77 last 27.00) - open/close 25395 (time 1709787632.52 total 883.78 last 28.68) - open/close 25688 (time 1709787642.53 total 893.79 last 29.25) - open/close 25950 (time 1709787652.56 total 903.82 last 26.14) - open/close 25974 (time 1709787664.60 total 915.87 last 1.99) - open/close 26233 (time 1709787674.61 total 925.88 last 25.87) - open/close 26481 (time 1709787684.64 total 935.91 last 24.73) - open/close 26739 (time 1709787694.70 total 945.96 last 25.65) - open/close 26954 (time 1709787704.71 total 955.97 last 21.48) - open/close 27187 (time 1709787714.72 total 965.99 last 23.27) - open/close 27480 (time 1709787724.75 total 976.01 last 29.22) - open/close 27755 (time 1709787734.76 total 986.02 last 27.48) - open/close 28008 (time 1709787744.79 total 996.05 last 25.23) - open/close 28253 (time 1709787754.82 total 1006.08 last 24.42) - open/close 28522 (time 1709787764.83 total 1016.09 last 26.87) - open/close 28785 (time 1709787774.84 total 1026.10 last 26.28) - open/close 28799 (time 1709787786.50 total 1037.76 last 1.20) - open/close 29073 (time 1709787796.51 total 1047.78 last 27.36) - open/close 29361 (time 1709787806.52 total 1057.78 last 28.78) - open/close 29657 (time 1709787816.56 total 1067.82 last 29.49) - open/close 29927 (time 1709787826.58 total 1077.85 last 26.93) - open/close 30000 (time 1709787828.94 total 1080.21 last 30.94) - open/close 30292 (time 1709787838.97 total 1090.23 last 29.13) - open/close 30588 (time 1709787848.99 total 1100.25 last 29.54) - open/close 30899 (time 1709787859.00 total 1110.27 last 31.06) - open/close 31174 (time 1709787869.02 total 1120.29 last 27.45) - open/close 31437 (time 1709787879.07 total 1130.33 last 26.17) - open/close 31622 (time 1709787896.47 total 1147.73 last 10.63) - open/close 31906 (time 1709787906.48 total 1157.74 last 28.37) - open/close 32184 (time 1709787916.50 total 1167.76 last 27.75) - open/close 32440 (time 1709787926.50 total 1177.76 last 25.59) - open/close 32705 (time 1709787936.51 total 1187.77 last 26.47) - open/close 32957 (time 1709787946.54 total 1197.80 last 25.14) - open/close 33231 (time 1709787956.54 total 1207.80 last 27.39) - open/close 33508 (time 1709787966.60 total 1217.86 last 27.54) - open/close 33779 (time 1709787976.60 total 1227.87 last 27.08) - open/close 34050 (time 1709787986.61 total 1237.87 last 27.09) - open/close 34339 (time 1709787996.61 total 1247.87 last 28.89) - open/close 34610 (time 1709788006.63 total 1257.89 last 27.04) - open/close 34884 (time 1709788016.63 total 1267.90 last 27.39) - open/close 35141 (time 1709788026.67 total 1277.94 last 25.60) - open/close 35388 (time 1709788036.71 total 1287.97 last 24.61) - open/close 35648 (time 1709788046.73 total 1298.00 last 25.94) - open/close 35900 (time 1709788056.77 total 1308.04 last 25.10) - open/close 36148 (time 1709788066.80 total 1318.07 last 24.72) - open/close 36441 (time 1709788076.81 total 1328.07 last 29.29) - open/close 36780 (time 1709788086.81 total 1338.07 last 33.89) - open/close 37093 (time 1709788096.87 total 1348.13 last 31.12) - open/close 37404 (time 1709788106.89 total 1358.15 last 31.05) - open/close 37720 (time 1709788116.91 total 1368.18 last 31.51) - open/close 38038 (time 1709788126.93 total 1378.19 last 31.75) - open/close 38352 (time 1709788136.94 total 1388.21 last 31.36) - open/close 38665 (time 1709788146.95 total 1398.21 last 31.29) - open/close 38967 (time 1709788156.97 total 1408.23 last 30.13) - open/close 39271 (time 1709788166.99 total 1418.25 last 30.36) - open/close 39572 (time 1709788176.99 total 1428.25 last 30.10) - open/close 39871 (time 1709788187.00 total 1438.27 last 29.84) - open/close 40000 (time 1709788191.54 total 1442.80 last 28.44) - open/close 40292 (time 1709788201.55 total 1452.81 last 29.17) - open/close 40586 (time 1709788211.56 total 1462.82 last 29.38) - open/close 40872 (time 1709788221.57 total 1472.84 last 28.55) - open/close 41165 (time 1709788231.58 total 1482.84 last 29.29) - open/close 41455 (time 1709788241.58 total 1492.84 last 28.99) - open/close 41741 (time 1709788251.59 total 1502.85 last 28.58) - open/close 42012 (time 1709788261.61 total 1512.87 last 27.05) - open/close 42291 (time 1709788271.63 total 1522.90 last 27.82) - open/close 42576 (time 1709788281.67 total 1532.93 last 28.41) - open/close 42842 (time 1709788291.70 total 1542.96 last 26.51) - open/close 43109 (time 1709788301.73 total 1552.99 last 26.62) - open/close 43374 (time 1709788311.75 total 1563.01 last 26.45) - open/close 43643 (time 1709788321.76 total 1573.02 last 26.86) - open/close 43924 (time 1709788331.77 total 1583.04 last 28.06) - open/close 44196 (time 1709788341.79 total 1593.06 last 27.15) - open/close 44465 (time 1709788351.82 total 1603.08 last 26.83) - open/close 44729 (time 1709788361.87 total 1613.13 last 26.27) - open/close 44988 (time 1709788371.90 total 1623.17 last 25.82) - open/close 45254 (time 1709788381.93 total 1633.20 last 26.52) - open/close 45514 (time 1709788391.94 total 1643.20 last 25.98) - open/close 45781 (time 1709788401.94 total 1653.21 last 26.69) - open/close 46039 (time 1709788411.96 total 1663.22 last 25.77) - open/close 46302 (time 1709788421.96 total 1673.23 last 26.28) - open/close 46559 (time 1709788432.00 total 1683.26 last 25.61) - open/close 46838 (time 1709788442.02 total 1693.28 last 27.85) - open/close 47097 (time 1709788452.02 total 1703.29 last 25.88) - open/close 47358 (time 1709788462.04 total 1713.31 last 26.05) - open/close 47613 (time 1709788472.05 total 1723.32 last 25.48) - open/close 47873 (time 1709788482.11 total 1733.38 last 25.84) - open/close 48113 (time 1709788492.13 total 1743.40 last 23.95) - open/close 48348 (time 1709788502.16 total 1753.42 last 23.44) - open/close 48604 (time 1709788512.19 total 1763.45 last 25.53) - open/close 48862 (time 1709788522.22 total 1773.49 last 25.71) - open/close 49109 (time 1709788532.23 total 1783.49 last 24.69) - open/close 49360 (time 1709788542.26 total 1793.52 last 25.03) - open/close 49609 (time 1709788552.27 total 1803.53 last 24.87) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1807.09 seconds: 27.50 ops/second - unlinked 0 (time 1709788557 ; total 0 ; last 0) - unlinked 10000 (time 1709789018 ; total 461 ; last 461) - unlinked 20000 (time 1709789393 ; total 836 ; last 375) - unlinked 30000 (time 1709789670 ; total 1113 ; last 277) - unlinked 40000 (time 1709789829 ; total 1272 ; last 159) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1326 seconds: 37.479637 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg351-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server unloading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 PASS 101a (3172s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg351-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg351-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg351-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 oleg351-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg351-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:38:27 (1709789907) start mds service on oleg351-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: detected 4 online CPUs by sysfs oleg351-server: Force libcfs to create 2 CPU partitions oleg351-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg351-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg351-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg351-client.virtnet: -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server start ost1 service on oleg351-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec oleg351-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800aeead000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800aeead000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg351-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server unloading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 PASS 101b (88s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg351-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg351-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg351-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg351-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 oleg351-server: oleg351-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: detected 4 online CPUs by sysfs oleg351-server: Force libcfs to create 2 CPU partitions oleg351-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg351-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg351-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3719 sec ========== 00:40:16 (1709790016) === conf-sanity: start cleanup 00:40:16 (1709790016) === === conf-sanity: finish cleanup 00:40:16 (1709790016) ===