-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:40:30 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg456-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg456-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg456-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg456-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg456-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg456-server oleg456-server: oleg456-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg456-server' oleg456-server: oleg456-server.virtnet: executing load_modules_local oleg456-server: Loading modules from /home/green/git/lustre-release/lustre oleg456-server: detected 4 online CPUs by sysfs oleg456-server: Force libcfs to create 2 CPU partitions oleg456-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg456-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg456-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg456-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg456-server: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg456-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg456-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg456-server stop mds service on oleg456-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg456-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:41:31 (1709786491) start mds service on oleg456-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg456-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-OST0000 oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg456-client.virtnet: -o user_xattr,flock oleg456-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h41m46s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff88012bfb5000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:42:29.000000000 -0500 Modify: 2024-03-06 23:42:29.000000000 -0500 Change: 2024-03-06 23:42:29.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 56 lines, 56 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg456-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg456-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg456-server stop mds service on oleg456-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg456-server unloading modules on: 'oleg456-server' oleg456-server: oleg456-server.virtnet: executing unload_modules_local modules unloaded. oleg456-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg456-server: Is the ldiskfs module available? oleg456-server: oleg456-server: tunefs.lustre FATAL: failed to write local files oleg456-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg456-client: oleg456-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg456-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg456-server: Is the ldiskfs module available? oleg456-server: oleg456-server: tunefs.lustre FATAL: failed to write local files oleg456-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg456-client: oleg456-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.156@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.156@tcp sys.timeout=20 oleg456-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg456-server: Is the ldiskfs module available? oleg456-server: oleg456-server: tunefs.lustre FATAL: failed to write local files oleg456-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg456-client: oleg456-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.156@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.156@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg456-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg456-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg456-client: oleg456-server: ssh exited with exit code 2 oleg456-server: oleg456-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg456-server' oleg456-server: oleg456-server.virtnet: executing load_modules_local oleg456-server: Loading modules from /home/green/git/lustre-release/lustre oleg456-server: detected 4 online CPUs by sysfs oleg456-server: Force libcfs to create 2 CPU partitions oleg456-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg456-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg456-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg456-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg456-server: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg456-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec stop ost1 service on oleg456-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg456-server stop mds service on oleg456-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg456-server PASS 35b (123s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:43:36 (1709786616) start mds service on oleg456-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg456-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-OST0000 oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg456-client.virtnet: -o user_xattr,flock oleg456-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg456-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg456-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg456-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg456-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg456-client.virtnet: -o user_xattr,flock oleg456-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg456-client.virtnet /mnt/lustre (opts:) stop mds service on oleg456-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg456-server 000000 2 000008 oleg456-server: 8+0 records in oleg456-server: 8+0 records out oleg456-server: 4096 bytes (4.1 kB) copied, 0.000349777 s, 11.7 MB/s start mds service on oleg456-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg456-client.virtnet: -o user_xattr,flock oleg456-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg456-client.virtnet /mnt/lustre (opts:) stop mds service on oleg456-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg456-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg456-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg456-server stop mds service on oleg456-server unloading modules on: 'oleg456-server' oleg456-server: oleg456-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg456-client: oleg456-server: ssh exited with exit code 2 pdsh@oleg456-client: oleg456-client: ssh exited with exit code 2 pdsh@oleg456-client: oleg456-client: ssh exited with exit code 2 pdsh@oleg456-client: oleg456-server: ssh exited with exit code 2 PASS 38 (82s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg456-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg456-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg456-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg456-client: oleg456-server: ssh exited with exit code 2 oleg456-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg456-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:45:01 (1709786701) Stopping clients: oleg456-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg456-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg456-client: oleg456-server: ssh exited with exit code 2 oleg456-server: oleg456-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg456-server' oleg456-server: oleg456-server.virtnet: executing load_modules_local oleg456-server: Loading modules from /home/green/git/lustre-release/lustre oleg456-server: detected 4 online CPUs by sysfs oleg456-server: Force libcfs to create 2 CPU partitions oleg456-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg456-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg456-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg456-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg456-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg456-client.virtnet: -o user_xattr,flock oleg456-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg456-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg456-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg456-server umount lustre on /mnt/lustre..... Stopping client oleg456-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg456-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg456-server stop mds service on oleg456-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg456-server unloading modules on: 'oleg456-server' oleg456-server: oleg456-server.virtnet: executing unload_modules_local modules unloaded. oleg456-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg456-server: Is the ldiskfs module available? oleg456-server: oleg456-server: tunefs.lustre FATAL: failed to write local files oleg456-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg456-client: oleg456-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg456-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg456-server: Is the ldiskfs module available? oleg456-server: oleg456-server: tunefs.lustre FATAL: failed to write local files oleg456-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg456-client: oleg456-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.156@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.156@tcp sys.timeout=20 oleg456-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg456-server: Is the ldiskfs module available? oleg456-server: oleg456-server: tunefs.lustre FATAL: failed to write local files oleg456-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg456-client: oleg456-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.156@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.156@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg456-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg456-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg456-client: oleg456-server: ssh exited with exit code 2 oleg456-server: oleg456-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg456-server' oleg456-server: oleg456-server.virtnet: executing load_modules_local oleg456-server: Loading modules from /home/green/git/lustre-release/lustre oleg456-server: detected 4 online CPUs by sysfs oleg456-server: Force libcfs to create 2 CPU partitions oleg456-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg456-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg456-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg456-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg456-server: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg456-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec stop ost1 service on oleg456-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg456-server stop mds service on oleg456-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg456-server PASS 51 (171s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:47:56 (1709786876) start mds service on oleg456-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg456-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-OST0000 oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg456-client.virtnet: -o user_xattr,flock oleg456-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 195 (time 1709786908.94 total 10.14 last 19.23) - open/close 400 (time 1709786918.95 total 20.15 last 20.48) - open/close 676 (time 1709786929.80 total 31.00 last 25.44) - open/close 826 (time 1709786939.80 total 41.00 last 15.00) - open/close 1067 (time 1709786950.04 total 51.23 last 23.54) - open/close 1365 (time 1709786960.44 total 61.64 last 28.64) - open/close 1489 (time 1709786971.19 total 72.39 last 11.54) - open/close 1710 (time 1709786982.12 total 83.32 last 20.22) - open/close 1893 (time 1709786992.45 total 93.65 last 17.72) - open/close 2046 (time 1709787002.70 total 103.90 last 14.92) - open/close 2196 (time 1709787012.72 total 113.92 last 14.98) - open/close 2500 (time 1709787023.34 total 124.54 last 28.61) - open/close 2666 (time 1709787033.55 total 134.75 last 16.26) - open/close 2831 (time 1709787043.84 total 145.04 last 16.03) - open/close 3014 (time 1709787054.38 total 155.58 last 17.36) - open/close 3250 (time 1709787064.83 total 166.03 last 22.59) - open/close 3533 (time 1709787075.60 total 176.80 last 26.28) - open/close 3706 (time 1709787086.21 total 187.41 last 16.31) - open/close 3914 (time 1709787096.22 total 197.42 last 20.78) - open/close 4190 (time 1709787106.86 total 208.06 last 25.95) - open/close 4456 (time 1709787117.16 total 218.36 last 25.81) - open/close 4693 (time 1709787127.50 total 228.70 last 22.93) - open/close 4856 (time 1709787137.50 total 238.70 last 16.30) - open/close 4998 (time 1709787148.21 total 249.41 last 13.26) - open/close 5244 (time 1709787160.69 total 261.89 last 19.70) - open/close 5437 (time 1709787171.73 total 272.93 last 17.49) - open/close 5668 (time 1709787182.26 total 283.46 last 21.93) - open/close 5782 (time 1709787193.40 total 294.60 last 10.24) - open/close 5902 (time 1709787203.40 total 304.60 last 12.00) - open/close 6165 (time 1709787213.41 total 314.60 last 26.28) - open/close 6440 (time 1709787224.39 total 325.59 last 25.03) - open/close 6600 (time 1709787235.34 total 336.54 last 14.61) - open/close 6822 (time 1709787245.35 total 346.55 last 22.19) - open/close 6975 (time 1709787255.35 total 356.55 last 15.29) - open/close 7144 (time 1709787265.36 total 366.56 last 16.88) - open/close 7317 (time 1709787275.48 total 376.68 last 17.10) - open/close 7490 (time 1709787285.49 total 386.69 last 17.28) - open/close 7623 (time 1709787295.58 total 396.78 last 13.19) - open/close 7890 (time 1709787306.30 total 407.50 last 24.90) - open/close 8044 (time 1709787316.83 total 418.02 last 14.63) - open/close 8127 (time 1709787326.84 total 428.03 last 8.29) - open/close 8334 (time 1709787337.59 total 438.79 last 19.25) - open/close 8560 (time 1709787348.05 total 449.25 last 21.61) - open/close 8723 (time 1709787358.25 total 459.45 last 15.98) - open/close 8852 (time 1709787368.26 total 469.46 last 12.88) - open/close 9077 (time 1709787378.27 total 479.47 last 22.47) - open/close 9201 (time 1709787388.30 total 489.50 last 12.37) - open/close 9369 (time 1709787398.33 total 499.53 last 16.75) - open/close 9530 (time 1709787409.18 total 510.38 last 14.84) - open/close 9670 (time 1709787419.69 total 520.88 last 13.33) - open/close 9836 (time 1709787430.12 total 531.32 last 15.91) - open/close 10000 (time 1709787435.67 total 536.87 last 29.53) - open/close 10435 (time 1709787455.10 total 556.29 last 22.40) - open/close 10851 (time 1709787465.12 total 566.32 last 41.49) - open/close 11280 (time 1709787475.13 total 576.32 last 42.89) - open/close 11750 (time 1709787485.16 total 586.35 last 46.86) - open/close 12200 (time 1709787495.17 total 596.36 last 44.96) - open/close 12624 (time 1709787505.18 total 606.38 last 42.33) - open/close 12992 (time 1709787515.19 total 616.39 last 36.78) - open/close 13421 (time 1709787525.20 total 626.40 last 42.85) - open/close 13885 (time 1709787535.21 total 636.41 last 46.36) - open/close 14288 (time 1709787545.22 total 646.42 last 40.25) - open/close 14731 (time 1709787555.24 total 656.44 last 44.19) - open/close 15196 (time 1709787565.25 total 666.45 last 46.46) - open/close 15674 (time 1709787575.27 total 676.47 last 47.72) - open/close 16121 (time 1709787585.27 total 686.47 last 44.70) - open/close 16590 (time 1709787595.29 total 696.49 last 46.83) - open/close 17065 (time 1709787605.30 total 706.50 last 47.45) - open/close 17572 (time 1709787615.32 total 716.52 last 50.59) - open/close 18028 (time 1709787625.32 total 726.52 last 45.60) - open/close 18461 (time 1709787635.32 total 736.52 last 43.29) - open/close 18854 (time 1709787645.33 total 746.53 last 39.26) - open/close 19277 (time 1709787655.35 total 756.55 last 42.22) - open/close 19720 (time 1709787665.37 total 766.57 last 44.20) - open/close 20000 (time 1709787671.28 total 772.48 last 47.40) - open/close 20334 (time 1709787681.30 total 782.50 last 33.33) - open/close 20740 (time 1709787691.30 total 792.50 last 40.60) - open/close 21100 (time 1709787701.32 total 802.52 last 35.94) - open/close 21446 (time 1709787711.33 total 812.53 last 34.56) - open/close 21772 (time 1709787721.35 total 822.55 last 32.53) - open/close 22146 (time 1709787731.37 total 832.56 last 37.35) - open/close 22565 (time 1709787741.39 total 842.58 last 41.81) - open/close 22950 (time 1709787751.39 total 852.59 last 38.47) - open/close 23262 (time 1709787761.41 total 862.61 last 31.14) - open/close 23556 (time 1709787771.42 total 872.62 last 29.37) - open/close 23833 (time 1709787781.44 total 882.63 last 27.67) - open/close 24103 (time 1709787791.44 total 892.63 last 27.00) - open/close 24398 (time 1709787801.44 total 902.64 last 29.49) - open/close 24720 (time 1709787811.45 total 912.65 last 32.17) - open/close 25017 (time 1709787821.46 total 922.65 last 29.68) - open/close 25314 (time 1709787831.47 total 932.67 last 29.65) - open/close 25655 (time 1709787841.49 total 942.69 last 34.03) - open/close 26014 (time 1709787851.50 total 952.70 last 35.88) - open/close 26389 (time 1709787861.51 total 962.71 last 37.45) - open/close 26765 (time 1709787871.53 total 972.73 last 37.55) - open/close 27137 (time 1709787881.55 total 982.75 last 37.10) - open/close 27507 (time 1709787891.56 total 992.76 last 36.97) - open/close 27882 (time 1709787901.57 total 1002.77 last 37.46) - open/close 28215 (time 1709787911.58 total 1012.78 last 33.26) - open/close 28572 (time 1709787921.59 total 1022.79 last 35.69) - open/close 28911 (time 1709787931.59 total 1032.79 last 33.88) - open/close 29264 (time 1709787941.61 total 1042.81 last 35.25) - open/close 29614 (time 1709787951.62 total 1052.82 last 34.97) - open/close 29979 (time 1709787961.62 total 1062.82 last 36.49) - open/close 30314 (time 1709787971.66 total 1072.86 last 33.37) - open/close 30639 (time 1709787981.67 total 1082.86 last 32.49) - open/close 30992 (time 1709787991.68 total 1092.88 last 35.25) - open/close 31328 (time 1709788001.69 total 1102.88 last 33.58) - open/close 31610 (time 1709788011.71 total 1112.91 last 28.14) - open/close 31890 (time 1709788021.72 total 1122.92 last 27.95) - open/close 32220 (time 1709788031.74 total 1132.94 last 32.96) - open/close 32560 (time 1709788041.74 total 1142.94 last 33.99) - open/close 32875 (time 1709788051.75 total 1152.95 last 31.45) - open/close 33198 (time 1709788061.78 total 1162.98 last 32.20) - open/close 33529 (time 1709788071.82 total 1173.02 last 32.99) - open/close 33852 (time 1709788081.83 total 1183.03 last 32.27) - open/close 34184 (time 1709788091.86 total 1193.06 last 33.10) - open/close 34504 (time 1709788101.86 total 1203.06 last 31.99) - open/close 34813 (time 1709788111.87 total 1213.07 last 30.86) - open/close 35124 (time 1709788121.90 total 1223.10 last 31.02) - open/close 35425 (time 1709788131.92 total 1233.12 last 30.03) - open/close 35707 (time 1709788141.95 total 1243.15 last 28.12) - open/close 35981 (time 1709788151.99 total 1253.19 last 27.29) - open/close 36219 (time 1709788162.07 total 1263.27 last 23.61) - open/close 36471 (time 1709788172.09 total 1273.29 last 25.14) - open/close 36764 (time 1709788182.11 total 1283.31 last 29.26) - open/close 37057 (time 1709788192.12 total 1293.31 last 29.28) - open/close 37299 (time 1709788202.13 total 1303.33 last 24.17) - open/close 37588 (time 1709788212.14 total 1313.33 last 28.87) - open/close 37866 (time 1709788222.15 total 1323.35 last 27.77) - open/close 38151 (time 1709788232.18 total 1333.38 last 28.40) - open/close 38405 (time 1709788242.23 total 1343.43 last 25.28) - open/close 38680 (time 1709788252.26 total 1353.46 last 27.40) - open/close 38986 (time 1709788262.28 total 1363.48 last 30.55) - open/close 39268 (time 1709788272.32 total 1373.52 last 28.09) - open/close 39524 (time 1709788282.33 total 1383.53 last 25.57) - open/close 39798 (time 1709788292.36 total 1393.55 last 27.33) - open/close 40000 (time 1709788299.43 total 1400.63 last 28.56) - open/close 40269 (time 1709788309.47 total 1410.67 last 26.79) - open/close 40536 (time 1709788319.47 total 1420.67 last 26.69) - open/close 40808 (time 1709788329.48 total 1430.68 last 27.18) - open/close 41065 (time 1709788339.51 total 1440.70 last 25.64) - open/close 41314 (time 1709788349.56 total 1450.76 last 24.77) - open/close 41553 (time 1709788359.61 total 1460.81 last 23.78) - open/close 41812 (time 1709788369.61 total 1470.81 last 25.90) - open/close 42075 (time 1709788379.62 total 1480.81 last 26.28) - open/close 42339 (time 1709788389.62 total 1490.82 last 26.40) - open/close 42603 (time 1709788399.63 total 1500.83 last 26.37) - open/close 42874 (time 1709788409.63 total 1510.83 last 27.09) - open/close 43114 (time 1709788419.64 total 1520.84 last 23.98) - open/close 43368 (time 1709788429.65 total 1530.85 last 25.37) - open/close 43614 (time 1709788439.66 total 1540.86 last 24.57) - open/close 43857 (time 1709788449.68 total 1550.88 last 24.25) - open/close 44093 (time 1709788459.68 total 1560.88 last 23.59) - open/close 44347 (time 1709788469.72 total 1570.91 last 25.32) - open/close 44597 (time 1709788479.72 total 1580.92 last 24.99) - open/close 44835 (time 1709788489.73 total 1590.93 last 23.77) - open/close 45080 (time 1709788499.78 total 1600.97 last 24.39) - open/close 45347 (time 1709788509.79 total 1610.98 last 26.67) - open/close 45593 (time 1709788519.81 total 1621.01 last 24.54) - open/close 45822 (time 1709788529.85 total 1631.04 last 22.82) - open/close 46062 (time 1709788539.88 total 1641.08 last 23.92) - open/close 46314 (time 1709788549.92 total 1651.12 last 25.08) - open/close 46540 (time 1709788559.93 total 1661.13 last 22.58) - open/close 46775 (time 1709788569.94 total 1671.14 last 23.49) - open/close 47020 (time 1709788579.97 total 1681.17 last 24.41) - open/close 47246 (time 1709788590.01 total 1691.21 last 22.52) - open/close 47509 (time 1709788600.01 total 1701.21 last 26.30) - open/close 47759 (time 1709788610.02 total 1711.22 last 24.97) - open/close 48006 (time 1709788620.02 total 1721.22 last 24.70) - open/close 48263 (time 1709788630.04 total 1731.24 last 25.65) - open/close 48519 (time 1709788640.06 total 1741.26 last 25.55) - open/close 48760 (time 1709788650.08 total 1751.28 last 24.04) - open/close 49002 (time 1709788660.11 total 1761.31 last 24.12) - open/close 49227 (time 1709788670.12 total 1771.32 last 22.49) - open/close 49406 (time 1709788680.13 total 1781.33 last 17.88) - open/close 49592 (time 1709788690.13 total 1791.33 last 18.60) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1796.65 seconds: 27.66 ops/second - unlinked 0 (time 1709788696 ; total 0 ; last 0) - unlinked 10000 (time 1709789155 ; total 459 ; last 459) - unlinked 20000 (time 1709789527 ; total 831 ; last 372) - unlinked 30000 (time 1709789786 ; total 1090 ; last 259) - unlinked 40000 (time 1709789906 ; total 1210 ; last 120) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1262 seconds: 39.380348 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg456-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg456-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg456-server stop mds service on oleg456-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg456-server unloading modules on: 'oleg456-server' oleg456-server: oleg456-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg456-client: oleg456-client: ssh exited with exit code 2 pdsh@oleg456-client: oleg456-server: ssh exited with exit code 2 pdsh@oleg456-client: oleg456-client: ssh exited with exit code 2 pdsh@oleg456-client: oleg456-server: ssh exited with exit code 2 PASS 101a (3104s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg456-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg456-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg456-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg456-client: oleg456-server: ssh exited with exit code 2 oleg456-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg456-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:39:42 (1709789982) start mds service on oleg456-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg456-server' oleg456-server: oleg456-server.virtnet: executing load_modules_local oleg456-server: Loading modules from /home/green/git/lustre-release/lustre oleg456-server: detected 4 online CPUs by sysfs oleg456-server: Force libcfs to create 2 CPU partitions oleg456-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg456-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg456-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-OST0000 oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg456-client.virtnet: -o user_xattr,flock oleg456-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg456-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg456-server start ost1 service on oleg456-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-OST0000 oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec oleg456-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012aaa2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012aaa2800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg456-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg456-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg456-server stop mds service on oleg456-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg456-server unloading modules on: 'oleg456-server' oleg456-server: oleg456-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg456-client: oleg456-client: ssh exited with exit code 2 pdsh@oleg456-client: oleg456-server: ssh exited with exit code 2 pdsh@oleg456-client: oleg456-client: ssh exited with exit code 2 pdsh@oleg456-client: oleg456-server: ssh exited with exit code 2 PASS 101b (86s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg456-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg456-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg456-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg456-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg456-client: oleg456-server: ssh exited with exit code 2 oleg456-server: oleg456-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg456-server' oleg456-server: oleg456-server.virtnet: executing load_modules_local oleg456-server: Loading modules from /home/green/git/lustre-release/lustre oleg456-server: detected 4 online CPUs by sysfs oleg456-server: Force libcfs to create 2 CPU partitions oleg456-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg456-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3658 sec ========== 00:41:28 (1709790088) === conf-sanity: start cleanup 00:41:29 (1709790089) === === conf-sanity: finish cleanup 00:41:29 (1709790089) ===