-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:45:42 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg219-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg219-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg219-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg219-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg219-server oleg219-server: oleg219-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing load_modules_local oleg219-server: Loading modules from /home/green/git/lustre-release/lustre oleg219-server: detected 4 online CPUs by sysfs oleg219-server: Force libcfs to create 2 CPU partitions oleg219-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg219-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg219-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg219-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg219-server: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg219-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg219-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server stop mds service on oleg219-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg219-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:46:48 (1709786808) start mds service on oleg219-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg219-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg219-client.virtnet: -o user_xattr,flock oleg219-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h47m08s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800ac546000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:47:56.000000000 -0500 Modify: 2024-03-06 23:47:56.000000000 -0500 Change: 2024-03-06 23:47:56.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 76 lines, 76 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg219-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg219-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server stop mds service on oleg219-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg219-server unloading modules on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing unload_modules_local modules unloaded. oleg219-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg219-server: Is the ldiskfs module available? oleg219-server: oleg219-server: tunefs.lustre FATAL: failed to write local files oleg219-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg219-client: oleg219-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg219-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg219-server: Is the ldiskfs module available? oleg219-server: oleg219-server: tunefs.lustre FATAL: failed to write local files oleg219-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg219-client: oleg219-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.119@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.119@tcp sys.timeout=20 oleg219-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg219-server: Is the ldiskfs module available? oleg219-server: oleg219-server: tunefs.lustre FATAL: failed to write local files oleg219-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg219-client: oleg219-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.119@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.119@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg219-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg219-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2 oleg219-server: oleg219-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing load_modules_local oleg219-server: Loading modules from /home/green/git/lustre-release/lustre oleg219-server: detected 4 online CPUs by sysfs oleg219-server: Force libcfs to create 2 CPU partitions oleg219-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg219-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg219-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg219-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg219-server: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg219-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg219-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server stop mds service on oleg219-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg219-server PASS 35b (163s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:49:34 (1709786974) start mds service on oleg219-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg219-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg219-client.virtnet: -o user_xattr,flock oleg219-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg219-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg219-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg219-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg219-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg219-client.virtnet: -o user_xattr,flock oleg219-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg219-client.virtnet /mnt/lustre (opts:) stop mds service on oleg219-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg219-server 000000 2 000008 oleg219-server: 8+0 records in oleg219-server: 8+0 records out oleg219-server: 4096 bytes (4.1 kB) copied, 0.000416885 s, 9.8 MB/s start mds service on oleg219-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg219-client.virtnet: -o user_xattr,flock oleg219-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg219-client.virtnet /mnt/lustre (opts:) stop mds service on oleg219-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg219-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg219-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server stop mds service on oleg219-server unloading modules on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-client: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-client: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2 PASS 38 (97s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg219-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg219-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg219-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2 oleg219-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg219-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:51:14 (1709787074) Stopping clients: oleg219-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg219-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2 oleg219-server: oleg219-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing load_modules_local oleg219-server: Loading modules from /home/green/git/lustre-release/lustre oleg219-server: detected 4 online CPUs by sysfs oleg219-server: Force libcfs to create 2 CPU partitions oleg219-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg219-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg219-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg219-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg219-client.virtnet: -o user_xattr,flock oleg219-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg219-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg219-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg219-server umount lustre on /mnt/lustre..... Stopping client oleg219-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg219-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server stop mds service on oleg219-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg219-server unloading modules on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing unload_modules_local modules unloaded. oleg219-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg219-server: Is the ldiskfs module available? oleg219-server: oleg219-server: tunefs.lustre FATAL: failed to write local files oleg219-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg219-client: oleg219-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg219-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg219-server: Is the ldiskfs module available? oleg219-server: oleg219-server: tunefs.lustre FATAL: failed to write local files oleg219-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg219-client: oleg219-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.119@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.119@tcp sys.timeout=20 oleg219-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg219-server: Is the ldiskfs module available? oleg219-server: oleg219-server: tunefs.lustre FATAL: failed to write local files oleg219-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg219-client: oleg219-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.119@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.119@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg219-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg219-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2 oleg219-server: oleg219-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing load_modules_local oleg219-server: Loading modules from /home/green/git/lustre-release/lustre oleg219-server: detected 4 online CPUs by sysfs oleg219-server: Force libcfs to create 2 CPU partitions oleg219-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg219-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg219-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg219-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg219-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg219-server: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg219-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg219-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server stop mds service on oleg219-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg219-server PASS 51 (203s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:54:42 (1709787282) start mds service on oleg219-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg219-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg219-client.virtnet: -o user_xattr,flock oleg219-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 139 (time 1709787318.07 total 10.18 last 13.65) - open/close 392 (time 1709787328.59 total 20.70 last 24.05) - open/close 491 (time 1709787339.18 total 31.29 last 9.35) - open/close 626 (time 1709787350.09 total 42.20 last 12.37) - open/close 875 (time 1709787360.40 total 52.51 last 24.14) - open/close 1018 (time 1709787370.82 total 62.93 last 13.73) - open/close 1181 (time 1709787380.85 total 72.96 last 16.24) - open/close 1398 (time 1709787391.77 total 83.88 last 19.88) - open/close 1653 (time 1709787402.23 total 94.34 last 24.37) - open/close 1900 (time 1709787413.12 total 105.23 last 22.69) - open/close 2114 (time 1709787423.64 total 115.75 last 20.34) - open/close 2244 (time 1709787434.30 total 126.41 last 12.20) - open/close 2477 (time 1709787444.58 total 136.69 last 22.67) - open/close 2703 (time 1709787455.55 total 147.66 last 20.59) - open/close 2904 (time 1709787466.16 total 158.27 last 18.94) - open/close 3069 (time 1709787476.34 total 168.45 last 16.21) - open/close 3190 (time 1709787486.34 total 178.46 last 12.10) - open/close 3383 (time 1709787497.03 total 189.14 last 18.07) - open/close 3612 (time 1709787508.03 total 200.14 last 20.81) - open/close 3703 (time 1709787523.79 total 215.90 last 5.77) - open/close 3945 (time 1709787533.80 total 225.91 last 24.17) - open/close 4097 (time 1709787543.90 total 236.01 last 15.05) - open/close 4269 (time 1709787554.20 total 246.31 last 16.71) - open/close 4457 (time 1709787564.92 total 257.03 last 17.53) - open/close 4607 (time 1709787575.51 total 267.62 last 14.16) - open/close 4764 (time 1709787585.55 total 277.66 last 15.65) - open/close 4971 (time 1709787595.66 total 287.77 last 20.48) - open/close 5113 (time 1709787605.71 total 297.82 last 14.13) - open/close 5333 (time 1709787616.16 total 308.27 last 21.04) - open/close 5538 (time 1709787626.78 total 318.89 last 19.30) - open/close 5686 (time 1709787636.79 total 328.90 last 14.79) - open/close 5874 (time 1709787646.80 total 338.91 last 18.78) - open/close 6090 (time 1709787656.81 total 348.92 last 21.57) - open/close 6198 (time 1709787667.12 total 359.23 last 10.48) - open/close 6323 (time 1709787677.17 total 369.28 last 12.44) - open/close 6505 (time 1709787687.19 total 379.30 last 18.17) - open/close 6657 (time 1709787697.90 total 390.01 last 14.18) - open/close 6820 (time 1709787708.37 total 400.48 last 15.57) - open/close 7033 (time 1709787718.65 total 410.76 last 20.73) - open/close 7162 (time 1709787728.93 total 421.04 last 12.54) - open/close 7283 (time 1709787738.96 total 431.07 last 12.07) - open/close 7421 (time 1709787748.98 total 441.09 last 13.77) - open/close 7548 (time 1709787758.99 total 451.10 last 12.68) - open/close 7765 (time 1709787769.00 total 461.11 last 21.68) - open/close 7947 (time 1709787779.02 total 471.13 last 18.17) - open/close 8089 (time 1709787789.04 total 481.15 last 14.18) - open/close 8090 (time 1709787800.12 total 492.23 last 0.09) - open/close 8281 (time 1709787811.11 total 503.22 last 17.38) - open/close 8455 (time 1709787822.04 total 514.15 last 15.92) - open/close 8599 (time 1709787832.07 total 524.18 last 14.36) - open/close 8752 (time 1709787842.98 total 535.09 last 14.02) - open/close 8964 (time 1709787853.66 total 545.77 last 19.84) - open/close 9066 (time 1709787863.67 total 555.78 last 10.19) - open/close 9185 (time 1709787873.70 total 565.81 last 11.87) - open/close 9344 (time 1709787883.72 total 575.83 last 15.87) - open/close 9509 (time 1709787893.73 total 585.84 last 16.48) - open/close 9648 (time 1709787904.77 total 596.88 last 12.59) - open/close 9775 (time 1709787915.25 total 607.36 last 12.11) - open/close 9935 (time 1709787925.27 total 617.38 last 15.98) - open/close 10091 (time 1709787939.73 total 631.84 last 10.79) - open/close 10640 (time 1709787949.73 total 641.84 last 54.88) - open/close 11219 (time 1709787959.75 total 651.86 last 57.83) - open/close 11787 (time 1709787969.76 total 661.87 last 56.72) - open/close 12331 (time 1709787979.78 total 671.89 last 54.31) - open/close 12852 (time 1709787989.78 total 681.89 last 52.08) - open/close 13408 (time 1709787999.79 total 691.90 last 55.58) - open/close 14014 (time 1709788009.80 total 701.91 last 60.53) - open/close 14644 (time 1709788019.81 total 711.92 last 62.95) - open/close 15279 (time 1709788029.81 total 721.92 last 63.48) - open/close 15944 (time 1709788039.81 total 731.92 last 66.48) - open/close 16568 (time 1709788049.83 total 741.94 last 62.27) - open/close 17186 (time 1709788059.84 total 751.95 last 61.76) - open/close 17745 (time 1709788069.85 total 761.96 last 55.84) - open/close 18304 (time 1709788079.85 total 771.96 last 55.88) - open/close 18893 (time 1709788089.86 total 781.97 last 58.86) - open/close 19410 (time 1709788099.87 total 791.98 last 51.66) - open/close 19954 (time 1709788109.89 total 802.00 last 54.28) - open/close 20475 (time 1709788119.90 total 812.01 last 52.04) - open/close 20986 (time 1709788129.92 total 822.03 last 51.02) - open/close 21502 (time 1709788139.93 total 832.04 last 51.52) - open/close 21995 (time 1709788149.94 total 842.05 last 49.25) - open/close 22484 (time 1709788159.94 total 852.05 last 48.89) - open/close 22995 (time 1709788169.96 total 862.07 last 51.02) - open/close 23494 (time 1709788179.96 total 872.07 last 49.88) - open/close 23980 (time 1709788189.97 total 882.08 last 48.57) - open/close 24457 (time 1709788199.99 total 892.10 last 47.61) - open/close 24936 (time 1709788209.99 total 902.10 last 47.89) - open/close 25416 (time 1709788219.99 total 912.10 last 47.99) - open/close 25883 (time 1709788230.01 total 922.12 last 46.61) - open/close 26325 (time 1709788240.03 total 932.14 last 44.11) - open/close 26786 (time 1709788250.04 total 942.15 last 46.05) - open/close 27225 (time 1709788260.05 total 952.16 last 43.87) - open/close 27641 (time 1709788270.05 total 962.16 last 41.59) - open/close 28052 (time 1709788280.07 total 972.18 last 41.01) - open/close 28426 (time 1709788290.07 total 982.18 last 37.40) - open/close 28826 (time 1709788300.08 total 992.19 last 39.96) - open/close 29211 (time 1709788310.10 total 1002.21 last 38.45) - open/close 29598 (time 1709788320.13 total 1012.24 last 38.59) - open/close 29974 (time 1709788330.15 total 1022.26 last 37.53) - open/close 30331 (time 1709788340.16 total 1032.27 last 35.67) - open/close 30700 (time 1709788350.18 total 1042.29 last 36.81) - open/close 31056 (time 1709788360.20 total 1052.31 last 35.55) - open/close 31417 (time 1709788370.22 total 1062.33 last 36.02) - open/close 31766 (time 1709788380.24 total 1072.35 last 34.82) - open/close 32089 (time 1709788390.24 total 1082.35 last 32.30) - open/close 32413 (time 1709788400.26 total 1092.37 last 32.35) - open/close 32774 (time 1709788410.28 total 1102.39 last 36.01) - open/close 33118 (time 1709788420.28 total 1112.39 last 34.40) - open/close 33474 (time 1709788430.29 total 1122.40 last 35.56) - open/close 33824 (time 1709788440.31 total 1132.42 last 34.93) - open/close 34171 (time 1709788450.32 total 1142.43 last 34.69) - open/close 34517 (time 1709788460.32 total 1152.43 last 34.58) - open/close 34856 (time 1709788470.33 total 1162.44 last 33.87) - open/close 35213 (time 1709788480.35 total 1172.46 last 35.64) - open/close 35554 (time 1709788490.37 total 1182.48 last 34.04) - open/close 35894 (time 1709788500.38 total 1192.49 last 33.96) - open/close 36232 (time 1709788510.38 total 1202.49 last 33.79) - open/close 36540 (time 1709788520.40 total 1212.51 last 30.75) - open/close 36851 (time 1709788530.40 total 1222.51 last 31.10) - open/close 37163 (time 1709788540.40 total 1232.51 last 31.18) - open/close 37478 (time 1709788550.43 total 1242.54 last 31.41) - open/close 37785 (time 1709788560.43 total 1252.54 last 30.69) - open/close 38087 (time 1709788570.45 total 1262.56 last 30.16) - open/close 38387 (time 1709788580.45 total 1272.56 last 29.99) - open/close 38700 (time 1709788590.48 total 1282.59 last 31.23) - open/close 39012 (time 1709788600.51 total 1292.62 last 31.10) - open/close 39317 (time 1709788610.53 total 1302.64 last 30.42) - open/close 39608 (time 1709788620.54 total 1312.65 last 29.07) - open/close 39896 (time 1709788630.58 total 1322.69 last 28.70) - open/close 40000 (time 1709788634.35 total 1326.46 last 27.56) - open/close 40266 (time 1709788644.37 total 1336.49 last 26.54) - open/close 40526 (time 1709788654.38 total 1346.49 last 25.98) - open/close 40800 (time 1709788664.41 total 1356.52 last 27.32) - open/close 41067 (time 1709788674.44 total 1366.55 last 26.63) - open/close 41334 (time 1709788684.47 total 1376.58 last 26.63) - open/close 41613 (time 1709788694.50 total 1386.61 last 27.81) - open/close 41894 (time 1709788704.51 total 1396.62 last 28.08) - open/close 42171 (time 1709788714.52 total 1406.63 last 27.65) - open/close 42459 (time 1709788724.55 total 1416.66 last 28.71) - open/close 42736 (time 1709788734.59 total 1426.70 last 27.61) - open/close 43029 (time 1709788744.62 total 1436.73 last 29.20) - open/close 43301 (time 1709788754.63 total 1446.74 last 27.16) - open/close 43574 (time 1709788764.66 total 1456.77 last 27.24) - open/close 43834 (time 1709788774.70 total 1466.81 last 25.90) - open/close 44096 (time 1709788784.70 total 1476.81 last 26.19) - open/close 44364 (time 1709788794.73 total 1486.84 last 26.72) - open/close 44635 (time 1709788804.75 total 1496.86 last 27.06) - open/close 44908 (time 1709788814.75 total 1506.86 last 27.29) - open/close 45182 (time 1709788824.77 total 1516.88 last 27.34) - open/close 45442 (time 1709788834.79 total 1526.90 last 25.95) - open/close 45715 (time 1709788844.83 total 1536.94 last 27.20) - open/close 45986 (time 1709788854.85 total 1546.96 last 27.03) - open/close 46257 (time 1709788864.88 total 1556.99 last 27.02) - open/close 46530 (time 1709788874.90 total 1567.01 last 27.25) - open/close 46799 (time 1709788884.92 total 1577.03 last 26.83) - open/close 47057 (time 1709788894.94 total 1587.05 last 25.76) - open/close 47317 (time 1709788904.95 total 1597.06 last 25.97) - open/close 47574 (time 1709788914.96 total 1607.07 last 25.68) - open/close 47822 (time 1709788924.97 total 1617.08 last 24.78) - open/close 48079 (time 1709788934.99 total 1627.10 last 25.65) - open/close 48327 (time 1709788945.00 total 1637.11 last 24.76) - open/close 48582 (time 1709788955.02 total 1647.13 last 25.46) - open/close 48837 (time 1709788965.02 total 1657.13 last 25.50) - open/close 49088 (time 1709788975.02 total 1667.13 last 25.08) - open/close 49335 (time 1709788985.02 total 1677.13 last 24.70) - open/close 49580 (time 1709788995.05 total 1687.16 last 24.44) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1691.92 seconds: 29.37 ops/second - unlinked 0 (time 1709789000 ; total 0 ; last 0) - unlinked 10000 (time 1709789484 ; total 484 ; last 484) - unlinked 20000 (time 1709789856 ; total 856 ; last 372) - unlinked 30000 (time 1709790075 ; total 1075 ; last 219) - unlinked 40000 (time 1709790144 ; total 1144 ; last 69) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1169 seconds: 42.513260 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg219-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg219-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server stop mds service on oleg219-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg219-server unloading modules on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg219-client: oleg219-client: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-client: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2 PASS 101a (2900s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg219-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg219-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg219-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2 oleg219-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg219-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:43:03 (1709790183) start mds service on oleg219-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing load_modules_local oleg219-server: Loading modules from /home/green/git/lustre-release/lustre oleg219-server: detected 4 online CPUs by sysfs oleg219-server: Force libcfs to create 2 CPU partitions oleg219-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg219-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg219-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg219-client.virtnet: -o user_xattr,flock oleg219-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg219-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server start ost1 service on oleg219-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 12 sec oleg219-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800ab0ad800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800ab0ad800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg219-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg219-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server stop mds service on oleg219-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg219-server unloading modules on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg219-client: oleg219-client: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-client: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2 PASS 101b (83s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg219-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg219-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg219-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg219-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2 oleg219-server: oleg219-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing load_modules_local oleg219-server: Loading modules from /home/green/git/lustre-release/lustre oleg219-server: detected 4 online CPUs by sysfs oleg219-server: Force libcfs to create 2 CPU partitions oleg219-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg219-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3539 sec ========== 00:44:42 (1709790282) === conf-sanity: start cleanup 00:44:43 (1709790283) === === conf-sanity: finish cleanup 00:44:43 (1709790283) ===