-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:45:48 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg427-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg427-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg427-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg427-server oleg427-server: oleg427-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: detected 4 online CPUs by sysfs oleg427-server: Force libcfs to create 2 CPU partitions oleg427-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg427-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg427-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg427-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg427-server: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg427-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:47:07 (1709786827) start mds service on oleg427-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg427-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg427-client.virtnet: -o user_xattr,flock oleg427-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h47m26s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff88012c0fa800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:48:10.000000000 -0500 Modify: 2024-03-06 23:48:10.000000000 -0500 Change: 2024-03-06 23:48:10.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 76 lines, 76 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg427-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server unloading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing unload_modules_local modules unloaded. oleg427-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg427-server: Is the ldiskfs module available? oleg427-server: oleg427-server: tunefs.lustre FATAL: failed to write local files oleg427-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg427-client: oleg427-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg427-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg427-server: Is the ldiskfs module available? oleg427-server: oleg427-server: tunefs.lustre FATAL: failed to write local files oleg427-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg427-client: oleg427-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.127@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.127@tcp sys.timeout=20 oleg427-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg427-server: Is the ldiskfs module available? oleg427-server: oleg427-server: tunefs.lustre FATAL: failed to write local files oleg427-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg427-client: oleg427-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.127@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.127@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg427-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg427-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 oleg427-server: oleg427-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: detected 4 online CPUs by sysfs oleg427-server: Force libcfs to create 2 CPU partitions oleg427-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg427-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg427-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg427-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg427-server: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg427-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server PASS 35b (160s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:49:50 (1709786990) start mds service on oleg427-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg427-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg427-client.virtnet: -o user_xattr,flock oleg427-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg427-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg427-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg427-client.virtnet: -o user_xattr,flock oleg427-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg427-client.virtnet /mnt/lustre (opts:) stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server 000000 2 000008 oleg427-server: 8+0 records in oleg427-server: 8+0 records out oleg427-server: 4096 bytes (4.1 kB) copied, 0.00167828 s, 2.4 MB/s start mds service on oleg427-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg427-client.virtnet: -o user_xattr,flock oleg427-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg427-client.virtnet /mnt/lustre (opts:) stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server stop mds service on oleg427-server unloading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 PASS 38 (91s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg427-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg427-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg427-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 oleg427-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg427-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:51:25 (1709787085) Stopping clients: oleg427-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg427-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 oleg427-server: oleg427-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: detected 4 online CPUs by sysfs oleg427-server: Force libcfs to create 2 CPU partitions oleg427-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg427-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg427-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg427-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg427-client.virtnet: -o user_xattr,flock oleg427-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg427-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg427-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg427-server umount lustre on /mnt/lustre..... Stopping client oleg427-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server unloading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing unload_modules_local modules unloaded. oleg427-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg427-server: Is the ldiskfs module available? oleg427-server: oleg427-server: tunefs.lustre FATAL: failed to write local files oleg427-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg427-client: oleg427-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg427-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg427-server: Is the ldiskfs module available? oleg427-server: oleg427-server: tunefs.lustre FATAL: failed to write local files oleg427-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg427-client: oleg427-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.127@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.127@tcp sys.timeout=20 oleg427-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg427-server: Is the ldiskfs module available? oleg427-server: oleg427-server: tunefs.lustre FATAL: failed to write local files oleg427-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg427-client: oleg427-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.127@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.127@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg427-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg427-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 oleg427-server: oleg427-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: detected 4 online CPUs by sysfs oleg427-server: Force libcfs to create 2 CPU partitions oleg427-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg427-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg427-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg427-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg427-server: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg427-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server PASS 51 (191s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:54:41 (1709787281) start mds service on oleg427-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg427-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg427-client.virtnet: -o user_xattr,flock oleg427-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 190 (time 1709787317.50 total 10.85 last 17.51) - open/close 398 (time 1709787328.03 total 21.38 last 19.75) - open/close 523 (time 1709787338.14 total 31.49 last 12.36) - open/close 671 (time 1709787348.30 total 41.65 last 14.57) - open/close 742 (time 1709787358.31 total 51.66 last 7.09) - open/close 941 (time 1709787369.21 total 62.56 last 18.26) - open/close 1061 (time 1709787379.21 total 72.56 last 12.00) - open/close 1211 (time 1709787389.43 total 82.78 last 14.68) - open/close 1329 (time 1709787401.05 total 94.40 last 10.15) - open/close 1497 (time 1709787411.05 total 104.40 last 16.80) - open/close 1648 (time 1709787421.49 total 114.84 last 14.46) - open/close 1837 (time 1709787431.57 total 124.92 last 18.75) - open/close 1963 (time 1709787441.63 total 134.98 last 12.52) - open/close 2148 (time 1709787452.54 total 145.89 last 16.96) - open/close 2344 (time 1709787462.99 total 156.34 last 18.75) - open/close 2572 (time 1709787473.02 total 166.37 last 22.74) - open/close 2741 (time 1709787483.58 total 176.93 last 16.00) - open/close 2989 (time 1709787493.59 total 186.94 last 24.78) - open/close 3210 (time 1709787503.60 total 196.95 last 22.07) - open/close 3370 (time 1709787513.64 total 206.99 last 15.94) - open/close 3573 (time 1709787523.69 total 217.04 last 20.20) - open/close 3761 (time 1709787533.69 total 227.04 last 18.80) - open/close 3865 (time 1709787543.70 total 237.05 last 10.39) - open/close 3983 (time 1709787553.75 total 247.10 last 11.74) - open/close 4293 (time 1709787564.20 total 257.55 last 29.66) - open/close 4524 (time 1709787576.02 total 269.37 last 19.55) - open/close 4688 (time 1709787586.02 total 279.37 last 16.39) - open/close 4888 (time 1709787596.03 total 289.38 last 19.99) - open/close 5103 (time 1709787606.04 total 299.39 last 21.48) - open/close 5305 (time 1709787616.06 total 309.41 last 20.15) - open/close 5531 (time 1709787626.40 total 319.75 last 21.85) - open/close 5733 (time 1709787636.68 total 330.03 last 19.64) - open/close 5997 (time 1709787647.05 total 340.40 last 25.46) - open/close 6212 (time 1709787657.08 total 350.43 last 21.44) - open/close 6419 (time 1709787667.09 total 360.44 last 20.69) - open/close 6598 (time 1709787677.10 total 370.45 last 17.88) - open/close 6808 (time 1709787687.73 total 381.08 last 19.75) - open/close 7027 (time 1709787697.74 total 391.09 last 21.88) - open/close 7124 (time 1709787708.44 total 401.79 last 9.07) - open/close 7338 (time 1709787718.61 total 411.96 last 21.04) - open/close 7564 (time 1709787729.18 total 422.53 last 21.37) - open/close 7770 (time 1709787740.17 total 433.52 last 18.75) - open/close 7958 (time 1709787750.19 total 443.54 last 18.77) - open/close 8191 (time 1709787770.32 total 463.67 last 11.57) - open/close 8396 (time 1709787780.77 total 474.12 last 19.61) - open/close 8615 (time 1709787790.93 total 484.28 last 21.57) - open/close 8814 (time 1709787801.74 total 495.09 last 18.40) - open/close 9004 (time 1709787811.77 total 505.12 last 18.94) - open/close 9214 (time 1709787822.75 total 516.10 last 19.12) - open/close 9414 (time 1709787832.84 total 526.19 last 19.82) - open/close 9982 (time 1709787842.86 total 536.21 last 56.71) - open/close 10583 (time 1709787852.86 total 546.21 last 60.05) - open/close 11230 (time 1709787862.88 total 556.23 last 64.63) - open/close 11859 (time 1709787872.88 total 566.23 last 62.86) - open/close 12453 (time 1709787882.88 total 576.23 last 59.40) - open/close 12999 (time 1709787892.90 total 586.25 last 54.53) - open/close 13551 (time 1709787902.90 total 596.25 last 55.16) - open/close 14046 (time 1709787912.92 total 606.27 last 49.39) - open/close 14538 (time 1709787922.94 total 616.29 last 49.14) - open/close 15007 (time 1709787932.94 total 626.29 last 46.88) - open/close 15475 (time 1709787942.94 total 636.29 last 46.78) - open/close 15927 (time 1709787952.96 total 646.31 last 45.15) - open/close 16381 (time 1709787962.99 total 656.34 last 45.24) - open/close 16813 (time 1709787973.00 total 666.35 last 43.16) - open/close 17229 (time 1709787983.02 total 676.37 last 41.53) - open/close 17674 (time 1709787993.03 total 686.38 last 44.44) - open/close 18119 (time 1709788003.04 total 696.40 last 44.44) - open/close 18586 (time 1709788013.05 total 706.40 last 46.69) - open/close 19017 (time 1709788023.07 total 716.42 last 42.99) - open/close 19474 (time 1709788033.09 total 726.44 last 45.62) - open/close 19898 (time 1709788043.10 total 736.46 last 42.34) - open/close 20000 (time 1709788045.80 total 739.15 last 37.83) - open/close 20377 (time 1709788055.81 total 749.16 last 37.68) - open/close 20783 (time 1709788065.81 total 759.16 last 40.57) - open/close 21202 (time 1709788075.84 total 769.19 last 41.78) - open/close 21613 (time 1709788085.84 total 779.19 last 41.10) - open/close 22029 (time 1709788095.86 total 789.21 last 41.53) - open/close 22450 (time 1709788105.88 total 799.23 last 42.02) - open/close 22850 (time 1709788115.90 total 809.25 last 39.93) - open/close 23254 (time 1709788125.90 total 819.25 last 40.39) - open/close 23619 (time 1709788135.92 total 829.27 last 36.42) - open/close 23990 (time 1709788145.94 total 839.29 last 37.03) - open/close 24339 (time 1709788155.95 total 849.30 last 34.87) - open/close 24677 (time 1709788165.96 total 859.31 last 33.75) - open/close 25003 (time 1709788175.98 total 869.33 last 32.54) - open/close 25325 (time 1709788185.99 total 879.34 last 32.19) - open/close 25657 (time 1709788196.01 total 889.36 last 33.11) - open/close 25959 (time 1709788206.04 total 899.39 last 30.11) - open/close 26291 (time 1709788216.08 total 909.43 last 33.07) - open/close 26605 (time 1709788226.10 total 919.45 last 31.35) - open/close 26942 (time 1709788236.13 total 929.48 last 33.58) - open/close 27245 (time 1709788246.14 total 939.49 last 30.28) - open/close 27596 (time 1709788256.16 total 949.51 last 35.04) - open/close 27949 (time 1709788266.19 total 959.54 last 35.18) - open/close 28247 (time 1709788276.20 total 969.55 last 29.79) - open/close 28578 (time 1709788286.21 total 979.56 last 33.04) - open/close 28912 (time 1709788296.23 total 989.58 last 33.34) - open/close 29230 (time 1709788306.23 total 999.58 last 31.79) - open/close 29563 (time 1709788316.25 total 1009.60 last 33.24) - open/close 29871 (time 1709788326.26 total 1019.61 last 30.79) - open/close 30000 (time 1709788329.96 total 1023.31 last 34.82) - open/close 30314 (time 1709788339.97 total 1033.33 last 31.36) - open/close 30615 (time 1709788349.98 total 1043.33 last 30.09) - open/close 30911 (time 1709788359.99 total 1053.34 last 29.55) - open/close 31210 (time 1709788370.01 total 1063.36 last 29.85) - open/close 31523 (time 1709788380.01 total 1073.36 last 31.30) - open/close 31824 (time 1709788390.02 total 1083.37 last 30.07) - open/close 32138 (time 1709788400.03 total 1093.38 last 31.36) - open/close 32445 (time 1709788410.04 total 1103.39 last 30.67) - open/close 32741 (time 1709788420.04 total 1113.39 last 29.60) - open/close 33033 (time 1709788430.07 total 1123.42 last 29.13) - open/close 33331 (time 1709788440.09 total 1133.44 last 29.72) - open/close 33634 (time 1709788450.12 total 1143.47 last 30.23) - open/close 33919 (time 1709788460.12 total 1153.47 last 28.49) - open/close 34222 (time 1709788470.13 total 1163.48 last 30.27) - open/close 34520 (time 1709788480.17 total 1173.52 last 29.70) - open/close 34620 (time 1709788494.48 total 1187.83 last 6.99) - open/close 34935 (time 1709788504.51 total 1197.86 last 31.40) - open/close 35247 (time 1709788514.54 total 1207.89 last 31.11) - open/close 35520 (time 1709788524.55 total 1217.90 last 27.28) - open/close 35805 (time 1709788534.56 total 1227.91 last 28.47) - open/close 36089 (time 1709788544.56 total 1237.91 last 28.38) - open/close 36351 (time 1709788554.59 total 1247.94 last 26.13) - open/close 36601 (time 1709788564.59 total 1257.94 last 24.99) - open/close 36875 (time 1709788574.62 total 1267.97 last 27.33) - open/close 37150 (time 1709788584.62 total 1277.97 last 27.49) - open/close 37433 (time 1709788594.64 total 1287.99 last 28.24) - open/close 37720 (time 1709788604.67 total 1298.02 last 28.62) - open/close 38003 (time 1709788614.68 total 1308.03 last 28.27) - open/close 38300 (time 1709788624.69 total 1318.04 last 29.69) - open/close 38605 (time 1709788634.71 total 1328.06 last 30.45) - open/close 38910 (time 1709788644.74 total 1338.09 last 30.41) - open/close 39205 (time 1709788654.77 total 1348.12 last 29.39) - open/close 39497 (time 1709788664.80 total 1358.15 last 29.12) - open/close 39785 (time 1709788674.82 total 1368.17 last 28.75) - open/close 40000 (time 1709788682.36 total 1375.71 last 28.49) - open/close 40301 (time 1709788692.38 total 1385.73 last 30.05) - open/close 40590 (time 1709788702.39 total 1395.74 last 28.87) - open/close 40869 (time 1709788712.42 total 1405.77 last 27.82) - open/close 41127 (time 1709788722.44 total 1415.79 last 25.75) - open/close 41397 (time 1709788732.44 total 1425.79 last 26.98) - open/close 41666 (time 1709788742.45 total 1435.80 last 26.89) - open/close 41921 (time 1709788752.48 total 1445.83 last 25.41) - open/close 42184 (time 1709788762.52 total 1455.87 last 26.20) - open/close 42456 (time 1709788772.55 total 1465.90 last 27.12) - open/close 42712 (time 1709788782.55 total 1475.90 last 25.59) - open/close 42964 (time 1709788792.58 total 1485.93 last 25.13) - open/close 43213 (time 1709788802.62 total 1495.97 last 24.81) - open/close 43461 (time 1709788812.63 total 1505.98 last 24.76) - open/close 43705 (time 1709788822.68 total 1516.03 last 24.28) - open/close 43956 (time 1709788832.70 total 1526.05 last 25.05) - open/close 44204 (time 1709788842.72 total 1536.07 last 24.75) - open/close 44433 (time 1709788852.76 total 1546.11 last 22.82) - open/close 44682 (time 1709788862.78 total 1556.13 last 24.86) - open/close 44932 (time 1709788872.81 total 1566.16 last 24.92) - open/close 45180 (time 1709788882.85 total 1576.20 last 24.71) - open/close 45412 (time 1709788892.86 total 1586.21 last 23.18) - open/close 45651 (time 1709788902.86 total 1596.21 last 23.89) - open/close 45891 (time 1709788912.87 total 1606.22 last 23.97) - open/close 46123 (time 1709788922.89 total 1616.24 last 23.15) - open/close 46360 (time 1709788932.93 total 1626.28 last 23.62) - open/close 46596 (time 1709788942.96 total 1636.31 last 23.52) - open/close 46830 (time 1709788952.99 total 1646.35 last 23.32) - open/close 47065 (time 1709788963.00 total 1656.35 last 23.50) - open/close 47294 (time 1709788973.02 total 1666.37 last 22.85) - open/close 47523 (time 1709788983.04 total 1676.39 last 22.84) - open/close 47753 (time 1709788993.08 total 1686.43 last 22.92) - open/close 47984 (time 1709789003.11 total 1696.46 last 23.02) - open/close 48216 (time 1709789013.12 total 1706.47 last 23.18) - open/close 48442 (time 1709789023.14 total 1716.49 last 22.56) - open/close 48671 (time 1709789033.16 total 1726.51 last 22.84) - open/close 48905 (time 1709789043.20 total 1736.55 last 23.32) - open/close 49144 (time 1709789053.21 total 1746.56 last 23.86) - open/close 49377 (time 1709789063.23 total 1756.58 last 23.26) - open/close 49619 (time 1709789073.23 total 1766.58 last 24.19) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1769.94 seconds: 28.08 ops/second - unlinked 0 (time 1709789077 ; total 0 ; last 0) - unlinked 10000 (time 1709789590 ; total 513 ; last 513) - unlinked 20000 (time 1709789988 ; total 911 ; last 398) - unlinked 30000 (time 1709790222 ; total 1145 ; last 234) - unlinked 40000 (time 1709790300 ; total 1223 ; last 78) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1245 seconds: 39.918072 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg427-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server unloading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 PASS 101a (3054s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg427-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg427-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg427-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 oleg427-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg427-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:45:36 (1709790336) start mds service on oleg427-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: detected 4 online CPUs by sysfs oleg427-server: Force libcfs to create 2 CPU partitions oleg427-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg427-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg427-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg427-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg427-client.virtnet: -o user_xattr,flock oleg427-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server start ost1 service on oleg427-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 12 sec oleg427-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800a906e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a906e000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg427-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server unloading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 PASS 101b (81s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg427-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg427-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg427-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg427-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 oleg427-server: oleg427-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: detected 4 online CPUs by sysfs oleg427-server: Force libcfs to create 2 CPU partitions oleg427-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg427-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3684 sec ========== 00:47:12 (1709790432) === conf-sanity: start cleanup 00:47:12 (1709790432) === === conf-sanity: finish cleanup 00:47:12 (1709790432) ===