-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:13:03 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg350-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg350-server oleg350-server: oleg350-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: detected 4 online CPUs by sysfs oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg350-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg350-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:14:05 (1709784845) start mds service on oleg350-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg350-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock oleg350-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h14m20s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800aa221000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:15:03.000000000 -0500 Modify: 2024-03-06 23:15:03.000000000 -0500 Change: 2024-03-06 23:15:03.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 54 lines, 54 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server unloading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local modules unloaded. oleg350-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg350-server: Is the ldiskfs module available? oleg350-server: oleg350-server: tunefs.lustre FATAL: failed to write local files oleg350-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg350-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg350-server: Is the ldiskfs module available? oleg350-server: oleg350-server: tunefs.lustre FATAL: failed to write local files oleg350-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 oleg350-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg350-server: Is the ldiskfs module available? oleg350-server: oleg350-server: tunefs.lustre FATAL: failed to write local files oleg350-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 oleg350-server: oleg350-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: detected 4 online CPUs by sysfs oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: libkmod: kmod_module_get_size: failed to read coresize from /sys/module/acpi_cpufreq oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg350-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg350-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server PASS 35b (110s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:15:57 (1709784957) start mds service on oleg350-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg350-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock oleg350-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg350-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock oleg350-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server 000000 2 000008 oleg350-server: 8+0 records in oleg350-server: 8+0 records out oleg350-server: 4096 bytes (4.1 kB) copied, 0.00038342 s, 10.7 MB/s start mds service on oleg350-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock oleg350-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server unloading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 38 (57s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg350-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg350-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg350-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 oleg350-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg350-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:16:56 (1709785016) Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 oleg350-server: oleg350-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: detected 4 online CPUs by sysfs oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg350-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg350-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock oleg350-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg350-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg350-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg350-server umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server unloading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local modules unloaded. oleg350-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg350-server: Is the ldiskfs module available? oleg350-server: oleg350-server: tunefs.lustre FATAL: failed to write local files oleg350-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg350-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg350-server: Is the ldiskfs module available? oleg350-server: oleg350-server: tunefs.lustre FATAL: failed to write local files oleg350-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 oleg350-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg350-server: Is the ldiskfs module available? oleg350-server: oleg350-server: tunefs.lustre FATAL: failed to write local files oleg350-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 oleg350-server: oleg350-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: detected 4 online CPUs by sysfs oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg350-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg350-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server PASS 51 (120s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:18:58 (1709785138) start mds service on oleg350-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg350-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock oleg350-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 695 (time 1709785164.35 total 10.35 last 67.18) - open/close 1368 (time 1709785174.86 total 20.85 last 64.06) - open/close 2152 (time 1709785185.71 total 31.70 last 72.25) - open/close 2929 (time 1709785196.56 total 42.56 last 71.60) - open/close 3599 (time 1709785207.30 total 53.29 last 62.41) - open/close 4223 (time 1709785217.97 total 63.97 last 58.44) - open/close 4813 (time 1709785228.68 total 74.67 last 55.11) - open/close 5412 (time 1709785239.36 total 85.35 last 56.11) - open/close 5978 (time 1709785249.36 total 95.35 last 56.60) - open/close 6349 (time 1709785259.76 total 105.75 last 35.66) - open/close 6828 (time 1709785270.60 total 116.59 last 44.20) - open/close 7019 (time 1709785281.48 total 127.47 last 17.55) - open/close 7365 (time 1709785292.00 total 137.99 last 32.91) - open/close 7662 (time 1709785302.28 total 148.27 last 28.88) - open/close 8196 (time 1709785312.28 total 158.27 last 53.40) - open/close 8665 (time 1709785322.34 total 168.33 last 46.61) - open/close 9165 (time 1709785332.63 total 178.63 last 48.58) - open/close 9628 (time 1709785342.67 total 188.66 last 46.14) - open/close 9947 (time 1709785353.28 total 199.27 last 30.06) - open/close 10430 (time 1709785364.72 total 210.72 last 42.21) - open/close 10775 (time 1709785376.56 total 222.56 last 29.13) - open/close 11019 (time 1709785386.95 total 232.94 last 23.50) - open/close 11431 (time 1709785397.43 total 243.42 last 39.30) - open/close 12550 (time 1709785407.43 total 253.43 last 111.85) - open/close 13938 (time 1709785417.44 total 263.43 last 138.74) - open/close 15263 (time 1709785427.44 total 273.43 last 132.49) - open/close 16546 (time 1709785437.44 total 283.44 last 128.24) - open/close 17739 (time 1709785447.45 total 293.44 last 119.23) - open/close 18913 (time 1709785457.45 total 303.45 last 117.36) - open/close 20000 (time 1709785467.26 total 313.25 last 110.84) - open/close 21031 (time 1709785477.27 total 323.26 last 103.02) - open/close 21978 (time 1709785487.27 total 333.27 last 94.65) - open/close 22872 (time 1709785497.28 total 343.28 last 89.30) - open/close 23718 (time 1709785507.28 total 353.28 last 84.60) - open/close 24542 (time 1709785517.29 total 363.29 last 82.32) - open/close 25448 (time 1709785527.30 total 373.29 last 90.55) - open/close 26274 (time 1709785537.31 total 383.31 last 82.49) - open/close 27077 (time 1709785547.32 total 393.31 last 80.26) - open/close 27836 (time 1709785557.32 total 403.31 last 75.88) - open/close 28614 (time 1709785567.33 total 413.32 last 77.76) - open/close 29409 (time 1709785577.34 total 423.33 last 79.42) - open/close 30000 (time 1709785584.96 total 430.95 last 77.56) - open/close 30748 (time 1709785594.96 total 440.96 last 74.74) - open/close 31455 (time 1709785604.98 total 450.97 last 70.62) - open/close 32144 (time 1709785614.99 total 460.98 last 68.82) - open/close 32787 (time 1709785624.99 total 470.98 last 64.29) - open/close 33463 (time 1709785635.00 total 480.99 last 67.54) - open/close 34045 (time 1709785645.01 total 491.00 last 58.13) - open/close 34648 (time 1709785655.02 total 501.01 last 60.25) - open/close 35223 (time 1709785665.03 total 511.02 last 57.47) - open/close 35870 (time 1709785675.04 total 521.03 last 64.62) - open/close 36443 (time 1709785685.04 total 531.03 last 57.30) - open/close 36931 (time 1709785695.04 total 541.03 last 48.80) - open/close 37457 (time 1709785705.05 total 551.04 last 52.56) - open/close 37960 (time 1709785715.06 total 561.05 last 50.23) - open/close 38524 (time 1709785725.07 total 571.06 last 56.37) - open/close 39087 (time 1709785735.08 total 581.07 last 56.20) - open/close 39646 (time 1709785745.09 total 591.08 last 55.86) - open/close 40000 (time 1709785751.32 total 597.31 last 56.82) - open/close 40544 (time 1709785761.32 total 607.31 last 54.40) - open/close 41086 (time 1709785771.33 total 617.33 last 54.13) - open/close 41622 (time 1709785781.34 total 627.33 last 53.55) - open/close 42132 (time 1709785791.35 total 637.35 last 50.94) - open/close 42664 (time 1709785801.35 total 647.35 last 53.20) - open/close 43166 (time 1709785811.37 total 657.36 last 50.11) - open/close 43689 (time 1709785821.37 total 667.37 last 52.29) - open/close 44161 (time 1709785831.38 total 677.38 last 47.15) - open/close 44693 (time 1709785841.39 total 687.38 last 53.19) - open/close 45204 (time 1709785851.40 total 697.39 last 51.06) - open/close 45713 (time 1709785861.40 total 707.39 last 50.87) - open/close 46214 (time 1709785871.41 total 717.41 last 50.04) - open/close 46740 (time 1709785881.42 total 727.41 last 52.60) - open/close 47201 (time 1709785891.42 total 737.41 last 46.09) - open/close 47679 (time 1709785901.43 total 747.42 last 47.75) - open/close 48149 (time 1709785911.43 total 757.43 last 46.97) - open/close 48620 (time 1709785921.44 total 767.43 last 47.09) - open/close 49081 (time 1709785931.45 total 777.44 last 46.03) - open/close 49544 (time 1709785941.45 total 787.45 last 46.29) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 790.67 seconds: 62.86 ops/second - unlinked 0 (time 1709785945 ; total 0 ; last 0) - unlinked 10000 (time 1709786210 ; total 265 ; last 265) - unlinked 20000 (time 1709786446 ; total 501 ; last 236) - unlinked 30000 (time 1709786643 ; total 698 ; last 197) - unlinked 40000 (time 1709786755 ; total 810 ; last 112) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 850 seconds: 58.468235 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server unloading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 101a (1674s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg350-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg350-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg350-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 oleg350-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg350-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 23:46:55 (1709786815) start mds service on oleg350-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: detected 4 online CPUs by sysfs oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg350-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock oleg350-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server start ost1 service on oleg350-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec oleg350-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b6f85000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6f85000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server unloading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 101b (94s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg350-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg350-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 oleg350-server: oleg350-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: detected 4 online CPUs by sysfs oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 2158 sec ========== 23:49:02 (1709786942) === conf-sanity: start cleanup 23:49:03 (1709786943) === === conf-sanity: finish cleanup 23:49:03 (1709786943) ===