-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:43:01 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg204-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg204-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg204-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg204-server oleg204-server: oleg204-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing load_modules_local oleg204-server: Loading modules from /home/green/git/lustre-release/lustre oleg204-server: detected 4 online CPUs by sysfs oleg204-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg204-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg204-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg204-server: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg204-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:44:17 (1709786657) start mds service on oleg204-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg204-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock oleg204-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h44m36s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800ab1a2800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:45:23.000000000 -0500 Modify: 2024-03-06 23:45:23.000000000 -0500 Change: 2024-03-06 23:45:23.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 76 lines, 76 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server unloading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing unload_modules_local modules unloaded. oleg204-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg204-server: Is the ldiskfs module available? oleg204-server: oleg204-server: tunefs.lustre FATAL: failed to write local files oleg204-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg204-client: oleg204-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg204-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg204-server: Is the ldiskfs module available? oleg204-server: oleg204-server: tunefs.lustre FATAL: failed to write local files oleg204-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg204-client: oleg204-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.104@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.104@tcp sys.timeout=20 oleg204-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg204-server: Is the ldiskfs module available? oleg204-server: oleg204-server: tunefs.lustre FATAL: failed to write local files oleg204-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg204-client: oleg204-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.104@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.104@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg204-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg204-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 oleg204-server: oleg204-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing load_modules_local oleg204-server: Loading modules from /home/green/git/lustre-release/lustre oleg204-server: detected 4 online CPUs by sysfs oleg204-server: Force libcfs to create 2 CPU partitions oleg204-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg204-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg204-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg204-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg204-server: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg204-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server PASS 35b (143s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:46:42 (1709786802) start mds service on oleg204-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg204-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock oleg204-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg204-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock oleg204-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server 000000 2 000008 oleg204-server: 8+0 records in oleg204-server: 8+0 records out oleg204-server: 4096 bytes (4.1 kB) copied, 0.00166642 s, 2.5 MB/s start mds service on oleg204-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock oleg204-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server unloading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 PASS 38 (96s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg204-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg204-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg204-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 oleg204-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg204-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:48:21 (1709786901) Stopping clients: oleg204-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg204-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 oleg204-server: oleg204-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing load_modules_local oleg204-server: Loading modules from /home/green/git/lustre-release/lustre oleg204-server: detected 4 online CPUs by sysfs oleg204-server: Force libcfs to create 2 CPU partitions oleg204-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg204-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg204-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg204-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock oleg204-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg204-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg204-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg204-server umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server unloading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing unload_modules_local modules unloaded. oleg204-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg204-server: Is the ldiskfs module available? oleg204-server: oleg204-server: tunefs.lustre FATAL: failed to write local files oleg204-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg204-client: oleg204-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg204-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg204-server: Is the ldiskfs module available? oleg204-server: oleg204-server: tunefs.lustre FATAL: failed to write local files oleg204-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg204-client: oleg204-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.104@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.104@tcp sys.timeout=20 oleg204-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg204-server: Is the ldiskfs module available? oleg204-server: oleg204-server: tunefs.lustre FATAL: failed to write local files oleg204-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg204-client: oleg204-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.104@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.104@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg204-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg204-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 oleg204-server: oleg204-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing load_modules_local oleg204-server: Loading modules from /home/green/git/lustre-release/lustre oleg204-server: detected 4 online CPUs by sysfs oleg204-server: Force libcfs to create 2 CPU partitions oleg204-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg204-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg204-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg204-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg204-server: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg204-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server PASS 51 (203s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:51:48 (1709787108) start mds service on oleg204-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg204-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock oleg204-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 147 (time 1709787145.59 total 10.95 last 13.42) - open/close 358 (time 1709787155.63 total 20.99 last 21.03) - open/close 587 (time 1709787166.46 total 31.82 last 21.14) - open/close 694 (time 1709787176.47 total 41.83 last 10.68) - open/close 875 (time 1709787186.56 total 51.92 last 17.95) - open/close 1076 (time 1709787196.58 total 61.94 last 20.05) - open/close 1314 (time 1709787207.16 total 72.52 last 22.51) - open/close 1508 (time 1709787217.73 total 83.09 last 18.34) - open/close 1740 (time 1709787227.90 total 93.26 last 22.82) - open/close 1944 (time 1709787238.21 total 103.57 last 19.79) - open/close 2227 (time 1709787249.12 total 114.48 last 25.94) - open/close 2377 (time 1709787259.14 total 124.50 last 14.97) - open/close 2543 (time 1709787270.00 total 135.36 last 15.27) - open/close 2757 (time 1709787280.66 total 146.02 last 20.08) - open/close 2933 (time 1709787291.64 total 157.00 last 16.04) - open/close 3105 (time 1709787301.64 total 167.00 last 17.20) - open/close 3219 (time 1709787312.58 total 177.94 last 10.42) - open/close 3446 (time 1709787322.99 total 188.35 last 21.81) - open/close 3629 (time 1709787333.01 total 198.37 last 18.27) - open/close 3831 (time 1709787343.71 total 209.07 last 18.87) - open/close 4025 (time 1709787354.51 total 219.87 last 17.96) - open/close 4149 (time 1709787364.79 total 230.16 last 12.06) - open/close 4295 (time 1709787374.81 total 240.17 last 14.58) - open/close 4452 (time 1709787385.01 total 250.37 last 15.39) - open/close 4621 (time 1709787395.28 total 260.64 last 16.45) - open/close 4801 (time 1709787405.28 total 270.64 last 18.00) - open/close 5019 (time 1709787415.28 total 280.64 last 21.80) - open/close 5180 (time 1709787425.50 total 290.86 last 15.76) - open/close 5421 (time 1709787435.52 total 300.88 last 24.06) - open/close 5628 (time 1709787446.14 total 311.50 last 19.49) - open/close 5815 (time 1709787457.10 total 322.47 last 17.05) - open/close 5958 (time 1709787467.12 total 332.48 last 14.27) - open/close 6128 (time 1709787477.82 total 343.18 last 15.90) - open/close 6243 (time 1709787487.83 total 353.19 last 11.48) - open/close 6432 (time 1709787497.84 total 363.20 last 18.89) - open/close 6761 (time 1709787507.84 total 373.20 last 32.90) - open/close 6903 (time 1709787517.86 total 383.22 last 14.17) - open/close 7128 (time 1709787528.26 total 393.62 last 21.63) - open/close 7284 (time 1709787538.45 total 403.81 last 15.32) - open/close 7457 (time 1709787548.57 total 413.93 last 17.08) - open/close 7636 (time 1709787558.83 total 424.19 last 17.45) - open/close 7868 (time 1709787569.80 total 435.16 last 21.15) - open/close 8093 (time 1709787579.81 total 445.17 last 22.49) - open/close 8256 (time 1709787589.83 total 455.19 last 16.27) - open/close 8379 (time 1709787599.85 total 465.21 last 12.27) - open/close 8537 (time 1709787610.54 total 475.90 last 14.79) - open/close 8645 (time 1709787629.65 total 495.01 last 5.65) - open/close 8823 (time 1709787639.66 total 505.02 last 17.77) - open/close 9023 (time 1709787649.69 total 515.05 last 19.94) - open/close 9244 (time 1709787660.61 total 525.97 last 20.25) - open/close 9410 (time 1709787670.61 total 535.97 last 16.59) - open/close 9569 (time 1709787681.43 total 546.79 last 14.70) - open/close 9769 (time 1709787691.54 total 556.90 last 19.79) - open/close 9915 (time 1709787701.54 total 566.90 last 14.60) - open/close 10070 (time 1709787711.56 total 576.92 last 15.47) - open/close 10206 (time 1709787721.80 total 587.16 last 13.27) - open/close 10321 (time 1709787731.81 total 597.17 last 11.49) - open/close 10436 (time 1709787741.83 total 607.19 last 11.48) - open/close 10872 (time 1709787751.84 total 617.20 last 43.54) - open/close 11332 (time 1709787761.85 total 627.21 last 45.96) - open/close 11775 (time 1709787771.86 total 637.23 last 44.22) - open/close 12188 (time 1709787781.87 total 647.23 last 41.28) - open/close 12608 (time 1709787791.89 total 657.25 last 41.92) - open/close 13044 (time 1709787801.90 total 667.26 last 43.54) - open/close 13448 (time 1709787811.91 total 677.27 last 40.37) - open/close 13833 (time 1709787821.93 total 687.29 last 38.43) - open/close 14229 (time 1709787831.95 total 697.31 last 39.52) - open/close 14611 (time 1709787841.97 total 707.33 last 38.11) - open/close 14944 (time 1709787852.00 total 717.36 last 33.20) - open/close 15266 (time 1709787862.01 total 727.37 last 32.17) - open/close 15601 (time 1709787872.04 total 737.40 last 33.41) - open/close 15941 (time 1709787882.04 total 747.40 last 33.98) - open/close 16256 (time 1709787892.05 total 757.41 last 31.48) - open/close 16527 (time 1709787902.05 total 767.41 last 27.10) - open/close 16844 (time 1709787912.05 total 777.41 last 31.68) - open/close 17175 (time 1709787922.05 total 787.41 last 33.10) - open/close 17488 (time 1709787932.06 total 797.42 last 31.28) - open/close 17813 (time 1709787942.07 total 807.44 last 32.45) - open/close 18173 (time 1709787952.09 total 817.45 last 35.94) - open/close 18592 (time 1709787962.11 total 827.47 last 41.82) - open/close 18978 (time 1709787972.13 total 837.49 last 38.53) - open/close 19340 (time 1709787982.14 total 847.51 last 36.14) - open/close 19716 (time 1709787992.15 total 857.51 last 37.59) - open/close 20000 (time 1709787999.07 total 864.43 last 41.04) - open/close 20452 (time 1709788009.08 total 874.44 last 45.14) - open/close 20913 (time 1709788019.09 total 884.45 last 46.06) - open/close 21390 (time 1709788029.10 total 894.46 last 47.64) - open/close 21887 (time 1709788039.11 total 904.47 last 49.66) - open/close 22355 (time 1709788049.13 total 914.49 last 46.73) - open/close 22837 (time 1709788059.14 total 924.50 last 48.12) - open/close 23274 (time 1709788069.15 total 934.51 last 43.68) - open/close 23708 (time 1709788079.16 total 944.52 last 43.35) - open/close 24163 (time 1709788089.17 total 954.53 last 45.45) - open/close 24589 (time 1709788099.18 total 964.54 last 42.58) - open/close 25016 (time 1709788109.18 total 974.54 last 42.69) - open/close 25437 (time 1709788119.18 total 984.54 last 42.08) - open/close 25856 (time 1709788129.20 total 994.56 last 41.83) - open/close 26273 (time 1709788139.21 total 1004.57 last 41.67) - open/close 26676 (time 1709788149.22 total 1014.58 last 40.26) - open/close 27083 (time 1709788159.23 total 1024.59 last 40.65) - open/close 27493 (time 1709788169.24 total 1034.60 last 40.97) - open/close 27907 (time 1709788179.26 total 1044.62 last 41.33) - open/close 28302 (time 1709788189.26 total 1054.62 last 39.49) - open/close 28692 (time 1709788199.26 total 1064.62 last 38.98) - open/close 29080 (time 1709788209.27 total 1074.63 last 38.77) - open/close 29482 (time 1709788219.27 total 1084.63 last 40.19) - open/close 29876 (time 1709788229.29 total 1094.65 last 39.33) - open/close 30000 (time 1709788232.61 total 1097.97 last 37.33) - open/close 30373 (time 1709788242.61 total 1107.98 last 37.30) - open/close 30756 (time 1709788252.63 total 1117.99 last 38.23) - open/close 31118 (time 1709788262.64 total 1128.00 last 36.18) - open/close 31480 (time 1709788272.64 total 1138.00 last 36.18) - open/close 31804 (time 1709788282.65 total 1148.02 last 32.36) - open/close 32121 (time 1709788292.66 total 1158.02 last 31.68) - open/close 32450 (time 1709788302.68 total 1168.04 last 32.84) - open/close 32782 (time 1709788312.69 total 1178.05 last 33.18) - open/close 33100 (time 1709788322.70 total 1188.06 last 31.75) - open/close 33414 (time 1709788332.71 total 1198.07 last 31.38) - open/close 33729 (time 1709788342.73 total 1208.09 last 31.44) - open/close 34039 (time 1709788352.75 total 1218.11 last 30.94) - open/close 34349 (time 1709788362.75 total 1228.11 last 30.98) - open/close 34660 (time 1709788372.76 total 1238.12 last 31.07) - open/close 34963 (time 1709788382.78 total 1248.14 last 30.26) - open/close 35248 (time 1709788392.81 total 1258.17 last 28.40) - open/close 35548 (time 1709788402.83 total 1268.19 last 29.95) - open/close 35845 (time 1709788412.84 total 1278.20 last 29.66) - open/close 36136 (time 1709788422.87 total 1288.23 last 29.02) - open/close 36435 (time 1709788432.89 total 1298.25 last 29.82) - open/close 36739 (time 1709788442.93 total 1308.29 last 30.30) - open/close 37038 (time 1709788452.96 total 1318.32 last 29.80) - open/close 37330 (time 1709788462.99 total 1328.35 last 29.11) - open/close 37636 (time 1709788473.01 total 1338.37 last 30.55) - open/close 37953 (time 1709788483.02 total 1348.38 last 31.66) - open/close 38263 (time 1709788493.05 total 1358.41 last 30.91) - open/close 38581 (time 1709788503.07 total 1368.43 last 31.74) - open/close 38900 (time 1709788513.07 total 1378.43 last 31.89) - open/close 39209 (time 1709788523.09 total 1388.45 last 30.84) - open/close 39517 (time 1709788533.09 total 1398.45 last 30.79) - open/close 39830 (time 1709788543.11 total 1408.47 last 31.24) - open/close 40000 (time 1709788548.72 total 1414.08 last 30.34) - open/close 40308 (time 1709788558.74 total 1424.10 last 30.73) - open/close 40616 (time 1709788568.75 total 1434.11 last 30.76) - open/close 40919 (time 1709788578.78 total 1444.14 last 30.20) - open/close 41224 (time 1709788588.79 total 1454.15 last 30.48) - open/close 41513 (time 1709788598.81 total 1464.17 last 28.83) - open/close 41808 (time 1709788608.85 total 1474.21 last 29.40) - open/close 42092 (time 1709788618.86 total 1484.22 last 28.37) - open/close 42371 (time 1709788628.87 total 1494.23 last 27.87) - open/close 42638 (time 1709788638.89 total 1504.25 last 26.64) - open/close 42895 (time 1709788648.91 total 1514.27 last 25.66) - open/close 43157 (time 1709788658.92 total 1524.28 last 26.18) - open/close 43423 (time 1709788668.94 total 1534.30 last 26.54) - open/close 43688 (time 1709788678.94 total 1544.30 last 26.49) - open/close 43957 (time 1709788688.95 total 1554.31 last 26.87) - open/close 44228 (time 1709788698.98 total 1564.34 last 27.04) - open/close 44494 (time 1709788708.98 total 1574.34 last 26.59) - open/close 44771 (time 1709788719.01 total 1584.37 last 27.62) - open/close 45043 (time 1709788729.03 total 1594.39 last 27.13) - open/close 45320 (time 1709788739.05 total 1604.41 last 27.65) - open/close 45591 (time 1709788749.06 total 1614.42 last 27.08) - open/close 45856 (time 1709788759.06 total 1624.42 last 26.49) - open/close 46124 (time 1709788769.09 total 1634.45 last 26.72) - open/close 46384 (time 1709788779.10 total 1644.46 last 25.97) - open/close 46643 (time 1709788789.10 total 1654.46 last 25.89) - open/close 46906 (time 1709788799.12 total 1664.48 last 26.26) - open/close 47169 (time 1709788809.14 total 1674.50 last 26.26) - open/close 47429 (time 1709788819.16 total 1684.52 last 25.93) - open/close 47675 (time 1709788829.20 total 1694.56 last 24.51) - open/close 47923 (time 1709788839.22 total 1704.58 last 24.74) - open/close 48170 (time 1709788849.25 total 1714.61 last 24.63) - open/close 48412 (time 1709788859.28 total 1724.64 last 24.12) - open/close 48644 (time 1709788869.30 total 1734.66 last 23.16) - open/close 48885 (time 1709788879.31 total 1744.67 last 24.09) - open/close 49133 (time 1709788889.33 total 1754.70 last 24.73) - open/close 49387 (time 1709788899.34 total 1764.70 last 25.39) - open/close 49636 (time 1709788909.35 total 1774.71 last 24.87) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1777.26 seconds: 27.96 ops/second - unlinked 0 (time 1709788913 ; total 1 ; last 1) - unlinked 10000 (time 1709789371 ; total 459 ; last 458) - unlinked 20000 (time 1709789752 ; total 840 ; last 381) - unlinked 30000 (time 1709790008 ; total 1096 ; last 256) - unlinked 40000 (time 1709790117 ; total 1205 ; last 109) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1233 seconds: 40.306568 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server unloading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 PASS 101a (3048s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg204-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg204-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg204-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 oleg204-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg204-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:42:37 (1709790157) start mds service on oleg204-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing load_modules_local oleg204-server: Loading modules from /home/green/git/lustre-release/lustre oleg204-server: detected 4 online CPUs by sysfs oleg204-server: Force libcfs to create 2 CPU partitions oleg204-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg204-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg204-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg204-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock oleg204-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server start ost1 service on oleg204-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 12 sec oleg204-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012bf92000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012bf92000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server unloading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 PASS 101b (83s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg204-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg204-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg204-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg204-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 oleg204-server: oleg204-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing load_modules_local oleg204-server: Loading modules from /home/green/git/lustre-release/lustre oleg204-server: detected 4 online CPUs by sysfs oleg204-server: Force libcfs to create 2 CPU partitions oleg204-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg204-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3676 sec ========== 00:44:17 (1709790257) === conf-sanity: start cleanup 00:44:17 (1709790257) === === conf-sanity: finish cleanup 00:44:17 (1709790257) ===