-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:39:19 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg359-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg359-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg359-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg359-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg359-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg359-server oleg359-server: oleg359-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing load_modules_local oleg359-server: Loading modules from /home/green/git/lustre-release/lustre oleg359-server: detected 4 online CPUs by sysfs oleg359-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg359-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg359-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg359-server: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg359-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg359-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg359-server stop mds service on oleg359-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg359-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:40:25 (1709786425) start mds service on oleg359-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg359-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg359-client.virtnet: -o user_xattr,flock oleg359-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h40m41s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff88012a494000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:41:25.000000000 -0500 Modify: 2024-03-06 23:41:25.000000000 -0500 Change: 2024-03-06 23:41:25.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 72 lines, 72 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg359-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg359-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg359-server stop mds service on oleg359-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg359-server unloading modules on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing unload_modules_local modules unloaded. oleg359-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg359-server: Is the ldiskfs module available? oleg359-server: oleg359-server: tunefs.lustre FATAL: failed to write local files oleg359-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg359-client: oleg359-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg359-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg359-server: Is the ldiskfs module available? oleg359-server: oleg359-server: tunefs.lustre FATAL: failed to write local files oleg359-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg359-client: oleg359-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.159@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.159@tcp sys.timeout=20 oleg359-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg359-server: Is the ldiskfs module available? oleg359-server: oleg359-server: tunefs.lustre FATAL: failed to write local files oleg359-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg359-client: oleg359-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.159@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.159@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg359-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg359-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2 oleg359-server: oleg359-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing load_modules_local oleg359-server: Loading modules from /home/green/git/lustre-release/lustre oleg359-server: detected 4 online CPUs by sysfs oleg359-server: Force libcfs to create 2 CPU partitions oleg359-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg359-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg359-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg359-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg359-server: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg359-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg359-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg359-server stop mds service on oleg359-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg359-server PASS 35b (119s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:42:27 (1709786547) start mds service on oleg359-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg359-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg359-client.virtnet: -o user_xattr,flock oleg359-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg359-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg359-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg359-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg359-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg359-client.virtnet: -o user_xattr,flock oleg359-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg359-client.virtnet /mnt/lustre (opts:) stop mds service on oleg359-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg359-server 000000 2 000008 oleg359-server: 8+0 records in oleg359-server: 8+0 records out oleg359-server: 4096 bytes (4.1 kB) copied, 0.00859296 s, 477 kB/s start mds service on oleg359-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg359-client.virtnet: -o user_xattr,flock oleg359-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg359-client.virtnet /mnt/lustre (opts:) stop mds service on oleg359-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg359-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg359-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg359-server stop mds service on oleg359-server unloading modules on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-client: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-client: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2 PASS 38 (94s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg359-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg359-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg359-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2 oleg359-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg359-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:44:05 (1709786645) Stopping clients: oleg359-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg359-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2 oleg359-server: oleg359-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing load_modules_local oleg359-server: Loading modules from /home/green/git/lustre-release/lustre oleg359-server: detected 4 online CPUs by sysfs oleg359-server: Force libcfs to create 2 CPU partitions oleg359-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg359-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg359-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg359-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg359-client.virtnet: -o user_xattr,flock oleg359-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg359-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg359-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg359-server umount lustre on /mnt/lustre..... Stopping client oleg359-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg359-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg359-server stop mds service on oleg359-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg359-server unloading modules on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing unload_modules_local modules unloaded. oleg359-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg359-server: Is the ldiskfs module available? oleg359-server: oleg359-server: tunefs.lustre FATAL: failed to write local files oleg359-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg359-client: oleg359-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg359-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg359-server: Is the ldiskfs module available? oleg359-server: oleg359-server: tunefs.lustre FATAL: failed to write local files oleg359-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg359-client: oleg359-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.159@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.159@tcp sys.timeout=20 oleg359-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg359-server: Is the ldiskfs module available? oleg359-server: oleg359-server: tunefs.lustre FATAL: failed to write local files oleg359-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg359-client: oleg359-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.159@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.159@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg359-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg359-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2 oleg359-server: oleg359-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing load_modules_local oleg359-server: Loading modules from /home/green/git/lustre-release/lustre oleg359-server: detected 4 online CPUs by sysfs oleg359-server: Force libcfs to create 2 CPU partitions oleg359-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg359-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg359-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg359-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg359-server: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg359-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg359-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg359-server stop mds service on oleg359-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg359-server PASS 51 (148s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:46:36 (1709786796) start mds service on oleg359-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg359-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg359-client.virtnet: -o user_xattr,flock oleg359-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 262 (time 1709786828.62 total 10.52 last 24.91) - open/close 530 (time 1709786839.10 total 20.99 last 25.58) - open/close 727 (time 1709786849.30 total 31.20 last 19.31) - open/close 1086 (time 1709786859.63 total 41.52 last 34.77) - open/close 1236 (time 1709786869.68 total 51.57 last 14.92) - open/close 1434 (time 1709786879.98 total 61.87 last 19.23) - open/close 1756 (time 1709786890.27 total 72.17 last 31.27) - open/close 1890 (time 1709786900.28 total 82.17 last 13.40) - open/close 2076 (time 1709786911.18 total 93.08 last 17.06) - open/close 2307 (time 1709786921.66 total 103.56 last 22.05) - open/close 2548 (time 1709786932.64 total 114.54 last 21.95) - open/close 2768 (time 1709786942.64 total 124.54 last 22.00) - open/close 2972 (time 1709786952.69 total 134.58 last 20.31) - open/close 3231 (time 1709786962.69 total 144.59 last 25.89) - open/close 3426 (time 1709786972.70 total 154.60 last 19.49) - open/close 3563 (time 1709786983.19 total 165.09 last 13.06) - open/close 3697 (time 1709786993.38 total 175.28 last 13.15) - open/close 3842 (time 1709787003.61 total 185.51 last 14.18) - open/close 4106 (time 1709787013.62 total 195.52 last 26.37) - open/close 4337 (time 1709787024.49 total 206.39 last 21.25) - open/close 4471 (time 1709787035.37 total 217.27 last 12.32) - open/close 4609 (time 1709787045.53 total 227.43 last 13.58) - open/close 4882 (time 1709787055.54 total 237.44 last 27.29) - open/close 5029 (time 1709787066.28 total 248.18 last 13.68) - open/close 5202 (time 1709787077.07 total 258.96 last 16.04) - open/close 5259 (time 1709787087.09 total 268.98 last 5.69) - open/close 5342 (time 1709787097.42 total 279.32 last 8.03) - open/close 5509 (time 1709787107.64 total 289.53 last 16.35) - open/close 5715 (time 1709787118.35 total 300.25 last 19.22) - open/close 5868 (time 1709787128.42 total 310.32 last 15.19) - open/close 6031 (time 1709787138.79 total 320.69 last 15.73) - open/close 6307 (time 1709787148.79 total 330.69 last 27.60) - open/close 6484 (time 1709787159.42 total 341.32 last 16.64) - open/close 6716 (time 1709787170.20 total 352.10 last 21.52) - open/close 6932 (time 1709787181.12 total 363.02 last 19.78) - open/close 7108 (time 1709787191.14 total 373.03 last 17.58) - open/close 7241 (time 1709787201.74 total 383.64 last 12.54) - open/close 7511 (time 1709787212.44 total 394.33 last 25.25) - open/close 7754 (time 1709787222.45 total 404.35 last 24.27) - open/close 7961 (time 1709787232.46 total 414.35 last 20.69) - open/close 8176 (time 1709787242.47 total 424.36 last 21.48) - open/close 8409 (time 1709787253.09 total 434.98 last 21.94) - open/close 8565 (time 1709787263.50 total 445.40 last 14.98) - open/close 8724 (time 1709787274.10 total 456.00 last 15.00) - open/close 8935 (time 1709787284.17 total 466.06 last 20.96) - open/close 9115 (time 1709787294.53 total 476.43 last 17.36) - open/close 9254 (time 1709787304.55 total 486.45 last 13.87) - open/close 9410 (time 1709787314.59 total 496.49 last 15.54) - open/close 9510 (time 1709787324.61 total 506.51 last 9.98) - open/close 9909 (time 1709787334.62 total 516.51 last 39.87) - open/close 10534 (time 1709787344.63 total 526.53 last 62.41) - open/close 11100 (time 1709787354.64 total 536.54 last 56.54) - open/close 11635 (time 1709787364.65 total 546.55 last 53.45) - open/close 12174 (time 1709787374.65 total 556.55 last 53.90) - open/close 12676 (time 1709787384.65 total 566.55 last 50.20) - open/close 13127 (time 1709787394.66 total 576.56 last 45.07) - open/close 13580 (time 1709787404.66 total 586.56 last 45.27) - open/close 13992 (time 1709787414.67 total 596.56 last 41.19) - open/close 14482 (time 1709787424.67 total 606.57 last 48.99) - open/close 14894 (time 1709787434.69 total 616.58 last 41.13) - open/close 15363 (time 1709787444.69 total 626.59 last 46.86) - open/close 15858 (time 1709787454.70 total 636.60 last 49.46) - open/close 16272 (time 1709787464.72 total 646.62 last 41.32) - open/close 16642 (time 1709787474.74 total 656.64 last 36.92) - open/close 17054 (time 1709787484.75 total 666.65 last 41.15) - open/close 17431 (time 1709787494.77 total 676.67 last 37.63) - open/close 17799 (time 1709787504.78 total 686.67 last 36.78) - open/close 18181 (time 1709787514.82 total 696.71 last 38.05) - open/close 18587 (time 1709787524.83 total 706.73 last 40.53) - open/close 18934 (time 1709787534.85 total 716.75 last 34.65) - open/close 19338 (time 1709787544.85 total 726.75 last 40.39) - open/close 19682 (time 1709787554.86 total 736.76 last 34.37) - open/close 19987 (time 1709787564.95 total 746.85 last 30.23) - open/close 20261 (time 1709787574.96 total 756.85 last 27.38) - open/close 20574 (time 1709787584.97 total 766.87 last 31.25) - open/close 20852 (time 1709787594.97 total 776.87 last 27.80) - open/close 21154 (time 1709787604.98 total 786.88 last 30.18) - open/close 21448 (time 1709787614.99 total 796.88 last 29.38) - open/close 21753 (time 1709787625.02 total 806.92 last 30.40) - open/close 22042 (time 1709787635.07 total 816.97 last 28.75) - open/close 22319 (time 1709787645.10 total 826.99 last 27.64) - open/close 22622 (time 1709787655.10 total 837.00 last 30.28) - open/close 22959 (time 1709787665.14 total 847.04 last 33.57) - open/close 23253 (time 1709787675.15 total 857.05 last 29.36) - open/close 23548 (time 1709787685.16 total 867.06 last 29.49) - open/close 23803 (time 1709787695.19 total 877.09 last 25.42) - open/close 24084 (time 1709787705.21 total 887.10 last 28.06) - open/close 24367 (time 1709787715.26 total 897.16 last 28.15) - open/close 24672 (time 1709787725.29 total 907.19 last 30.39) - open/close 24983 (time 1709787735.31 total 917.21 last 31.04) - open/close 25255 (time 1709787745.33 total 927.22 last 27.17) - open/close 25502 (time 1709787755.33 total 937.23 last 24.69) - open/close 25820 (time 1709787765.34 total 947.24 last 31.75) - open/close 26070 (time 1709787784.87 total 966.77 last 12.80) - open/close 26355 (time 1709787794.89 total 976.78 last 28.45) - open/close 26685 (time 1709787804.89 total 986.78 last 32.99) - open/close 27003 (time 1709787814.89 total 996.79 last 31.78) - open/close 27288 (time 1709787824.91 total 1006.80 last 28.46) - open/close 27625 (time 1709787834.93 total 1016.82 last 33.63) - open/close 27940 (time 1709787844.94 total 1026.84 last 31.45) - open/close 28269 (time 1709787854.95 total 1036.84 last 32.89) - open/close 28584 (time 1709787864.96 total 1046.85 last 31.46) - open/close 28860 (time 1709787874.97 total 1056.87 last 27.55) - open/close 29147 (time 1709787885.02 total 1066.91 last 28.58) - open/close 29423 (time 1709787895.03 total 1076.93 last 27.56) - open/close 29732 (time 1709787905.07 total 1086.96 last 30.79) - open/close 30000 (time 1709787914.16 total 1096.06 last 29.47) - open/close 30278 (time 1709787924.16 total 1106.06 last 27.79) - open/close 30558 (time 1709787934.18 total 1116.07 last 27.96) - open/close 30822 (time 1709787944.18 total 1126.08 last 26.38) - open/close 31093 (time 1709787954.21 total 1136.11 last 27.03) - open/close 31161 (time 1709787967.59 total 1149.48 last 5.08) - open/close 31440 (time 1709787977.59 total 1159.49 last 27.89) - open/close 31746 (time 1709787987.62 total 1169.52 last 30.51) - open/close 32051 (time 1709787997.64 total 1179.54 last 30.42) - open/close 32354 (time 1709788007.67 total 1189.57 last 30.23) - open/close 32663 (time 1709788017.68 total 1199.57 last 30.87) - open/close 32943 (time 1709788027.70 total 1209.59 last 27.95) - open/close 33211 (time 1709788037.72 total 1219.61 last 26.75) - open/close 33463 (time 1709788047.73 total 1229.63 last 25.16) - open/close 33745 (time 1709788057.73 total 1239.63 last 28.19) - open/close 34022 (time 1709788067.77 total 1249.67 last 27.59) - open/close 34336 (time 1709788077.78 total 1259.68 last 31.39) - open/close 34686 (time 1709788087.78 total 1269.68 last 34.99) - open/close 35019 (time 1709788097.80 total 1279.69 last 33.25) - open/close 35348 (time 1709788107.81 total 1289.71 last 32.84) - open/close 35686 (time 1709788117.82 total 1299.71 last 33.79) - open/close 36018 (time 1709788127.82 total 1309.72 last 33.19) - open/close 36341 (time 1709788137.83 total 1319.73 last 32.26) - open/close 36666 (time 1709788147.84 total 1329.73 last 32.49) - open/close 36981 (time 1709788157.86 total 1339.76 last 31.42) - open/close 37302 (time 1709788167.87 total 1349.76 last 32.08) - open/close 37622 (time 1709788177.89 total 1359.79 last 31.92) - open/close 37941 (time 1709788187.90 total 1369.79 last 31.88) - open/close 38238 (time 1709788197.93 total 1379.83 last 29.59) - open/close 38550 (time 1709788207.94 total 1389.84 last 31.18) - open/close 38857 (time 1709788217.97 total 1399.87 last 30.60) - open/close 39164 (time 1709788228.00 total 1409.90 last 30.61) - open/close 39468 (time 1709788238.02 total 1419.91 last 30.36) - open/close 39776 (time 1709788248.03 total 1429.92 last 30.77) - open/close 40000 (time 1709788255.88 total 1437.78 last 28.51) - open/close 40292 (time 1709788265.91 total 1447.81 last 29.12) - open/close 40584 (time 1709788275.93 total 1457.83 last 29.15) - open/close 40865 (time 1709788285.98 total 1467.87 last 27.98) - open/close 41136 (time 1709788295.99 total 1477.89 last 27.06) - open/close 41416 (time 1709788306.01 total 1487.91 last 27.95) - open/close 41697 (time 1709788316.03 total 1497.93 last 28.04) - open/close 41972 (time 1709788326.06 total 1507.96 last 27.42) - open/close 42258 (time 1709788336.06 total 1517.96 last 28.59) - open/close 42534 (time 1709788346.07 total 1527.96 last 27.59) - open/close 42825 (time 1709788356.07 total 1537.96 last 29.10) - open/close 43105 (time 1709788366.07 total 1547.97 last 27.99) - open/close 43377 (time 1709788376.09 total 1557.98 last 27.16) - open/close 43656 (time 1709788386.10 total 1567.99 last 27.87) - open/close 43927 (time 1709788396.11 total 1578.00 last 27.07) - open/close 44201 (time 1709788406.13 total 1588.03 last 27.33) - open/close 44479 (time 1709788416.17 total 1598.07 last 27.70) - open/close 44759 (time 1709788426.19 total 1608.09 last 27.94) - open/close 45041 (time 1709788436.20 total 1618.10 last 28.17) - open/close 45319 (time 1709788446.21 total 1628.11 last 27.77) - open/close 45586 (time 1709788456.21 total 1638.11 last 26.69) - open/close 45858 (time 1709788466.23 total 1648.12 last 27.17) - open/close 46130 (time 1709788476.26 total 1658.15 last 27.12) - open/close 46379 (time 1709788486.26 total 1668.16 last 24.89) - open/close 46626 (time 1709788496.29 total 1678.19 last 24.62) - open/close 46887 (time 1709788506.34 total 1688.24 last 25.97) - open/close 47148 (time 1709788516.35 total 1698.25 last 26.08) - open/close 47411 (time 1709788526.35 total 1708.25 last 26.29) - open/close 47667 (time 1709788536.39 total 1718.29 last 25.50) - open/close 47926 (time 1709788546.40 total 1728.29 last 25.88) - open/close 48184 (time 1709788556.43 total 1738.32 last 25.72) - open/close 48437 (time 1709788566.44 total 1748.33 last 25.27) - open/close 48700 (time 1709788576.48 total 1758.38 last 26.19) - open/close 48955 (time 1709788586.49 total 1768.39 last 25.47) - open/close 49210 (time 1709788596.52 total 1778.42 last 25.42) - open/close 49469 (time 1709788606.54 total 1788.43 last 25.86) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1797.41 seconds: 27.65 ops/second - unlinked 0 (time 1709788616 ; total 0 ; last 0) - unlinked 10000 (time 1709789076 ; total 460 ; last 460) - unlinked 20000 (time 1709789449 ; total 833 ; last 373) - unlinked 30000 (time 1709789719 ; total 1103 ; last 270) - unlinked 40000 (time 1709789870 ; total 1254 ; last 151) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1302 seconds: 38.170506 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg359-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg359-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg359-server stop mds service on oleg359-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg359-server unloading modules on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg359-client: oleg359-client: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-client: ssh exited with exit code 2 PASS 101a (3134s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg359-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg359-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg359-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2 oleg359-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg359-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:38:53 (1709789933) start mds service on oleg359-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing load_modules_local oleg359-server: Loading modules from /home/green/git/lustre-release/lustre oleg359-server: detected 4 online CPUs by sysfs oleg359-server: Force libcfs to create 2 CPU partitions oleg359-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg359-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg359-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg359-client.virtnet: -o user_xattr,flock oleg359-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg359-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg359-server start ost1 service on oleg359-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec oleg359-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800a9d7a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a9d7a000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg359-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg359-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg359-server stop mds service on oleg359-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg359-server unloading modules on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg359-client: oleg359-client: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-client: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2 PASS 101b (88s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg359-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg359-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg359-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg359-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2 oleg359-server: oleg359-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing load_modules_local oleg359-server: Loading modules from /home/green/git/lustre-release/lustre oleg359-server: detected 4 online CPUs by sysfs oleg359-server: Force libcfs to create 2 CPU partitions oleg359-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg359-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3682 sec ========== 00:40:42 (1709790042) === conf-sanity: start cleanup 00:40:42 (1709790042) === === conf-sanity: finish cleanup 00:40:42 (1709790042) ===