-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:40:53 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg330-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg330-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg330-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg330-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg330-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg330-server oleg330-server: oleg330-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: detected 4 online CPUs by sysfs oleg330-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg330-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg330-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg330-server: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg330-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg330-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg330-server stop mds service on oleg330-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg330-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:41:57 (1709786517) start mds service on oleg330-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg330-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Started lustre-OST0000 oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg330-client.virtnet: -o user_xattr,flock oleg330-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h42m15s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff88012aa9c800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:43:01.000000000 -0500 Modify: 2024-03-06 23:43:01.000000000 -0500 Change: 2024-03-06 23:43:01.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 76 lines, 76 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg330-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg330-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg330-server stop mds service on oleg330-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg330-server unloading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing unload_modules_local modules unloaded. oleg330-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg330-server: Is the ldiskfs module available? oleg330-server: oleg330-server: tunefs.lustre FATAL: failed to write local files oleg330-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg330-client: oleg330-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg330-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg330-server: Is the ldiskfs module available? oleg330-server: oleg330-server: tunefs.lustre FATAL: failed to write local files oleg330-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg330-client: oleg330-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.130@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.130@tcp sys.timeout=20 oleg330-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg330-server: Is the ldiskfs module available? oleg330-server: oleg330-server: tunefs.lustre FATAL: failed to write local files oleg330-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg330-client: oleg330-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.130@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.130@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg330-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg330-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2 oleg330-server: oleg330-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: detected 4 online CPUs by sysfs oleg330-server: Force libcfs to create 2 CPU partitions oleg330-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg330-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg330-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg330-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg330-server: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg330-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg330-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg330-server stop mds service on oleg330-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg330-server PASS 35b (156s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:44:36 (1709786676) start mds service on oleg330-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg330-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Started lustre-OST0000 oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg330-client.virtnet: -o user_xattr,flock oleg330-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg330-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg330-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg330-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg330-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg330-client.virtnet: -o user_xattr,flock oleg330-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg330-client.virtnet /mnt/lustre (opts:) stop mds service on oleg330-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg330-server 000000 2 000008 oleg330-server: 8+0 records in oleg330-server: 8+0 records out oleg330-server: 4096 bytes (4.1 kB) copied, 0.000395595 s, 10.4 MB/s start mds service on oleg330-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg330-client.virtnet: -o user_xattr,flock oleg330-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg330-client.virtnet /mnt/lustre (opts:) stop mds service on oleg330-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg330-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg330-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg330-server stop mds service on oleg330-server unloading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2 pdsh@oleg330-client: oleg330-client: ssh exited with exit code 2 pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2 pdsh@oleg330-client: oleg330-client: ssh exited with exit code 2 PASS 38 (76s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg330-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg330-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg330-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2 oleg330-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg330-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:45:55 (1709786755) Stopping clients: oleg330-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg330-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2 oleg330-server: oleg330-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: detected 4 online CPUs by sysfs oleg330-server: Force libcfs to create 2 CPU partitions oleg330-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg330-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg330-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg330-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg330-client.virtnet: -o user_xattr,flock oleg330-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg330-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg330-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg330-server umount lustre on /mnt/lustre..... Stopping client oleg330-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg330-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg330-server stop mds service on oleg330-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg330-server unloading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing unload_modules_local modules unloaded. oleg330-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg330-server: Is the ldiskfs module available? oleg330-server: oleg330-server: tunefs.lustre FATAL: failed to write local files oleg330-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg330-client: oleg330-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg330-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg330-server: Is the ldiskfs module available? oleg330-server: oleg330-server: tunefs.lustre FATAL: failed to write local files oleg330-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg330-client: oleg330-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.130@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.130@tcp sys.timeout=20 oleg330-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg330-server: Is the ldiskfs module available? oleg330-server: oleg330-server: tunefs.lustre FATAL: failed to write local files oleg330-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg330-client: oleg330-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.130@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.130@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg330-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg330-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2 oleg330-server: oleg330-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: detected 4 online CPUs by sysfs oleg330-server: Force libcfs to create 2 CPU partitions oleg330-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg330-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg330-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg330-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg330-server: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg330-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg330-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg330-server stop mds service on oleg330-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg330-server PASS 51 (144s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:48:23 (1709786903) start mds service on oleg330-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg330-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Started lustre-OST0000 oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg330-client.virtnet: -o user_xattr,flock oleg330-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 235 (time 1709786934.71 total 10.00 last 23.50) - open/close 578 (time 1709786945.26 total 20.55 last 32.51) - open/close 830 (time 1709786955.48 total 30.77 last 24.65) - open/close 1000 (time 1709786966.01 total 41.30 last 16.15) - open/close 1181 (time 1709786976.01 total 51.30 last 18.10) - open/close 1388 (time 1709786986.26 total 61.55 last 20.19) - open/close 1672 (time 1709786996.27 total 71.55 last 28.38) - open/close 1786 (time 1709787006.51 total 81.80 last 11.13) - open/close 2025 (time 1709787016.91 total 92.20 last 22.98) - open/close 2299 (time 1709787028.27 total 103.56 last 24.12) - open/close 2469 (time 1709787040.66 total 115.95 last 13.72) - open/close 2798 (time 1709787051.49 total 126.78 last 30.37) - open/close 3038 (time 1709787061.98 total 137.27 last 22.87) - open/close 3288 (time 1709787072.96 total 148.24 last 22.79) - open/close 3560 (time 1709787082.96 total 158.25 last 27.20) - open/close 3798 (time 1709787093.19 total 168.48 last 23.26) - open/close 4073 (time 1709787103.99 total 179.28 last 25.45) - open/close 4261 (time 1709787114.86 total 190.15 last 17.30) - open/close 4509 (time 1709787126.36 total 201.65 last 21.56) - open/close 4701 (time 1709787137.31 total 212.60 last 17.53) - open/close 4764 (time 1709787149.51 total 224.80 last 5.17) - open/close 4873 (time 1709787160.30 total 235.59 last 10.10) - open/close 5101 (time 1709787170.32 total 245.61 last 22.75) - open/close 5212 (time 1709787180.58 total 255.87 last 10.81) - open/close 5437 (time 1709787190.70 total 265.99 last 22.25) - open/close 5552 (time 1709787200.99 total 276.28 last 11.17) - open/close 5668 (time 1709787211.34 total 286.63 last 11.21) - open/close 5859 (time 1709787221.36 total 296.65 last 19.06) - open/close 6083 (time 1709787232.39 total 307.68 last 20.31) - open/close 6264 (time 1709787243.35 total 318.63 last 16.52) - open/close 6397 (time 1709787253.35 total 328.63 last 13.30) - open/close 6587 (time 1709787263.98 total 339.27 last 17.87) - open/close 6711 (time 1709787274.61 total 349.90 last 11.66) - open/close 6853 (time 1709787285.18 total 360.47 last 13.43) - open/close 7059 (time 1709787296.18 total 371.47 last 18.73) - open/close 7261 (time 1709787306.19 total 381.48 last 20.19) - open/close 7397 (time 1709787316.19 total 391.48 last 13.59) - open/close 7623 (time 1709787326.77 total 402.06 last 21.36) - open/close 7843 (time 1709787337.01 total 412.30 last 21.49) - open/close 7938 (time 1709787347.02 total 422.30 last 9.49) - open/close 8094 (time 1709787357.98 total 433.27 last 14.23) - open/close 8293 (time 1709787368.65 total 443.94 last 18.65) - open/close 8481 (time 1709787378.66 total 453.95 last 18.78) - open/close 8603 (time 1709787388.66 total 463.95 last 12.20) - open/close 8726 (time 1709787398.66 total 473.95 last 12.30) - open/close 8944 (time 1709787409.35 total 484.64 last 20.39) - open/close 9081 (time 1709787419.45 total 494.74 last 13.56) - open/close 9220 (time 1709787430.40 total 505.69 last 12.69) - open/close 9393 (time 1709787440.53 total 515.82 last 17.07) - open/close 9539 (time 1709787450.61 total 525.90 last 14.49) - open/close 9627 (time 1709787461.04 total 536.33 last 8.43) - open/close 9757 (time 1709787471.05 total 546.34 last 12.99) - open/close 9943 (time 1709787481.93 total 557.22 last 17.11) - open/close 10000 (time 1709787485.49 total 560.78 last 15.98) - open/close 10127 (time 1709787495.50 total 570.79 last 12.68) - open/close 10661 (time 1709787505.53 total 580.82 last 53.28) - open/close 11224 (time 1709787515.53 total 590.82 last 56.30) - open/close 11730 (time 1709787525.54 total 600.83 last 50.53) - open/close 11822 (time 1709787538.42 total 613.71 last 7.14) - open/close 12364 (time 1709787548.43 total 623.72 last 54.17) - open/close 12875 (time 1709787558.43 total 633.72 last 51.09) - open/close 13322 (time 1709787568.44 total 643.73 last 44.68) - open/close 13770 (time 1709787578.44 total 653.73 last 44.78) - open/close 14218 (time 1709787588.45 total 663.73 last 44.78) - open/close 14643 (time 1709787598.45 total 673.73 last 42.50) - open/close 15123 (time 1709787608.45 total 683.74 last 47.98) - open/close 15561 (time 1709787618.47 total 693.76 last 43.73) - open/close 16050 (time 1709787628.47 total 703.76 last 48.88) - open/close 16499 (time 1709787638.48 total 713.77 last 44.87) - open/close 16905 (time 1709787648.48 total 723.77 last 40.59) - open/close 17329 (time 1709787658.49 total 733.78 last 42.38) - open/close 17735 (time 1709787668.53 total 743.82 last 40.43) - open/close 18121 (time 1709787678.53 total 753.82 last 38.60) - open/close 18476 (time 1709787688.55 total 763.84 last 35.43) - open/close 18817 (time 1709787698.55 total 773.84 last 34.10) - open/close 19162 (time 1709787708.57 total 783.86 last 34.42) - open/close 19486 (time 1709787718.60 total 793.89 last 32.31) - open/close 19876 (time 1709787728.61 total 803.90 last 38.99) - open/close 20000 (time 1709787731.87 total 807.16 last 37.98) - open/close 20346 (time 1709787741.88 total 817.17 last 34.57) - open/close 20693 (time 1709787751.90 total 827.19 last 34.63) - open/close 21051 (time 1709787761.90 total 837.19 last 35.79) - open/close 21399 (time 1709787771.92 total 847.21 last 34.73) - open/close 21733 (time 1709787781.95 total 857.24 last 33.33) - open/close 22086 (time 1709787791.96 total 867.24 last 35.27) - open/close 22479 (time 1709787801.96 total 877.25 last 39.29) - open/close 22865 (time 1709787811.99 total 887.27 last 38.50) - open/close 23207 (time 1709787821.99 total 897.28 last 34.17) - open/close 23559 (time 1709787832.01 total 907.30 last 35.16) - open/close 23912 (time 1709787842.01 total 917.30 last 35.28) - open/close 24257 (time 1709787852.03 total 927.32 last 34.43) - open/close 24602 (time 1709787862.07 total 937.36 last 34.38) - open/close 24923 (time 1709787872.08 total 947.37 last 32.05) - open/close 25227 (time 1709787882.11 total 957.40 last 30.33) - open/close 25557 (time 1709787892.13 total 967.42 last 32.94) - open/close 25906 (time 1709787902.15 total 977.44 last 34.84) - open/close 26262 (time 1709787912.16 total 987.45 last 35.53) - open/close 26578 (time 1709787922.18 total 997.47 last 31.54) - open/close 26887 (time 1709787932.19 total 1007.48 last 30.87) - open/close 27186 (time 1709787942.21 total 1017.50 last 29.85) - open/close 27460 (time 1709787952.22 total 1027.51 last 27.36) - open/close 27761 (time 1709787962.24 total 1037.53 last 30.04) - open/close 28058 (time 1709787972.27 total 1047.56 last 29.63) - open/close 28361 (time 1709787982.27 total 1057.56 last 30.29) - open/close 28702 (time 1709787992.30 total 1067.59 last 34.00) - open/close 29025 (time 1709788002.31 total 1077.60 last 32.26) - open/close 29335 (time 1709788012.33 total 1087.62 last 30.95) - open/close 29650 (time 1709788022.34 total 1097.63 last 31.46) - open/close 29931 (time 1709788032.34 total 1107.63 last 28.10) - open/close 30000 (time 1709788034.96 total 1110.25 last 26.36) - open/close 30292 (time 1709788044.98 total 1120.27 last 29.13) - open/close 30556 (time 1709788055.04 total 1130.33 last 26.26) - open/close 30840 (time 1709788065.08 total 1140.37 last 28.29) - open/close 31132 (time 1709788075.09 total 1150.38 last 29.17) - open/close 31492 (time 1709788085.09 total 1160.38 last 36.00) - open/close 31867 (time 1709788095.13 total 1170.42 last 37.35) - open/close 32221 (time 1709788105.15 total 1180.44 last 35.33) - open/close 32561 (time 1709788115.15 total 1190.44 last 34.00) - open/close 32889 (time 1709788125.17 total 1200.46 last 32.74) - open/close 33233 (time 1709788135.18 total 1210.47 last 34.34) - open/close 33587 (time 1709788145.19 total 1220.48 last 35.38) - open/close 33919 (time 1709788155.22 total 1230.51 last 33.11) - open/close 34256 (time 1709788165.24 total 1240.53 last 33.63) - open/close 34576 (time 1709788175.26 total 1250.55 last 31.94) - open/close 34912 (time 1709788185.29 total 1260.58 last 33.51) - open/close 35233 (time 1709788195.31 total 1270.60 last 32.02) - open/close 35553 (time 1709788205.32 total 1280.61 last 31.98) - open/close 35859 (time 1709788215.33 total 1290.62 last 30.56) - open/close 36194 (time 1709788225.36 total 1300.65 last 33.41) - open/close 36521 (time 1709788235.37 total 1310.66 last 32.68) - open/close 36825 (time 1709788245.38 total 1320.67 last 30.36) - open/close 37105 (time 1709788255.38 total 1330.67 last 27.99) - open/close 37412 (time 1709788265.40 total 1340.69 last 30.65) - open/close 37702 (time 1709788275.42 total 1350.71 last 28.95) - open/close 37983 (time 1709788285.42 total 1360.71 last 28.09) - open/close 38248 (time 1709788295.45 total 1370.74 last 26.43) - open/close 38525 (time 1709788305.47 total 1380.76 last 27.65) - open/close 38809 (time 1709788315.49 total 1390.78 last 28.33) - open/close 39097 (time 1709788325.51 total 1400.80 last 28.74) - open/close 39410 (time 1709788335.55 total 1410.84 last 31.19) - open/close 39708 (time 1709788345.55 total 1420.84 last 29.79) - open/close 40000 (time 1709788355.34 total 1430.63 last 29.82) - open/close 40274 (time 1709788365.35 total 1440.64 last 27.37) - open/close 40539 (time 1709788375.37 total 1450.66 last 26.45) - open/close 40797 (time 1709788385.39 total 1460.68 last 25.75) - open/close 41086 (time 1709788395.40 total 1470.69 last 28.86) - open/close 41373 (time 1709788405.44 total 1480.73 last 28.60) - open/close 41647 (time 1709788415.46 total 1490.75 last 27.34) - open/close 41945 (time 1709788425.47 total 1500.76 last 29.77) - open/close 42204 (time 1709788435.47 total 1510.76 last 25.90) - open/close 42469 (time 1709788445.48 total 1520.77 last 26.48) - open/close 42725 (time 1709788455.49 total 1530.78 last 25.57) - open/close 42974 (time 1709788465.51 total 1540.80 last 24.86) - open/close 43231 (time 1709788475.52 total 1550.81 last 25.67) - open/close 43495 (time 1709788485.52 total 1560.81 last 26.40) - open/close 43750 (time 1709788495.53 total 1570.82 last 25.46) - open/close 43998 (time 1709788505.56 total 1580.85 last 24.73) - open/close 44252 (time 1709788515.57 total 1590.86 last 25.38) - open/close 44519 (time 1709788525.61 total 1600.89 last 26.61) - open/close 44768 (time 1709788535.63 total 1610.91 last 24.85) - open/close 45037 (time 1709788545.66 total 1620.95 last 26.81) - open/close 45300 (time 1709788555.69 total 1630.98 last 26.21) - open/close 45554 (time 1709788565.71 total 1641.00 last 25.36) - open/close 45824 (time 1709788575.73 total 1651.02 last 26.94) - open/close 46096 (time 1709788585.76 total 1661.05 last 27.12) - open/close 46358 (time 1709788595.76 total 1671.05 last 26.18) - open/close 46626 (time 1709788605.78 total 1681.07 last 26.77) - open/close 46895 (time 1709788615.80 total 1691.09 last 26.84) - open/close 47171 (time 1709788625.83 total 1701.12 last 27.53) - open/close 47432 (time 1709788635.83 total 1711.12 last 26.10) - open/close 47700 (time 1709788645.85 total 1721.14 last 26.74) - open/close 47965 (time 1709788655.85 total 1731.14 last 26.50) - open/close 48227 (time 1709788665.88 total 1741.17 last 26.12) - open/close 48481 (time 1709788675.89 total 1751.18 last 25.39) - open/close 48723 (time 1709788685.91 total 1761.19 last 24.16) - open/close 48956 (time 1709788695.93 total 1771.21 last 23.25) - open/close 49198 (time 1709788705.96 total 1781.25 last 24.11) - open/close 49437 (time 1709788715.97 total 1791.26 last 23.88) - open/close 49677 (time 1709788726.00 total 1801.29 last 23.93) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1802.10 seconds: 27.58 ops/second - unlinked 0 (time 1709788728 ; total 0 ; last 0) - unlinked 10000 (time 1709789205 ; total 477 ; last 477) - unlinked 20000 (time 1709789634 ; total 906 ; last 429) - unlinked 30000 (time 1709789977 ; total 1249 ; last 343) - unlinked 40000 (time 1709790164 ; total 1436 ; last 187) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1498 seconds: 33.176235 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg330-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg330-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg330-server stop mds service on oleg330-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg330-server unloading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg330-client: oleg330-client: ssh exited with exit code 2 pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2 pdsh@oleg330-client: oleg330-client: ssh exited with exit code 2 pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2 PASS 101a (3341s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg330-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg330-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg330-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2 oleg330-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg330-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:44:06 (1709790246) start mds service on oleg330-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: detected 4 online CPUs by sysfs oleg330-server: Force libcfs to create 2 CPU partitions oleg330-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg330-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg330-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Started lustre-OST0000 oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg330-client.virtnet: -o user_xattr,flock oleg330-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg330-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg330-server start ost1 service on oleg330-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Started lustre-OST0000 oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec oleg330-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800ac6f9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800ac6f9000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg330-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg330-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg330-server stop mds service on oleg330-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg330-server unloading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg330-client: oleg330-client: ssh exited with exit code 2 pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2 pdsh@oleg330-client: oleg330-client: ssh exited with exit code 2 pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2 PASS 101b (91s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg330-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg330-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg330-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg330-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2 oleg330-server: oleg330-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: detected 4 online CPUs by sysfs oleg330-server: Force libcfs to create 2 CPU partitions oleg330-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg330-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3903 sec ========== 00:45:57 (1709790357) === conf-sanity: start cleanup 00:45:58 (1709790358) === === conf-sanity: finish cleanup 00:45:58 (1709790358) ===