-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:40:40 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg139-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg139-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg139-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg139-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg139-server oleg139-server: oleg139-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing load_modules_local oleg139-server: Loading modules from /home/green/git/lustre-release/lustre oleg139-server: detected 4 online CPUs by sysfs oleg139-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg139-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg139-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg139-server: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg139-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg139-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server stop mds service on oleg139-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg139-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:41:44 (1709786504) start mds service on oleg139-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg139-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0000 oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg139-client.virtnet: -o user_xattr,flock oleg139-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h42m00s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800aa970000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:42:45.000000000 -0500 Modify: 2024-03-06 23:42:45.000000000 -0500 Change: 2024-03-06 23:42:45.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 76 lines, 76 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg139-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg139-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server stop mds service on oleg139-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg139-server unloading modules on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing unload_modules_local modules unloaded. oleg139-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg139-server: Is the ldiskfs module available? oleg139-server: oleg139-server: tunefs.lustre FATAL: failed to write local files oleg139-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg139-client: oleg139-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg139-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg139-server: Is the ldiskfs module available? oleg139-server: oleg139-server: tunefs.lustre FATAL: failed to write local files oleg139-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg139-client: oleg139-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.139@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.139@tcp sys.timeout=20 oleg139-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg139-server: Is the ldiskfs module available? oleg139-server: oleg139-server: tunefs.lustre FATAL: failed to write local files oleg139-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg139-client: oleg139-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.139@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.139@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg139-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg139-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2 oleg139-server: oleg139-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing load_modules_local oleg139-server: Loading modules from /home/green/git/lustre-release/lustre oleg139-server: detected 4 online CPUs by sysfs oleg139-server: Force libcfs to create 2 CPU partitions oleg139-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg139-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg139-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg139-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg139-server: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg139-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg139-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server stop mds service on oleg139-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg139-server PASS 35b (125s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:43:51 (1709786631) start mds service on oleg139-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg139-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0000 oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg139-client.virtnet: -o user_xattr,flock oleg139-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg139-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg139-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg139-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg139-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg139-client.virtnet: -o user_xattr,flock oleg139-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg139-client.virtnet /mnt/lustre (opts:) stop mds service on oleg139-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg139-server 000000 2 000008 oleg139-server: 8+0 records in oleg139-server: 8+0 records out oleg139-server: 4096 bytes (4.1 kB) copied, 0.00121273 s, 3.4 MB/s start mds service on oleg139-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg139-client.virtnet: -o user_xattr,flock oleg139-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg139-client.virtnet /mnt/lustre (opts:) stop mds service on oleg139-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg139-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg139-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server stop mds service on oleg139-server unloading modules on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2 PASS 38 (78s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg139-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg139-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg139-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2 oleg139-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg139-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:45:12 (1709786712) Stopping clients: oleg139-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg139-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2 oleg139-server: oleg139-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing load_modules_local oleg139-server: Loading modules from /home/green/git/lustre-release/lustre oleg139-server: detected 4 online CPUs by sysfs oleg139-server: Force libcfs to create 2 CPU partitions oleg139-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg139-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg139-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg139-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg139-client.virtnet: -o user_xattr,flock oleg139-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg139-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg139-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg139-server umount lustre on /mnt/lustre..... Stopping client oleg139-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg139-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server stop mds service on oleg139-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg139-server unloading modules on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing unload_modules_local modules unloaded. oleg139-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg139-server: Is the ldiskfs module available? oleg139-server: oleg139-server: tunefs.lustre FATAL: failed to write local files oleg139-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg139-client: oleg139-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg139-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg139-server: Is the ldiskfs module available? oleg139-server: oleg139-server: tunefs.lustre FATAL: failed to write local files oleg139-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg139-client: oleg139-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.139@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.139@tcp sys.timeout=20 oleg139-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg139-server: Is the ldiskfs module available? oleg139-server: oleg139-server: tunefs.lustre FATAL: failed to write local files oleg139-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg139-client: oleg139-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.139@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.139@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg139-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg139-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2 oleg139-server: oleg139-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing load_modules_local oleg139-server: Loading modules from /home/green/git/lustre-release/lustre oleg139-server: detected 4 online CPUs by sysfs oleg139-server: Force libcfs to create 2 CPU partitions oleg139-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg139-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg139-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg139-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg139-server: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg139-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg139-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server stop mds service on oleg139-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg139-server PASS 51 (172s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:48:09 (1709786889) start mds service on oleg139-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg139-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0000 oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg139-client.virtnet: -o user_xattr,flock oleg139-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 159 (time 1709786924.84 total 10.03 last 15.86) - open/close 466 (time 1709786935.23 total 20.42 last 29.55) - open/close 702 (time 1709786945.81 total 31.00 last 22.31) - open/close 870 (time 1709786955.82 total 41.00 last 16.79) - open/close 990 (time 1709786966.41 total 51.59 last 11.33) - open/close 1136 (time 1709786977.13 total 62.32 last 13.62) - open/close 1447 (time 1709786987.93 total 73.11 last 28.81) - open/close 1711 (time 1709786998.56 total 83.74 last 24.84) - open/close 2098 (time 1709787009.05 total 94.24 last 36.86) - open/close 2379 (time 1709787019.79 total 104.97 last 26.18) - open/close 2522 (time 1709787029.79 total 114.98 last 14.29) - open/close 2704 (time 1709787039.80 total 124.98 last 18.19) - open/close 3021 (time 1709787050.83 total 136.01 last 28.73) - open/close 3311 (time 1709787061.01 total 146.19 last 28.49) - open/close 3501 (time 1709787071.35 total 156.54 last 18.37) - open/close 3800 (time 1709787082.14 total 167.32 last 27.72) - open/close 3924 (time 1709787092.40 total 177.58 last 12.09) - open/close 4065 (time 1709787102.66 total 187.85 last 13.74) - open/close 4283 (time 1709787113.01 total 198.19 last 21.07) - open/close 4502 (time 1709787123.04 total 208.22 last 21.83) - open/close 4664 (time 1709787133.06 total 218.24 last 16.17) - open/close 4866 (time 1709787143.07 total 228.25 last 20.18) - open/close 5049 (time 1709787153.74 total 238.93 last 17.14) - open/close 5215 (time 1709787164.71 total 249.89 last 15.14) - open/close 5269 (time 1709787174.88 total 260.07 last 5.31) - open/close 5555 (time 1709787185.29 total 270.47 last 27.49) - open/close 5749 (time 1709787195.92 total 281.11 last 18.25) - open/close 5927 (time 1709787206.73 total 291.91 last 16.47) - open/close 6094 (time 1709787217.56 total 302.74 last 15.42) - open/close 6271 (time 1709787227.83 total 313.01 last 17.23) - open/close 6472 (time 1709787238.07 total 323.25 last 19.63) - open/close 6620 (time 1709787248.33 total 333.51 last 14.43) - open/close 6751 (time 1709787258.82 total 344.01 last 12.48) - open/close 6874 (time 1709787269.03 total 354.22 last 12.05) - open/close 7021 (time 1709787279.69 total 364.87 last 13.79) - open/close 7146 (time 1709787289.98 total 375.16 last 12.15) - open/close 7419 (time 1709787299.99 total 385.18 last 27.26) - open/close 7634 (time 1709787311.02 total 396.20 last 19.50) - open/close 7813 (time 1709787321.06 total 406.25 last 17.82) - open/close 8033 (time 1709787332.04 total 417.22 last 20.04) - open/close 8189 (time 1709787342.34 total 427.53 last 15.14) - open/close 8333 (time 1709787352.80 total 437.99 last 13.77) - open/close 8496 (time 1709787363.71 total 448.89 last 14.94) - open/close 8727 (time 1709787373.71 total 458.89 last 23.10) - open/close 8915 (time 1709787383.98 total 469.16 last 18.31) - open/close 9125 (time 1709787394.53 total 479.71 last 19.91) - open/close 9272 (time 1709787404.54 total 489.72 last 14.68) - open/close 9448 (time 1709787414.54 total 499.73 last 17.60) - open/close 9624 (time 1709787424.58 total 509.77 last 17.53) - open/close 9781 (time 1709787435.54 total 520.72 last 14.33) - open/close 9962 (time 1709787446.42 total 531.61 last 16.63) - open/close 10177 (time 1709787456.49 total 541.67 last 21.36) - open/close 10302 (time 1709787466.89 total 552.08 last 12.01) - open/close 10486 (time 1709787476.90 total 562.09 last 18.38) - open/close 10896 (time 1709787486.93 total 572.12 last 40.89) - open/close 11353 (time 1709787496.94 total 582.12 last 45.67) - open/close 11752 (time 1709787506.94 total 592.13 last 39.87) - open/close 12180 (time 1709787516.96 total 602.14 last 42.75) - open/close 12614 (time 1709787526.97 total 612.16 last 43.32) - open/close 13029 (time 1709787536.99 total 622.18 last 41.42) - open/close 13406 (time 1709787547.00 total 632.19 last 37.66) - open/close 13734 (time 1709787566.55 total 651.73 last 16.78) - open/close 14096 (time 1709787576.56 total 661.75 last 36.15) - open/close 14474 (time 1709787586.58 total 671.77 last 37.72) - open/close 14838 (time 1709787596.59 total 681.77 last 36.39) - open/close 15188 (time 1709787606.59 total 691.77 last 34.99) - open/close 15593 (time 1709787616.60 total 701.78 last 40.47) - open/close 15953 (time 1709787626.60 total 711.79 last 35.98) - open/close 16296 (time 1709787636.60 total 721.79 last 34.29) - open/close 16651 (time 1709787646.64 total 731.82 last 35.39) - open/close 17028 (time 1709787656.65 total 741.83 last 37.65) - open/close 17326 (time 1709787666.65 total 751.83 last 29.80) - open/close 17642 (time 1709787676.67 total 761.85 last 31.55) - open/close 17975 (time 1709787686.68 total 771.87 last 33.25) - open/close 18282 (time 1709787696.69 total 781.87 last 30.68) - open/close 18638 (time 1709787706.69 total 791.87 last 35.59) - open/close 18981 (time 1709787716.71 total 801.90 last 34.23) - open/close 19345 (time 1709787726.73 total 811.91 last 36.34) - open/close 19698 (time 1709787736.75 total 821.93 last 35.22) - open/close 19998 (time 1709787746.76 total 831.94 last 29.98) - open/close 20307 (time 1709787756.76 total 841.94 last 30.89) - open/close 20624 (time 1709787766.78 total 851.97 last 31.62) - open/close 20941 (time 1709787776.79 total 861.98 last 31.68) - open/close 21261 (time 1709787786.81 total 872.00 last 31.93) - open/close 21535 (time 1709787796.82 total 882.00 last 27.38) - open/close 21840 (time 1709787806.83 total 892.01 last 30.47) - open/close 22175 (time 1709787816.84 total 902.03 last 33.46) - open/close 22477 (time 1709787826.88 total 912.06 last 30.09) - open/close 22729 (time 1709787836.89 total 922.07 last 25.17) - open/close 22983 (time 1709787846.89 total 932.07 last 25.39) - open/close 23260 (time 1709787856.91 total 942.09 last 27.64) - open/close 23554 (time 1709787866.93 total 952.11 last 29.35) - open/close 23847 (time 1709787876.94 total 962.13 last 29.26) - open/close 24109 (time 1709787886.96 total 972.15 last 26.14) - open/close 24395 (time 1709787896.97 total 982.16 last 28.58) - open/close 24658 (time 1709787906.99 total 992.17 last 26.26) - open/close 24920 (time 1709787916.99 total 1002.18 last 26.18) - open/close 25170 (time 1709787927.02 total 1012.21 last 24.92) - open/close 25421 (time 1709787937.03 total 1022.22 last 25.08) - open/close 25668 (time 1709787947.06 total 1032.25 last 24.63) - open/close 25985 (time 1709787957.06 total 1042.25 last 31.69) - open/close 26261 (time 1709787967.07 total 1052.26 last 27.58) - open/close 26531 (time 1709787977.08 total 1062.27 last 26.97) - open/close 26850 (time 1709787987.08 total 1072.27 last 31.90) - open/close 27179 (time 1709787997.11 total 1082.29 last 32.82) - open/close 27481 (time 1709788007.12 total 1092.31 last 30.16) - open/close 27792 (time 1709788017.14 total 1102.33 last 31.03) - open/close 27913 (time 1709788032.28 total 1117.46 last 8.00) - open/close 28216 (time 1709788042.31 total 1127.50 last 30.19) - open/close 28518 (time 1709788052.33 total 1137.51 last 30.16) - open/close 28823 (time 1709788062.35 total 1147.54 last 30.42) - open/close 29119 (time 1709788072.38 total 1157.57 last 29.52) - open/close 29427 (time 1709788082.38 total 1167.57 last 30.79) - open/close 29717 (time 1709788092.40 total 1177.59 last 28.95) - open/close 30000 (time 1709788102.25 total 1187.44 last 28.73) - open/close 30272 (time 1709788112.26 total 1197.44 last 27.19) - open/close 30527 (time 1709788122.28 total 1207.46 last 25.45) - open/close 30806 (time 1709788132.30 total 1217.49 last 27.82) - open/close 31089 (time 1709788142.33 total 1227.52 last 28.22) - open/close 31345 (time 1709788152.34 total 1237.53 last 25.58) - open/close 31607 (time 1709788162.38 total 1247.56 last 26.11) - open/close 31858 (time 1709788172.41 total 1257.59 last 25.02) - open/close 32114 (time 1709788182.42 total 1267.61 last 25.56) - open/close 32379 (time 1709788192.44 total 1277.62 last 26.47) - open/close 32645 (time 1709788202.47 total 1287.65 last 26.52) - open/close 32931 (time 1709788212.48 total 1297.67 last 28.56) - open/close 33204 (time 1709788222.50 total 1307.68 last 27.25) - open/close 33497 (time 1709788232.53 total 1317.72 last 29.20) - open/close 33770 (time 1709788242.57 total 1327.75 last 27.22) - open/close 34018 (time 1709788252.60 total 1337.79 last 24.71) - open/close 34233 (time 1709788262.61 total 1347.80 last 21.48) - open/close 34486 (time 1709788272.64 total 1357.82 last 25.24) - open/close 34739 (time 1709788282.67 total 1367.85 last 25.22) - open/close 35005 (time 1709788292.72 total 1377.91 last 26.46) - open/close 35251 (time 1709788302.73 total 1387.91 last 24.58) - open/close 35504 (time 1709788312.73 total 1397.91 last 25.30) - open/close 35760 (time 1709788322.74 total 1407.92 last 25.57) - open/close 36024 (time 1709788332.75 total 1417.93 last 26.37) - open/close 36275 (time 1709788342.75 total 1427.94 last 25.10) - open/close 36541 (time 1709788352.76 total 1437.94 last 26.58) - open/close 36796 (time 1709788362.78 total 1447.96 last 25.46) - open/close 37052 (time 1709788372.79 total 1457.97 last 25.58) - open/close 37311 (time 1709788382.80 total 1467.99 last 25.86) - open/close 37554 (time 1709788392.81 total 1478.00 last 24.27) - open/close 37806 (time 1709788402.83 total 1488.01 last 25.17) - open/close 38070 (time 1709788412.85 total 1498.04 last 26.32) - open/close 38306 (time 1709788422.87 total 1508.05 last 23.57) - open/close 38555 (time 1709788432.89 total 1518.08 last 24.84) - open/close 38792 (time 1709788442.92 total 1528.10 last 23.64) - open/close 39030 (time 1709788452.94 total 1538.12 last 23.75) - open/close 39266 (time 1709788462.96 total 1548.15 last 23.54) - open/close 39475 (time 1709788473.00 total 1558.18 last 20.83) - open/close 39717 (time 1709788483.02 total 1568.21 last 24.13) - open/close 39953 (time 1709788493.05 total 1578.23 last 23.55) - open/close 40196 (time 1709788503.05 total 1588.24 last 24.28) - open/close 40446 (time 1709788513.08 total 1598.26 last 24.94) - open/close 40690 (time 1709788523.10 total 1608.29 last 24.34) - open/close 40937 (time 1709788533.14 total 1618.33 last 24.61) - open/close 41176 (time 1709788543.17 total 1628.36 last 23.83) - open/close 41414 (time 1709788553.20 total 1638.39 last 23.73) - open/close 41661 (time 1709788563.21 total 1648.40 last 24.68) - open/close 41907 (time 1709788573.24 total 1658.43 last 24.52) - open/close 42121 (time 1709788583.27 total 1668.46 last 21.34) - open/close 42352 (time 1709788593.32 total 1678.50 last 22.99) - open/close 42586 (time 1709788603.32 total 1688.50 last 23.40) - open/close 42819 (time 1709788613.32 total 1698.50 last 23.30) - open/close 43072 (time 1709788623.35 total 1708.54 last 25.22) - open/close 43328 (time 1709788633.39 total 1718.57 last 25.51) - open/close 43568 (time 1709788643.42 total 1728.61 last 23.91) - open/close 43803 (time 1709788653.45 total 1738.64 last 23.44) - open/close 44035 (time 1709788663.49 total 1748.67 last 23.12) - open/close 44256 (time 1709788673.51 total 1758.69 last 22.05) - open/close 44476 (time 1709788683.51 total 1768.69 last 22.00) - open/close 44690 (time 1709788693.56 total 1778.74 last 21.30) - open/close 44911 (time 1709788703.56 total 1788.74 last 22.09) - open/close 45133 (time 1709788713.58 total 1798.76 last 22.16) - open/close 45363 (time 1709788723.60 total 1808.79 last 22.94) - open/close 45584 (time 1709788733.62 total 1818.81 last 22.06) - open/close 45797 (time 1709788743.65 total 1828.84 last 21.23) - open/close 45989 (time 1709788753.67 total 1838.86 last 19.16) - open/close 46180 (time 1709788763.71 total 1848.90 last 19.02) - open/close 46370 (time 1709788773.77 total 1858.96 last 18.89) - open/close 46570 (time 1709788783.77 total 1868.96 last 20.00) - open/close 46794 (time 1709788793.81 total 1879.00 last 22.32) - open/close 47007 (time 1709788803.81 total 1889.00 last 21.30) - open/close 47214 (time 1709788813.83 total 1899.01 last 20.66) - open/close 47410 (time 1709788823.88 total 1909.06 last 19.51) - open/close 47606 (time 1709788833.90 total 1919.08 last 19.56) - open/close 47797 (time 1709788843.94 total 1929.13 last 19.02) - open/close 47993 (time 1709788853.96 total 1939.15 last 19.56) - open/close 48179 (time 1709788863.99 total 1949.18 last 18.54) - open/close 48376 (time 1709788874.03 total 1959.21 last 19.64) - open/close 48572 (time 1709788884.04 total 1969.22 last 19.58) - open/close 48773 (time 1709788894.06 total 1979.25 last 20.05) - open/close 48980 (time 1709788904.08 total 1989.26 last 20.67) - open/close 49189 (time 1709788914.12 total 1999.30 last 20.81) - open/close 49388 (time 1709788924.17 total 2009.35 last 19.80) - open/close 49586 (time 1709788934.17 total 2019.36 last 19.79) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 2025.35 seconds: 24.54 ops/second - unlinked 0 (time 1709788941 ; total 0 ; last 0) - unlinked 10000 (time 1709789508 ; total 567 ; last 567) - unlinked 20000 (time 1709789978 ; total 1037 ; last 470) - unlinked 30000 (time 1709790275 ; total 1334 ; last 297) - unlinked 40000 (time 1709790396 ; total 1455 ; last 121) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1492 seconds: 33.309650 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg139-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg139-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server stop mds service on oleg139-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg139-server unloading modules on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2 PASS 101a (3558s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg139-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg139-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg139-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2 oleg139-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg139-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:47:29 (1709790449) start mds service on oleg139-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing load_modules_local oleg139-server: Loading modules from /home/green/git/lustre-release/lustre oleg139-server: detected 4 online CPUs by sysfs oleg139-server: Force libcfs to create 2 CPU partitions oleg139-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg139-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg139-server