-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:39:21 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg418-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg418-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg418-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg418-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg418-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg418-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg418-server oleg418-server: oleg418-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing load_modules_local oleg418-server: Loading modules from /home/green/git/lustre-release/lustre oleg418-server: detected 4 online CPUs by sysfs oleg418-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg418-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg418-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg418-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg418-server: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg418-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg418-server: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg418-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg418-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg418-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:40:44 (1709786444) start mds service on oleg418-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg418-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg418-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-OST0000 oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg418-client.virtnet: -o user_xattr,flock oleg418-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h41m02s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800b6701000.stats=clear mdc.lustre-MDT0001-mdc-ffff8800b6701000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:41:45.000000000 -0500 Modify: 2024-03-06 23:41:45.000000000 -0500 Change: 2024-03-06 23:41:45.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 63 lines, 63 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg418-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg418-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg418-server unloading modules on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing unload_modules_local modules unloaded. oleg418-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg418-server: Is the ldiskfs module available? oleg418-server: oleg418-server: tunefs.lustre FATAL: failed to write local files oleg418-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg418-client: oleg418-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg418-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg418-server: Is the ldiskfs module available? oleg418-server: oleg418-server: tunefs.lustre FATAL: failed to write local files oleg418-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg418-client: oleg418-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.204.118@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.204.118@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg418-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg418-server: Is the ldiskfs module available? oleg418-server: oleg418-server: tunefs.lustre FATAL: failed to write local files oleg418-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg418-client: oleg418-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.118@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.118@tcp sys.timeout=20 oleg418-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg418-server: Is the ldiskfs module available? oleg418-server: oleg418-server: tunefs.lustre FATAL: failed to write local files oleg418-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg418-client: oleg418-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.118@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.118@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg418-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg418-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2 oleg418-server: oleg418-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing load_modules_local oleg418-server: Loading modules from /home/green/git/lustre-release/lustre oleg418-server: detected 4 online CPUs by sysfs oleg418-server: Force libcfs to create 2 CPU partitions oleg418-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg418-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg418-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg418-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg418-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg418-server: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg418-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg418-server: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg418-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg418-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg418-server PASS 35b (144s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:43:10 (1709786590) start mds service on oleg418-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg418-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg418-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-OST0000 oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg418-client.virtnet: -o user_xattr,flock oleg418-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg418-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg418-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg418-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg418-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg418-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg418-client.virtnet: -o user_xattr,flock oleg418-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg418-client.virtnet /mnt/lustre (opts:) stop mds service on oleg418-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg418-server 000000 2 000008 oleg418-server: 8+0 records in oleg418-server: 8+0 records out oleg418-server: 4096 bytes (4.1 kB) copied, 0.000337708 s, 12.1 MB/s start mds service on oleg418-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg418-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg418-client.virtnet: -o user_xattr,flock oleg418-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg418-client.virtnet /mnt/lustre (opts:) stop mds service on oleg418-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg418-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg418-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg418-server stop mds service on oleg418-server stop mds service on oleg418-server unloading modules on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg418-client: oleg418-client: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-client: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2 PASS 38 (142s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg418-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg418-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg418-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2 oleg418-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg418-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:45:35 (1709786735) Stopping clients: oleg418-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg418-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2 oleg418-server: oleg418-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing load_modules_local oleg418-server: Loading modules from /home/green/git/lustre-release/lustre oleg418-server: detected 4 online CPUs by sysfs oleg418-server: Force libcfs to create 2 CPU partitions oleg418-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg418-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg418-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg418-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg418-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg418-client.virtnet: -o user_xattr,flock oleg418-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg418-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg418-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg418-server umount lustre on /mnt/lustre..... Stopping client oleg418-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg418-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg418-server unloading modules on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing unload_modules_local modules unloaded. oleg418-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg418-server: Is the ldiskfs module available? oleg418-server: oleg418-server: tunefs.lustre FATAL: failed to write local files oleg418-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg418-client: oleg418-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg418-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg418-server: Is the ldiskfs module available? oleg418-server: oleg418-server: tunefs.lustre FATAL: failed to write local files oleg418-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg418-client: oleg418-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.204.118@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.204.118@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg418-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg418-server: Is the ldiskfs module available? oleg418-server: oleg418-server: tunefs.lustre FATAL: failed to write local files oleg418-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg418-client: oleg418-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.118@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.118@tcp sys.timeout=20 oleg418-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg418-server: Is the ldiskfs module available? oleg418-server: oleg418-server: tunefs.lustre FATAL: failed to write local files oleg418-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg418-client: oleg418-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.118@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.118@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg418-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg418-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2 oleg418-server: oleg418-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing load_modules_local oleg418-server: Loading modules from /home/green/git/lustre-release/lustre oleg418-server: detected 4 online CPUs by sysfs oleg418-server: Force libcfs to create 2 CPU partitions oleg418-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg418-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg418-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg418-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg418-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg418-server: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg418-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg418-server: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg418-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg418-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg418-server PASS 51 (225s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:49:25 (1709786965) start mds service on oleg418-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg418-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg418-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-OST0000 oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg418-client.virtnet: -o user_xattr,flock oleg418-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 142 (time 1709787012.11 total 11.02 last 12.89) - open/close 287 (time 1709787022.45 total 21.36 last 14.02) - open/close 499 (time 1709787033.59 total 32.50 last 19.03) - open/close 709 (time 1709787043.61 total 42.52 last 20.97) - open/close 889 (time 1709787053.97 total 52.89 last 17.36) - open/close 1177 (time 1709787064.18 total 63.09 last 28.23) - open/close 1360 (time 1709787074.21 total 73.12 last 18.24) - open/close 1558 (time 1709787084.70 total 83.61 last 18.86) - open/close 1703 (time 1709787094.70 total 93.62 last 14.50) - open/close 1879 (time 1709787104.76 total 103.67 last 17.50) - open/close 2139 (time 1709787114.96 total 113.87 last 25.49) - open/close 2263 (time 1709787125.69 total 124.60 last 11.56) - open/close 2489 (time 1709787136.65 total 135.56 last 20.62) - open/close 2632 (time 1709787146.69 total 145.60 last 14.24) - open/close 2830 (time 1709787156.82 total 155.73 last 19.54) - open/close 3176 (time 1709787167.07 total 165.99 last 33.74) - open/close 3436 (time 1709787177.92 total 176.83 last 23.98) - open/close 3636 (time 1709787187.93 total 186.84 last 19.98) - open/close 3925 (time 1709787198.45 total 197.36 last 27.46) - open/close 4129 (time 1709787209.19 total 208.10 last 18.99) - open/close 4337 (time 1709787220.03 total 218.94 last 19.18) - open/close 4477 (time 1709787230.33 total 229.25 last 13.59) - open/close 4620 (time 1709787240.91 total 239.82 last 13.52) - open/close 4841 (time 1709787250.95 total 249.86 last 22.01) - open/close 5002 (time 1709787260.96 total 259.87 last 16.09) - open/close 5141 (time 1709787271.37 total 270.28 last 13.35) - open/close 5356 (time 1709787281.38 total 280.29 last 21.49) - open/close 5519 (time 1709787291.76 total 290.67 last 15.70) - open/close 5689 (time 1709787301.77 total 300.68 last 16.98) - open/close 5990 (time 1709787312.67 total 311.58 last 27.62) - open/close 6127 (time 1709787322.69 total 321.60 last 13.67) - open/close 6328 (time 1709787333.40 total 332.31 last 18.78) - open/close 6540 (time 1709787344.39 total 343.31 last 19.27) - open/close 6757 (time 1709787354.52 total 353.43 last 21.43) - open/close 6914 (time 1709787365.52 total 364.43 last 14.27) - open/close 7087 (time 1709787376.07 total 374.99 last 16.40) - open/close 7268 (time 1709787386.11 total 385.02 last 18.04) - open/close 7381 (time 1709787396.57 total 395.48 last 10.80) - open/close 7518 (time 1709787406.64 total 405.55 last 13.60) - open/close 7668 (time 1709787417.43 total 416.34 last 13.91) - open/close 7910 (time 1709787427.46 total 426.37 last 24.12) - open/close 8119 (time 1709787438.50 total 437.41 last 18.93) - open/close 8240 (time 1709787448.54 total 447.45 last 12.06) - open/close 8399 (time 1709787459.42 total 458.33 last 14.61) - open/close 8611 (time 1709787469.47 total 468.38 last 21.09) - open/close 8762 (time 1709787480.24 total 479.15 last 14.02) - open/close 8889 (time 1709787491.11 total 490.02 last 11.69) - open/close 9097 (time 1709787501.52 total 500.44 last 19.97) - open/close 9250 (time 1709787511.54 total 510.45 last 15.28) - open/close 9385 (time 1709787530.43 total 529.34 last 7.15) - open/close 9468 (time 1709787540.46 total 539.37 last 8.28) - open/close 9757 (time 1709787550.48 total 549.39 last 28.85) - open/close 9982 (time 1709787565.88 total 564.79 last 14.61) - open/close 10487 (time 1709787575.89 total 574.80 last 50.43) - open/close 11005 (time 1709787585.90 total 584.81 last 51.75) - open/close 11578 (time 1709787595.91 total 594.82 last 57.28) - open/close 12098 (time 1709787605.92 total 604.83 last 51.95) - open/close 12578 (time 1709787615.93 total 614.84 last 47.94) - open/close 13063 (time 1709787625.95 total 624.86 last 48.42) - open/close 13520 (time 1709787635.95 total 634.86 last 45.69) - open/close 13978 (time 1709787645.95 total 644.86 last 45.78) - open/close 14458 (time 1709787655.96 total 654.87 last 47.97) - open/close 14915 (time 1709787665.97 total 664.88 last 45.67) - open/close 15335 (time 1709787675.97 total 674.88 last 41.98) - open/close 15730 (time 1709787685.98 total 684.89 last 39.47) - open/close 16153 (time 1709787695.98 total 694.90 last 42.28) - open/close 16551 (time 1709787705.99 total 704.91 last 39.76) - open/close 16899 (time 1709787716.00 total 714.91 last 34.78) - open/close 17254 (time 1709787726.02 total 724.93 last 35.42) - open/close 17653 (time 1709787736.03 total 734.94 last 39.87) - open/close 18089 (time 1709787746.03 total 744.94 last 43.59) - open/close 18468 (time 1709787756.04 total 754.95 last 37.88) - open/close 18791 (time 1709787766.09 total 765.01 last 32.12) - open/close 19095 (time 1709787776.13 total 775.04 last 30.30) - open/close 19394 (time 1709787786.14 total 785.05 last 29.88) - open/close 19693 (time 1709787796.14 total 795.05 last 29.88) - open/close 20000 (time 1709787805.74 total 804.66 last 31.97) - open/close 20352 (time 1709787815.78 total 814.69 last 35.09) - open/close 20675 (time 1709787825.79 total 824.70 last 32.27) - open/close 21015 (time 1709787835.80 total 834.71 last 33.95) - open/close 21395 (time 1709787845.81 total 844.72 last 37.95) - open/close 21787 (time 1709787855.82 total 854.73 last 39.17) - open/close 22208 (time 1709787865.83 total 864.74 last 42.06) - open/close 22597 (time 1709787875.84 total 874.75 last 38.84) - open/close 23006 (time 1709787885.85 total 884.76 last 40.86) - open/close 23391 (time 1709787895.88 total 894.79 last 38.41) - open/close 23772 (time 1709787905.88 total 904.79 last 38.09) - open/close 24135 (time 1709787915.90 total 914.81 last 36.22) - open/close 24514 (time 1709787925.90 total 924.82 last 37.90) - open/close 24880 (time 1709787935.91 total 934.82 last 36.59) - open/close 25234 (time 1709787945.93 total 944.84 last 35.32) - open/close 25599 (time 1709787955.96 total 954.87 last 36.40) - open/close 25804 (time 1709787972.40 total 971.31 last 12.47) - open/close 26145 (time 1709787982.41 total 981.32 last 34.07) - open/close 26508 (time 1709787992.41 total 991.32 last 36.30) - open/close 26858 (time 1709788002.43 total 1001.35 last 34.92) - open/close 27163 (time 1709788012.45 total 1011.36 last 30.46) - open/close 27460 (time 1709788022.46 total 1021.37 last 29.65) - open/close 27802 (time 1709788032.46 total 1031.38 last 34.20) - open/close 28151 (time 1709788042.47 total 1041.38 last 34.88) - open/close 28485 (time 1709788052.50 total 1051.41 last 33.31) - open/close 28822 (time 1709788062.51 total 1061.42 last 33.66) - open/close 29154 (time 1709788072.53 total 1071.44 last 33.12) - open/close 29477 (time 1709788082.54 total 1081.45 last 32.27) - open/close 29819 (time 1709788092.56 total 1091.47 last 34.14) - open/close 30000 (time 1709788098.09 total 1097.00 last 32.75) - open/close 30323 (time 1709788108.11 total 1107.02 last 32.23) - open/close 30644 (time 1709788118.12 total 1117.03 last 32.06) - open/close 30970 (time 1709788128.14 total 1127.05 last 32.54) - open/close 31264 (time 1709788138.15 total 1137.06 last 29.38) - open/close 31565 (time 1709788148.18 total 1147.09 last 30.00) - open/close 31842 (time 1709788158.20 total 1157.11 last 27.64) - open/close 32121 (time 1709788168.20 total 1167.11 last 27.89) - open/close 32414 (time 1709788178.22 total 1177.13 last 29.25) - open/close 32722 (time 1709788188.26 total 1187.17 last 30.68) - open/close 32995 (time 1709788198.26 total 1197.17 last 27.30) - open/close 33268 (time 1709788208.26 total 1207.17 last 27.30) - open/close 33560 (time 1709788218.27 total 1217.18 last 29.17) - open/close 33848 (time 1709788228.31 total 1227.22 last 28.69) - open/close 34130 (time 1709788238.35 total 1237.26 last 28.10) - open/close 34396 (time 1709788248.36 total 1247.27 last 26.56) - open/close 34702 (time 1709788258.38 total 1257.29 last 30.55) - open/close 34991 (time 1709788268.39 total 1267.30 last 28.87) - open/close 35267 (time 1709788278.39 total 1277.30 last 27.60) - open/close 35553 (time 1709788288.42 total 1287.34 last 28.49) - open/close 35844 (time 1709788298.48 total 1297.39 last 28.94) - open/close 36108 (time 1709788308.50 total 1307.41 last 26.35) - open/close 36371 (time 1709788318.52 total 1317.43 last 26.25) - open/close 36651 (time 1709788328.52 total 1327.43 last 27.99) - open/close 36917 (time 1709788338.56 total 1337.47 last 26.50) - open/close 37165 (time 1709788348.56 total 1347.47 last 24.79) - open/close 37412 (time 1709788358.60 total 1357.51 last 24.62) - open/close 37658 (time 1709788368.60 total 1367.51 last 24.59) - open/close 37902 (time 1709788378.64 total 1377.55 last 24.31) - open/close 38145 (time 1709788388.65 total 1387.56 last 24.28) - open/close 38390 (time 1709788398.68 total 1397.59 last 24.43) - open/close 38631 (time 1709788408.70 total 1407.61 last 24.04) - open/close 38890 (time 1709788418.71 total 1417.63 last 25.87) - open/close 39131 (time 1709788428.75 total 1427.66 last 24.02) - open/close 39366 (time 1709788438.77 total 1437.68 last 23.45) - open/close 39612 (time 1709788448.79 total 1447.70 last 24.54) - open/close 39852 (time 1709788458.83 total 1457.74 last 23.92) - open/close 40000 (time 1709788464.98 total 1463.90 last 24.03) - open/close 40257 (time 1709788475.01 total 1473.92 last 25.64) - open/close 40493 (time 1709788485.03 total 1483.95 last 23.54) - open/close 40747 (time 1709788495.04 total 1493.95 last 25.38) - open/close 41007 (time 1709788505.07 total 1503.98 last 25.93) - open/close 41272 (time 1709788515.11 total 1514.02 last 26.39) - open/close 41511 (time 1709788525.15 total 1524.06 last 23.82) - open/close 41757 (time 1709788535.18 total 1534.09 last 24.53) - open/close 42014 (time 1709788545.21 total 1544.12 last 25.62) - open/close 42266 (time 1709788555.24 total 1554.15 last 25.11) - open/close 42499 (time 1709788565.25 total 1564.16 last 23.28) - open/close 42747 (time 1709788575.28 total 1574.19 last 24.74) - open/close 42987 (time 1709788585.31 total 1584.22 last 23.92) - open/close 43246 (time 1709788595.34 total 1594.25 last 25.81) - open/close 43506 (time 1709788605.36 total 1604.27 last 25.96) - open/close 43750 (time 1709788615.36 total 1614.27 last 24.40) - open/close 44000 (time 1709788625.39 total 1624.30 last 24.93) - open/close 44259 (time 1709788635.39 total 1634.30 last 25.89) - open/close 44521 (time 1709788645.41 total 1644.32 last 26.16) - open/close 44774 (time 1709788655.43 total 1654.34 last 25.25) - open/close 45022 (time 1709788665.44 total 1664.35 last 24.76) - open/close 45241 (time 1709788675.46 total 1674.37 last 21.86) - open/close 45460 (time 1709788685.46 total 1684.37 last 21.89) - open/close 45691 (time 1709788695.47 total 1694.39 last 23.07) - open/close 45928 (time 1709788705.49 total 1704.41 last 23.65) - open/close 46164 (time 1709788715.54 total 1714.45 last 23.50) - open/close 46392 (time 1709788725.55 total 1724.46 last 22.77) - open/close 46625 (time 1709788735.56 total 1734.47 last 23.28) - open/close 46849 (time 1709788745.57 total 1744.48 last 22.38) - open/close 47072 (time 1709788755.59 total 1754.50 last 22.25) - open/close 47307 (time 1709788765.61 total 1764.52 last 23.45) - open/close 47543 (time 1709788775.62 total 1774.53 last 23.59) - open/close 47765 (time 1709788785.63 total 1784.54 last 22.16) - open/close 47992 (time 1709788795.67 total 1794.58 last 22.61) - open/close 48213 (time 1709788805.71 total 1804.62 last 22.02) - open/close 48436 (time 1709788815.71 total 1814.63 last 22.28) - open/close 48654 (time 1709788825.75 total 1824.66 last 21.72) - open/close 48881 (time 1709788835.80 total 1834.71 last 22.60) - open/close 49095 (time 1709788845.83 total 1844.74 last 21.32) - open/close 49306 (time 1709788855.83 total 1854.75 last 21.10) - open/close 49534 (time 1709788865.85 total 1864.76 last 22.76) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1867.68 seconds: 26.56 ops/second - unlinked 0 (time 1709788869 ; total 0 ; last 0) - unlinked 10000 (time 1709789374 ; total 505 ; last 505) - unlinked 20000 (time 1709789780 ; total 911 ; last 406) - unlinked 30000 (time 1709790057 ; total 1188 ; last 277) - unlinked 40000 (time 1709790191 ; total 1322 ; last 134) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1358 seconds: 36.524300 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg418-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg418-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg418-server unloading modules on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-client: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-client: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2 PASS 101a (3284s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg418-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg418-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg418-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2 oleg418-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg418-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:44:10 (1709790250) start mds service on oleg418-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing load_modules_local oleg418-server: Loading modules from /home/green/git/lustre-release/lustre oleg418-server: detected 4 online CPUs by sysfs oleg418-server: Force libcfs to create 2 CPU partitions oleg418-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg418-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg418-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg418-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-OST0000 oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg418-client.virtnet: -o user_xattr,flock oleg418-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg418-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg418-server start ost1 service on oleg418-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-OST0000 oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec oleg418-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800ab579000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800ab579000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg418-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg418-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg418-server unloading modules on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg418-client: oleg418-client: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-client: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2 PASS 101b (81s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg418-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg418-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg418-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg418-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2 oleg418-server: oleg418-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing load_modules_local oleg418-server: Loading modules from /home/green/git/lustre-release/lustre oleg418-server: detected 4 online CPUs by sysfs oleg418-server: Force libcfs to create 2 CPU partitions oleg418-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg418-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg418-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3989 sec ========== 00:45:50 (1709790350) === conf-sanity: start cleanup 00:45:50 (1709790350) === === conf-sanity: finish cleanup 00:45:50 (1709790350) ===