-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:43:44 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg151-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg151-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg151-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg151-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg151-server oleg151-server: oleg151-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing load_modules_local oleg151-server: Loading modules from /home/green/git/lustre-release/lustre oleg151-server: detected 4 online CPUs by sysfs oleg151-server: Force libcfs to create 2 CPU partitions oleg151-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg151-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg151-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg151-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg151-server: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg151-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg151-server: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg151-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg151-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:45:31 (1709786731) start mds service on oleg151-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg151-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg151-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg151-client.virtnet: -o user_xattr,flock oleg151-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h46m05s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800aaea5000.stats=clear mdc.lustre-MDT0001-mdc-ffff8800aaea5000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:46:52.000000000 -0500 Modify: 2024-03-06 23:46:52.000000000 -0500 Change: 2024-03-06 23:46:52.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 70 lines, 70 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg151-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg151-server unloading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing unload_modules_local modules unloaded. oleg151-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg151-server: Is the ldiskfs module available? oleg151-server: oleg151-server: tunefs.lustre FATAL: failed to write local files oleg151-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg151-client: oleg151-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg151-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg151-server: Is the ldiskfs module available? oleg151-server: oleg151-server: tunefs.lustre FATAL: failed to write local files oleg151-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg151-client: oleg151-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.151@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.151@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg151-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg151-server: Is the ldiskfs module available? oleg151-server: oleg151-server: tunefs.lustre FATAL: failed to write local files oleg151-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg151-client: oleg151-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.151@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.151@tcp sys.timeout=20 oleg151-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg151-server: Is the ldiskfs module available? oleg151-server: oleg151-server: tunefs.lustre FATAL: failed to write local files oleg151-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg151-client: oleg151-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.151@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.151@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg151-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg151-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 oleg151-server: oleg151-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing load_modules_local oleg151-server: Loading modules from /home/green/git/lustre-release/lustre oleg151-server: detected 4 online CPUs by sysfs oleg151-server: Force libcfs to create 2 CPU partitions oleg151-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg151-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg151-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg151-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg151-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg151-server: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg151-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg151-server: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg151-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg151-server PASS 35b (217s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:49:12 (1709786952) start mds service on oleg151-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg151-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg151-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg151-client.virtnet: -o user_xattr,flock oleg151-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg151-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg151-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg151-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg151-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg151-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg151-client.virtnet: -o user_xattr,flock oleg151-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg151-client.virtnet /mnt/lustre (opts:) stop mds service on oleg151-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg151-server 000000 2 000008 oleg151-server: 8+0 records in oleg151-server: 8+0 records out oleg151-server: 4096 bytes (4.1 kB) copied, 0.000468212 s, 8.7 MB/s start mds service on oleg151-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg151-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg151-client.virtnet: -o user_xattr,flock oleg151-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg151-client.virtnet /mnt/lustre (opts:) stop mds service on oleg151-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg151-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server stop mds service on oleg151-server stop mds service on oleg151-server unloading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2 PASS 38 (159s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg151-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg151-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg151-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 oleg151-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg151-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:51:54 (1709787114) Stopping clients: oleg151-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg151-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 oleg151-server: oleg151-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing load_modules_local oleg151-server: Loading modules from /home/green/git/lustre-release/lustre oleg151-server: detected 4 online CPUs by sysfs oleg151-server: Force libcfs to create 2 CPU partitions oleg151-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg151-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg151-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg151-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg151-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg151-client.virtnet: -o user_xattr,flock oleg151-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg151-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg151-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg151-server umount lustre on /mnt/lustre..... Stopping client oleg151-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg151-server unloading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing unload_modules_local modules unloaded. oleg151-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg151-server: Is the ldiskfs module available? oleg151-server: oleg151-server: tunefs.lustre FATAL: failed to write local files oleg151-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg151-client: oleg151-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg151-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg151-server: Is the ldiskfs module available? oleg151-server: oleg151-server: tunefs.lustre FATAL: failed to write local files oleg151-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg151-client: oleg151-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.151@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.151@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg151-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg151-server: Is the ldiskfs module available? oleg151-server: oleg151-server: tunefs.lustre FATAL: failed to write local files oleg151-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg151-client: oleg151-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.151@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.151@tcp sys.timeout=20 oleg151-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg151-server: Is the ldiskfs module available? oleg151-server: oleg151-server: tunefs.lustre FATAL: failed to write local files oleg151-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg151-client: oleg151-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.151@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.151@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg151-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg151-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 oleg151-server: oleg151-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing load_modules_local oleg151-server: Loading modules from /home/green/git/lustre-release/lustre oleg151-server: detected 4 online CPUs by sysfs oleg151-server: Force libcfs to create 2 CPU partitions oleg151-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg151-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg151-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg151-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg151-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg151-server: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg151-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg151-server: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg151-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg151-server PASS 51 (258s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:56:16 (1709787376) start mds service on oleg151-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg151-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg151-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg151-client.virtnet: -o user_xattr,flock oleg151-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 319 (time 1709787431.16 total 10.54 last 30.26) - open/close 495 (time 1709787441.50 total 20.89 last 17.01) - open/close 682 (time 1709787451.51 total 30.89 last 18.70) - open/close 825 (time 1709787462.46 total 41.85 last 13.05) - open/close 1025 (time 1709787472.47 total 51.85 last 20.00) - open/close 1298 (time 1709787482.60 total 61.99 last 26.93) - open/close 1441 (time 1709787493.50 total 72.88 last 13.13) - open/close 1649 (time 1709787503.51 total 82.89 last 20.77) - open/close 1794 (time 1709787514.50 total 93.89 last 13.19) - open/close 1982 (time 1709787524.71 total 104.10 last 18.42) - open/close 2074 (time 1709787535.00 total 114.39 last 8.94) - open/close 2311 (time 1709787545.03 total 124.42 last 23.63) - open/close 2452 (time 1709787555.45 total 134.84 last 13.53) - open/close 2606 (time 1709787565.90 total 145.29 last 14.74) - open/close 2783 (time 1709787575.91 total 155.30 last 17.68) - open/close 2980 (time 1709787586.19 total 165.57 last 19.17) - open/close 3171 (time 1709787596.24 total 175.63 last 18.99) - open/close 3417 (time 1709787606.25 total 185.63 last 24.59) - open/close 3679 (time 1709787616.41 total 195.79 last 25.79) - open/close 3839 (time 1709787627.37 total 206.75 last 14.60) - open/close 4003 (time 1709787638.22 total 217.60 last 15.11) - open/close 4267 (time 1709787649.22 total 228.61 last 23.98) - open/close 4523 (time 1709787659.23 total 238.62 last 25.57) - open/close 4748 (time 1709787669.25 total 248.63 last 22.47) - open/close 4994 (time 1709787679.95 total 259.34 last 22.98) - open/close 5133 (time 1709787690.80 total 270.19 last 12.81) - open/close 5248 (time 1709787702.90 total 282.29 last 9.50) - open/close 5518 (time 1709787712.90 total 292.29 last 26.99) - open/close 5715 (time 1709787723.65 total 303.03 last 18.34) - open/close 5984 (time 1709787734.34 total 313.72 last 25.16) - open/close 6234 (time 1709787745.63 total 325.02 last 22.13) - open/close 6439 (time 1709787755.79 total 335.18 last 20.18) - open/close 6662 (time 1709787765.79 total 345.18 last 22.30) - open/close 6796 (time 1709787776.27 total 355.65 last 12.80) - open/close 7031 (time 1709787786.37 total 365.75 last 23.27) - open/close 7155 (time 1709787796.91 total 376.29 last 11.76) - open/close 7325 (time 1709787807.86 total 387.25 last 15.52) - open/close 7542 (time 1709787818.01 total 397.39 last 21.38) - open/close 7785 (time 1709787828.10 total 407.48 last 24.08) - open/close 7929 (time 1709787838.83 total 418.22 last 13.42) - open/close 8136 (time 1709787848.85 total 428.23 last 20.67) - open/close 8232 (time 1709787859.76 total 439.14 last 8.80) - open/close 8460 (time 1709787870.77 total 450.15 last 20.70) - open/close 8645 (time 1709787880.77 total 460.16 last 18.49) - open/close 8752 (time 1709787890.79 total 470.18 last 10.68) - open/close 8917 (time 1709787900.83 total 480.22 last 16.44) - open/close 9106 (time 1709787911.63 total 491.01 last 17.51) - open/close 9254 (time 1709787922.67 total 502.05 last 13.41) - open/close 9422 (time 1709787934.81 total 514.19 last 13.84) - open/close 9572 (time 1709787945.74 total 525.13 last 13.72) - open/close 9900 (time 1709787955.75 total 535.14 last 32.75) - open/close 10545 (time 1709787965.76 total 545.15 last 64.44) - open/close 11156 (time 1709787975.77 total 555.15 last 61.09) - open/close 11786 (time 1709787985.77 total 565.15 last 62.98) - open/close 12420 (time 1709787995.78 total 575.16 last 63.33) - open/close 13023 (time 1709788005.79 total 585.17 last 60.26) - open/close 13620 (time 1709788015.80 total 595.19 last 59.59) - open/close 13931 (time 1709788032.48 total 611.86 last 18.65) - open/close 14502 (time 1709788042.49 total 621.87 last 57.06) - open/close 15063 (time 1709788052.49 total 631.87 last 56.09) - open/close 15602 (time 1709788062.49 total 641.88 last 53.88) - open/close 16115 (time 1709788072.51 total 651.89 last 51.23) - open/close 16634 (time 1709788082.51 total 661.90 last 51.85) - open/close 17144 (time 1709788092.53 total 671.91 last 50.94) - open/close 17642 (time 1709788102.53 total 681.92 last 49.75) - open/close 18090 (time 1709788112.56 total 691.95 last 44.67) - open/close 18528 (time 1709788122.57 total 701.95 last 43.77) - open/close 19008 (time 1709788132.59 total 711.97 last 47.92) - open/close 19452 (time 1709788142.60 total 721.98 last 44.34) - open/close 19863 (time 1709788152.62 total 732.00 last 41.03) - open/close 20000 (time 1709788155.86 total 735.24 last 42.26) - open/close 20408 (time 1709788165.88 total 745.27 last 40.70) - open/close 20821 (time 1709788175.90 total 755.28 last 41.25) - open/close 21215 (time 1709788185.93 total 765.31 last 39.27) - open/close 21607 (time 1709788195.94 total 775.32 last 39.15) - open/close 22034 (time 1709788205.96 total 785.35 last 42.60) - open/close 22449 (time 1709788215.98 total 795.37 last 41.43) - open/close 22874 (time 1709788225.99 total 805.38 last 42.45) - open/close 23331 (time 1709788236.01 total 815.40 last 45.62) - open/close 23749 (time 1709788246.03 total 825.42 last 41.70) - open/close 24144 (time 1709788256.05 total 835.43 last 39.45) - open/close 24522 (time 1709788266.07 total 845.45 last 37.73) - open/close 24896 (time 1709788276.09 total 855.47 last 37.32) - open/close 25282 (time 1709788286.09 total 865.48 last 38.58) - open/close 25686 (time 1709788296.12 total 875.51 last 40.28) - open/close 26064 (time 1709788306.15 total 885.54 last 37.69) - open/close 26443 (time 1709788316.16 total 895.55 last 37.86) - open/close 26839 (time 1709788326.17 total 905.55 last 39.59) - open/close 27208 (time 1709788336.19 total 915.57 last 36.83) - open/close 27576 (time 1709788346.20 total 925.59 last 36.74) - open/close 27951 (time 1709788356.21 total 935.59 last 37.48) - open/close 28319 (time 1709788366.21 total 945.60 last 36.78) - open/close 28685 (time 1709788376.24 total 955.62 last 36.51) - open/close 29039 (time 1709788386.27 total 965.65 last 35.29) - open/close 29385 (time 1709788396.28 total 975.67 last 34.55) - open/close 29751 (time 1709788406.29 total 985.68 last 36.56) - open/close 30000 (time 1709788413.19 total 992.58 last 36.09) - open/close 30335 (time 1709788423.20 total 1002.59 last 33.47) - open/close 30666 (time 1709788433.22 total 1012.60 last 33.06) - open/close 30994 (time 1709788443.23 total 1022.62 last 32.75) - open/close 31319 (time 1709788453.25 total 1032.64 last 32.43) - open/close 31647 (time 1709788463.28 total 1042.67 last 32.70) - open/close 31940 (time 1709788473.29 total 1052.67 last 29.29) - open/close 32246 (time 1709788483.30 total 1062.68 last 30.58) - open/close 32563 (time 1709788493.31 total 1072.70 last 31.64) - open/close 32892 (time 1709788503.33 total 1082.72 last 32.84) - open/close 33216 (time 1709788513.35 total 1092.73 last 32.36) - open/close 33535 (time 1709788523.37 total 1102.75 last 31.83) - open/close 33854 (time 1709788533.39 total 1112.78 last 31.83) - open/close 34140 (time 1709788543.39 total 1122.78 last 28.60) - open/close 34440 (time 1709788553.40 total 1132.79 last 29.96) - open/close 34748 (time 1709788563.43 total 1142.82 last 30.71) - open/close 35050 (time 1709788573.43 total 1152.82 last 30.19) - open/close 35326 (time 1709788583.44 total 1162.83 last 27.58) - open/close 35625 (time 1709788593.46 total 1172.84 last 29.85) - open/close 35928 (time 1709788603.48 total 1182.87 last 30.23) - open/close 36227 (time 1709788613.51 total 1192.89 last 29.82) - open/close 36536 (time 1709788623.52 total 1202.90 last 30.88) - open/close 36851 (time 1709788633.55 total 1212.93 last 31.40) - open/close 37146 (time 1709788643.55 total 1222.94 last 29.49) - open/close 37439 (time 1709788653.55 total 1232.94 last 29.30) - open/close 37732 (time 1709788663.55 total 1242.94 last 29.30) - open/close 38015 (time 1709788673.57 total 1252.96 last 28.25) - open/close 38296 (time 1709788683.60 total 1262.98 last 28.02) - open/close 38564 (time 1709788693.62 total 1273.01 last 26.74) - open/close 38839 (time 1709788703.63 total 1283.01 last 27.48) - open/close 39114 (time 1709788713.65 total 1293.04 last 27.43) - open/close 39395 (time 1709788723.66 total 1303.05 last 28.08) - open/close 39677 (time 1709788733.66 total 1313.05 last 28.20) - open/close 39941 (time 1709788743.70 total 1323.08 last 26.31) - open/close 40000 (time 1709788745.95 total 1325.33 last 26.24) - open/close 40238 (time 1709788755.96 total 1335.34 last 23.77) - open/close 40487 (time 1709788765.96 total 1345.34 last 24.89) - open/close 40734 (time 1709788775.97 total 1355.35 last 24.68) - open/close 40978 (time 1709788785.97 total 1365.36 last 24.40) - open/close 41248 (time 1709788795.99 total 1375.37 last 26.95) - open/close 41510 (time 1709788806.01 total 1385.40 last 26.14) - open/close 41775 (time 1709788816.02 total 1395.40 last 26.48) - open/close 42024 (time 1709788826.04 total 1405.43 last 24.84) - open/close 42286 (time 1709788836.07 total 1415.46 last 26.13) - open/close 42545 (time 1709788846.08 total 1425.46 last 25.88) - open/close 42806 (time 1709788856.10 total 1435.49 last 26.04) - open/close 43062 (time 1709788866.13 total 1445.51 last 25.53) - open/close 43331 (time 1709788876.14 total 1455.52 last 26.87) - open/close 43593 (time 1709788886.14 total 1465.52 last 26.20) - open/close 43856 (time 1709788896.15 total 1475.53 last 26.28) - open/close 44113 (time 1709788906.15 total 1485.54 last 25.69) - open/close 44367 (time 1709788916.17 total 1495.56 last 25.34) - open/close 44608 (time 1709788926.19 total 1505.57 last 24.07) - open/close 44848 (time 1709788936.20 total 1515.59 last 23.97) - open/close 45066 (time 1709788946.24 total 1525.63 last 21.71) - open/close 45290 (time 1709788956.27 total 1535.65 last 22.34) - open/close 45520 (time 1709788966.27 total 1545.66 last 22.99) - open/close 45753 (time 1709788976.31 total 1555.69 last 23.23) - open/close 45984 (time 1709788986.32 total 1565.70 last 23.07) - open/close 46218 (time 1709788996.34 total 1575.73 last 23.35) - open/close 46451 (time 1709789006.38 total 1585.77 last 23.20) - open/close 46673 (time 1709789016.39 total 1595.77 last 22.19) - open/close 46905 (time 1709789026.43 total 1605.81 last 23.11) - open/close 47123 (time 1709789036.43 total 1615.82 last 21.79) - open/close 47338 (time 1709789046.43 total 1625.82 last 21.49) - open/close 47558 (time 1709789056.44 total 1635.83 last 21.98) - open/close 47772 (time 1709789066.47 total 1645.86 last 21.34) - open/close 47983 (time 1709789076.49 total 1655.87 last 21.06) - open/close 48188 (time 1709789086.49 total 1665.88 last 20.49) - open/close 48400 (time 1709789096.52 total 1675.90 last 21.14) - open/close 48613 (time 1709789106.54 total 1685.93 last 21.25) - open/close 48819 (time 1709789116.56 total 1695.94 last 20.57) - open/close 49021 (time 1709789126.58 total 1705.96 last 20.16) - open/close 49225 (time 1709789136.62 total 1716.01 last 20.31) - open/close 49429 (time 1709789146.62 total 1726.01 last 20.40) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1734.32 seconds: 28.60 ops/second - unlinked 0 (time 1709789156 ; total 0 ; last 0) - unlinked 10000 (time 1709789668 ; total 512 ; last 512) - unlinked 20000 (time 1709790055 ; total 899 ; last 387) - unlinked 30000 (time 1709790299 ; total 1143 ; last 244) - unlinked 40000 (time 1709790396 ; total 1240 ; last 97) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1275 seconds: 38.901962 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg151-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg151-server unloading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2 PASS 101a (3076s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg151-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg151-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg151-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 oleg151-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg151-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:47:36 (1709790456) start mds service on oleg151-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing load_modules_local oleg151-server: Loading modules from /home/green/git/lustre-release/lustre oleg151-server: detected 4 online CPUs by sysfs oleg151-server: Force libcfs to create 2 CPU partitions oleg151-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg151-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg151-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg151-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg151-client.virtnet: -o user_xattr,flock oleg151-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server start ost1 service on oleg151-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 11 sec oleg151-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b6c93800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6c93800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg151-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg151-server unloading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 PASS 101b (106s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg151-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg151-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg151-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg151-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 oleg151-server: oleg151-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing load_modules_local oleg151-server: Loading modules from /home/green/git/lustre-release/lustre oleg151-server: detected 4 online CPUs by sysfs oleg151-server: Force libcfs to create 2 CPU partitions oleg151-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg151-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3962 sec ========== 00:49:46 (1709790586) === conf-sanity: start cleanup 00:49:47 (1709790587) === === conf-sanity: finish cleanup 00:49:47 (1709790587) ===