-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:44:21 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg403-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg403-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg403-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg403-server oleg403-server: oleg403-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: detected 4 online CPUs by sysfs oleg403-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg403-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg403-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg403-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg403-server: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg403-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg403-server: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg403-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:46:21 (1709786781) start mds service on oleg403-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg403-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg403-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock oleg403-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h46m54s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800b33c8000.stats=clear mdc.lustre-MDT0001-mdc-ffff8800b33c8000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:47:40.000000000 -0500 Modify: 2024-03-06 23:47:40.000000000 -0500 Change: 2024-03-06 23:47:40.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 69 lines, 69 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg403-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server unloading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing unload_modules_local modules unloaded. oleg403-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg403-server: Is the ldiskfs module available? oleg403-server: oleg403-server: tunefs.lustre FATAL: failed to write local files oleg403-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg403-client: oleg403-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg403-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg403-server: Is the ldiskfs module available? oleg403-server: oleg403-server: tunefs.lustre FATAL: failed to write local files oleg403-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg403-client: oleg403-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.204.103@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.204.103@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg403-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg403-server: Is the ldiskfs module available? oleg403-server: oleg403-server: tunefs.lustre FATAL: failed to write local files oleg403-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg403-client: oleg403-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.103@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.103@tcp sys.timeout=20 oleg403-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg403-server: Is the ldiskfs module available? oleg403-server: oleg403-server: tunefs.lustre FATAL: failed to write local files oleg403-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg403-client: oleg403-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.103@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.103@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg403-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg403-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 oleg403-server: oleg403-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: detected 4 online CPUs by sysfs oleg403-server: Force libcfs to create 2 CPU partitions oleg403-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg403-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg403-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg403-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg403-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg403-server: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg403-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg403-server: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg403-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server PASS 35b (204s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:49:48 (1709786988) start mds service on oleg403-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg403-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg403-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock oleg403-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg403-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg403-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg403-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock oleg403-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg403-client.virtnet /mnt/lustre (opts:) stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server 000000 2 000008 oleg403-server: 8+0 records in oleg403-server: 8+0 records out oleg403-server: 4096 bytes (4.1 kB) copied, 0.00108512 s, 3.8 MB/s start mds service on oleg403-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg403-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock oleg403-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg403-client.virtnet /mnt/lustre (opts:) stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server stop mds service on oleg403-server unloading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 PASS 38 (146s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg403-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg403-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg403-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 oleg403-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg403-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:52:17 (1709787137) Stopping clients: oleg403-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg403-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 oleg403-server: oleg403-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: detected 4 online CPUs by sysfs oleg403-server: Force libcfs to create 2 CPU partitions oleg403-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg403-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg403-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg403-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg403-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock oleg403-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg403-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg403-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg403-server umount lustre on /mnt/lustre..... Stopping client oleg403-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server unloading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing unload_modules_local modules unloaded. oleg403-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg403-server: Is the ldiskfs module available? oleg403-server: oleg403-server: tunefs.lustre FATAL: failed to write local files oleg403-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg403-client: oleg403-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg403-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg403-server: Is the ldiskfs module available? oleg403-server: oleg403-server: tunefs.lustre FATAL: failed to write local files oleg403-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg403-client: oleg403-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.204.103@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.204.103@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg403-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg403-server: Is the ldiskfs module available? oleg403-server: oleg403-server: tunefs.lustre FATAL: failed to write local files oleg403-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg403-client: oleg403-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.103@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.103@tcp sys.timeout=20 oleg403-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg403-server: Is the ldiskfs module available? oleg403-server: oleg403-server: tunefs.lustre FATAL: failed to write local files oleg403-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg403-client: oleg403-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.103@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.103@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg403-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg403-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 oleg403-server: oleg403-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: detected 4 online CPUs by sysfs oleg403-server: Force libcfs to create 2 CPU partitions oleg403-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg403-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg403-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg403-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg403-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg403-server: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg403-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg403-server: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg403-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server PASS 51 (246s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:56:28 (1709787388) start mds service on oleg403-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg403-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg403-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock oleg403-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 154 (time 1709787443.49 total 10.42 last 14.78) - open/close 339 (time 1709787454.24 total 21.17 last 17.21) - open/close 539 (time 1709787464.26 total 31.19 last 19.96) - open/close 674 (time 1709787474.34 total 41.26 last 13.40) - open/close 897 (time 1709787484.47 total 51.40 last 22.01) - open/close 1083 (time 1709787494.47 total 61.40 last 18.60) - open/close 1253 (time 1709787504.49 total 71.42 last 16.96) - open/close 1419 (time 1709787515.25 total 82.18 last 15.43) - open/close 1594 (time 1709787526.12 total 93.05 last 16.09) - open/close 1693 (time 1709787536.94 total 103.87 last 9.15) - open/close 1820 (time 1709787546.95 total 113.87 last 12.70) - open/close 1891 (time 1709787557.03 total 123.96 last 7.04) - open/close 2005 (time 1709787567.44 total 134.36 last 10.96) - open/close 2238 (time 1709787577.74 total 144.67 last 22.61) - open/close 2565 (time 1709787588.03 total 154.96 last 31.79) - open/close 2880 (time 1709787598.03 total 164.96 last 31.49) - open/close 3070 (time 1709787608.51 total 175.43 last 18.14) - open/close 3271 (time 1709787618.84 total 185.77 last 19.44) - open/close 3469 (time 1709787629.60 total 196.53 last 18.41) - open/close 3745 (time 1709787639.61 total 206.54 last 27.57) - open/close 3902 (time 1709787649.97 total 216.90 last 15.16) - open/close 4063 (time 1709787659.98 total 226.90 last 16.09) - open/close 4288 (time 1709787669.99 total 236.92 last 22.46) - open/close 4483 (time 1709787680.01 total 246.94 last 19.46) - open/close 4674 (time 1709787690.85 total 257.78 last 17.62) - open/close 4834 (time 1709787701.31 total 268.24 last 15.30) - open/close 5050 (time 1709787711.36 total 278.28 last 21.50) - open/close 5209 (time 1709787722.23 total 289.16 last 14.62) - open/close 5502 (time 1709787732.55 total 299.47 last 28.40) - open/close 5763 (time 1709787742.97 total 309.90 last 25.03) - open/close 6002 (time 1709787754.45 total 321.38 last 20.82) - open/close 6272 (time 1709787764.46 total 331.39 last 26.97) - open/close 6396 (time 1709787774.80 total 341.72 last 12.00) - open/close 6653 (time 1709787784.85 total 351.78 last 25.56) - open/close 6848 (time 1709787794.87 total 361.79 last 19.47) - open/close 7135 (time 1709787805.79 total 372.72 last 26.27) - open/close 7433 (time 1709787816.72 total 383.65 last 27.26) - open/close 7707 (time 1709787826.73 total 393.66 last 27.39) - open/close 7947 (time 1709787836.93 total 403.86 last 23.52) - open/close 8225 (time 1709787847.48 total 414.40 last 26.37) - open/close 8434 (time 1709787857.61 total 424.54 last 20.62) - open/close 8660 (time 1709787867.83 total 434.76 last 22.11) - open/close 8849 (time 1709787877.84 total 444.76 last 18.89) - open/close 9130 (time 1709787888.54 total 455.47 last 26.26) - open/close 9438 (time 1709787899.40 total 466.33 last 28.35) - open/close 9615 (time 1709787909.46 total 476.38 last 17.61) - open/close 10000 (time 1709787918.44 total 485.37 last 42.86) - open/close 10644 (time 1709787928.45 total 495.38 last 64.30) - open/close 11251 (time 1709787938.46 total 505.38 last 60.70) - open/close 11833 (time 1709787948.46 total 515.39 last 58.15) - open/close 12399 (time 1709787958.48 total 525.41 last 56.52) - open/close 12914 (time 1709787968.49 total 535.42 last 51.44) - open/close 13409 (time 1709787978.50 total 545.42 last 49.48) - open/close 13919 (time 1709787988.50 total 555.43 last 50.97) - open/close 14449 (time 1709787998.52 total 565.44 last 52.92) - open/close 14901 (time 1709788008.53 total 575.46 last 45.14) - open/close 15348 (time 1709788018.54 total 585.47 last 44.63) - open/close 15815 (time 1709788028.56 total 595.49 last 46.61) - open/close 16275 (time 1709788038.56 total 605.49 last 46.00) - open/close 16712 (time 1709788048.58 total 615.51 last 43.62) - open/close 17117 (time 1709788058.60 total 625.53 last 40.42) - open/close 17541 (time 1709788068.62 total 635.54 last 42.34) - open/close 18004 (time 1709788078.64 total 645.57 last 46.19) - open/close 18464 (time 1709788088.64 total 655.57 last 45.98) - open/close 18907 (time 1709788098.65 total 665.58 last 44.28) - open/close 19330 (time 1709788108.65 total 675.58 last 42.28) - open/close 19751 (time 1709788118.66 total 685.58 last 42.08) - open/close 20000 (time 1709788124.22 total 691.15 last 44.77) - open/close 20380 (time 1709788134.22 total 701.15 last 37.99) - open/close 20763 (time 1709788144.24 total 711.16 last 38.24) - open/close 21126 (time 1709788154.25 total 721.17 last 36.26) - open/close 21473 (time 1709788164.28 total 731.21 last 34.58) - open/close 21836 (time 1709788174.29 total 741.22 last 36.25) - open/close 22231 (time 1709788184.30 total 751.23 last 39.47) - open/close 22626 (time 1709788194.31 total 761.24 last 39.46) - open/close 22945 (time 1709788204.33 total 771.26 last 31.84) - open/close 23310 (time 1709788214.35 total 781.27 last 36.44) - open/close 23665 (time 1709788224.37 total 791.30 last 35.43) - open/close 24010 (time 1709788234.38 total 801.31 last 34.46) - open/close 24319 (time 1709788244.39 total 811.32 last 30.86) - open/close 24674 (time 1709788254.41 total 821.34 last 35.42) - open/close 25068 (time 1709788264.42 total 831.35 last 39.35) - open/close 25398 (time 1709788274.44 total 841.36 last 32.96) - open/close 25734 (time 1709788284.45 total 851.38 last 33.55) - open/close 26074 (time 1709788294.46 total 861.39 last 33.96) - open/close 26417 (time 1709788304.48 total 871.40 last 34.24) - open/close 26754 (time 1709788314.49 total 881.41 last 33.67) - open/close 27072 (time 1709788324.49 total 891.42 last 31.78) - open/close 27414 (time 1709788334.51 total 901.44 last 34.14) - open/close 27733 (time 1709788344.53 total 911.46 last 31.84) - open/close 28023 (time 1709788354.56 total 921.48 last 28.93) - open/close 28300 (time 1709788364.56 total 931.49 last 27.68) - open/close 28611 (time 1709788374.58 total 941.51 last 31.04) - open/close 28927 (time 1709788384.59 total 951.52 last 31.56) - open/close 29234 (time 1709788394.60 total 961.52 last 30.69) - open/close 29546 (time 1709788404.61 total 971.54 last 31.16) - open/close 29842 (time 1709788414.62 total 981.55 last 29.56) - open/close 30000 (time 1709788420.50 total 987.43 last 26.88) - open/close 30286 (time 1709788430.51 total 997.43 last 28.59) - open/close 30564 (time 1709788440.52 total 1007.45 last 27.77) - open/close 30827 (time 1709788450.54 total 1017.47 last 26.24) - open/close 31102 (time 1709788460.57 total 1027.50 last 27.41) - open/close 31375 (time 1709788470.58 total 1037.51 last 27.27) - open/close 31661 (time 1709788480.60 total 1047.53 last 28.54) - open/close 31960 (time 1709788490.61 total 1057.53 last 29.89) - open/close 32253 (time 1709788500.61 total 1067.54 last 29.28) - open/close 32580 (time 1709788510.64 total 1077.56 last 32.63) - open/close 32881 (time 1709788520.66 total 1087.59 last 30.01) - open/close 33135 (time 1709788530.67 total 1097.60 last 25.38) - open/close 33421 (time 1709788540.68 total 1107.61 last 28.58) - open/close 33725 (time 1709788550.70 total 1117.63 last 30.32) - open/close 34005 (time 1709788560.73 total 1127.66 last 27.93) - open/close 34291 (time 1709788570.76 total 1137.69 last 28.51) - open/close 34589 (time 1709788580.77 total 1147.70 last 29.78) - open/close 34879 (time 1709788590.77 total 1157.70 last 28.99) - open/close 35199 (time 1709788600.80 total 1167.73 last 31.92) - open/close 35506 (time 1709788610.80 total 1177.73 last 30.70) - open/close 35800 (time 1709788620.82 total 1187.75 last 29.33) - open/close 36141 (time 1709788630.85 total 1197.78 last 34.00) - open/close 36475 (time 1709788640.86 total 1207.78 last 33.39) - open/close 36779 (time 1709788650.86 total 1217.79 last 30.38) - open/close 37087 (time 1709788660.89 total 1227.81 last 30.72) - open/close 37363 (time 1709788670.92 total 1237.85 last 27.50) - open/close 37569 (time 1709788680.95 total 1247.88 last 20.54) - open/close 37785 (time 1709788690.99 total 1257.92 last 21.52) - open/close 38023 (time 1709788700.99 total 1267.92 last 23.79) - open/close 38324 (time 1709788711.02 total 1277.94 last 30.02) - open/close 38582 (time 1709788721.02 total 1287.95 last 25.78) - open/close 38884 (time 1709788731.05 total 1297.98 last 30.12) - open/close 39166 (time 1709788741.05 total 1307.98 last 28.20) - open/close 39416 (time 1709788751.08 total 1318.01 last 24.93) - open/close 39688 (time 1709788761.08 total 1328.01 last 27.19) - open/close 40000 (time 1709788771.07 total 1337.99 last 31.25) - open/close 40299 (time 1709788781.08 total 1348.01 last 29.86) - open/close 40566 (time 1709788791.10 total 1358.03 last 26.65) - open/close 40849 (time 1709788801.13 total 1368.06 last 28.20) - open/close 41135 (time 1709788811.15 total 1378.08 last 28.55) - open/close 41429 (time 1709788821.16 total 1388.09 last 29.37) - open/close 41722 (time 1709788831.19 total 1398.12 last 29.21) - open/close 42000 (time 1709788841.20 total 1408.12 last 27.79) - open/close 42251 (time 1709788851.20 total 1418.13 last 25.09) - open/close 42539 (time 1709788861.23 total 1428.16 last 28.72) - open/close 42850 (time 1709788871.26 total 1438.18 last 31.01) - open/close 43150 (time 1709788881.28 total 1448.21 last 29.93) - open/close 43441 (time 1709788891.29 total 1458.22 last 29.08) - open/close 43733 (time 1709788901.30 total 1468.23 last 29.17) - open/close 44020 (time 1709788911.32 total 1478.25 last 28.64) - open/close 44307 (time 1709788921.34 total 1488.26 last 28.66) - open/close 44589 (time 1709788931.36 total 1498.29 last 28.13) - open/close 44879 (time 1709788941.39 total 1508.31 last 28.93) - open/close 45157 (time 1709788951.40 total 1518.33 last 27.76) - open/close 45439 (time 1709788961.41 total 1528.34 last 28.16) - open/close 45720 (time 1709788971.42 total 1538.35 last 28.08) - open/close 45992 (time 1709788981.45 total 1548.38 last 27.12) - open/close 46246 (time 1709788991.47 total 1558.40 last 25.34) - open/close 46519 (time 1709789001.50 total 1568.43 last 27.23) - open/close 46771 (time 1709789011.53 total 1578.46 last 25.13) - open/close 47018 (time 1709789021.53 total 1588.46 last 24.69) - open/close 47263 (time 1709789031.56 total 1598.49 last 24.43) - open/close 47503 (time 1709789041.60 total 1608.52 last 23.92) - open/close 47766 (time 1709789051.60 total 1618.53 last 26.29) - open/close 48022 (time 1709789061.62 total 1628.55 last 25.54) - open/close 48285 (time 1709789071.66 total 1638.58 last 26.21) - open/close 48544 (time 1709789081.67 total 1648.59 last 25.87) - open/close 48796 (time 1709789091.67 total 1658.60 last 25.18) - open/close 49056 (time 1709789101.70 total 1668.63 last 25.93) - open/close 49317 (time 1709789111.72 total 1678.64 last 26.07) - open/close 49575 (time 1709789121.73 total 1688.66 last 25.77) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1689.64 seconds: 29.36 ops/second - unlinked 0 (time 1709789123 ; total 0 ; last 0) - unlinked 10000 (time 1709789559 ; total 436 ; last 436) - unlinked 20000 (time 1709789859 ; total 736 ; last 300) - unlinked 30000 (time 1709790016 ; total 893 ; last 157) - unlinked 40000 (time 1709790088 ; total 965 ; last 72) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1007 seconds: 49.255215 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg403-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server unloading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 PASS 101a (2760s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg403-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg403-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg403-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 oleg403-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg403-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:42:30 (1709790150) start mds service on oleg403-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: detected 4 online CPUs by sysfs oleg403-server: Force libcfs to create 2 CPU partitions oleg403-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg403-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg403-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg403-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock oleg403-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server start ost1 service on oleg403-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 7 sec oleg403-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012aa66000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012aa66000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg403-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server unloading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 PASS 101b (104s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg403-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg403-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg403-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg403-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 oleg403-server: oleg403-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: detected 4 online CPUs by sysfs oleg403-server: Force libcfs to create 2 CPU partitions oleg403-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg403-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3614 sec ========== 00:44:35 (1709790275) === conf-sanity: start cleanup 00:44:35 (1709790275) === === conf-sanity: finish cleanup 00:44:35 (1709790275) ===