-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:40:49 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg356-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg356-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg356-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg356-server oleg356-server: oleg356-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing load_modules_local oleg356-server: Loading modules from /home/green/git/lustre-release/lustre oleg356-server: detected 4 online CPUs by sysfs oleg356-server: Force libcfs to create 2 CPU partitions oleg356-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg356-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg356-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg356-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg356-server: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg356-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg356-server: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg356-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg356-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:42:16 (1709786536) start mds service on oleg356-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg356-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg356-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg356-client.virtnet: -o user_xattr,flock oleg356-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h42m48s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff88012a42c000.stats=clear mdc.lustre-MDT0001-mdc-ffff88012a42c000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:43:34.000000000 -0500 Modify: 2024-03-06 23:43:34.000000000 -0500 Change: 2024-03-06 23:43:34.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 88 lines, 88 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg356-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg356-server unloading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing unload_modules_local modules unloaded. oleg356-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg356-server: Is the ldiskfs module available? oleg356-server: oleg356-server: tunefs.lustre FATAL: failed to write local files oleg356-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg356-client: oleg356-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg356-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg356-server: Is the ldiskfs module available? oleg356-server: oleg356-server: tunefs.lustre FATAL: failed to write local files oleg356-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg356-client: oleg356-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.156@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.156@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg356-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg356-server: Is the ldiskfs module available? oleg356-server: oleg356-server: tunefs.lustre FATAL: failed to write local files oleg356-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg356-client: oleg356-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.156@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.156@tcp sys.timeout=20 oleg356-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg356-server: Is the ldiskfs module available? oleg356-server: oleg356-server: tunefs.lustre FATAL: failed to write local files oleg356-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg356-client: oleg356-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.156@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.156@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg356-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg356-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 oleg356-server: oleg356-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing load_modules_local oleg356-server: Loading modules from /home/green/git/lustre-release/lustre oleg356-server: detected 4 online CPUs by sysfs oleg356-server: Force libcfs to create 2 CPU partitions oleg356-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg356-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg356-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg356-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg356-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg356-server: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg356-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg356-server: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg356-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg356-server PASS 35b (188s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:45:27 (1709786727) start mds service on oleg356-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg356-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg356-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg356-client.virtnet: -o user_xattr,flock oleg356-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg356-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg356-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg356-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg356-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg356-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg356-client.virtnet: -o user_xattr,flock oleg356-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg356-client.virtnet /mnt/lustre (opts:) stop mds service on oleg356-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg356-server 000000 2 000008 oleg356-server: 8+0 records in oleg356-server: 8+0 records out oleg356-server: 4096 bytes (4.1 kB) copied, 0.000288981 s, 14.2 MB/s start mds service on oleg356-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg356-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg356-client.virtnet: -o user_xattr,flock oleg356-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg356-client.virtnet /mnt/lustre (opts:) stop mds service on oleg356-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg356-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server stop mds service on oleg356-server stop mds service on oleg356-server unloading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 PASS 38 (138s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg356-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg356-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg356-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 oleg356-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg356-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:47:48 (1709786868) Stopping clients: oleg356-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg356-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 oleg356-server: oleg356-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing load_modules_local oleg356-server: Loading modules from /home/green/git/lustre-release/lustre oleg356-server: detected 4 online CPUs by sysfs oleg356-server: Force libcfs to create 2 CPU partitions oleg356-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg356-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg356-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg356-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg356-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg356-client.virtnet: -o user_xattr,flock oleg356-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg356-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg356-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg356-server umount lustre on /mnt/lustre..... Stopping client oleg356-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg356-server unloading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing unload_modules_local modules unloaded. oleg356-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg356-server: Is the ldiskfs module available? oleg356-server: oleg356-server: tunefs.lustre FATAL: failed to write local files oleg356-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg356-client: oleg356-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg356-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg356-server: Is the ldiskfs module available? oleg356-server: oleg356-server: tunefs.lustre FATAL: failed to write local files oleg356-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg356-client: oleg356-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.156@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.156@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg356-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg356-server: Is the ldiskfs module available? oleg356-server: oleg356-server: tunefs.lustre FATAL: failed to write local files oleg356-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg356-client: oleg356-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.156@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.156@tcp sys.timeout=20 oleg356-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg356-server: Is the ldiskfs module available? oleg356-server: oleg356-server: tunefs.lustre FATAL: failed to write local files oleg356-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg356-client: oleg356-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.156@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.156@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg356-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg356-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 oleg356-server: oleg356-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing load_modules_local oleg356-server: Loading modules from /home/green/git/lustre-release/lustre oleg356-server: detected 4 online CPUs by sysfs oleg356-server: Force libcfs to create 2 CPU partitions oleg356-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg356-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg356-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg356-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg356-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg356-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg356-server: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg356-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg356-server: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg356-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg356-server PASS 51 (235s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:51:46 (1709787106) start mds service on oleg356-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg356-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg356-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg356-client.virtnet: -o user_xattr,flock oleg356-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 98 (time 1709787161.17 total 11.39 last 8.60) - open/close 301 (time 1709787171.19 total 21.40 last 20.27) - open/close 372 (time 1709787181.47 total 31.69 last 6.90) - open/close 667 (time 1709787192.34 total 42.56 last 27.14) - open/close 843 (time 1709787202.35 total 52.57 last 17.59) - open/close 1096 (time 1709787212.36 total 62.58 last 25.27) - open/close 1352 (time 1709787222.38 total 72.60 last 25.55) - open/close 1598 (time 1709787233.28 total 83.50 last 22.57) - open/close 1709 (time 1709787243.99 total 94.20 last 10.37) - open/close 1938 (time 1709787254.12 total 104.34 last 22.60) - open/close 2181 (time 1709787264.61 total 114.83 last 23.16) - open/close 2375 (time 1709787274.97 total 125.19 last 18.72) - open/close 2532 (time 1709787285.60 total 135.82 last 14.78) - open/close 2766 (time 1709787296.50 total 146.72 last 21.47) - open/close 2939 (time 1709787307.28 total 157.49 last 16.05) - open/close 3129 (time 1709787317.40 total 167.62 last 18.76) - open/close 3327 (time 1709787327.42 total 177.63 last 19.77) - open/close 3617 (time 1709787338.02 total 188.23 last 27.36) - open/close 3807 (time 1709787349.05 total 199.26 last 17.22) - open/close 4034 (time 1709787359.05 total 209.27 last 22.70) - open/close 4268 (time 1709787369.60 total 219.81 last 22.19) - open/close 4472 (time 1709787380.31 total 230.53 last 19.04) - open/close 4652 (time 1709787391.17 total 241.39 last 16.57) - open/close 4754 (time 1709787401.22 total 251.44 last 10.15) - open/close 4964 (time 1709787411.27 total 261.49 last 20.90) - open/close 5109 (time 1709787421.30 total 271.51 last 14.46) - open/close 5359 (time 1709787431.32 total 281.54 last 24.95) - open/close 5459 (time 1709787442.24 total 292.46 last 9.15) - open/close 5691 (time 1709787452.25 total 302.47 last 23.17) - open/close 5832 (time 1709787462.26 total 312.48 last 14.09) - open/close 6017 (time 1709787472.27 total 322.49 last 18.47) - open/close 6132 (time 1709787483.22 total 333.44 last 10.51) - open/close 6290 (time 1709787493.93 total 344.15 last 14.75) - open/close 6398 (time 1709787504.49 total 354.71 last 10.23) - open/close 6530 (time 1709787514.52 total 364.74 last 13.16) - open/close 6647 (time 1709787524.55 total 374.77 last 11.67) - open/close 6879 (time 1709787535.39 total 385.60 last 21.41) - open/close 7048 (time 1709787545.39 total 395.61 last 16.89) - open/close 7307 (time 1709787556.27 total 406.48 last 23.82) - open/close 7489 (time 1709787566.30 total 416.52 last 18.14) - open/close 7691 (time 1709787577.23 total 427.45 last 18.48) - open/close 7930 (time 1709787588.09 total 438.31 last 22.01) - open/close 8095 (time 1709787598.61 total 448.83 last 15.69) - open/close 8278 (time 1709787608.63 total 458.84 last 18.27) - open/close 8433 (time 1709787618.63 total 468.85 last 15.49) - open/close 8547 (time 1709787628.67 total 478.89 last 11.36) - open/close 8723 (time 1709787638.70 total 488.92 last 17.54) - open/close 8912 (time 1709787649.38 total 499.60 last 17.70) - open/close 9092 (time 1709787660.08 total 510.30 last 16.82) - open/close 9258 (time 1709787670.08 total 520.30 last 16.60) - open/close 9445 (time 1709787680.10 total 530.31 last 18.67) - open/close 9603 (time 1709787690.44 total 540.66 last 15.27) - open/close 9803 (time 1709787700.45 total 550.66 last 19.99) - open/close 10000 (time 1709787710.30 total 560.52 last 19.99) - open/close 10127 (time 1709787720.30 total 570.52 last 12.70) - open/close 10268 (time 1709787730.30 total 580.52 last 14.10) - open/close 10469 (time 1709787740.64 total 590.86 last 19.44) - open/close 10661 (time 1709787750.65 total 600.87 last 19.18) - open/close 10817 (time 1709787760.67 total 610.89 last 15.58) - open/close 11075 (time 1709787771.05 total 621.27 last 24.85) - open/close 11178 (time 1709787781.32 total 631.54 last 10.02) - open/close 11328 (time 1709787792.28 total 642.50 last 13.69) - open/close 11809 (time 1709787802.30 total 652.52 last 48.02) - open/close 12298 (time 1709787812.31 total 662.52 last 48.88) - open/close 12791 (time 1709787822.31 total 672.53 last 49.28) - open/close 13255 (time 1709787832.33 total 682.54 last 46.32) - open/close 13694 (time 1709787842.33 total 692.55 last 43.89) - open/close 13851 (time 1709787857.41 total 707.63 last 10.41) - open/close 14286 (time 1709787867.42 total 717.63 last 43.48) - open/close 14724 (time 1709787877.44 total 727.66 last 43.69) - open/close 15129 (time 1709787887.45 total 737.67 last 40.46) - open/close 15432 (time 1709787905.75 total 755.97 last 16.56) - open/close 15852 (time 1709787915.77 total 765.99 last 41.93) - open/close 16227 (time 1709787925.78 total 776.00 last 37.45) - open/close 16449 (time 1709787942.89 total 793.11 last 12.98) - open/close 16821 (time 1709787952.89 total 803.11 last 37.20) - open/close 17191 (time 1709787962.91 total 813.13 last 36.91) - open/close 17554 (time 1709787972.91 total 823.13 last 36.30) - open/close 17923 (time 1709787982.93 total 833.14 last 36.86) - open/close 18333 (time 1709787992.94 total 843.16 last 40.95) - open/close 18704 (time 1709788002.96 total 853.18 last 37.03) - open/close 19065 (time 1709788012.97 total 863.18 last 36.07) - open/close 19427 (time 1709788022.97 total 873.19 last 36.18) - open/close 19761 (time 1709788032.99 total 883.21 last 33.33) - open/close 20000 (time 1709788040.04 total 890.25 last 33.95) - open/close 20323 (time 1709788050.04 total 900.26 last 32.29) - open/close 20644 (time 1709788060.07 total 910.29 last 31.99) - open/close 20967 (time 1709788070.10 total 920.31 last 32.23) - open/close 21349 (time 1709788080.11 total 930.32 last 38.16) - open/close 21770 (time 1709788090.13 total 940.35 last 42.00) - open/close 22200 (time 1709788100.16 total 950.38 last 42.88) - open/close 22606 (time 1709788110.18 total 960.40 last 40.52) - open/close 23021 (time 1709788120.20 total 970.42 last 41.40) - open/close 23432 (time 1709788130.21 total 980.43 last 41.05) - open/close 23858 (time 1709788140.24 total 990.45 last 42.50) - open/close 24248 (time 1709788150.26 total 1000.48 last 38.90) - open/close 24634 (time 1709788160.28 total 1010.49 last 38.54) - open/close 25025 (time 1709788170.30 total 1020.52 last 39.01) - open/close 25392 (time 1709788180.31 total 1030.53 last 36.66) - open/close 25783 (time 1709788190.33 total 1040.55 last 39.02) - open/close 26131 (time 1709788200.36 total 1050.57 last 34.72) - open/close 26486 (time 1709788210.38 total 1060.60 last 35.40) - open/close 26817 (time 1709788220.39 total 1070.61 last 33.08) - open/close 27163 (time 1709788230.40 total 1080.61 last 34.58) - open/close 27503 (time 1709788240.43 total 1090.64 last 33.89) - open/close 27829 (time 1709788250.47 total 1100.68 last 32.48) - open/close 28132 (time 1709788260.47 total 1110.69 last 30.28) - open/close 28425 (time 1709788270.47 total 1120.69 last 29.30) - open/close 28743 (time 1709788280.48 total 1130.70 last 31.77) - open/close 29041 (time 1709788290.49 total 1140.71 last 29.77) - open/close 29342 (time 1709788300.52 total 1150.74 last 30.01) - open/close 29634 (time 1709788310.54 total 1160.75 last 29.16) - open/close 29942 (time 1709788320.54 total 1170.76 last 30.80) - open/close 30276 (time 1709788330.56 total 1180.78 last 33.33) - open/close 30605 (time 1709788340.58 total 1190.80 last 32.84) - open/close 30938 (time 1709788350.60 total 1200.82 last 33.23) - open/close 31265 (time 1709788360.62 total 1210.83 last 32.65) - open/close 31561 (time 1709788370.63 total 1220.85 last 29.56) - open/close 31869 (time 1709788380.63 total 1230.85 last 30.79) - open/close 32176 (time 1709788390.66 total 1240.88 last 30.62) - open/close 32487 (time 1709788400.68 total 1250.90 last 31.03) - open/close 32788 (time 1709788410.70 total 1260.91 last 30.06) - open/close 33110 (time 1709788420.72 total 1270.94 last 32.13) - open/close 33406 (time 1709788430.73 total 1280.95 last 29.56) - open/close 33710 (time 1709788440.74 total 1290.96 last 30.36) - open/close 33995 (time 1709788450.77 total 1300.99 last 28.43) - open/close 34281 (time 1709788460.78 total 1311.00 last 28.57) - open/close 34574 (time 1709788470.80 total 1321.02 last 29.25) - open/close 34886 (time 1709788480.83 total 1331.05 last 31.10) - open/close 35150 (time 1709788490.87 total 1341.09 last 26.30) - open/close 35404 (time 1709788500.87 total 1351.09 last 25.40) - open/close 35672 (time 1709788510.88 total 1361.10 last 26.77) - open/close 35960 (time 1709788520.91 total 1371.12 last 28.73) - open/close 36235 (time 1709788530.92 total 1381.13 last 27.47) - open/close 36525 (time 1709788540.93 total 1391.15 last 28.97) - open/close 36816 (time 1709788550.96 total 1401.17 last 29.01) - open/close 37096 (time 1709788560.97 total 1411.19 last 27.96) - open/close 37387 (time 1709788570.97 total 1421.19 last 29.09) - open/close 37685 (time 1709788580.98 total 1431.20 last 29.77) - open/close 37980 (time 1709788591.01 total 1441.23 last 29.42) - open/close 38274 (time 1709788601.03 total 1451.25 last 29.32) - open/close 38563 (time 1709788611.05 total 1461.27 last 28.84) - open/close 38855 (time 1709788621.09 total 1471.30 last 29.11) - open/close 39146 (time 1709788631.11 total 1481.33 last 29.03) - open/close 39422 (time 1709788641.13 total 1491.35 last 27.53) - open/close 39681 (time 1709788651.17 total 1501.39 last 25.80) - open/close 39924 (time 1709788661.21 total 1511.43 last 24.19) - open/close 40000 (time 1709788664.25 total 1514.47 last 25.01) - open/close 40250 (time 1709788674.26 total 1524.47 last 24.99) - open/close 40504 (time 1709788684.26 total 1534.48 last 25.38) - open/close 40752 (time 1709788694.27 total 1544.48 last 24.79) - open/close 41009 (time 1709788704.29 total 1554.50 last 25.64) - open/close 41273 (time 1709788714.29 total 1564.51 last 26.38) - open/close 41531 (time 1709788724.31 total 1574.53 last 25.75) - open/close 41812 (time 1709788734.34 total 1584.55 last 28.03) - open/close 42088 (time 1709788744.36 total 1594.57 last 27.55) - open/close 42347 (time 1709788754.36 total 1604.58 last 25.89) - open/close 42582 (time 1709788764.40 total 1614.62 last 23.41) - open/close 42789 (time 1709788774.41 total 1624.63 last 20.68) - open/close 42995 (time 1709788784.43 total 1634.65 last 20.55) - open/close 43218 (time 1709788794.46 total 1644.68 last 22.23) - open/close 43425 (time 1709788804.50 total 1654.71 last 20.63) - open/close 43639 (time 1709788814.52 total 1664.74 last 21.34) - open/close 43845 (time 1709788824.54 total 1674.76 last 20.57) - open/close 44053 (time 1709788834.58 total 1684.79 last 20.72) - open/close 44291 (time 1709788844.60 total 1694.81 last 23.76) - open/close 44511 (time 1709788854.62 total 1704.84 last 21.94) - open/close 44746 (time 1709788864.64 total 1714.86 last 23.45) - open/close 44978 (time 1709788874.65 total 1724.87 last 23.19) - open/close 45202 (time 1709788884.67 total 1734.89 last 22.36) - open/close 45458 (time 1709788894.69 total 1744.91 last 25.54) - open/close 45705 (time 1709788904.70 total 1754.92 last 24.67) - open/close 45947 (time 1709788914.71 total 1764.93 last 24.19) - open/close 46196 (time 1709788924.74 total 1774.96 last 24.83) - open/close 46433 (time 1709788934.75 total 1784.97 last 23.66) - open/close 46651 (time 1709788944.77 total 1794.98 last 21.78) - open/close 46876 (time 1709788954.78 total 1805.00 last 22.47) - open/close 47090 (time 1709788964.80 total 1815.02 last 21.35) - open/close 47331 (time 1709788974.82 total 1825.04 last 24.05) - open/close 47571 (time 1709788984.83 total 1835.05 last 23.97) - open/close 47802 (time 1709788994.84 total 1845.05 last 23.10) - open/close 48026 (time 1709789004.84 total 1855.06 last 22.39) - open/close 48255 (time 1709789014.87 total 1865.09 last 22.83) - open/close 48481 (time 1709789024.90 total 1875.11 last 22.54) - open/close 48699 (time 1709789034.94 total 1885.16 last 21.71) - open/close 48914 (time 1709789044.96 total 1895.18 last 21.46) - open/close 49142 (time 1709789054.98 total 1905.20 last 22.75) - open/close 49370 (time 1709789065.02 total 1915.24 last 22.71) - open/close 49581 (time 1709789075.07 total 1925.28 last 21.01) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1926.24 seconds: 25.75 ops/second - unlinked 0 (time 1709789077 ; total 0 ; last 0) - unlinked 10000 (time 1709789590 ; total 513 ; last 513) - unlinked 20000 (time 1709790035 ; total 958 ; last 445) - unlinked 30000 (time 1709790319 ; total 1242 ; last 284) - unlinked 40000 (time 1709790423 ; total 1346 ; last 104) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1372 seconds: 36.151604 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg356-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg356-server unloading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 PASS 101a (3362s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg356-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg356-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg356-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 oleg356-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg356-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:47:50 (1709790470) start mds service on oleg356-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing load_modules_local oleg356-server: Loading modules from /home/green/git/lustre-release/lustre oleg356-server: detected 4 online CPUs by sysfs oleg356-server: Force libcfs to create 2 CPU partitions oleg356-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg356-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg356-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg356-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg356-client.virtnet: -o user_xattr,flock oleg356-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server start ost1 service on oleg356-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 12 sec oleg356-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b639d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b639d800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg356-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg356-server unloading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 PASS 101b (98s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg356-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg356-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg356-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg356-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 oleg356-server: oleg356-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing load_modules_local oleg356-server: Loading modules from /home/green/git/lustre-release/lustre oleg356-server: detected 4 online CPUs by sysfs oleg356-server: Force libcfs to create 2 CPU partitions oleg356-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg356-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 4144 sec ========== 00:49:53 (1709790593) === conf-sanity: start cleanup 00:49:54 (1709790594) === === conf-sanity: finish cleanup 00:49:54 (1709790594) ===