-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:39:20 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg131-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg131-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg131-server oleg131-server: oleg131-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg131-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg131-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg131-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg131-server: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg131-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg131-server: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg131-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg131-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:40:44 (1709786444) start mds service on oleg131-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg131-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg131-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg131-client.virtnet: -o user_xattr,flock oleg131-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h41m02s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff88012a5a2000.stats=clear mdc.lustre-MDT0001-mdc-ffff88012a5a2000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:41:45.000000000 -0500 Modify: 2024-03-06 23:41:45.000000000 -0500 Change: 2024-03-06 23:41:45.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 86 lines, 86 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg131-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg131-server unloading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing unload_modules_local modules unloaded. oleg131-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg131-server: Is the ldiskfs module available? oleg131-server: oleg131-server: tunefs.lustre FATAL: failed to write local files oleg131-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg131-client: oleg131-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg131-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg131-server: Is the ldiskfs module available? oleg131-server: oleg131-server: tunefs.lustre FATAL: failed to write local files oleg131-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg131-client: oleg131-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.131@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.131@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg131-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg131-server: Is the ldiskfs module available? oleg131-server: oleg131-server: tunefs.lustre FATAL: failed to write local files oleg131-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg131-client: oleg131-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.131@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.131@tcp sys.timeout=20 oleg131-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg131-server: Is the ldiskfs module available? oleg131-server: oleg131-server: tunefs.lustre FATAL: failed to write local files oleg131-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg131-client: oleg131-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.131@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.131@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg131-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 oleg131-server: oleg131-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg131-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg131-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg131-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg131-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg131-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg131-server: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg131-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg131-server: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg131-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg131-server PASS 35b (148s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:43:14 (1709786594) start mds service on oleg131-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg131-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg131-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg131-client.virtnet: -o user_xattr,flock oleg131-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg131-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg131-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg131-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg131-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg131-client.virtnet: -o user_xattr,flock oleg131-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg131-client.virtnet /mnt/lustre (opts:) stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg131-server 000000 2 000008 oleg131-server: 8+0 records in oleg131-server: 8+0 records out oleg131-server: 4096 bytes (4.1 kB) copied, 0.00038888 s, 10.5 MB/s start mds service on oleg131-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg131-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg131-client.virtnet: -o user_xattr,flock oleg131-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg131-client.virtnet /mnt/lustre (opts:) stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg131-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server stop mds service on oleg131-server stop mds service on oleg131-server unloading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 PASS 38 (140s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg131-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg131-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg131-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 oleg131-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg131-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:45:37 (1709786737) Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg131-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 oleg131-server: oleg131-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg131-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg131-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg131-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg131-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg131-client.virtnet: -o user_xattr,flock oleg131-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg131-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec stop ost2 service on oleg131-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg131-server umount lustre on /mnt/lustre..... Stopping client oleg131-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg131-server unloading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing unload_modules_local modules unloaded. oleg131-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg131-server: Is the ldiskfs module available? oleg131-server: oleg131-server: tunefs.lustre FATAL: failed to write local files oleg131-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg131-client: oleg131-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg131-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg131-server: Is the ldiskfs module available? oleg131-server: oleg131-server: tunefs.lustre FATAL: failed to write local files oleg131-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg131-client: oleg131-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.131@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.131@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg131-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg131-server: Is the ldiskfs module available? oleg131-server: oleg131-server: tunefs.lustre FATAL: failed to write local files oleg131-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg131-client: oleg131-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.131@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.131@tcp sys.timeout=20 oleg131-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg131-server: Is the ldiskfs module available? oleg131-server: oleg131-server: tunefs.lustre FATAL: failed to write local files oleg131-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg131-client: oleg131-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.131@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.131@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg131-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 oleg131-server: oleg131-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg131-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg131-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg131-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg131-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg131-server: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg131-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg131-server: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg131-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg131-server PASS 51 (238s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:49:38 (1709786978) start mds service on oleg131-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg131-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg131-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg131-client.virtnet: -o user_xattr,flock oleg131-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 155 (time 1709787025.58 total 10.00 last 15.49) - open/close 383 (time 1709787036.02 total 20.45 last 21.83) - open/close 651 (time 1709787046.03 total 30.45 last 26.79) - open/close 817 (time 1709787056.03 total 40.46 last 16.59) - open/close 1062 (time 1709787066.28 total 50.71 last 23.91) - open/close 1253 (time 1709787076.28 total 60.71 last 19.09) - open/close 1406 (time 1709787095.35 total 79.78 last 8.02) - open/close 1571 (time 1709787105.55 total 89.98 last 16.18) - open/close 1751 (time 1709787115.88 total 100.31 last 17.42) - open/close 1878 (time 1709787125.90 total 110.33 last 12.67) - open/close 2101 (time 1709787135.98 total 120.40 last 22.14) - open/close 2285 (time 1709787146.00 total 130.43 last 18.36) - open/close 2471 (time 1709787156.31 total 140.74 last 18.03) - open/close 2618 (time 1709787166.74 total 151.17 last 14.10) - open/close 2707 (time 1709787177.19 total 161.62 last 8.52) - open/close 2864 (time 1709787188.05 total 172.48 last 14.45) - open/close 2981 (time 1709787198.13 total 182.55 last 11.61) - open/close 3162 (time 1709787208.68 total 193.11 last 17.14) - open/close 3316 (time 1709787219.37 total 203.80 last 14.41) - open/close 3472 (time 1709787229.38 total 213.80 last 15.59) - open/close 3638 (time 1709787239.38 total 223.81 last 16.59) - open/close 3849 (time 1709787250.07 total 234.50 last 19.73) - open/close 4052 (time 1709787261.61 total 246.04 last 17.60) - open/close 4285 (time 1709787271.62 total 256.05 last 23.28) - open/close 4463 (time 1709787281.95 total 266.38 last 17.22) - open/close 4628 (time 1709787291.96 total 276.38 last 16.49) - open/close 4855 (time 1709787302.57 total 286.99 last 21.40) - open/close 5019 (time 1709787312.80 total 297.22 last 16.03) - open/close 5217 (time 1709787323.88 total 308.31 last 17.86) - open/close 5357 (time 1709787333.89 total 318.32 last 13.99) - open/close 5496 (time 1709787343.90 total 328.33 last 13.88) - open/close 5696 (time 1709787354.70 total 339.13 last 18.52) - open/close 5884 (time 1709787364.71 total 349.14 last 18.78) - open/close 5981 (time 1709787375.56 total 359.99 last 8.94) - open/close 6187 (time 1709787386.19 total 370.61 last 19.39) - open/close 6351 (time 1709787396.99 total 381.42 last 15.18) - open/close 6529 (time 1709787406.99 total 391.42 last 17.80) - open/close 6675 (time 1709787417.78 total 402.20 last 13.54) - open/close 6820 (time 1709787427.94 total 412.36 last 14.27) - open/close 7011 (time 1709787437.96 total 422.38 last 19.06) - open/close 7164 (time 1709787448.48 total 432.91 last 14.54) - open/close 7371 (time 1709787458.49 total 442.92 last 20.69) - open/close 7494 (time 1709787469.08 total 453.51 last 11.61) - open/close 7661 (time 1709787479.08 total 463.51 last 16.69) - open/close 7865 (time 1709787489.92 total 474.35 last 18.82) - open/close 7984 (time 1709787499.93 total 484.36 last 11.89) - open/close 8226 (time 1709787509.98 total 494.41 last 24.08) - open/close 8384 (time 1709787519.99 total 504.42 last 15.78) - open/close 8521 (time 1709787530.81 total 515.24 last 12.66) - open/close 8736 (time 1709787540.83 total 525.26 last 21.45) - open/close 8828 (time 1709787550.85 total 535.27 last 9.19) - open/close 8987 (time 1709787560.86 total 545.29 last 15.88) - open/close 9157 (time 1709787571.37 total 555.79 last 16.18) - open/close 9281 (time 1709787581.38 total 565.81 last 12.38) - open/close 9360 (time 1709787591.46 total 575.88 last 7.84) - open/close 9628 (time 1709787601.46 total 585.89 last 26.79) - open/close 9927 (time 1709787618.75 total 603.18 last 17.30) - open/close 10396 (time 1709787628.80 total 613.23 last 46.66) - open/close 10880 (time 1709787638.80 total 623.23 last 48.39) - open/close 11339 (time 1709787648.81 total 633.24 last 45.85) - open/close 11788 (time 1709787658.82 total 643.24 last 44.88) - open/close 12209 (time 1709787668.83 total 653.25 last 42.06) - open/close 12684 (time 1709787678.85 total 663.28 last 47.39) - open/close 13090 (time 1709787688.88 total 673.30 last 40.49) - open/close 13511 (time 1709787698.88 total 683.31 last 42.07) - open/close 13938 (time 1709787708.90 total 693.32 last 42.64) - open/close 14386 (time 1709787718.91 total 703.34 last 44.73) - open/close 14844 (time 1709787728.93 total 713.36 last 45.70) - open/close 15277 (time 1709787738.95 total 723.38 last 43.24) - open/close 15666 (time 1709787748.97 total 733.40 last 38.81) - open/close 16032 (time 1709787758.98 total 743.41 last 36.58) - open/close 16403 (time 1709787768.98 total 753.41 last 37.10) - open/close 16783 (time 1709787778.98 total 763.41 last 37.99) - open/close 17086 (time 1709787788.99 total 773.42 last 30.27) - open/close 17401 (time 1709787799.00 total 783.43 last 31.47) - open/close 17785 (time 1709787809.00 total 793.43 last 38.39) - open/close 18183 (time 1709787819.01 total 803.44 last 39.78) - open/close 18526 (time 1709787829.08 total 813.51 last 34.05) - open/close 18818 (time 1709787839.10 total 823.53 last 29.14) - open/close 19099 (time 1709787849.19 total 833.62 last 27.85) - open/close 19449 (time 1709787859.21 total 843.64 last 34.92) - open/close 19769 (time 1709787869.22 total 853.65 last 31.97) - open/close 20000 (time 1709787876.22 total 860.65 last 33.01) - open/close 20319 (time 1709787886.22 total 870.65 last 31.90) - open/close 20647 (time 1709787896.24 total 880.67 last 32.73) - open/close 20935 (time 1709787906.25 total 890.68 last 28.79) - open/close 21228 (time 1709787916.26 total 900.68 last 29.28) - open/close 21500 (time 1709787926.28 total 910.71 last 27.12) - open/close 21789 (time 1709787936.30 total 920.72 last 28.86) - open/close 22070 (time 1709787946.32 total 930.75 last 28.03) - open/close 22422 (time 1709787956.34 total 940.77 last 35.13) - open/close 22758 (time 1709787966.39 total 950.82 last 33.44) - open/close 23073 (time 1709787976.40 total 960.82 last 31.49) - open/close 23426 (time 1709787986.42 total 970.85 last 35.21) - open/close 23775 (time 1709787996.45 total 980.88 last 34.80) - open/close 24105 (time 1709788006.45 total 990.88 last 32.98) - open/close 24439 (time 1709788016.48 total 1000.91 last 33.31) - open/close 24759 (time 1709788026.48 total 1010.91 last 31.99) - open/close 25106 (time 1709788036.50 total 1020.93 last 34.62) - open/close 25423 (time 1709788046.54 total 1030.97 last 31.59) - open/close 25758 (time 1709788056.54 total 1040.97 last 33.49) - open/close 26076 (time 1709788066.54 total 1050.97 last 31.80) - open/close 26415 (time 1709788076.56 total 1060.99 last 33.85) - open/close 26744 (time 1709788086.58 total 1071.01 last 32.83) - open/close 27062 (time 1709788096.59 total 1081.02 last 31.76) - open/close 27378 (time 1709788106.62 total 1091.05 last 31.50) - open/close 27693 (time 1709788116.66 total 1101.09 last 31.37) - open/close 27996 (time 1709788126.68 total 1111.11 last 30.24) - open/close 28311 (time 1709788136.71 total 1121.14 last 31.41) - open/close 28597 (time 1709788146.73 total 1131.15 last 28.55) - open/close 28893 (time 1709788156.75 total 1141.18 last 29.54) - open/close 29165 (time 1709788166.78 total 1151.21 last 27.11) - open/close 29457 (time 1709788176.81 total 1161.24 last 29.12) - open/close 29743 (time 1709788186.84 total 1171.27 last 28.50) - open/close 30000 (time 1709788196.45 total 1180.88 last 26.75) - open/close 30304 (time 1709788206.46 total 1190.88 last 30.38) - open/close 30603 (time 1709788216.49 total 1200.91 last 29.81) - open/close 30895 (time 1709788226.50 total 1210.93 last 29.15) - open/close 31211 (time 1709788236.53 total 1220.96 last 31.50) - open/close 31500 (time 1709788246.58 total 1231.00 last 28.78) - open/close 31782 (time 1709788256.58 total 1241.01 last 28.18) - open/close 32041 (time 1709788266.60 total 1251.03 last 25.85) - open/close 32316 (time 1709788276.64 total 1261.07 last 27.40) - open/close 32605 (time 1709788286.65 total 1271.08 last 28.88) - open/close 32894 (time 1709788296.67 total 1281.10 last 28.84) - open/close 33165 (time 1709788306.68 total 1291.10 last 27.08) - open/close 33439 (time 1709788316.71 total 1301.13 last 27.32) - open/close 33729 (time 1709788326.74 total 1311.16 last 28.92) - open/close 34008 (time 1709788336.75 total 1321.18 last 27.87) - open/close 34272 (time 1709788346.77 total 1331.20 last 26.34) - open/close 34540 (time 1709788356.79 total 1341.22 last 26.75) - open/close 34650 (time 1709788371.93 total 1356.36 last 7.27) - open/close 34924 (time 1709788381.97 total 1366.39 last 27.30) - open/close 35190 (time 1709788392.00 total 1376.42 last 26.52) - open/close 35451 (time 1709788402.00 total 1386.43 last 26.09) - open/close 35731 (time 1709788412.04 total 1396.47 last 27.90) - open/close 35982 (time 1709788422.04 total 1406.47 last 25.09) - open/close 36236 (time 1709788432.05 total 1416.48 last 25.38) - open/close 36483 (time 1709788442.06 total 1426.49 last 24.68) - open/close 36742 (time 1709788452.08 total 1436.51 last 25.83) - open/close 36996 (time 1709788462.10 total 1446.53 last 25.35) - open/close 37236 (time 1709788472.14 total 1456.57 last 23.90) - open/close 37490 (time 1709788482.19 total 1466.62 last 25.27) - open/close 37738 (time 1709788492.23 total 1476.66 last 24.71) - open/close 37986 (time 1709788502.25 total 1486.68 last 24.76) - open/close 38246 (time 1709788512.27 total 1496.70 last 25.94) - open/close 38501 (time 1709788522.29 total 1506.72 last 25.45) - open/close 38755 (time 1709788532.32 total 1516.75 last 25.34) - open/close 39008 (time 1709788542.34 total 1526.76 last 25.25) - open/close 39255 (time 1709788552.36 total 1536.79 last 24.64) - open/close 39507 (time 1709788562.37 total 1546.79 last 25.19) - open/close 39767 (time 1709788572.37 total 1556.80 last 26.00) - open/close 39995 (time 1709788582.39 total 1566.81 last 22.76) - open/close 40237 (time 1709788592.40 total 1576.83 last 24.16) - open/close 40483 (time 1709788602.41 total 1586.84 last 24.58) - open/close 40732 (time 1709788612.43 total 1596.86 last 24.84) - open/close 40993 (time 1709788622.45 total 1606.88 last 26.05) - open/close 41258 (time 1709788632.47 total 1616.89 last 26.46) - open/close 41501 (time 1709788642.48 total 1626.90 last 24.28) - open/close 41742 (time 1709788652.49 total 1636.92 last 24.06) - open/close 41980 (time 1709788662.52 total 1646.95 last 23.74) - open/close 42211 (time 1709788672.56 total 1656.99 last 23.01) - open/close 42425 (time 1709788682.57 total 1666.99 last 21.38) - open/close 42652 (time 1709788692.58 total 1677.00 last 22.68) - open/close 42882 (time 1709788702.59 total 1687.02 last 22.96) - open/close 43109 (time 1709788712.60 total 1697.03 last 22.69) - open/close 43345 (time 1709788722.62 total 1707.04 last 23.56) - open/close 43576 (time 1709788732.62 total 1717.05 last 23.08) - open/close 43797 (time 1709788742.67 total 1727.10 last 22.00) - open/close 44007 (time 1709788752.69 total 1737.12 last 20.95) - open/close 44215 (time 1709788762.71 total 1747.13 last 20.77) - open/close 44422 (time 1709788772.72 total 1757.15 last 20.67) - open/close 44625 (time 1709788782.75 total 1767.18 last 20.24) - open/close 44847 (time 1709788792.76 total 1777.19 last 22.18) - open/close 45070 (time 1709788802.79 total 1787.22 last 22.22) - open/close 45298 (time 1709788812.82 total 1797.25 last 22.74) - open/close 45519 (time 1709788822.86 total 1807.29 last 22.01) - open/close 45741 (time 1709788832.90 total 1817.33 last 22.11) - open/close 45965 (time 1709788842.92 total 1827.35 last 22.36) - open/close 46194 (time 1709788852.94 total 1837.37 last 22.85) - open/close 46406 (time 1709788862.97 total 1847.40 last 21.15) - open/close 46634 (time 1709788873.00 total 1857.43 last 22.73) - open/close 46855 (time 1709788883.04 total 1867.47 last 22.01) - open/close 47071 (time 1709788893.08 total 1877.51 last 21.51) - open/close 47291 (time 1709788903.11 total 1887.53 last 21.95) - open/close 47508 (time 1709788913.14 total 1897.57 last 21.63) - open/close 47711 (time 1709788923.14 total 1907.57 last 20.30) - open/close 47910 (time 1709788933.18 total 1917.61 last 19.82) - open/close 48110 (time 1709788943.22 total 1927.64 last 19.93) - open/close 48304 (time 1709788953.22 total 1937.65 last 19.40) - open/close 48502 (time 1709788963.25 total 1947.68 last 19.74) - open/close 48699 (time 1709788973.26 total 1957.68 last 19.69) - open/close 48898 (time 1709788983.27 total 1967.70 last 19.87) - open/close 49095 (time 1709788993.28 total 1977.71 last 19.68) - open/close 49293 (time 1709789003.29 total 1987.72 last 19.78) - open/close 49487 (time 1709789013.33 total 1997.75 last 19.33) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 2003.26 seconds: 24.76 ops/second - unlinked 0 (time 1709789020 ; total 0 ; last 0) - unlinked 10000 (time 1709789564 ; total 544 ; last 544) - unlinked 20000 (time 1709789997 ; total 977 ; last 433) - unlinked 30000 (time 1709790280 ; total 1260 ; last 283) - unlinked 40000 (time 1709790398 ; total 1378 ; last 118) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1419 seconds: 34.954193 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg131-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg131-server unloading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 PASS 101a (3488s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg131-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg131-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg131-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 oleg131-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg131-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:47:50 (1709790470) start mds service on oleg131-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg131-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg131-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg131-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg131-client.virtnet: -o user_xattr,flock oleg131-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server start ost1 service on oleg131-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 10 sec oleg131-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012a609000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a609000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg131-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg131-server unloading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 PASS 101b (101s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg131-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg131-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg131-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 oleg131-server: oleg131-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: Force libcfs to create 2 CPU partitions