-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:39:08 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg347-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg347-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg347-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg347-server oleg347-server: oleg347-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing load_modules_local oleg347-server: Loading modules from /home/green/git/lustre-release/lustre oleg347-server: detected 4 online CPUs by sysfs oleg347-server: Force libcfs to create 2 CPU partitions oleg347-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg347-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg347-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg347-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg347-server: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg347-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg347-server: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg347-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg347-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:40:38 (1709786438) start mds service on oleg347-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg347-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg347-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-OST0000 oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg347-client.virtnet: -o user_xattr,flock oleg347-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h40m58s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800b40ab000.stats=clear mdc.lustre-MDT0001-mdc-ffff8800b40ab000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:41:41.000000000 -0500 Modify: 2024-03-06 23:41:41.000000000 -0500 Change: 2024-03-06 23:41:41.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 86 lines, 86 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg347-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg347-server unloading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing unload_modules_local modules unloaded. oleg347-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg347-server: Is the ldiskfs module available? oleg347-server: oleg347-server: tunefs.lustre FATAL: failed to write local files oleg347-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg347-client: oleg347-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg347-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg347-server: Is the ldiskfs module available? oleg347-server: oleg347-server: tunefs.lustre FATAL: failed to write local files oleg347-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg347-client: oleg347-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.147@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.147@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg347-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg347-server: Is the ldiskfs module available? oleg347-server: oleg347-server: tunefs.lustre FATAL: failed to write local files oleg347-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg347-client: oleg347-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.147@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.147@tcp sys.timeout=20 oleg347-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg347-server: Is the ldiskfs module available? oleg347-server: oleg347-server: tunefs.lustre FATAL: failed to write local files oleg347-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg347-client: oleg347-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.147@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.147@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg347-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg347-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 oleg347-server: oleg347-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing load_modules_local oleg347-server: Loading modules from /home/green/git/lustre-release/lustre oleg347-server: detected 4 online CPUs by sysfs oleg347-server: Force libcfs to create 2 CPU partitions oleg347-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg347-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg347-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg347-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg347-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg347-server: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg347-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg347-server: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg347-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg347-server PASS 35b (185s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:43:47 (1709786627) start mds service on oleg347-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg347-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg347-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-OST0000 oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg347-client.virtnet: -o user_xattr,flock oleg347-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg347-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg347-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg347-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg347-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg347-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg347-client.virtnet: -o user_xattr,flock oleg347-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg347-client.virtnet /mnt/lustre (opts:) stop mds service on oleg347-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg347-server 000000 2 000008 oleg347-server: 8+0 records in oleg347-server: 8+0 records out oleg347-server: 4096 bytes (4.1 kB) copied, 0.000522812 s, 7.8 MB/s start mds service on oleg347-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg347-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg347-client.virtnet: -o user_xattr,flock oleg347-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg347-client.virtnet /mnt/lustre (opts:) stop mds service on oleg347-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg347-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server stop mds service on oleg347-server stop mds service on oleg347-server unloading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg347-client: oleg347-client: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-client: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 PASS 38 (132s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg347-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg347-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg347-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 oleg347-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg347-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:46:01 (1709786761) Stopping clients: oleg347-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg347-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 oleg347-server: oleg347-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing load_modules_local oleg347-server: Loading modules from /home/green/git/lustre-release/lustre oleg347-server: detected 4 online CPUs by sysfs oleg347-server: Force libcfs to create 2 CPU partitions oleg347-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg347-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg347-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg347-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg347-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg347-client.virtnet: -o user_xattr,flock oleg347-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg347-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg347-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg347-server umount lustre on /mnt/lustre..... Stopping client oleg347-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg347-server unloading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing unload_modules_local modules unloaded. oleg347-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg347-server: Is the ldiskfs module available? oleg347-server: oleg347-server: tunefs.lustre FATAL: failed to write local files oleg347-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg347-client: oleg347-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg347-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg347-server: Is the ldiskfs module available? oleg347-server: oleg347-server: tunefs.lustre FATAL: failed to write local files oleg347-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg347-client: oleg347-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.147@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.147@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg347-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg347-server: Is the ldiskfs module available? oleg347-server: oleg347-server: tunefs.lustre FATAL: failed to write local files oleg347-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg347-client: oleg347-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.147@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.147@tcp sys.timeout=20 oleg347-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg347-server: Is the ldiskfs module available? oleg347-server: oleg347-server: tunefs.lustre FATAL: failed to write local files oleg347-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg347-client: oleg347-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.147@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.147@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg347-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg347-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 oleg347-server: oleg347-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing load_modules_local oleg347-server: Loading modules from /home/green/git/lustre-release/lustre oleg347-server: detected 4 online CPUs by sysfs oleg347-server: Force libcfs to create 2 CPU partitions oleg347-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg347-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg347-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg347-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg347-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg347-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg347-server: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg347-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg347-server: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg347-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg347-server PASS 51 (204s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:49:29 (1709786969) start mds service on oleg347-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg347-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg347-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-OST0000 oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg347-client.virtnet: -o user_xattr,flock oleg347-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 385 (time 1709787021.27 total 10.00 last 38.50) - open/close 539 (time 1709787031.28 total 20.01 last 15.39) - open/close 670 (time 1709787041.88 total 30.61 last 12.36) - open/close 976 (time 1709787052.27 total 41.00 last 29.44) - open/close 1260 (time 1709787063.17 total 51.89 last 26.07) - open/close 1533 (time 1709787073.21 total 61.94 last 27.18) - open/close 1704 (time 1709787083.64 total 72.37 last 16.39) - open/close 1847 (time 1709787093.65 total 82.37 last 14.29) - open/close 2074 (time 1709787103.65 total 92.38 last 22.69) - open/close 2294 (time 1709787114.68 total 103.40 last 19.96) - open/close 2369 (time 1709787124.88 total 113.61 last 7.35) - open/close 2489 (time 1709787134.89 total 123.61 last 11.99) - open/close 2632 (time 1709787145.83 total 134.55 last 13.07) - open/close 2765 (time 1709787156.53 total 145.26 last 12.43) - open/close 2950 (time 1709787167.37 total 156.10 last 17.07) - open/close 3198 (time 1709787177.38 total 166.10 last 24.78) - open/close 3422 (time 1709787188.14 total 176.86 last 20.82) - open/close 3517 (time 1709787198.67 total 187.40 last 9.02) - open/close 3693 (time 1709787209.38 total 198.10 last 16.44) - open/close 3907 (time 1709787219.46 total 208.19 last 21.22) - open/close 4187 (time 1709787230.38 total 219.11 last 25.64) - open/close 4393 (time 1709787240.42 total 229.15 last 20.52) - open/close 4520 (time 1709787250.89 total 239.61 last 12.14) - open/close 4647 (time 1709787261.24 total 249.97 last 12.27) - open/close 4813 (time 1709787271.26 total 259.98 last 16.57) - open/close 5027 (time 1709787281.91 total 270.63 last 20.10) - open/close 5129 (time 1709787291.93 total 280.66 last 10.17) - open/close 5260 (time 1709787303.52 total 292.25 last 11.31) - open/close 5491 (time 1709787313.68 total 302.40 last 22.74) - open/close 5628 (time 1709787324.30 total 313.02 last 12.90) - open/close 5828 (time 1709787334.33 total 323.05 last 19.94) - open/close 5963 (time 1709787344.88 total 333.60 last 12.80) - open/close 6077 (time 1709787355.18 total 343.91 last 11.06) - open/close 6201 (time 1709787365.88 total 354.61 last 11.59) - open/close 6340 (time 1709787375.89 total 364.61 last 13.90) - open/close 6433 (time 1709787385.92 total 374.64 last 9.27) - open/close 6614 (time 1709787396.06 total 384.78 last 17.85) - open/close 6804 (time 1709787406.06 total 394.79 last 18.99) - open/close 7029 (time 1709787416.09 total 404.82 last 22.43) - open/close 7142 (time 1709787426.46 total 415.19 last 10.90) - open/close 7321 (time 1709787436.48 total 425.21 last 17.87) - open/close 7509 (time 1709787446.48 total 435.21 last 18.79) - open/close 7789 (time 1709787456.49 total 445.22 last 27.98) - open/close 7990 (time 1709787466.81 total 455.53 last 19.48) - open/close 8168 (time 1709787477.16 total 465.89 last 17.19) - open/close 8331 (time 1709787487.65 total 476.38 last 15.54) - open/close 8572 (time 1709787498.25 total 486.98 last 22.73) - open/close 8743 (time 1709787508.26 total 496.99 last 17.08) - open/close 8994 (time 1709787518.28 total 507.01 last 25.05) - open/close 9184 (time 1709787529.16 total 517.89 last 17.47) - open/close 9311 (time 1709787539.17 total 527.90 last 12.68) - open/close 9396 (time 1709787549.18 total 537.91 last 8.49) - open/close 9547 (time 1709787559.20 total 547.92 last 15.08) - open/close 9671 (time 1709787579.85 total 568.57 last 6.01) - open/close 9776 (time 1709787589.86 total 578.58 last 10.49) - open/close 9941 (time 1709787599.89 total 588.62 last 16.45) - open/close 10000 (time 1709787602.29 total 591.01 last 24.59) - open/close 10359 (time 1709787612.30 total 601.03 last 35.86) - open/close 10789 (time 1709787622.31 total 611.04 last 42.94) - open/close 11241 (time 1709787632.33 total 621.06 last 45.13) - open/close 11642 (time 1709787642.35 total 631.07 last 40.03) - open/close 12039 (time 1709787652.35 total 641.08 last 39.68) - open/close 12445 (time 1709787662.36 total 651.09 last 40.55) - open/close 12828 (time 1709787672.38 total 661.11 last 38.22) - open/close 13161 (time 1709787682.39 total 671.12 last 33.27) - open/close 13525 (time 1709787692.41 total 681.14 last 36.34) - open/close 13845 (time 1709787702.46 total 691.19 last 31.83) - open/close 14157 (time 1709787712.49 total 701.22 last 31.10) - open/close 14422 (time 1709787731.15 total 719.88 last 14.20) - open/close 14759 (time 1709787741.17 total 729.89 last 33.65) - open/close 15081 (time 1709787751.18 total 739.90 last 32.16) - open/close 15420 (time 1709787761.22 total 749.94 last 33.76) - open/close 15769 (time 1709787771.23 total 759.95 last 34.87) - open/close 16079 (time 1709787781.25 total 769.98 last 30.92) - open/close 16409 (time 1709787791.26 total 779.98 last 32.99) - open/close 16777 (time 1709787801.27 total 790.00 last 36.75) - open/close 17134 (time 1709787811.27 total 800.00 last 35.68) - open/close 17461 (time 1709787821.28 total 810.01 last 32.67) - open/close 17637 (time 1709787837.70 total 826.42 last 10.72) - open/close 17970 (time 1709787847.73 total 836.45 last 33.21) - open/close 18314 (time 1709787868.27 total 856.99 last 16.75) - open/close 18638 (time 1709787878.28 total 867.00 last 32.37) - open/close 18958 (time 1709787888.29 total 877.02 last 31.95) - open/close 19272 (time 1709787898.30 total 887.03 last 31.37) - open/close 19632 (time 1709787908.31 total 897.04 last 35.96) - open/close 19959 (time 1709787918.32 total 907.05 last 32.68) - open/close 20266 (time 1709787928.33 total 917.06 last 30.67) - open/close 20559 (time 1709787938.37 total 927.09 last 29.19) - open/close 20858 (time 1709787948.37 total 937.10 last 29.89) - open/close 21182 (time 1709787958.40 total 947.13 last 32.31) - open/close 21500 (time 1709787968.42 total 957.14 last 31.75) - open/close 21813 (time 1709787978.43 total 967.15 last 31.26) - open/close 22144 (time 1709787988.43 total 977.16 last 33.08) - open/close 22494 (time 1709787998.46 total 987.18 last 34.93) - open/close 22809 (time 1709788008.48 total 997.21 last 31.42) - open/close 23137 (time 1709788018.50 total 1007.23 last 32.73) - open/close 23457 (time 1709788028.52 total 1017.25 last 31.94) - open/close 23756 (time 1709788038.55 total 1027.27 last 29.83) - open/close 24055 (time 1709788048.57 total 1037.30 last 29.81) - open/close 24345 (time 1709788058.60 total 1047.33 last 28.92) - open/close 24588 (time 1709788078.24 total 1066.96 last 12.37) - open/close 24980 (time 1709788088.27 total 1076.99 last 39.09) - open/close 25356 (time 1709788098.30 total 1087.03 last 37.47) - open/close 25731 (time 1709788108.33 total 1097.05 last 37.40) - open/close 26085 (time 1709788118.35 total 1107.07 last 35.34) - open/close 26451 (time 1709788128.35 total 1117.08 last 36.57) - open/close 26837 (time 1709788138.38 total 1127.10 last 38.51) - open/close 27204 (time 1709788148.40 total 1137.12 last 36.63) - open/close 27549 (time 1709788158.40 total 1147.12 last 34.50) - open/close 27907 (time 1709788168.41 total 1157.14 last 35.75) - open/close 28250 (time 1709788178.43 total 1167.15 last 34.25) - open/close 28606 (time 1709788188.43 total 1177.16 last 35.58) - open/close 28934 (time 1709788198.43 total 1187.16 last 32.79) - open/close 29256 (time 1709788208.45 total 1197.18 last 32.14) - open/close 29584 (time 1709788218.46 total 1207.19 last 32.77) - open/close 29922 (time 1709788228.49 total 1217.22 last 33.71) - open/close 30000 (time 1709788230.90 total 1219.63 last 32.32) - open/close 30326 (time 1709788240.92 total 1229.65 last 32.53) - open/close 30640 (time 1709788250.95 total 1239.68 last 31.32) - open/close 30941 (time 1709788260.97 total 1249.70 last 30.04) - open/close 31248 (time 1709788271.00 total 1259.73 last 30.61) - open/close 31556 (time 1709788281.01 total 1269.74 last 30.76) - open/close 31844 (time 1709788291.05 total 1279.77 last 28.70) - open/close 32138 (time 1709788301.09 total 1289.82 last 29.27) - open/close 32421 (time 1709788311.12 total 1299.84 last 28.23) - open/close 32716 (time 1709788321.13 total 1309.86 last 29.47) - open/close 33020 (time 1709788331.16 total 1319.88 last 30.32) - open/close 33318 (time 1709788341.17 total 1329.90 last 29.76) - open/close 33620 (time 1709788351.18 total 1339.91 last 30.16) - open/close 33918 (time 1709788361.21 total 1349.93 last 29.72) - open/close 34211 (time 1709788371.22 total 1359.94 last 29.27) - open/close 34504 (time 1709788381.23 total 1369.96 last 29.26) - open/close 34796 (time 1709788391.26 total 1379.98 last 29.13) - open/close 35087 (time 1709788401.27 total 1389.99 last 29.06) - open/close 35366 (time 1709788411.29 total 1400.01 last 27.85) - open/close 35669 (time 1709788421.29 total 1410.02 last 30.29) - open/close 35946 (time 1709788431.29 total 1420.02 last 27.69) - open/close 36244 (time 1709788441.31 total 1430.04 last 29.74) - open/close 36522 (time 1709788451.33 total 1440.06 last 27.74) - open/close 36804 (time 1709788461.36 total 1450.08 last 28.13) - open/close 37075 (time 1709788471.39 total 1460.11 last 27.02) - open/close 37357 (time 1709788481.41 total 1470.13 last 28.14) - open/close 37613 (time 1709788491.41 total 1480.14 last 25.60) - open/close 37869 (time 1709788501.41 total 1490.14 last 25.59) - open/close 38138 (time 1709788511.42 total 1500.15 last 26.88) - open/close 38419 (time 1709788521.44 total 1510.16 last 28.06) - open/close 38686 (time 1709788531.45 total 1520.18 last 26.65) - open/close 38963 (time 1709788541.49 total 1530.22 last 27.60) - open/close 39238 (time 1709788551.53 total 1540.25 last 27.40) - open/close 39496 (time 1709788561.53 total 1550.25 last 25.80) - open/close 39765 (time 1709788571.53 total 1560.26 last 26.88) - open/close 40000 (time 1709788580.53 total 1569.25 last 26.14) - open/close 40258 (time 1709788590.55 total 1579.28 last 25.74) - open/close 40529 (time 1709788600.58 total 1589.31 last 27.01) - open/close 40810 (time 1709788610.61 total 1599.34 last 28.02) - open/close 41091 (time 1709788620.61 total 1609.34 last 28.10) - open/close 41365 (time 1709788630.63 total 1619.35 last 27.36) - open/close 41623 (time 1709788640.66 total 1629.38 last 25.72) - open/close 41848 (time 1709788650.68 total 1639.41 last 22.44) - open/close 42063 (time 1709788660.69 total 1649.41 last 21.49) - open/close 42286 (time 1709788670.69 total 1659.41 last 22.30) - open/close 42532 (time 1709788680.71 total 1669.43 last 24.56) - open/close 42775 (time 1709788690.72 total 1679.44 last 24.27) - open/close 43028 (time 1709788700.73 total 1689.45 last 25.28) - open/close 43277 (time 1709788710.74 total 1699.47 last 24.86) - open/close 43514 (time 1709788720.79 total 1709.52 last 23.58) - open/close 43782 (time 1709788730.81 total 1719.53 last 26.77) - open/close 44054 (time 1709788740.82 total 1729.55 last 27.16) - open/close 44323 (time 1709788750.85 total 1739.58 last 26.81) - open/close 44563 (time 1709788760.88 total 1749.61 last 23.92) - open/close 44793 (time 1709788770.92 total 1759.65 last 22.92) - open/close 45017 (time 1709788780.96 total 1769.69 last 22.31) - open/close 45244 (time 1709788790.99 total 1779.72 last 22.62) - open/close 45464 (time 1709788801.00 total 1789.73 last 21.98) - open/close 45684 (time 1709788811.04 total 1799.77 last 21.91) - open/close 45911 (time 1709788821.05 total 1809.78 last 22.68) - open/close 46129 (time 1709788831.06 total 1819.78 last 21.78) - open/close 46366 (time 1709788841.09 total 1829.81 last 23.64) - open/close 46596 (time 1709788851.12 total 1839.85 last 22.92) - open/close 46817 (time 1709788861.13 total 1849.86 last 22.08) - open/close 47041 (time 1709788871.13 total 1859.86 last 22.39) - open/close 47260 (time 1709788881.16 total 1869.88 last 21.85) - open/close 47494 (time 1709788891.16 total 1879.89 last 23.39) - open/close 47728 (time 1709788901.19 total 1889.92 last 23.32) - open/close 47957 (time 1709788911.23 total 1899.96 last 22.81) - open/close 48195 (time 1709788921.26 total 1909.99 last 23.73) - open/close 48429 (time 1709788931.27 total 1920.00 last 23.37) - open/close 48646 (time 1709788941.27 total 1930.00 last 21.70) - open/close 48869 (time 1709788951.28 total 1940.00 last 22.30) - open/close 49087 (time 1709788961.32 total 1950.04 last 21.71) - open/close 49311 (time 1709788971.34 total 1960.06 last 22.36) - open/close 49546 (time 1709788981.36 total 1970.09 last 23.44) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1972.43 seconds: 25.15 ops/second - unlinked 0 (time 1709788985 ; total 0 ; last 0) - unlinked 10000 (time 1709789479 ; total 494 ; last 494) - unlinked 20000 (time 1709789911 ; total 926 ; last 432) - unlinked 30000 (time 1709790211 ; total 1226 ; last 300) - unlinked 40000 (time 1709790345 ; total 1360 ; last 134) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1401 seconds: 35.403282 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg347-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg347-server unloading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg347-client: oleg347-client: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-client: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 PASS 101a (3430s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg347-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg347-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg347-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 oleg347-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg347-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:46:40 (1709790400) start mds service on oleg347-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing load_modules_local oleg347-server: Loading modules from /home/green/git/lustre-release/lustre oleg347-server: detected 4 online CPUs by sysfs oleg347-server: Force libcfs to create 2 CPU partitions oleg347-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg347-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg347-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg347-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-OST0000 oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg347-client.virtnet: -o user_xattr,flock oleg347-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server start ost1 service on oleg347-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-OST0000 oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 10 sec oleg347-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b5c61000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5c61000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg347-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg347-server unloading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg347-client: oleg347-client: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-client: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 PASS 101b (96s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg347-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg347-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg347-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg347-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 oleg347-server: oleg347-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing load_modules_local oleg347-server: Loading modules from /home/green/git/lustre-release/lustre oleg347-server: detected 4 online CPUs by sysfs oleg347-server: Force libcfs to create 2 CPU partitions oleg347-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg347-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 4170 sec ========== 00:48:39 (1709790519) === conf-sanity: start cleanup 00:48:39 (1709790519) === === conf-sanity: finish cleanup 00:48:39 (1709790519) ===