-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:38:35 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg430-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg430-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg430-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg430-server oleg430-server: oleg430-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing load_modules_local oleg430-server: Loading modules from /home/green/git/lustre-release/lustre oleg430-server: detected 4 online CPUs by sysfs oleg430-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg430-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg430-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg430-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg430-server: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg430-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg430-server: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg430-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg430-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:40:00 (1709786400) start mds service on oleg430-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg430-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg430-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg430-client.virtnet: -o user_xattr,flock oleg430-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h40m18s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff88012b022800.stats=clear mdc.lustre-MDT0001-mdc-ffff88012b022800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:41:01.000000000 -0500 Modify: 2024-03-06 23:41:01.000000000 -0500 Change: 2024-03-06 23:41:01.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 82 lines, 82 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg430-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg430-server unloading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing unload_modules_local modules unloaded. oleg430-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg430-server: Is the ldiskfs module available? oleg430-server: oleg430-server: tunefs.lustre FATAL: failed to write local files oleg430-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg430-client: oleg430-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg430-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg430-server: Is the ldiskfs module available? oleg430-server: oleg430-server: tunefs.lustre FATAL: failed to write local files oleg430-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg430-client: oleg430-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.204.130@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.204.130@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg430-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg430-server: Is the ldiskfs module available? oleg430-server: oleg430-server: tunefs.lustre FATAL: failed to write local files oleg430-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg430-client: oleg430-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.130@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.130@tcp sys.timeout=20 oleg430-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg430-server: Is the ldiskfs module available? oleg430-server: oleg430-server: tunefs.lustre FATAL: failed to write local files oleg430-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg430-client: oleg430-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.130@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.130@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg430-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg430-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 oleg430-server: oleg430-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing load_modules_local oleg430-server: Loading modules from /home/green/git/lustre-release/lustre oleg430-server: detected 4 online CPUs by sysfs oleg430-server: Force libcfs to create 2 CPU partitions oleg430-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg430-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg430-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg430-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg430-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg430-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg430-server: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg430-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg430-server: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg430-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg430-server PASS 35b (137s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:42:18 (1709786538) start mds service on oleg430-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg430-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg430-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg430-client.virtnet: -o user_xattr,flock oleg430-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg430-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg430-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg430-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg430-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg430-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg430-client.virtnet: -o user_xattr,flock oleg430-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg430-client.virtnet /mnt/lustre (opts:) stop mds service on oleg430-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg430-server 000000 2 000008 oleg430-server: 8+0 records in oleg430-server: 8+0 records out oleg430-server: 4096 bytes (4.1 kB) copied, 0.000437135 s, 9.4 MB/s start mds service on oleg430-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg430-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg430-client.virtnet: -o user_xattr,flock oleg430-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg430-client.virtnet /mnt/lustre (opts:) stop mds service on oleg430-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg430-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server stop mds service on oleg430-server stop mds service on oleg430-server unloading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 PASS 38 (113s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg430-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg430-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg430-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 oleg430-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg430-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:44:13 (1709786653) Stopping clients: oleg430-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg430-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 oleg430-server: oleg430-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing load_modules_local oleg430-server: Loading modules from /home/green/git/lustre-release/lustre oleg430-server: detected 4 online CPUs by sysfs oleg430-server: Force libcfs to create 2 CPU partitions oleg430-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg430-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg430-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg430-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg430-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg430-client.virtnet: -o user_xattr,flock oleg430-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg430-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec stop ost2 service on oleg430-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg430-server umount lustre on /mnt/lustre..... Stopping client oleg430-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg430-server unloading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing unload_modules_local modules unloaded. oleg430-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg430-server: Is the ldiskfs module available? oleg430-server: oleg430-server: tunefs.lustre FATAL: failed to write local files oleg430-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg430-client: oleg430-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg430-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg430-server: Is the ldiskfs module available? oleg430-server: oleg430-server: tunefs.lustre FATAL: failed to write local files oleg430-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg430-client: oleg430-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.204.130@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.204.130@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg430-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg430-server: Is the ldiskfs module available? oleg430-server: oleg430-server: tunefs.lustre FATAL: failed to write local files oleg430-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg430-client: oleg430-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.130@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.130@tcp sys.timeout=20 oleg430-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg430-server: Is the ldiskfs module available? oleg430-server: oleg430-server: tunefs.lustre FATAL: failed to write local files oleg430-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg430-client: oleg430-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.130@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.130@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg430-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg430-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 oleg430-server: oleg430-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing load_modules_local oleg430-server: Loading modules from /home/green/git/lustre-release/lustre oleg430-server: detected 4 online CPUs by sysfs oleg430-server: Force libcfs to create 2 CPU partitions oleg430-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg430-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg430-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg430-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg430-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg430-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg430-server: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg430-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec oleg430-server: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg430-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg430-server PASS 51 (204s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:47:40 (1709786860) start mds service on oleg430-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg430-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg430-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg430-client.virtnet: -o user_xattr,flock oleg430-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 198 (time 1709786904.93 total 11.00 last 18.00) - open/close 291 (time 1709786915.10 total 21.17 last 9.15) - open/close 364 (time 1709786925.10 total 31.17 last 7.30) - open/close 508 (time 1709786937.61 total 43.68 last 11.51) - open/close 811 (time 1709786947.62 total 53.69 last 30.27) - open/close 1132 (time 1709786958.39 total 64.46 last 29.80) - open/close 1281 (time 1709786968.47 total 74.54 last 14.78) - open/close 1340 (time 1709786978.50 total 84.57 last 5.89) - open/close 1615 (time 1709786989.46 total 95.53 last 25.07) - open/close 1971 (time 1709786999.50 total 105.57 last 35.48) - open/close 2133 (time 1709787009.67 total 115.74 last 15.93) - open/close 2316 (time 1709787019.88 total 125.95 last 17.92) - open/close 2569 (time 1709787029.89 total 135.96 last 25.28) - open/close 2722 (time 1709787040.21 total 146.28 last 14.83) - open/close 2979 (time 1709787050.22 total 156.29 last 25.66) - open/close 3199 (time 1709787061.14 total 167.21 last 20.14) - open/close 3554 (time 1709787072.03 total 178.10 last 32.61) - open/close 3753 (time 1709787082.93 total 189.00 last 18.26) - open/close 3945 (time 1709787093.66 total 199.73 last 17.89) - open/close 4165 (time 1709787103.69 total 209.76 last 21.93) - open/close 4423 (time 1709787114.30 total 220.37 last 24.32) - open/close 4624 (time 1709787124.36 total 230.43 last 19.97) - open/close 4840 (time 1709787134.40 total 240.47 last 21.52) - open/close 5034 (time 1709787154.59 total 260.66 last 9.61) - open/close 5345 (time 1709787165.01 total 271.08 last 29.85) - open/close 5600 (time 1709787175.03 total 281.10 last 25.44) - open/close 5781 (time 1709787185.76 total 291.83 last 16.88) - open/close 5999 (time 1709787196.06 total 302.13 last 21.16) - open/close 6093 (time 1709787206.06 total 312.13 last 9.40) - open/close 6290 (time 1709787216.07 total 322.14 last 19.69) - open/close 6525 (time 1709787226.62 total 332.69 last 22.27) - open/close 6727 (time 1709787237.35 total 343.42 last 18.82) - open/close 6976 (time 1709787247.96 total 354.03 last 23.47) - open/close 7118 (time 1709787257.98 total 364.05 last 14.18) - open/close 7247 (time 1709787268.73 total 374.80 last 12.00) - open/close 7468 (time 1709787278.73 total 384.80 last 22.09) - open/close 7729 (time 1709787289.60 total 395.67 last 24.00) - open/close 7970 (time 1709787299.62 total 405.69 last 24.06) - open/close 8117 (time 1709787309.72 total 415.79 last 14.55) - open/close 8335 (time 1709787320.05 total 426.12 last 21.10) - open/close 8437 (time 1709787330.06 total 436.13 last 10.19) - open/close 8663 (time 1709787340.99 total 447.06 last 20.68) - open/close 8860 (time 1709787350.99 total 457.06 last 19.68) - open/close 8993 (time 1709787366.86 total 472.93 last 8.38) - open/close 9139 (time 1709787377.10 total 483.17 last 14.25) - open/close 9351 (time 1709787387.94 total 494.01 last 19.56) - open/close 9508 (time 1709787397.95 total 504.02 last 15.69) - open/close 9588 (time 1709787414.84 total 520.91 last 4.73) - open/close 10000 (time 1709787424.42 total 530.49 last 43.04) - open/close 10455 (time 1709787434.43 total 540.50 last 45.45) - open/close 10908 (time 1709787444.43 total 550.50 last 45.28) - open/close 11318 (time 1709787454.44 total 560.51 last 40.97) - open/close 11748 (time 1709787464.46 total 570.53 last 42.93) - open/close 12156 (time 1709787474.49 total 580.56 last 40.68) - open/close 12584 (time 1709787484.51 total 590.58 last 42.69) - open/close 12980 (time 1709787494.52 total 600.59 last 39.58) - open/close 13387 (time 1709787504.52 total 610.59 last 40.68) - open/close 13470 (time 1709787518.22 total 624.29 last 6.06) - open/close 13814 (time 1709787528.22 total 634.29 last 34.39) - open/close 14224 (time 1709787538.23 total 644.30 last 40.98) - open/close 14609 (time 1709787548.24 total 654.31 last 38.44) - open/close 15006 (time 1709787558.25 total 664.32 last 39.68) - open/close 15464 (time 1709787568.26 total 674.33 last 45.76) - open/close 15857 (time 1709787578.27 total 684.34 last 39.24) - open/close 16298 (time 1709787588.30 total 694.37 last 43.99) - open/close 16710 (time 1709787598.30 total 704.37 last 41.20) - open/close 17112 (time 1709787608.31 total 714.38 last 40.14) - open/close 17516 (time 1709787618.35 total 724.42 last 40.25) - open/close 17889 (time 1709787628.35 total 734.42 last 37.29) - open/close 18251 (time 1709787638.36 total 744.43 last 36.18) - open/close 18641 (time 1709787648.39 total 754.46 last 38.90) - open/close 19053 (time 1709787658.39 total 764.46 last 41.19) - open/close 19447 (time 1709787668.39 total 774.46 last 39.40) - open/close 19796 (time 1709787678.40 total 784.47 last 34.87) - open/close 20000 (time 1709787684.31 total 790.38 last 34.49) - open/close 20378 (time 1709787694.32 total 800.39 last 37.76) - open/close 20747 (time 1709787704.33 total 810.40 last 36.88) - open/close 21053 (time 1709787714.33 total 820.40 last 30.57) - open/close 21366 (time 1709787724.36 total 830.43 last 31.22) - open/close 21720 (time 1709787734.37 total 840.44 last 35.38) - open/close 22106 (time 1709787744.40 total 850.47 last 38.47) - open/close 22200 (time 1709787758.13 total 864.20 last 6.85) - open/close 22476 (time 1709787768.13 total 874.20 last 27.59) - open/close 22767 (time 1709787778.16 total 884.23 last 29.01) - open/close 23034 (time 1709787788.19 total 894.26 last 26.64) - open/close 23299 (time 1709787798.21 total 904.28 last 26.45) - open/close 23638 (time 1709787808.21 total 914.28 last 33.89) - open/close 23950 (time 1709787818.23 total 924.30 last 31.15) - open/close 24229 (time 1709787828.23 total 934.30 last 27.89) - open/close 24553 (time 1709787838.24 total 944.31 last 32.37) - open/close 24885 (time 1709787848.26 total 954.33 last 33.13) - open/close 25235 (time 1709787858.28 total 964.35 last 34.91) - open/close 25611 (time 1709787868.30 total 974.37 last 37.54) - open/close 25965 (time 1709787878.31 total 984.38 last 35.36) - open/close 26347 (time 1709787888.32 total 994.39 last 38.17) - open/close 26708 (time 1709787898.32 total 1004.39 last 36.10) - open/close 27046 (time 1709787908.36 total 1014.43 last 33.65) - open/close 27366 (time 1709787918.37 total 1024.44 last 31.96) - open/close 27696 (time 1709787928.40 total 1034.47 last 32.91) - open/close 28028 (time 1709787938.43 total 1044.50 last 33.10) - open/close 28356 (time 1709787948.44 total 1054.51 last 32.75) - open/close 28684 (time 1709787958.45 total 1064.52 last 32.77) - open/close 28994 (time 1709787968.47 total 1074.54 last 30.95) - open/close 29310 (time 1709787978.48 total 1084.55 last 31.56) - open/close 29638 (time 1709787988.51 total 1094.58 last 32.72) - open/close 29979 (time 1709787998.52 total 1104.59 last 34.07) - open/close 30276 (time 1709788008.52 total 1114.59 last 29.69) - open/close 30532 (time 1709788028.28 total 1134.35 last 12.96) - open/close 30855 (time 1709788038.29 total 1144.36 last 32.26) - open/close 31155 (time 1709788048.32 total 1154.39 last 29.91) - open/close 31463 (time 1709788058.34 total 1164.41 last 30.73) - open/close 31770 (time 1709788068.36 total 1174.43 last 30.66) - open/close 32090 (time 1709788078.36 total 1184.43 last 31.98) - open/close 32394 (time 1709788088.39 total 1194.46 last 30.31) - open/close 32712 (time 1709788098.42 total 1204.49 last 31.70) - open/close 33014 (time 1709788108.45 total 1214.52 last 30.13) - open/close 33317 (time 1709788118.46 total 1224.53 last 30.27) - open/close 33621 (time 1709788128.47 total 1234.54 last 30.36) - open/close 33901 (time 1709788138.50 total 1244.57 last 27.92) - open/close 34191 (time 1709788148.52 total 1254.59 last 28.94) - open/close 34465 (time 1709788158.57 total 1264.64 last 27.27) - open/close 34711 (time 1709788168.60 total 1274.67 last 24.53) - open/close 34984 (time 1709788178.61 total 1284.68 last 27.27) - open/close 35278 (time 1709788188.63 total 1294.70 last 29.34) - open/close 35553 (time 1709788198.64 total 1304.71 last 27.47) - open/close 35832 (time 1709788208.66 total 1314.73 last 27.85) - open/close 36097 (time 1709788218.66 total 1324.73 last 26.49) - open/close 36361 (time 1709788228.66 total 1334.73 last 26.40) - open/close 36619 (time 1709788238.68 total 1344.75 last 25.75) - open/close 36868 (time 1709788248.72 total 1354.79 last 24.81) - open/close 37025 (time 1709788265.54 total 1371.61 last 9.33) - open/close 37281 (time 1709788275.55 total 1381.62 last 25.55) - open/close 37545 (time 1709788285.59 total 1391.66 last 26.31) - open/close 37815 (time 1709788295.61 total 1401.68 last 26.94) - open/close 38061 (time 1709788305.63 total 1411.70 last 24.55) - open/close 38310 (time 1709788315.68 total 1421.75 last 24.77) - open/close 38564 (time 1709788325.70 total 1431.77 last 25.36) - open/close 38819 (time 1709788335.70 total 1441.77 last 25.50) - open/close 39058 (time 1709788345.73 total 1451.80 last 23.83) - open/close 39313 (time 1709788355.75 total 1461.82 last 25.43) - open/close 39558 (time 1709788365.77 total 1471.84 last 24.46) - open/close 39816 (time 1709788375.81 total 1481.88 last 25.71) - open/close 40000 (time 1709788383.27 total 1489.34 last 24.66) - open/close 40248 (time 1709788393.30 total 1499.37 last 24.72) - open/close 40501 (time 1709788403.32 total 1509.39 last 25.25) - open/close 40753 (time 1709788413.35 total 1519.42 last 25.12) - open/close 40993 (time 1709788423.36 total 1529.43 last 24.00) - open/close 41241 (time 1709788433.39 total 1539.46 last 24.72) - open/close 41485 (time 1709788443.41 total 1549.48 last 24.36) - open/close 41730 (time 1709788453.43 total 1559.50 last 24.43) - open/close 41970 (time 1709788463.44 total 1569.51 last 23.97) - open/close 42214 (time 1709788473.47 total 1579.54 last 24.34) - open/close 42448 (time 1709788483.51 total 1589.58 last 23.31) - open/close 42696 (time 1709788493.54 total 1599.61 last 24.72) - open/close 42945 (time 1709788503.57 total 1609.64 last 24.83) - open/close 43204 (time 1709788513.59 total 1619.66 last 25.86) - open/close 43434 (time 1709788523.60 total 1629.67 last 22.96) - open/close 43657 (time 1709788533.61 total 1639.68 last 22.29) - open/close 43877 (time 1709788543.63 total 1649.70 last 21.94) - open/close 44100 (time 1709788553.64 total 1659.71 last 22.29) - open/close 44313 (time 1709788563.69 total 1669.76 last 21.20) - open/close 44534 (time 1709788573.71 total 1679.78 last 22.04) - open/close 44753 (time 1709788583.73 total 1689.80 last 21.87) - open/close 44976 (time 1709788593.74 total 1699.81 last 22.27) - open/close 45207 (time 1709788603.77 total 1709.84 last 23.03) - open/close 45430 (time 1709788613.78 total 1719.85 last 22.29) - open/close 45667 (time 1709788623.78 total 1729.85 last 23.70) - open/close 45914 (time 1709788633.80 total 1739.87 last 24.65) - open/close 46152 (time 1709788643.81 total 1749.88 last 23.78) - open/close 46383 (time 1709788653.84 total 1759.91 last 23.01) - open/close 46611 (time 1709788663.88 total 1769.95 last 22.71) - open/close 46825 (time 1709788673.91 total 1779.98 last 21.34) - open/close 47036 (time 1709788683.94 total 1790.01 last 21.04) - open/close 47258 (time 1709788693.96 total 1800.03 last 22.15) - open/close 47487 (time 1709788704.00 total 1810.07 last 22.82) - open/close 47722 (time 1709788714.00 total 1820.07 last 23.49) - open/close 47945 (time 1709788724.04 total 1830.11 last 22.22) - open/close 48170 (time 1709788734.08 total 1840.15 last 22.41) - open/close 48392 (time 1709788744.11 total 1850.18 last 22.13) - open/close 48614 (time 1709788754.14 total 1860.21 last 22.14) - open/close 48844 (time 1709788764.15 total 1870.22 last 22.96) - open/close 49078 (time 1709788774.16 total 1880.23 last 23.37) - open/close 49291 (time 1709788784.17 total 1890.24 last 21.27) - open/close 49509 (time 1709788794.18 total 1900.25 last 21.79) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1904.56 seconds: 26.04 ops/second - unlinked 0 (time 1709788799 ; total 0 ; last 0) - unlinked 10000 (time 1709789307 ; total 508 ; last 508) - unlinked 20000 (time 1709789725 ; total 926 ; last 418) - unlinked 30000 (time 1709790020 ; total 1221 ; last 295) - unlinked 40000 (time 1709790155 ; total 1356 ; last 135) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1399 seconds: 35.453896 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg430-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg430-server unloading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2 PASS 101a (3360s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg430-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg430-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg430-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 oleg430-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg430-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:43:41 (1709790221) start mds service on oleg430-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing load_modules_local oleg430-server: Loading modules from /home/green/git/lustre-release/lustre oleg430-server: detected 4 online CPUs by sysfs oleg430-server: Force libcfs to create 2 CPU partitions oleg430-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg430-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg430-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg430-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg430-client.virtnet: -o user_xattr,flock oleg430-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server start ost1 service on oleg430-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 8 sec oleg430-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b5bc9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5bc9000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg430-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg430-server unloading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 PASS 101b (94s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg430-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg430-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg430-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg430-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 oleg430-server: oleg430-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing load_modules_local oleg430-server: Loading modules from /home/green/git/lustre-release/lustre oleg430-server: detected 4 online CPUs by sysfs oleg430-server: Force libcfs to create 2 CPU partitions oleg430-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg430-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 4018 sec ========== 00:45:34 (1709790334) === conf-sanity: start cleanup 00:45:34 (1709790334) === === conf-sanity: finish cleanup 00:45:34 (1709790334) ===