-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:45:23 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg357-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg357-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg357-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg357-server oleg357-server: oleg357-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg357-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg357-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg357-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg357-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg357-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:47:02 (1709786822) start mds service on oleg357-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg357-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg357-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock oleg357-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h47m24s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff880136ab2800.stats=clear mdc.lustre-MDT0001-mdc-ffff880136ab2800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:48:08.000000000 -0500 Modify: 2024-03-06 23:48:08.000000000 -0500 Change: 2024-03-06 23:48:08.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 86 lines, 86 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server unloading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing unload_modules_local modules unloaded. oleg357-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg357-server: Is the ldiskfs module available? oleg357-server: oleg357-server: tunefs.lustre FATAL: failed to write local files oleg357-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg357-client: oleg357-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg357-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg357-server: Is the ldiskfs module available? oleg357-server: oleg357-server: tunefs.lustre FATAL: failed to write local files oleg357-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg357-client: oleg357-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.157@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.157@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg357-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg357-server: Is the ldiskfs module available? oleg357-server: oleg357-server: tunefs.lustre FATAL: failed to write local files oleg357-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg357-client: oleg357-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.157@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.157@tcp sys.timeout=20 oleg357-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg357-server: Is the ldiskfs module available? oleg357-server: oleg357-server: tunefs.lustre FATAL: failed to write local files oleg357-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg357-client: oleg357-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.157@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.157@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg357-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg357-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 oleg357-server: oleg357-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg357-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg357-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg357-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg357-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg357-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg357-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server PASS 35b (202s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:50:27 (1709787027) start mds service on oleg357-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg357-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg357-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock oleg357-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg357-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg357-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock oleg357-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:) stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server 000000 2 000008 oleg357-server: 8+0 records in oleg357-server: 8+0 records out oleg357-server: 4096 bytes (4.1 kB) copied, 0.000392134 s, 10.4 MB/s start mds service on oleg357-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg357-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock oleg357-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:) stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server stop mds service on oleg357-server unloading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 PASS 38 (162s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg357-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg357-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg357-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 oleg357-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg357-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:53:13 (1709787193) Stopping clients: oleg357-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg357-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 oleg357-server: oleg357-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg357-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg357-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg357-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg357-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock oleg357-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg357-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg357-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg357-server umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server unloading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing unload_modules_local modules unloaded. oleg357-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg357-server: Is the ldiskfs module available? oleg357-server: oleg357-server: tunefs.lustre FATAL: failed to write local files oleg357-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg357-client: oleg357-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg357-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg357-server: Is the ldiskfs module available? oleg357-server: oleg357-server: tunefs.lustre FATAL: failed to write local files oleg357-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg357-client: oleg357-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.157@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.157@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg357-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg357-server: Is the ldiskfs module available? oleg357-server: oleg357-server: tunefs.lustre FATAL: failed to write local files oleg357-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg357-client: oleg357-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.157@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.157@tcp sys.timeout=20 oleg357-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg357-server: Is the ldiskfs module available? oleg357-server: oleg357-server: tunefs.lustre FATAL: failed to write local files oleg357-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg357-client: oleg357-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.157@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.157@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg357-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg357-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 oleg357-server: oleg357-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg357-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg357-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg357-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg357-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg357-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg357-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server PASS 51 (272s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:57:50 (1709787470) start mds service on oleg357-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg357-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg357-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock oleg357-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 168 (time 1709787521.82 total 10.02 last 16.76) - open/close 383 (time 1709787532.44 total 20.65 last 20.24) - open/close 594 (time 1709787543.41 total 31.62 last 19.23) - open/close 775 (time 1709787553.53 total 41.73 last 17.89) - open/close 987 (time 1709787564.07 total 52.27 last 20.11) - open/close 1180 (time 1709787574.57 total 62.78 last 18.38) - open/close 1460 (time 1709787584.57 total 72.78 last 27.99) - open/close 1729 (time 1709787594.58 total 82.79 last 26.88) - open/close 1944 (time 1709787604.60 total 92.81 last 21.45) - open/close 2122 (time 1709787614.61 total 102.81 last 17.79) - open/close 2366 (time 1709787624.62 total 112.83 last 24.37) - open/close 2615 (time 1709787634.63 total 122.83 last 24.89) - open/close 2793 (time 1709787645.33 total 133.53 last 16.64) - open/close 3015 (time 1709787655.34 total 143.55 last 22.16) - open/close 3278 (time 1709787665.38 total 153.58 last 26.21) - open/close 3433 (time 1709787676.27 total 164.48 last 14.22) - open/close 3619 (time 1709787686.67 total 174.87 last 17.89) - open/close 3835 (time 1709787697.04 total 185.24 last 20.83) - open/close 4055 (time 1709787707.37 total 195.57 last 21.30) - open/close 4377 (time 1709787717.86 total 206.06 last 30.69) - open/close 4464 (time 1709787728.37 total 216.57 last 8.28) - open/close 4609 (time 1709787739.30 total 227.50 last 13.27) - open/close 4830 (time 1709787750.27 total 238.47 last 20.15) - open/close 5023 (time 1709787760.50 total 248.70 last 18.86) - open/close 5142 (time 1709787770.50 total 258.70 last 11.90) - open/close 5372 (time 1709787781.15 total 269.35 last 21.59) - open/close 5585 (time 1709787791.16 total 279.37 last 21.27) - open/close 5754 (time 1709787802.12 total 290.32 last 15.42) - open/close 5984 (time 1709787812.91 total 301.11 last 21.32) - open/close 6298 (time 1709787823.50 total 311.70 last 29.65) - open/close 6511 (time 1709787834.26 total 322.46 last 19.80) - open/close 6721 (time 1709787844.35 total 332.55 last 20.81) - open/close 7007 (time 1709787854.85 total 343.05 last 27.24) - open/close 7231 (time 1709787865.69 total 353.89 last 20.66) - open/close 7345 (time 1709787875.70 total 363.90 last 11.39) - open/close 7567 (time 1709787885.71 total 373.92 last 22.17) - open/close 7770 (time 1709787896.51 total 384.71 last 18.81) - open/close 7986 (time 1709787907.01 total 395.21 last 20.58) - open/close 8123 (time 1709787917.35 total 405.56 last 13.24) - open/close 8311 (time 1709787927.79 total 416.00 last 18.01) - open/close 8525 (time 1709787937.80 total 426.00 last 21.39) - open/close 8804 (time 1709787947.81 total 436.01 last 27.87) - open/close 8902 (time 1709787957.82 total 446.02 last 9.79) - open/close 9149 (time 1709787968.03 total 456.24 last 24.18) - open/close 9387 (time 1709787978.42 total 466.62 last 22.91) - open/close 9589 (time 1709787988.67 total 476.88 last 19.70) - open/close 9820 (time 1709788001.50 total 489.71 last 18.00) - open/close 10000 (time 1709788006.69 total 494.89 last 34.70) - open/close 10250 (time 1709788016.88 total 505.09 last 24.53) - open/close 10471 (time 1709788027.06 total 515.27 last 21.71) - open/close 10689 (time 1709788037.20 total 525.40 last 21.51) - open/close 10931 (time 1709788047.66 total 535.86 last 23.14) - open/close 11053 (time 1709788058.28 total 546.48 last 11.49) - open/close 11281 (time 1709788068.52 total 556.72 last 22.26) - open/close 11692 (time 1709788078.54 total 566.74 last 41.01) - open/close 12388 (time 1709788088.56 total 576.76 last 69.47) - open/close 13055 (time 1709788098.57 total 586.77 last 66.64) - open/close 13697 (time 1709788108.57 total 596.78 last 64.16) - open/close 14315 (time 1709788118.58 total 606.79 last 61.72) - open/close 14946 (time 1709788128.59 total 616.79 last 63.09) - open/close 15538 (time 1709788138.59 total 626.80 last 59.16) - open/close 16103 (time 1709788148.60 total 636.81 last 56.45) - open/close 16663 (time 1709788158.61 total 646.82 last 55.93) - open/close 17232 (time 1709788168.62 total 656.83 last 56.85) - open/close 17764 (time 1709788178.62 total 666.83 last 53.20) - open/close 18303 (time 1709788188.68 total 676.88 last 53.62) - open/close 18797 (time 1709788198.68 total 686.88 last 49.39) - open/close 19288 (time 1709788208.69 total 696.90 last 49.03) - open/close 19779 (time 1709788218.69 total 706.90 last 49.09) - open/close 20000 (time 1709788222.97 total 711.17 last 51.71) - open/close 20490 (time 1709788232.97 total 721.18 last 48.97) - open/close 20966 (time 1709788242.99 total 731.20 last 47.52) - open/close 21420 (time 1709788253.00 total 741.21 last 45.35) - open/close 21868 (time 1709788263.01 total 751.22 last 44.74) - open/close 22309 (time 1709788273.03 total 761.24 last 44.02) - open/close 22752 (time 1709788283.05 total 771.25 last 44.24) - open/close 23155 (time 1709788293.06 total 781.27 last 40.23) - open/close 23566 (time 1709788303.07 total 791.28 last 41.06) - open/close 23971 (time 1709788313.09 total 801.29 last 40.44) - open/close 24373 (time 1709788323.11 total 811.31 last 40.11) - open/close 24800 (time 1709788333.15 total 821.35 last 42.54) - open/close 25216 (time 1709788343.16 total 831.36 last 41.54) - open/close 25629 (time 1709788353.16 total 841.37 last 41.28) - open/close 26047 (time 1709788363.18 total 851.38 last 41.75) - open/close 26443 (time 1709788373.19 total 861.40 last 39.52) - open/close 26850 (time 1709788383.20 total 871.40 last 40.68) - open/close 27239 (time 1709788393.22 total 881.42 last 38.84) - open/close 27638 (time 1709788403.24 total 891.44 last 39.81) - open/close 28007 (time 1709788413.25 total 901.45 last 36.85) - open/close 28411 (time 1709788423.26 total 911.46 last 40.36) - open/close 28796 (time 1709788433.28 total 921.49 last 38.40) - open/close 29199 (time 1709788443.31 total 931.51 last 40.21) - open/close 29571 (time 1709788453.31 total 941.52 last 37.18) - open/close 29940 (time 1709788463.34 total 951.54 last 36.81) - open/close 30314 (time 1709788473.35 total 961.56 last 37.35) - open/close 30687 (time 1709788483.36 total 971.56 last 37.27) - open/close 31024 (time 1709788493.37 total 981.57 last 33.66) - open/close 31361 (time 1709788503.38 total 991.58 last 33.68) - open/close 31719 (time 1709788513.39 total 1001.60 last 35.73) - open/close 32069 (time 1709788523.41 total 1011.61 last 34.95) - open/close 32412 (time 1709788533.43 total 1021.64 last 34.22) - open/close 32771 (time 1709788543.45 total 1031.66 last 35.82) - open/close 33115 (time 1709788553.47 total 1041.67 last 34.34) - open/close 33449 (time 1709788563.49 total 1051.69 last 33.34) - open/close 33806 (time 1709788573.51 total 1061.71 last 35.63) - open/close 34141 (time 1709788583.51 total 1071.71 last 33.50) - open/close 34478 (time 1709788593.51 total 1081.72 last 33.68) - open/close 34814 (time 1709788603.54 total 1091.74 last 33.52) - open/close 35155 (time 1709788613.56 total 1101.77 last 34.01) - open/close 35488 (time 1709788623.58 total 1111.79 last 33.25) - open/close 35809 (time 1709788633.61 total 1121.81 last 32.02) - open/close 36089 (time 1709788643.63 total 1131.83 last 27.94) - open/close 36327 (time 1709788653.67 total 1141.87 last 23.71) - open/close 36563 (time 1709788663.69 total 1151.90 last 23.54) - open/close 36835 (time 1709788673.72 total 1161.93 last 27.12) - open/close 37130 (time 1709788683.74 total 1171.95 last 29.44) - open/close 37426 (time 1709788693.76 total 1181.97 last 29.54) - open/close 37725 (time 1709788703.78 total 1191.98 last 29.86) - open/close 38018 (time 1709788713.79 total 1201.99 last 29.27) - open/close 38314 (time 1709788723.81 total 1212.01 last 29.54) - open/close 38627 (time 1709788733.83 total 1222.03 last 31.24) - open/close 38940 (time 1709788743.83 total 1232.04 last 31.29) - open/close 39249 (time 1709788753.86 total 1242.06 last 30.82) - open/close 39546 (time 1709788763.91 total 1252.11 last 29.55) - open/close 39834 (time 1709788773.91 total 1262.11 last 28.79) - open/close 40000 (time 1709788779.88 total 1268.09 last 27.80) - open/close 40285 (time 1709788789.90 total 1278.10 last 28.45) - open/close 40553 (time 1709788799.93 total 1288.13 last 26.72) - open/close 40830 (time 1709788809.93 total 1298.14 last 27.70) - open/close 41102 (time 1709788819.93 total 1308.14 last 27.20) - open/close 41352 (time 1709788829.95 total 1318.16 last 24.95) - open/close 41637 (time 1709788839.96 total 1328.16 last 28.49) - open/close 41909 (time 1709788849.97 total 1338.17 last 27.17) - open/close 42174 (time 1709788859.98 total 1348.19 last 26.45) - open/close 42435 (time 1709788870.01 total 1358.21 last 26.04) - open/close 42698 (time 1709788880.01 total 1368.21 last 26.30) - open/close 42976 (time 1709788890.04 total 1378.24 last 27.72) - open/close 43244 (time 1709788900.05 total 1388.25 last 26.77) - open/close 43512 (time 1709788910.08 total 1398.28 last 26.72) - open/close 43786 (time 1709788920.09 total 1408.30 last 27.36) - open/close 44049 (time 1709788930.10 total 1418.30 last 26.29) - open/close 44309 (time 1709788940.11 total 1428.32 last 25.97) - open/close 44571 (time 1709788950.14 total 1438.34 last 26.13) - open/close 44821 (time 1709788960.14 total 1448.35 last 24.99) - open/close 45081 (time 1709788970.16 total 1458.36 last 25.96) - open/close 45352 (time 1709788980.19 total 1468.40 last 27.02) - open/close 45621 (time 1709788990.19 total 1478.40 last 26.89) - open/close 45876 (time 1709789000.20 total 1488.41 last 25.47) - open/close 46135 (time 1709789010.21 total 1498.41 last 25.89) - open/close 46387 (time 1709789020.21 total 1508.42 last 25.18) - open/close 46640 (time 1709789030.22 total 1518.42 last 25.29) - open/close 46888 (time 1709789040.25 total 1528.45 last 24.72) - open/close 47142 (time 1709789050.27 total 1538.47 last 25.36) - open/close 47407 (time 1709789060.29 total 1548.50 last 26.43) - open/close 47658 (time 1709789070.31 total 1558.51 last 25.07) - open/close 47894 (time 1709789080.31 total 1568.51 last 23.59) - open/close 48128 (time 1709789090.32 total 1578.52 last 23.38) - open/close 48370 (time 1709789100.34 total 1588.54 last 24.15) - open/close 48619 (time 1709789110.37 total 1598.57 last 24.83) - open/close 48860 (time 1709789120.38 total 1608.58 last 24.07) - open/close 49096 (time 1709789130.40 total 1618.60 last 23.56) - open/close 49335 (time 1709789140.40 total 1628.61 last 23.89) - open/close 49578 (time 1709789150.44 total 1638.65 last 24.21) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1639.53 seconds: 30.25 ops/second - unlinked 0 (time 1709789152 ; total 0 ; last 0) - unlinked 10000 (time 1709789599 ; total 447 ; last 447) - unlinked 20000 (time 1709789940 ; total 788 ; last 341) - unlinked 30000 (time 1709790150 ; total 998 ; last 210) - unlinked 40000 (time 1709790263 ; total 1111 ; last 113) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1153 seconds: 43.018215 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server unloading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 PASS 101a (2856s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg357-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg357-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg357-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 oleg357-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg357-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:45:28 (1709790328) start mds service on oleg357-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg357-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg357-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg357-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock oleg357-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server start ost1 service on oleg357-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec oleg357-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800a860c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a860c000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server unloading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 PASS 101b (86s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg357-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg357-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg357-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg357-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 oleg357-server: oleg357-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg357-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3710 sec ========== 00:47:13 (1709790433) === conf-sanity: start cleanup 00:47:13 (1709790433) === === conf-sanity: finish cleanup 00:47:13 (1709790433) ===