-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:13:05 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg222-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg222-server oleg222-server: oleg222-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: detected 4 online CPUs by sysfs oleg222-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg222-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg222-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg222-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg222-server: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg222-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg222-server: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg222-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg222-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:14:30 (1709784870) start mds service on oleg222-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg222-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg222-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock oleg222-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h14m48s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff880130e67800.stats=clear mdc.lustre-MDT0001-mdc-ffff880130e67800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:15:31.000000000 -0500 Modify: 2024-03-06 23:15:31.000000000 -0500 Change: 2024-03-06 23:15:31.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 66 lines, 66 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg222-server unloading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local modules unloaded. oleg222-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg222-server: Is the ldiskfs module available? oleg222-server: oleg222-server: tunefs.lustre FATAL: failed to write local files oleg222-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg222-client: oleg222-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg222-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg222-server: Is the ldiskfs module available? oleg222-server: oleg222-server: tunefs.lustre FATAL: failed to write local files oleg222-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg222-client: oleg222-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg222-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg222-server: Is the ldiskfs module available? oleg222-server: oleg222-server: tunefs.lustre FATAL: failed to write local files oleg222-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg222-client: oleg222-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.122@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.122@tcp sys.timeout=20 oleg222-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg222-server: Is the ldiskfs module available? oleg222-server: oleg222-server: tunefs.lustre FATAL: failed to write local files oleg222-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg222-client: oleg222-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.122@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.122@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 oleg222-server: oleg222-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: detected 4 online CPUs by sysfs oleg222-server: Force libcfs to create 2 CPU partitions oleg222-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg222-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg222-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg222-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg222-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg222-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg222-server: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg222-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec oleg222-server: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg222-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg222-server PASS 35b (138s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:16:49 (1709785009) start mds service on oleg222-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg222-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg222-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock oleg222-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg222-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg222-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg222-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock oleg222-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg222-server 000000 2 000008 oleg222-server: 8+0 records in oleg222-server: 8+0 records out oleg222-server: 4096 bytes (4.1 kB) copied, 0.000275016 s, 14.9 MB/s start mds service on oleg222-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg222-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock oleg222-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg222-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server stop mds service on oleg222-server unloading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 PASS 38 (105s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg222-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg222-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg222-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 oleg222-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg222-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:18:35 (1709785115) Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 oleg222-server: oleg222-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: detected 4 online CPUs by sysfs oleg222-server: Force libcfs to create 2 CPU partitions oleg222-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg222-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg222-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg222-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg222-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg222-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock oleg222-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg222-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 6 sec stop ost2 service on oleg222-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg222-server umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg222-server unloading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local modules unloaded. oleg222-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg222-server: Is the ldiskfs module available? oleg222-server: oleg222-server: tunefs.lustre FATAL: failed to write local files oleg222-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg222-client: oleg222-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg222-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg222-server: Is the ldiskfs module available? oleg222-server: oleg222-server: tunefs.lustre FATAL: failed to write local files oleg222-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg222-client: oleg222-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg222-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg222-server: Is the ldiskfs module available? oleg222-server: oleg222-server: tunefs.lustre FATAL: failed to write local files oleg222-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg222-client: oleg222-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.122@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.122@tcp sys.timeout=20 oleg222-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg222-server: Is the ldiskfs module available? oleg222-server: oleg222-server: tunefs.lustre FATAL: failed to write local files oleg222-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg222-client: oleg222-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.122@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.122@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 oleg222-server: oleg222-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: detected 4 online CPUs by sysfs oleg222-server: Force libcfs to create 2 CPU partitions oleg222-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg222-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg222-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg222-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg222-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg222-server: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg222-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg222-server: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg222-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg222-server PASS 51 (160s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:21:17 (1709785277) start mds service on oleg222-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg222-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg222-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock oleg222-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 706 (time 1709785307.34 total 10.59 last 66.65) - open/close 1230 (time 1709785317.35 total 20.59 last 52.39) - open/close 1728 (time 1709785327.35 total 30.60 last 49.79) - open/close 2221 (time 1709785337.43 total 40.68 last 48.88) - open/close 2933 (time 1709785348.19 total 51.44 last 66.20) - open/close 3611 (time 1709785358.93 total 62.18 last 63.09) - open/close 4132 (time 1709785369.40 total 72.64 last 49.79) - open/close 4393 (time 1709785380.22 total 83.47 last 24.10) - open/close 4825 (time 1709785390.73 total 93.97 last 41.14) - open/close 5272 (time 1709785401.29 total 104.54 last 42.30) - open/close 5848 (time 1709785411.30 total 114.55 last 57.58) - open/close 6313 (time 1709785421.84 total 125.09 last 44.10) - open/close 6856 (time 1709785432.79 total 136.04 last 49.61) - open/close 7334 (time 1709785443.50 total 146.75 last 44.61) - open/close 7834 (time 1709785454.39 total 157.63 last 45.95) - open/close 8311 (time 1709785465.16 total 168.41 last 44.28) - open/close 8807 (time 1709785475.16 total 178.41 last 49.58) - open/close 9036 (time 1709785488.33 total 191.58 last 17.39) - open/close 9520 (time 1709785498.81 total 202.06 last 46.18) - open/close 9724 (time 1709785509.00 total 212.25 last 20.02) - open/close 10000 (time 1709785517.32 total 220.57 last 33.18) - open/close 10343 (time 1709785528.16 total 231.41 last 31.65) - open/close 10702 (time 1709785539.03 total 242.28 last 33.00) - open/close 12470 (time 1709785549.04 total 252.29 last 176.77) - open/close 14015 (time 1709785559.04 total 262.29 last 154.41) - open/close 15460 (time 1709785569.05 total 272.30 last 144.41) - open/close 16782 (time 1709785579.05 total 282.30 last 132.19) - open/close 17928 (time 1709785589.06 total 292.30 last 114.52) - open/close 18998 (time 1709785599.06 total 302.31 last 106.92) - open/close 20000 (time 1709785608.29 total 311.54 last 108.62) - open/close 20905 (time 1709785618.29 total 321.54 last 90.45) - open/close 21807 (time 1709785628.30 total 331.55 last 90.16) - open/close 22705 (time 1709785638.30 total 341.55 last 89.73) - open/close 23555 (time 1709785648.31 total 351.56 last 84.98) - open/close 24445 (time 1709785658.31 total 361.56 last 88.94) - open/close 25227 (time 1709785668.32 total 371.56 last 78.19) - open/close 25983 (time 1709785678.32 total 381.57 last 75.58) - open/close 26685 (time 1709785688.33 total 391.58 last 70.13) - open/close 27382 (time 1709785698.34 total 401.58 last 69.64) - open/close 28082 (time 1709785708.35 total 411.60 last 69.90) - open/close 28681 (time 1709785718.36 total 421.61 last 59.82) - open/close 29263 (time 1709785728.38 total 431.63 last 58.10) - open/close 29828 (time 1709785738.40 total 441.65 last 56.42) - open/close 30000 (time 1709785741.63 total 444.88 last 53.18) - open/close 30634 (time 1709785751.64 total 454.89 last 63.34) - open/close 31263 (time 1709785761.65 total 464.90 last 62.84) - open/close 31934 (time 1709785771.65 total 474.90 last 67.10) - open/close 32538 (time 1709785781.67 total 484.92 last 60.28) - open/close 33176 (time 1709785791.68 total 494.93 last 63.73) - open/close 33821 (time 1709785801.69 total 504.94 last 64.45) - open/close 34406 (time 1709785811.69 total 514.94 last 58.48) - open/close 34926 (time 1709785821.70 total 524.94 last 51.98) - open/close 35398 (time 1709785831.70 total 534.95 last 47.19) - open/close 35891 (time 1709785841.70 total 544.95 last 49.30) - open/close 36366 (time 1709785851.70 total 554.95 last 47.48) - open/close 36848 (time 1709785861.71 total 564.95 last 48.19) - open/close 37318 (time 1709785871.71 total 574.96 last 46.98) - open/close 37814 (time 1709785881.73 total 584.97 last 49.53) - open/close 38296 (time 1709785891.74 total 594.99 last 48.12) - open/close 38771 (time 1709785901.76 total 605.00 last 47.43) - open/close 39212 (time 1709785911.76 total 615.01 last 44.07) - open/close 39646 (time 1709785921.78 total 625.03 last 43.32) - open/close 40000 (time 1709785930.14 total 633.39 last 42.35) - open/close 40415 (time 1709785940.16 total 643.41 last 41.43) - open/close 40828 (time 1709785950.17 total 653.42 last 41.23) - open/close 41256 (time 1709785960.18 total 663.43 last 42.78) - open/close 41676 (time 1709785970.19 total 673.43 last 41.96) - open/close 42087 (time 1709785980.19 total 683.44 last 41.08) - open/close 42557 (time 1709785990.21 total 693.46 last 46.92) - open/close 43015 (time 1709786000.22 total 703.46 last 45.77) - open/close 43450 (time 1709786010.22 total 713.47 last 43.48) - open/close 43874 (time 1709786020.22 total 723.47 last 42.38) - open/close 44254 (time 1709786030.24 total 733.49 last 37.95) - open/close 44626 (time 1709786040.25 total 743.50 last 37.16) - open/close 45042 (time 1709786050.26 total 753.51 last 41.55) - open/close 45474 (time 1709786060.28 total 763.53 last 43.11) - open/close 45880 (time 1709786070.29 total 773.53 last 40.59) - open/close 46287 (time 1709786080.31 total 783.55 last 40.62) - open/close 46717 (time 1709786090.31 total 793.56 last 42.99) - open/close 47107 (time 1709786100.32 total 803.57 last 38.94) - open/close 47485 (time 1709786110.33 total 813.57 last 37.79) - open/close 47840 (time 1709786120.34 total 823.58 last 35.47) - open/close 48185 (time 1709786130.35 total 833.60 last 34.44) - open/close 48575 (time 1709786140.36 total 843.61 last 38.96) - open/close 48960 (time 1709786150.38 total 853.63 last 38.44) - open/close 49341 (time 1709786160.39 total 863.64 last 38.06) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 870.20 seconds: 57.00 ops/second - unlinked 0 (time 1709786167 ; total 0 ; last 0) - unlinked 10000 (time 1709786474 ; total 307 ; last 307) - unlinked 20000 (time 1709786738 ; total 571 ; last 264) - unlinked 30000 (time 1709786953 ; total 786 ; last 215) - unlinked 40000 (time 1709787096 ; total 929 ; last 143) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1014 seconds: 48.915188 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg222-server unloading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 PASS 101a (1932s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg222-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg222-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg222-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 oleg222-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg222-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 23:53:33 (1709787213) start mds service on oleg222-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: detected 4 online CPUs by sysfs oleg222-server: Force libcfs to create 2 CPU partitions oleg222-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg222-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg222-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg222-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock oleg222-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server start ost1 service on oleg222-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec oleg222-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800ace54000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800ace54000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg222-server unloading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 PASS 101b (137s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg222-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg222-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 oleg222-server: oleg222-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: detected 4 online CPUs by sysfs oleg222-server: Force libcfs to create 2 CPU partitions oleg222-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg222-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 2609 sec ========== 23:56:34 (1709787394) === conf-sanity: start cleanup 23:56:35 (1709787395) === === conf-sanity: finish cleanup 23:56:35 (1709787395) ===