-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:39:52 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg260-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg260-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg260-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg260-server oleg260-server: oleg260-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: detected 4 online CPUs by sysfs oleg260-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg260-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg260-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg260-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg260-server: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg260-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg260-server: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg260-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg260-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:41:12 (1709786472) start mds service on oleg260-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg260-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg260-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg260-client.virtnet: -o user_xattr,flock oleg260-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h41m30s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800a9ea2800.stats=clear mdc.lustre-MDT0001-mdc-ffff8800a9ea2800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:42:14.000000000 -0500 Modify: 2024-03-06 23:42:14.000000000 -0500 Change: 2024-03-06 23:42:14.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 85 lines, 85 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg260-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg260-server unloading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing unload_modules_local modules unloaded. oleg260-server: oleg260-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg260-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg260-client: oleg260-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg260-server: oleg260-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg260-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg260-client: oleg260-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg260-server: oleg260-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg260-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg260-client: oleg260-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg260-server: oleg260-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg260-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg260-client: oleg260-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg260-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg260-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 oleg260-server: oleg260-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: detected 4 online CPUs by sysfs oleg260-server: Force libcfs to create 2 CPU partitions oleg260-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg260-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg260-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg260-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg260-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg260-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg260-server: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg260-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg260-server: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg260-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg260-server PASS 35b (159s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:43:54 (1709786634) start mds service on oleg260-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg260-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg260-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg260-client.virtnet: -o user_xattr,flock oleg260-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg260-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg260-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg260-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg260-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg260-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg260-client.virtnet: -o user_xattr,flock oleg260-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg260-client.virtnet /mnt/lustre (opts:) stop mds service on oleg260-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg260-server 000000 2 000008 oleg260-server: 8+0 records in oleg260-server: 8+0 records out oleg260-server: 4096 bytes (4.1 kB) copied, 0.00528379 s, 775 kB/s start mds service on oleg260-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg260-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg260-client.virtnet: -o user_xattr,flock oleg260-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg260-client.virtnet /mnt/lustre (opts:) stop mds service on oleg260-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg260-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server stop mds service on oleg260-server stop mds service on oleg260-server unloading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 PASS 38 (169s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg260-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg260-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg260-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 oleg260-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg260-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:46:45 (1709786805) Stopping clients: oleg260-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg260-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 oleg260-server: oleg260-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: detected 4 online CPUs by sysfs oleg260-server: Force libcfs to create 2 CPU partitions oleg260-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg260-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg260-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg260-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg260-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg260-client.virtnet: -o user_xattr,flock oleg260-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg260-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg260-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg260-server umount lustre on /mnt/lustre..... Stopping client oleg260-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg260-server unloading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing unload_modules_local modules unloaded. oleg260-server: oleg260-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg260-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg260-client: oleg260-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg260-server: oleg260-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg260-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg260-client: oleg260-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg260-server: oleg260-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg260-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg260-client: oleg260-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg260-server: oleg260-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg260-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg260-client: oleg260-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg260-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg260-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 oleg260-server: oleg260-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: detected 4 online CPUs by sysfs oleg260-server: Force libcfs to create 2 CPU partitions oleg260-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg260-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg260-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg260-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg260-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg260-server: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg260-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg260-server: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg260-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg260-server PASS 51 (252s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:51:02 (1709787062) start mds service on oleg260-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg260-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg260-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg260-client.virtnet: -o user_xattr,flock oleg260-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 137 (time 1709787108.75 total 10.79 last 12.69) - open/close 335 (time 1709787119.37 total 21.41 last 18.65) - open/close 502 (time 1709787129.65 total 31.68 last 16.25) - open/close 620 (time 1709787140.38 total 42.42 last 10.99) - open/close 805 (time 1709787151.31 total 53.35 last 16.93) - open/close 1034 (time 1709787161.93 total 63.97 last 21.56) - open/close 1175 (time 1709787172.21 total 74.24 last 13.72) - open/close 1361 (time 1709787183.96 total 85.99 last 15.83) - open/close 1480 (time 1709787194.71 total 96.75 last 11.06) - open/close 1727 (time 1709787204.91 total 106.95 last 24.22) - open/close 1843 (time 1709787215.60 total 117.64 last 10.85) - open/close 2069 (time 1709787226.90 total 128.94 last 19.99) - open/close 2258 (time 1709787237.31 total 139.35 last 18.16) - open/close 2429 (time 1709787247.49 total 149.52 last 16.81) - open/close 2663 (time 1709787257.50 total 159.54 last 23.36) - open/close 2833 (time 1709787267.66 total 169.70 last 16.74) - open/close 3013 (time 1709787278.47 total 180.50 last 16.65) - open/close 3153 (time 1709787289.02 total 191.06 last 13.27) - open/close 3273 (time 1709787299.53 total 201.57 last 11.42) - open/close 3365 (time 1709787310.27 total 212.30 last 8.57) - open/close 3491 (time 1709787320.72 total 222.76 last 12.05) - open/close 3647 (time 1709787330.73 total 232.76 last 15.60) - open/close 3765 (time 1709787341.15 total 243.18 last 11.32) - open/close 3888 (time 1709787351.57 total 253.61 last 11.80) - open/close 4073 (time 1709787361.58 total 263.61 last 18.49) - open/close 4231 (time 1709787371.58 total 273.62 last 15.79) - open/close 4349 (time 1709787381.61 total 283.65 last 11.77) - open/close 4465 (time 1709787392.32 total 294.36 last 10.83) - open/close 4645 (time 1709787402.32 total 304.36 last 18.00) - open/close 4903 (time 1709787413.10 total 315.14 last 23.93) - open/close 5031 (time 1709787429.82 total 331.86 last 7.65) - open/close 5188 (time 1709787440.53 total 342.57 last 14.67) - open/close 5377 (time 1709787451.16 total 353.20 last 17.78) - open/close 5555 (time 1709787461.17 total 363.20 last 17.79) - open/close 5721 (time 1709787471.19 total 373.22 last 16.56) - open/close 5857 (time 1709787481.76 total 383.80 last 12.86) - open/close 5994 (time 1709787492.71 total 394.75 last 12.51) - open/close 6155 (time 1709787502.80 total 404.83 last 15.96) - open/close 6423 (time 1709787513.49 total 415.52 last 25.07) - open/close 6633 (time 1709787523.97 total 426.00 last 20.04) - open/close 6790 (time 1709787533.98 total 436.02 last 15.67) - open/close 6916 (time 1709787544.26 total 446.29 last 12.26) - open/close 7039 (time 1709787554.31 total 456.34 last 12.24) - open/close 7241 (time 1709787565.40 total 467.44 last 18.21) - open/close 7381 (time 1709787576.09 total 478.13 last 13.10) - open/close 7489 (time 1709787586.79 total 488.83 last 10.09) - open/close 7631 (time 1709787597.55 total 499.59 last 13.19) - open/close 7774 (time 1709787608.04 total 510.08 last 13.64) - open/close 7988 (time 1709787618.08 total 520.12 last 21.32) - open/close 8111 (time 1709787628.09 total 530.12 last 12.29) - open/close 8211 (time 1709787638.24 total 540.28 last 9.85) - open/close 8354 (time 1709787659.04 total 561.07 last 6.88) - open/close 8493 (time 1709787669.27 total 571.31 last 13.58) - open/close 8719 (time 1709787680.02 total 582.06 last 21.02) - open/close 8884 (time 1709787690.03 total 592.07 last 16.49) - open/close 9189 (time 1709787700.04 total 602.08 last 30.47) - open/close 9636 (time 1709787710.04 total 612.08 last 44.68) - open/close 10000 (time 1709787719.37 total 621.40 last 39.04) - open/close 10425 (time 1709787729.38 total 631.42 last 42.44) - open/close 10849 (time 1709787739.39 total 641.43 last 42.36) - open/close 11235 (time 1709787749.42 total 651.46 last 38.49) - open/close 11625 (time 1709787759.42 total 661.46 last 38.98) - open/close 12049 (time 1709787769.45 total 671.49 last 42.29) - open/close 12407 (time 1709787789.12 total 691.16 last 18.20) - open/close 12827 (time 1709787799.13 total 701.17 last 41.98) - open/close 13251 (time 1709787809.15 total 711.19 last 42.30) - open/close 13627 (time 1709787819.16 total 721.20 last 37.57) - open/close 14026 (time 1709787829.17 total 731.21 last 39.86) - open/close 14435 (time 1709787839.19 total 741.23 last 40.82) - open/close 14761 (time 1709787849.24 total 751.28 last 32.42) - open/close 15055 (time 1709787859.25 total 761.28 last 29.39) - open/close 15362 (time 1709787869.25 total 771.29 last 30.68) - open/close 15691 (time 1709787879.28 total 781.31 last 32.82) - open/close 15976 (time 1709787889.28 total 791.32 last 28.48) - open/close 16261 (time 1709787899.37 total 801.41 last 28.26) - open/close 16557 (time 1709787909.38 total 811.42 last 29.56) - open/close 16872 (time 1709787919.41 total 821.44 last 31.43) - open/close 17097 (time 1709787938.45 total 840.49 last 11.82) - open/close 17436 (time 1709787948.45 total 850.49 last 33.89) - open/close 17831 (time 1709787958.46 total 860.50 last 39.48) - open/close 18222 (time 1709787968.48 total 870.51 last 39.03) - open/close 18581 (time 1709787978.48 total 880.52 last 35.87) - open/close 18944 (time 1709787988.49 total 890.52 last 36.30) - open/close 19311 (time 1709787998.50 total 900.54 last 36.65) - open/close 19738 (time 1709788008.50 total 910.54 last 42.69) - open/close 20000 (time 1709788014.59 total 916.63 last 43.04) - open/close 20460 (time 1709788024.59 total 926.63 last 45.99) - open/close 20914 (time 1709788034.61 total 936.65 last 45.31) - open/close 21378 (time 1709788044.63 total 946.67 last 46.31) - open/close 21838 (time 1709788054.64 total 956.68 last 45.96) - open/close 22279 (time 1709788064.66 total 966.70 last 44.00) - open/close 22699 (time 1709788074.69 total 976.73 last 41.90) - open/close 23140 (time 1709788084.70 total 986.74 last 44.06) - open/close 23556 (time 1709788094.70 total 996.74 last 41.58) - open/close 23979 (time 1709788104.72 total 1006.76 last 42.22) - open/close 24386 (time 1709788114.74 total 1016.78 last 40.63) - open/close 24797 (time 1709788124.75 total 1026.79 last 41.05) - open/close 25206 (time 1709788134.75 total 1036.79 last 40.90) - open/close 25598 (time 1709788144.75 total 1046.79 last 39.20) - open/close 26011 (time 1709788154.77 total 1056.81 last 41.23) - open/close 26421 (time 1709788164.78 total 1066.82 last 40.95) - open/close 26854 (time 1709788174.79 total 1076.83 last 43.26) - open/close 27263 (time 1709788184.79 total 1086.83 last 40.89) - open/close 27663 (time 1709788194.79 total 1096.83 last 39.99) - open/close 28047 (time 1709788204.81 total 1106.84 last 38.34) - open/close 28455 (time 1709788214.81 total 1116.85 last 40.80) - open/close 28857 (time 1709788224.82 total 1126.86 last 40.13) - open/close 29247 (time 1709788234.84 total 1136.88 last 38.92) - open/close 29630 (time 1709788244.87 total 1146.91 last 38.21) - open/close 30000 (time 1709788254.60 total 1156.63 last 38.03) - open/close 30356 (time 1709788264.60 total 1166.64 last 35.58) - open/close 30735 (time 1709788274.62 total 1176.66 last 37.84) - open/close 31074 (time 1709788284.62 total 1186.66 last 33.89) - open/close 31407 (time 1709788294.63 total 1196.67 last 33.26) - open/close 31743 (time 1709788304.66 total 1206.70 last 33.51) - open/close 32074 (time 1709788314.68 total 1216.72 last 33.04) - open/close 32402 (time 1709788324.70 total 1226.74 last 32.73) - open/close 32718 (time 1709788334.71 total 1236.75 last 31.56) - open/close 33007 (time 1709788344.73 total 1246.77 last 28.86) - open/close 33307 (time 1709788354.76 total 1256.80 last 29.90) - open/close 33614 (time 1709788364.79 total 1266.83 last 30.61) - open/close 33903 (time 1709788374.80 total 1276.84 last 28.87) - open/close 34191 (time 1709788384.82 total 1286.85 last 28.75) - open/close 34470 (time 1709788394.84 total 1296.88 last 27.82) - open/close 34794 (time 1709788404.85 total 1306.89 last 32.38) - open/close 35115 (time 1709788414.87 total 1316.91 last 32.03) - open/close 35438 (time 1709788424.89 total 1326.92 last 32.25) - open/close 35772 (time 1709788434.90 total 1336.94 last 33.35) - open/close 36098 (time 1709788444.91 total 1346.94 last 32.58) - open/close 36422 (time 1709788454.91 total 1356.95 last 32.39) - open/close 36723 (time 1709788464.94 total 1366.98 last 30.00) - open/close 37038 (time 1709788474.97 total 1377.00 last 31.42) - open/close 37342 (time 1709788484.98 total 1387.02 last 30.37) - open/close 37664 (time 1709788494.99 total 1397.03 last 32.15) - open/close 37984 (time 1709788505.00 total 1407.04 last 31.98) - open/close 38303 (time 1709788515.02 total 1417.05 last 31.85) - open/close 38622 (time 1709788525.04 total 1427.07 last 31.84) - open/close 38936 (time 1709788535.06 total 1437.10 last 31.31) - open/close 39250 (time 1709788545.08 total 1447.12 last 31.34) - open/close 39570 (time 1709788555.10 total 1457.14 last 31.95) - open/close 39883 (time 1709788565.11 total 1467.14 last 31.28) - open/close 40000 (time 1709788568.94 total 1470.97 last 30.55) - open/close 40304 (time 1709788578.94 total 1480.97 last 30.40) - open/close 40617 (time 1709788588.95 total 1490.99 last 31.26) - open/close 40929 (time 1709788598.97 total 1501.00 last 31.15) - open/close 41232 (time 1709788608.99 total 1511.02 last 30.23) - open/close 41537 (time 1709788619.00 total 1521.03 last 30.48) - open/close 41841 (time 1709788629.01 total 1531.05 last 30.34) - open/close 42129 (time 1709788639.03 total 1541.07 last 28.75) - open/close 42405 (time 1709788649.04 total 1551.07 last 27.59) - open/close 42690 (time 1709788659.07 total 1561.10 last 28.42) - open/close 42987 (time 1709788669.09 total 1571.12 last 29.64) - open/close 43266 (time 1709788679.11 total 1581.14 last 27.85) - open/close 43551 (time 1709788689.12 total 1591.15 last 28.47) - open/close 43842 (time 1709788699.13 total 1601.17 last 29.07) - open/close 44123 (time 1709788709.16 total 1611.20 last 28.01) - open/close 44394 (time 1709788719.20 total 1621.23 last 27.00) - open/close 44653 (time 1709788729.21 total 1631.25 last 25.85) - open/close 44908 (time 1709788739.22 total 1641.26 last 25.48) - open/close 45165 (time 1709788749.26 total 1651.29 last 25.61) - open/close 45415 (time 1709788759.27 total 1661.30 last 24.97) - open/close 45671 (time 1709788769.28 total 1671.32 last 25.57) - open/close 45919 (time 1709788779.29 total 1681.33 last 24.77) - open/close 46186 (time 1709788789.30 total 1691.34 last 26.68) - open/close 46459 (time 1709788799.30 total 1701.34 last 27.29) - open/close 46737 (time 1709788809.33 total 1711.36 last 27.74) - open/close 47012 (time 1709788819.34 total 1721.38 last 27.46) - open/close 47274 (time 1709788829.37 total 1731.41 last 26.12) - open/close 47540 (time 1709788839.38 total 1741.42 last 26.58) - open/close 47806 (time 1709788849.39 total 1751.43 last 26.57) - open/close 48066 (time 1709788859.41 total 1761.44 last 25.96) - open/close 48323 (time 1709788869.41 total 1771.45 last 25.68) - open/close 48574 (time 1709788879.43 total 1781.47 last 25.05) - open/close 48824 (time 1709788889.44 total 1791.48 last 24.98) - open/close 49074 (time 1709788899.45 total 1801.49 last 24.97) - open/close 49321 (time 1709788909.47 total 1811.51 last 24.65) - open/close 49561 (time 1709788919.50 total 1821.54 last 23.93) - open/close 49808 (time 1709788929.53 total 1831.56 last 24.64) total: 50000 open/close in 1839.87 seconds: 27.18 ops/second - unlinked 0 (time 1709788938 ; total 0 ; last 0) - unlinked 10000 (time 1709789404 ; total 466 ; last 466) - unlinked 20000 (time 1709789783 ; total 845 ; last 379) - unlinked 30000 (time 1709790036 ; total 1098 ; last 253) - unlinked 40000 (time 1709790149 ; total 1211 ; last 113) total: 50000 unlinks in 1244 seconds: 40.192924 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg260-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg260-server unloading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 PASS 101a (3140s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg260-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg260-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg260-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 oleg260-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg260-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:43:24 (1709790204) start mds service on oleg260-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: detected 4 online CPUs by sysfs oleg260-server: Force libcfs to create 2 CPU partitions oleg260-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg260-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg260-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg260-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg260-client.virtnet: -o user_xattr,flock oleg260-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server start ost1 service on oleg260-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 8 sec oleg260-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b6567800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6567800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg260-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg260-server unloading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 PASS 101b (87s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg260-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg260-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg260-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg260-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 oleg260-server: oleg260-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: detected 4 online CPUs by sysfs oleg260-server: Force libcfs to create 2 CPU partitions oleg260-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg260-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3921 sec ========== 00:45:13 (1709790313) === conf-sanity: start cleanup 00:45:13 (1709790313) === === conf-sanity: finish cleanup 00:45:13 (1709790313) ===