-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:39:39 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg459-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg459-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg459-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg459-server oleg459-server: oleg459-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing load_modules_local oleg459-server: Loading modules from /home/green/git/lustre-release/lustre oleg459-server: detected 4 online CPUs by sysfs oleg459-server: Force libcfs to create 2 CPU partitions oleg459-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg459-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg459-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg459-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg459-server: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg459-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg459-server: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg459-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg459-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:41:02 (1709786462) start mds service on oleg459-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg459-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg459-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-OST0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg459-client.virtnet: -o user_xattr,flock oleg459-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h41m18s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800a9bda800.stats=clear mdc.lustre-MDT0001-mdc-ffff8800a9bda800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:42:01.000000000 -0500 Modify: 2024-03-06 23:42:01.000000000 -0500 Change: 2024-03-06 23:42:01.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 63 lines, 63 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg459-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg459-server unloading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing unload_modules_local modules unloaded. oleg459-server: oleg459-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg459-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg459-client: oleg459-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg459-server: oleg459-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg459-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg459-client: oleg459-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg459-server: oleg459-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg459-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg459-client: oleg459-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg459-server: oleg459-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg459-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg459-client: oleg459-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg459-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg459-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 oleg459-server: oleg459-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing load_modules_local oleg459-server: Loading modules from /home/green/git/lustre-release/lustre oleg459-server: detected 4 online CPUs by sysfs oleg459-server: Force libcfs to create 2 CPU partitions oleg459-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg459-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg459-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg459-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg459-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg459-server: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg459-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec oleg459-server: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg459-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg459-server PASS 35b (147s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:43:31 (1709786611) start mds service on oleg459-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg459-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg459-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-OST0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg459-client.virtnet: -o user_xattr,flock oleg459-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg459-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg459-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg459-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg459-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg459-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg459-client.virtnet: -o user_xattr,flock oleg459-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg459-client.virtnet /mnt/lustre (opts:) stop mds service on oleg459-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg459-server 000000 2 000008 oleg459-server: 8+0 records in oleg459-server: 8+0 records out oleg459-server: 4096 bytes (4.1 kB) copied, 0.00084476 s, 4.8 MB/s start mds service on oleg459-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg459-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg459-client.virtnet: -o user_xattr,flock oleg459-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg459-client.virtnet /mnt/lustre (opts:) stop mds service on oleg459-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg459-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server stop mds service on oleg459-server stop mds service on oleg459-server unloading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 PASS 38 (145s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg459-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg459-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg459-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 oleg459-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg459-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:45:59 (1709786759) Stopping clients: oleg459-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg459-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 oleg459-server: oleg459-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing load_modules_local oleg459-server: Loading modules from /home/green/git/lustre-release/lustre oleg459-server: detected 4 online CPUs by sysfs oleg459-server: Force libcfs to create 2 CPU partitions oleg459-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg459-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg459-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg459-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg459-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg459-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg459-client.virtnet: -o user_xattr,flock oleg459-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg459-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg459-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg459-server umount lustre on /mnt/lustre..... Stopping client oleg459-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg459-server unloading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing unload_modules_local modules unloaded. oleg459-server: oleg459-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg459-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg459-client: oleg459-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg459-server: oleg459-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg459-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg459-client: oleg459-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg459-server: oleg459-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg459-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg459-client: oleg459-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg459-server: oleg459-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg459-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg459-client: oleg459-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg459-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg459-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 oleg459-server: oleg459-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing load_modules_local oleg459-server: Loading modules from /home/green/git/lustre-release/lustre oleg459-server: detected 4 online CPUs by sysfs oleg459-server: Force libcfs to create 2 CPU partitions oleg459-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg459-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg459-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg459-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg459-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg459-server: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg459-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg459-server: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg459-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg459-server PASS 51 (237s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:50:00 (1709787000) start mds service on oleg459-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg459-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg459-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-OST0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg459-client.virtnet: -o user_xattr,flock oleg459-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 197 (time 1709787047.48 total 10.43 last 18.89) - open/close 358 (time 1709787057.49 total 20.44 last 16.08) - open/close 684 (time 1709787068.18 total 31.14 last 30.49) - open/close 883 (time 1709787080.46 total 43.41 last 16.21) - open/close 1066 (time 1709787091.14 total 54.10 last 17.13) - open/close 1310 (time 1709787101.78 total 64.74 last 22.93) - open/close 1407 (time 1709787111.84 total 74.80 last 9.64) - open/close 1825 (time 1709787121.85 total 84.80 last 41.78) - open/close 2093 (time 1709787132.18 total 95.13 last 25.94) - open/close 2328 (time 1709787142.19 total 105.15 last 23.47) - open/close 2489 (time 1709787152.37 total 115.33 last 15.81) - open/close 2765 (time 1709787163.17 total 126.12 last 25.57) - open/close 3012 (time 1709787173.73 total 136.68 last 23.39) - open/close 3220 (time 1709787184.34 total 147.30 last 19.60) - open/close 3471 (time 1709787194.73 total 157.69 last 24.15) - open/close 3653 (time 1709787204.75 total 167.70 last 18.18) - open/close 3870 (time 1709787214.75 total 177.71 last 21.69) - open/close 4099 (time 1709787224.76 total 187.72 last 22.87) - open/close 4340 (time 1709787234.77 total 197.72 last 24.09) - open/close 4531 (time 1709787244.77 total 207.73 last 19.09) - open/close 4780 (time 1709787255.71 total 218.66 last 22.77) - open/close 4952 (time 1709787265.71 total 228.67 last 17.20) - open/close 5092 (time 1709787275.97 total 238.92 last 13.65) - open/close 5249 (time 1709787286.32 total 249.27 last 15.17) - open/close 5359 (time 1709787296.33 total 259.29 last 10.98) - open/close 5473 (time 1709787306.62 total 269.57 last 11.08) - open/close 5575 (time 1709787316.82 total 279.77 last 10.00) - open/close 5702 (time 1709787326.97 total 289.93 last 12.51) - open/close 5893 (time 1709787336.98 total 299.93 last 19.09) - open/close 6021 (time 1709787347.23 total 310.18 last 12.49) - open/close 6276 (time 1709787357.23 total 320.19 last 25.49) - open/close 6458 (time 1709787367.44 total 330.40 last 17.82) - open/close 6630 (time 1709787377.93 total 340.88 last 16.41) - open/close 6797 (time 1709787388.24 total 351.20 last 16.18) - open/close 6916 (time 1709787398.40 total 361.35 last 11.72) - open/close 7088 (time 1709787409.29 total 372.25 last 15.78) - open/close 7225 (time 1709787419.30 total 382.25 last 13.69) - open/close 7488 (time 1709787429.31 total 392.26 last 26.28) - open/close 7746 (time 1709787439.80 total 402.76 last 24.58) - open/close 7936 (time 1709787449.82 total 412.77 last 18.97) - open/close 8012 (time 1709787460.61 total 423.57 last 7.04) - open/close 8145 (time 1709787470.62 total 433.57 last 13.29) - open/close 8283 (time 1709787480.63 total 443.59 last 13.78) - open/close 8410 (time 1709787491.43 total 454.38 last 11.76) - open/close 8553 (time 1709787501.44 total 464.39 last 14.29) - open/close 8728 (time 1709787511.46 total 474.41 last 17.46) - open/close 8931 (time 1709787522.45 total 485.41 last 18.46) - open/close 9132 (time 1709787533.02 total 495.98 last 19.02) - open/close 9303 (time 1709787543.89 total 506.85 last 15.73) - open/close 9524 (time 1709787553.92 total 516.88 last 22.03) - open/close 9711 (time 1709787563.93 total 526.88 last 18.69) - open/close 9913 (time 1709787573.94 total 536.89 last 20.18) - open/close 10523 (time 1709787583.95 total 546.90 last 60.94) - open/close 11130 (time 1709787593.95 total 556.90 last 60.68) - open/close 11732 (time 1709787603.97 total 566.93 last 60.07) - open/close 12305 (time 1709787613.97 total 576.93 last 57.29) - open/close 12855 (time 1709787624.02 total 586.97 last 54.74) - open/close 13376 (time 1709787634.03 total 596.99 last 52.02) - open/close 13444 (time 1709787646.52 total 609.47 last 5.45) - open/close 14003 (time 1709787656.53 total 619.48 last 55.84) - open/close 14533 (time 1709787666.54 total 629.50 last 52.93) - open/close 15012 (time 1709787676.55 total 639.50 last 47.87) - open/close 15442 (time 1709787686.56 total 649.51 last 42.96) - open/close 15887 (time 1709787696.60 total 659.55 last 44.33) - open/close 16317 (time 1709787706.60 total 669.56 last 42.98) - open/close 16736 (time 1709787716.62 total 679.57 last 41.84) - open/close 17150 (time 1709787726.63 total 689.59 last 41.33) - open/close 17608 (time 1709787736.65 total 699.61 last 45.70) - open/close 18093 (time 1709787746.66 total 709.62 last 48.46) - open/close 18506 (time 1709787756.68 total 719.64 last 41.22) - open/close 18892 (time 1709787766.69 total 729.65 last 38.56) - open/close 19265 (time 1709787776.69 total 739.65 last 37.30) - open/close 19625 (time 1709787786.71 total 749.66 last 35.94) - open/close 19984 (time 1709787796.72 total 759.67 last 35.86) - open/close 20377 (time 1709787806.72 total 769.68 last 39.28) - open/close 20740 (time 1709787816.75 total 779.71 last 36.20) - open/close 21067 (time 1709787826.77 total 789.73 last 32.64) - open/close 21407 (time 1709787836.78 total 799.74 last 33.96) - open/close 21792 (time 1709787846.79 total 809.74 last 38.48) - open/close 22175 (time 1709787856.79 total 819.75 last 38.29) - open/close 22569 (time 1709787866.79 total 829.75 last 39.39) - open/close 22971 (time 1709787876.80 total 839.76 last 40.17) - open/close 23382 (time 1709787886.81 total 849.77 last 41.06) - open/close 23767 (time 1709787896.83 total 859.78 last 38.44) - open/close 24148 (time 1709787906.84 total 869.80 last 38.04) - open/close 24512 (time 1709787916.85 total 879.80 last 36.38) - open/close 24884 (time 1709787926.87 total 889.82 last 37.12) - open/close 25246 (time 1709787936.87 total 899.83 last 36.19) - open/close 25595 (time 1709787946.89 total 909.84 last 34.84) - open/close 25963 (time 1709787956.91 total 919.86 last 36.73) - open/close 26310 (time 1709787966.91 total 929.87 last 34.67) - open/close 26667 (time 1709787976.92 total 939.88 last 35.66) - open/close 26990 (time 1709787986.94 total 949.90 last 32.24) - open/close 27337 (time 1709787996.95 total 959.91 last 34.67) - open/close 27716 (time 1709788006.96 total 969.91 last 37.88) - open/close 28077 (time 1709788016.98 total 979.93 last 36.02) - open/close 28415 (time 1709788027.00 total 989.96 last 33.72) - open/close 28763 (time 1709788037.02 total 999.97 last 34.75) - open/close 29084 (time 1709788047.05 total 1010.00 last 32.01) - open/close 29394 (time 1709788057.07 total 1020.02 last 30.93) - open/close 29708 (time 1709788067.08 total 1030.03 last 31.38) - open/close 30000 (time 1709788075.94 total 1038.89 last 32.96) - open/close 30309 (time 1709788085.95 total 1048.91 last 30.86) - open/close 30652 (time 1709788095.97 total 1058.93 last 34.22) - open/close 30982 (time 1709788105.98 total 1068.93 last 32.99) - open/close 31312 (time 1709788116.01 total 1078.96 last 32.91) - open/close 31645 (time 1709788126.03 total 1088.98 last 33.24) - open/close 31940 (time 1709788136.05 total 1099.00 last 29.43) - open/close 32237 (time 1709788146.06 total 1109.01 last 29.68) - open/close 32536 (time 1709788156.08 total 1119.03 last 29.84) - open/close 32818 (time 1709788166.10 total 1129.06 last 28.13) - open/close 33106 (time 1709788176.13 total 1139.08 last 28.73) - open/close 33407 (time 1709788186.15 total 1149.11 last 30.03) - open/close 33720 (time 1709788196.18 total 1159.14 last 31.21) - open/close 33997 (time 1709788206.19 total 1169.15 last 27.66) - open/close 34289 (time 1709788216.23 total 1179.19 last 29.09) - open/close 34543 (time 1709788226.24 total 1189.19 last 25.39) - open/close 34818 (time 1709788236.25 total 1199.20 last 27.47) - open/close 35066 (time 1709788246.26 total 1209.22 last 24.77) - open/close 35339 (time 1709788256.27 total 1219.23 last 27.27) - open/close 35642 (time 1709788266.28 total 1229.24 last 30.28) - open/close 35918 (time 1709788276.28 total 1239.24 last 27.60) - open/close 36190 (time 1709788286.30 total 1249.25 last 27.16) - open/close 36469 (time 1709788296.32 total 1259.28 last 27.83) - open/close 36753 (time 1709788306.35 total 1269.30 last 28.33) - open/close 37024 (time 1709788316.37 total 1279.33 last 27.03) - open/close 37292 (time 1709788326.38 total 1289.33 last 26.79) - open/close 37562 (time 1709788336.38 total 1299.33 last 26.99) - open/close 37817 (time 1709788346.40 total 1309.36 last 25.44) - open/close 38079 (time 1709788356.42 total 1319.37 last 26.16) - open/close 38332 (time 1709788366.42 total 1329.38 last 25.29) - open/close 38591 (time 1709788376.45 total 1339.41 last 25.82) - open/close 38848 (time 1709788386.48 total 1349.43 last 25.63) - open/close 39101 (time 1709788396.51 total 1359.47 last 25.21) - open/close 39353 (time 1709788406.54 total 1369.49 last 25.14) - open/close 39608 (time 1709788416.54 total 1379.50 last 25.48) - open/close 39861 (time 1709788426.54 total 1389.50 last 25.30) - open/close 40000 (time 1709788431.96 total 1394.92 last 25.65) - open/close 40255 (time 1709788441.97 total 1404.93 last 25.47) - open/close 40511 (time 1709788452.01 total 1414.96 last 25.51) - open/close 40748 (time 1709788462.03 total 1424.99 last 23.65) - open/close 40989 (time 1709788472.07 total 1435.02 last 24.01) - open/close 41238 (time 1709788482.10 total 1445.06 last 24.82) - open/close 41497 (time 1709788492.12 total 1455.07 last 25.85) - open/close 41757 (time 1709788502.15 total 1465.10 last 25.92) - open/close 42024 (time 1709788512.15 total 1475.10 last 26.70) - open/close 42265 (time 1709788522.19 total 1485.14 last 24.01) - open/close 42507 (time 1709788532.20 total 1495.16 last 24.16) - open/close 42754 (time 1709788542.22 total 1505.17 last 24.66) - open/close 42999 (time 1709788552.22 total 1515.18 last 24.49) - open/close 43244 (time 1709788562.23 total 1525.19 last 24.48) - open/close 43491 (time 1709788572.24 total 1535.20 last 24.66) - open/close 43735 (time 1709788582.25 total 1545.20 last 24.39) - open/close 43985 (time 1709788592.27 total 1555.23 last 24.94) - open/close 44239 (time 1709788602.30 total 1565.26 last 25.33) - open/close 44484 (time 1709788612.34 total 1575.29 last 24.41) - open/close 44730 (time 1709788622.36 total 1585.31 last 24.56) - open/close 44986 (time 1709788632.38 total 1595.33 last 25.55) - open/close 45245 (time 1709788642.38 total 1605.33 last 25.90) - open/close 45503 (time 1709788652.38 total 1615.34 last 25.79) - open/close 45752 (time 1709788662.41 total 1625.37 last 24.83) - open/close 46000 (time 1709788672.44 total 1635.39 last 24.73) - open/close 46255 (time 1709788682.45 total 1645.40 last 25.48) - open/close 46518 (time 1709788692.47 total 1655.42 last 26.25) - open/close 46771 (time 1709788702.49 total 1665.44 last 25.25) - open/close 47017 (time 1709788712.52 total 1675.47 last 24.52) - open/close 47246 (time 1709788722.55 total 1685.51 last 22.82) - open/close 47483 (time 1709788732.59 total 1695.55 last 23.60) - open/close 47718 (time 1709788742.60 total 1705.55 last 23.50) - open/close 47944 (time 1709788752.61 total 1715.56 last 22.57) - open/close 48178 (time 1709788762.64 total 1725.60 last 23.31) - open/close 48416 (time 1709788772.66 total 1735.62 last 23.75) - open/close 48645 (time 1709788782.72 total 1745.67 last 22.78) - open/close 48862 (time 1709788792.73 total 1755.68 last 21.68) - open/close 49079 (time 1709788802.74 total 1765.69 last 21.68) - open/close 49297 (time 1709788812.74 total 1775.70 last 21.79) - open/close 49517 (time 1709788822.77 total 1785.72 last 21.94) - open/close 49738 (time 1709788832.79 total 1795.75 last 22.05) - open/close 49957 (time 1709788842.81 total 1805.77 last 21.86) total: 50000 open/close in 1807.89 seconds: 27.66 ops/second - unlinked 0 (time 1709788846 ; total 0 ; last 0) - unlinked 10000 (time 1709789354 ; total 508 ; last 508) - unlinked 20000 (time 1709789776 ; total 930 ; last 422) - unlinked 30000 (time 1709790074 ; total 1228 ; last 298) - unlinked 40000 (time 1709790208 ; total 1362 ; last 134) total: 50000 unlinks in 1407 seconds: 35.536602 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg459-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg459-server unloading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 PASS 101a (3280s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg459-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg459-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg459-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 oleg459-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg459-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:44:42 (1709790282) start mds service on oleg459-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing load_modules_local oleg459-server: Loading modules from /home/green/git/lustre-release/lustre oleg459-server: detected 4 online CPUs by sysfs oleg459-server: Force libcfs to create 2 CPU partitions oleg459-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg459-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg459-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg459-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-OST0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg459-client.virtnet: -o user_xattr,flock oleg459-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server start ost1 service on oleg459-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-OST0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 8 sec oleg459-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff880137003000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff880137003000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg459-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg459-server unloading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 PASS 101b (89s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg459-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg459-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg459-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg459-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 oleg459-server: oleg459-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing load_modules_local oleg459-server: Loading modules from /home/green/git/lustre-release/lustre oleg459-server: detected 4 online CPUs by sysfs oleg459-server: Force libcfs to create 2 CPU partitions oleg459-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg459-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4012 sec ========== 00:46:32 (1709790392) === conf-sanity: start cleanup 00:46:33 (1709790393) === === conf-sanity: finish cleanup 00:46:33 (1709790393) ===