-----============= acceptance-small: conf-sanity-1 ============----- Sun Dec 15 04:31:26 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg317-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity-1.*ex: No such file or directory excepting tests: skipping tests SLOW=no: 106 Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg317-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg317-server oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server == conf-sanity-1 test 17: Verify failed mds_postsetup won't fail assertion (2936) (should return errs) ========================================================== 04:33:06 (1734255186) SKIP: conf-sanity-1 test_17 ldiskfs only test SKIP 17 (2s) == conf-sanity-1 test 18: check mkfs creates large journals ========================================================== 04:33:08 (1734255188) SKIP: conf-sanity-1 test_18 ldiskfs only test SKIP 18 (1s) == conf-sanity-1 test 21c: start mds between two osts, stop mds last ========================================================== 04:33:09 (1734255189) start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost2 service on oleg317-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop ost2 service on oleg317-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg317-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server PASS 21c (163s) == conf-sanity-1 test 29: permanently remove an OST ====== 04:35:52 (1734255352) start ost2 service on oleg317-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Setting lustre-OST0001.osc.active from 1 to 0 Waiting 90s for '0' Updated after 2s: want '0' got '0' Live client success: got lustre-OST0001_UUID FULL DEACTIVATED osc.lustre-OST0001-osc-MDT0000.active updated after 0 sec (got 0) osc.lustre-OST0001-osc-MDT0001.active updated after 0 sec (got 0) umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre New client success: got 'lustre-OST0001_UUID NEW DEACTIVATED' Setting lustre-OST0001.osc.active from 0 to 1 Waiting 90s for '1' umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg317-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg317-server stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local modules unloaded. oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server PASS 29 (139s) == conf-sanity-1 test 35a: Reconnect to the last active server first ========================================================== 04:38:12 (1734255492) start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre debug=ha Set up a fake failnode for the MDS Wait for RECONNECT_INTERVAL seconds (10s) conf-sanity.sh test_35a 2024-12-15 4h38m43s Stopping the MDT: lustre-MDT0000 stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server Restarting the MDT: lustre-MDT0000 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 Wait for df (29178) ... done debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout Debug log: 79 lines, 79 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local modules unloaded. oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server PASS 35a (115s) == conf-sanity-1 test 35b: Continue reconnection retries, if the active server is busy ========================================================== 04:40:06 (1734255606) start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-12-15 4h40m40s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800aa506800.stats=clear mdc.lustre-MDT0001-mdc-ffff8800aa506800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity-1/f35b.conf-sanity-1' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-15 04:41:23.000000000 -0500 Modify: 2024-12-15 04:41:23.000000000 -0500 Change: 2024-12-15 04:41:23.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 74 lines, 74 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local modules unloaded. oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server PASS 35b (178s) == conf-sanity-1 test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 04:43:04 (1734255784) Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov -o nosvc -n lustre-mdt1/mdt1 /mnt/lustre-mds1 nomtab: 1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all Start lustre-mdt1/mdt1 without service Commit the device label on lustre-mdt1/mdt1 Started lustre:MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg317-server Starting mds1: -o localrecov -o nomgs,force lustre-mdt1/mdt1 /mnt/lustre-mds1 force: 1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre blah blah umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server PASS 41b (88s) == conf-sanity-1 test 46a: handle ost additional - wide striped file ========================================================== 04:44:32 (1734255872) Testing with 2 OSTs start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec oleg317-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012a64c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a64c000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre2..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 /mnt/lustre2 stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre2/f41b.conf-sanity-1 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2 0x2 0x280000400 /mnt/lustre2/widestripe lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2 0x2 0x2c0000401 0 34 0x22 0x280000400 File: '/mnt/lustre/widestripe' Size: 3 Blocks: 2 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826934273 Links: 1 Access: (0674/-rw-rwxr--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-15 04:45:16.000000000 -0500 Modify: 2024-12-15 04:45:16.000000000 -0500 Change: 2024-12-15 04:45:17.000000000 -0500 Birth: - umount lustre on /mnt/lustre2..... Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost2 (opts:-f) on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server stop ost1 service on oleg317-server stop mds service on oleg317-server stop mds service on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local modules unloaded. oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server PASS 46a (145s) == conf-sanity-1 test 47a: server restart does not lose lru_resize ========================================================== 04:46:57 (1734256017) Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre setup single mount lustre success ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a9845000.lru_size=100 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a9845000.lru_size=100 ldlm.namespaces.lustre-OST0000-osc-ffff8800a9845000.lru_size=100 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a9845000.lru_size=100 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a9845000.lru_size=100 Failing ost1 on oleg317-server Stopping /mnt/lustre-ost1 (opts:) on oleg317-server 04:47:39 (1734256059) shut down facet: ost1 facet_host: oleg317-server facet_failover_host: oleg317-server Failover ost1 to oleg317-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 04:47:52 (1734256072) targets are mounted 04:47:52 (1734256072) facet_failover done Failing mds1 on oleg317-server Stopping /mnt/lustre-mds1 (opts:) on oleg317-server 04:47:53 (1734256073) shut down facet: mds1 facet_host: oleg317-server facet_failover_host: oleg317-server Failover mds1 to oleg317-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 04:48:06 (1734256086) targets are mounted 04:48:06 (1734256086) facet_failover done ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a9845000.lru_size=100 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a9845000.lru_size=100 umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 PASS 47a (92s) == conf-sanity-1 test 47b: client restart does not lose lru_resize ========================================================== 04:48:29 (1734256109) Failing mds1 on oleg317-server 04:48:30 (1734256110) shut down facet: mds1 facet_host: oleg317-server facet_failover_host: oleg317-server Failover mds1 to oleg317-server mount facets: mds1 Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 04:48:53 (1734256133) targets are mounted 04:48:53 (1734256133) facet_failover done pdsh@oleg317-client: oleg317-client: ssh exited with exit code 95 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid pdsh@oleg317-client: oleg317-client: ssh exited with exit code 95 Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre setup single mount lustre success ln: failed to create symbolic link '/home/green/git/lustre-release/lustre/utils/lctl': File exists SKIP: conf-sanity-1 test_47b Can not create symlink /usr/sbin/lctl SKIP 47b (71s) == conf-sanity-1 test 48: too many acls on file ========== 04:49:40 (1734256180) start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: mount.lustre: according to /etc/mtab lustre-mdt1/mdt1 is already mounted on /mnt/lustre-mds1 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 17 Start of lustre-mdt1/mdt1 on mds1 failed 17 start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg317-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 17 seq.cli-lustre-OST0000-super.width=65536 Start of lustre-ost1/ost1 on ost1 failed 17 mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre mount.lustre: according to /etc/mtab 192.168.203.117@tcp:/lustre is already mounted on /mnt/lustre setup single mount lustre success debug=0 debug=0 /mnt/lustre stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre/widestripe lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 4 0x4 0x280000401 It is expected to hold at least 4500 ACL entries File: '/mnt/lustre/widestripe' Size: 3 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205289279491 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-15 04:49:45.000000000 -0500 Modify: 2024-12-15 04:49:45.000000000 -0500 Change: 2024-12-15 04:50:57.000000000 -0500 Birth: - getfacl: Removing leading '/' from absolute path names Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg317-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg317-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg317-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server PASS 48 (172s) == conf-sanity-1 test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:52:32 (1734256352) Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre setup single mount lustre success check ldlm_timout... umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server PASS 49a (67s) == conf-sanity-1 test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:53:39 (1734256419) Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 PASS 49b (57s) == conf-sanity-1 test 50c: lazystatfs one server down ==== 04:54:36 (1734256476) start mds service on oleg317-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg317-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre llite.lustre-ffff88012a64a000.lazystatfs=1 stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server oleg317-server: oleg317-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.7732 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3328 378240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 383616 3072 378496 1% /mnt/lustre[MDT:1] lustre-OST0001_UUID 382976 4096 376832 2% /mnt/lustre[OST:1] filesystem_summary: 382976 4096 376832 2% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity-framework.sh: line 241: kill: (21693) - No such process umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg317-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server PASS 50c (124s) == conf-sanity-1 test 50d: lazystatfs client/server conn race ========================================================== 04:56:40 (1734256600) start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg317-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre llite.lustre-ffff8800b62cf800.lazystatfs=1 stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.7732 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3328 378240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 383616 3072 378496 1% /mnt/lustre[MDT:1] lustre-OST0001_UUID 382976 4096 376832 2% /mnt/lustre[OST:1] filesystem_summary: 382976 4096 376832 2% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity-framework.sh: line 241: kill: (28187) - No such process umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg317-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg317-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server PASS 50d (113s) == conf-sanity-1 test 50f: normal statfs one server in down ========================================================== 04:58:33 (1734256713) start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec start ost2 service on oleg317-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg317-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg317-server oleg317-server: oleg317-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre llite.lustre-ffff8800b6c65800.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.7732 start ost2 service on oleg317-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0001 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg317-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg317-server umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server PASS 50f (209s) == conf-sanity-1 test 50g: deactivated OST should not cause panic ========================================================== 05:02:02 (1734256922) start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre start ost2 service on oleg317-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec oleg317-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b63de000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b63de000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.117@tcp:/lustre 382976 4096 376832 2% /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg317-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg317-server stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server PASS 50g (151s) == conf-sanity-1 test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 05:04:33 (1734257073) Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg317-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec stop ost2 service on oleg317-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg317-server umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local modules unloaded. oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server PASS 51 (177s) == conf-sanity-1 test 54a: test llverdev and partial verify of device ========================================================== 05:07:30 (1734257250) SKIP: conf-sanity-1 test_54a ldiskfs only test SKIP 54a (1s) == conf-sanity-1 test 55: check lov_objid size =========== 05:07:31 (1734257251) SKIP: conf-sanity-1 test_55 ldiskfs only test SKIP 55 (1s) == conf-sanity-1 test 56a: check big OST indexes and out-of-index-order start ========================================================== 05:07:32 (1734257252) Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 Permanent disk data: Target: lustre:OST2710 Index: 10000 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.117@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.117@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=10000 lustre:fsname=lustre lustre:svname=lustre:OST2710 Permanent disk data: Target: lustre:OST03e8 Index: 1000 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.117@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost2 /dev/vdf mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2/ost2 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost2/ost2 properties lustre:mgsnode=192.168.203.117@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=1000 lustre:fsname=lustre lustre:svname=lustre:OST03e8 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST2710-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST2710 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg317-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST03e8-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST03e8 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre ok OBDS: 1000: lustre-OST03e8_UUID ACTIVE 10000: lustre-OST2710_UUID ACTIVE 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00897333 s, 456 kB/s oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre opts: Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg317-server Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server PASS 56a (186s) == conf-sanity-1 test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 05:10:38 (1734257438) SKIP: conf-sanity-1 test_56b needs >= 3 MDTs SKIP 56b (1s) == conf-sanity-1 test 57a: initial registration from failnode should fail (should return errs) ========================================================== 05:10:39 (1734257439) oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.203.117@tcp Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.203.117@tcp failover.node=192.168.203.117@tcp autodegrade=on Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.203.117@tcp lustre:failover.node=192.168.203.117@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg317-server: mount.lustre: mount lustre-ost1/ost1 at /mnt/lustre-ost1 failed: Cannot assign requested address pdsh@oleg317-client: oleg317-server: ssh exited with exit code 99 oleg317-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory oleg317-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 Start of lustre-ost1/ost1 on ost1 failed 99 umount lustre on /mnt/lustre..... stop ost1 service on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 PASS 57a (124s) == conf-sanity-1 test 57b: initial registration from servicenode should not fail ========================================================== 05:12:43 (1734257563) oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.203.117@tcp Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x1042 (OST update no_primnode ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.203.117@tcp failover.node=192.168.203.117@tcp autodegrade=on Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.203.117@tcp lustre:failover.node=192.168.203.117@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4162 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid umount lustre on /mnt/lustre..... stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 PASS 57b (133s) == conf-sanity-1 test 59: writeconf mount option ========= 05:14:56 (1734257696) oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server original ost count: 1 (expect > 0) oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid after mdt writeconf count: 0 (expect 0) OST start without writeconf should fail: oleg317-server: mount.lustre: mount lustre-ost1/ost1 at /mnt/lustre-ost1 failed: No such file or directory oleg317-server: Is the MGS specification correct? oleg317-server: Is the filesystem name correct? oleg317-server: If upgrading, is the copied client log valid? (see upgrade docs) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 oleg317-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory oleg317-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 OST start with writeconf should succeed: oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid after ost writeconf count: 1 (expect 1) oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid after ost2 writeconf count: 2 (expect 2) oleg317-server: oleg317-server.virtnet: executing unload_modules_local oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server PASS 59 (167s) == conf-sanity-1 test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 05:17:43 (1734257863) SKIP: conf-sanity-1 test_60a ldiskfs only test SKIP 60a (1s) == conf-sanity-1 test 62: start with disabled journal ==== 05:17:44 (1734257864) SKIP: conf-sanity-1 test_62 ldiskfs only test SKIP 62 (1s) == conf-sanity-1 test 64: check lfs df --lazy ============ 05:17:45 (1734257865) start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg317-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre stop ost2 service on oleg317-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg317-server /home/green/git/lustre-release/lustre/utils/lfs df UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3328 378240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 383616 3072 378496 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 382976 4096 376832 2% /mnt/lustre[OST:0] lustre-OST0001_UUID : Transport endpoint is not connected filesystem_summary: 382976 4096 376832 2% /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local modules unloaded. oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg317-server: oleg317-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg317-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server PASS 64 (155s) == conf-sanity-1 test 66: replace nids =================== 05:20:20 (1734258020) client=34615898 MDS=34615898 OSS=34615898 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Setting lustre-OST0000.osc.active from 1 to 0 Waiting 90s for '0' Updated after 10s: want '0' got '0' replace_nids should fail if MDS, OSTs and clients are UP oleg317-server: error: replace_nids: Operation now in progress pdsh@oleg317-client: oleg317-server: ssh exited with exit code 115 umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) replace_nids should fail if MDS and OSTs are UP oleg317-server: error: replace_nids: Operation now in progress pdsh@oleg317-client: oleg317-server: ssh exited with exit code 115 stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server replace_nids should fail if MDS is UP oleg317-server: error: replace_nids: Operation now in progress pdsh@oleg317-client: oleg317-server: ssh exited with exit code 115 stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server start mds service on oleg317-server Starting mds1: -o localrecov -o nosvc lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all Start lustre-mdt1/mdt1 without service Started lustre-MDT0000 command should accept two parameters pdsh@oleg317-client: oleg317-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] correct device name should be passed oleg317-server: error: replace_nids: No such device or address pdsh@oleg317-client: oleg317-server: ssh exited with exit code 6 wrong nids list should not destroy the system pdsh@oleg317-client: oleg317-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] pdsh@oleg317-client: oleg317-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] replace OST nid command should accept two parameters pdsh@oleg317-client: oleg317-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] wrong nids list should not destroy the system pdsh@oleg317-client: oleg317-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] set NIDs with failover replace MDS nid stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Setting lustre-OST0000.osc.active from 0 to 1 Waiting 90s for '1' Waiting 80s for '1' Waiting 70s for '1' Updated after 22s: want '1' got '1' start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local modules unloaded. Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server PASS 66 (262s) == conf-sanity-1 test 70e: Sync-on-Cancel will be enabled by default on DNE ========================================================== 05:24:42 (1734258282) Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions Permanent disk data: Target: lustre:MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x65 (MDT MGS first_time update ) Persistent mount opts: Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mkfs_cmd = zpool create -f -O canmount=off lustre-mdt1 /dev/vdc mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt1/mdt1 xattr=sa dnodesize=auto Writing lustre-mdt1/mdt1 properties lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=101 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:MDT0000 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.117@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.117@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Permanent disk data: Target: lustre:MDT0001 Index: 1 Lustre FS: lustre Mount type: zfs Flags: 0x61 (MDT first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.117@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mkfs_cmd = zpool create -f -O canmount=off lustre-mdt2 /dev/vdd mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt2/mdt2 xattr=sa dnodesize=auto Writing lustre-mdt2/mdt2 properties lustre:mgsnode=192.168.203.117@tcp lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=97 lustre:index=1 lustre:fsname=lustre lustre:svname=lustre:MDT0001 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-server: oleg317-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 oleg317-server: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 oleg317-server: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 2 sec stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server umount lustre on /mnt/lustre..... stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 PASS 70e (107s) == conf-sanity-1 test 75: The order of --index should be irrelevant ========================================================== 05:26:29 (1734258389) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' Permanent disk data: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x45 (MDT MGS update ) Persistent mount opts: Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mkfs_cmd = zpool create -f -O canmount=off lustre-mdt1 /dev/vdc mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt1/mdt1 xattr=sa dnodesize=auto Writing lustre-mdt1/mdt1 properties lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=69 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-MDT0000 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.117@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.117@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 Permanent disk data: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x45 (MDT MGS update ) Persistent mount opts: Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mkfs_cmd = zpool create -f -O canmount=off lustre-mdt1 /dev/vdc mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt1/mdt1 xattr=sa dnodesize=auto Writing lustre-mdt1/mdt1 properties lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=69 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-MDT0000 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.117@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.117@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server PASS 75 (95s) == conf-sanity-1 test 78: run resize2fs on MDT and OST filesystems ========================================================== 05:28:04 (1734258484) SKIP: conf-sanity-1 test_78 ldiskfs only test SKIP 78 (1s) == conf-sanity-1 test 79: format MDT/OST without mgs option (should return errors) ========================================================== 05:28:05 (1734258485) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: oleg317-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode oleg317-server: mkfs.lustre: exiting with 22 (Invalid argument) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: oleg317-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode oleg317-server: mkfs.lustre: exiting with 22 (Invalid argument) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: oleg317-server: mkfs.lustre FATAL: Must specify --mgsnode oleg317-server: mkfs.lustre: exiting with 22 (Invalid argument) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server PASS 79 (73s) == conf-sanity-1 test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 05:29:19 (1734258559) SKIP: conf-sanity-1 test_83 ldiskfs only test SKIP 83 (2s) == conf-sanity-1 test 86: Replacing mkfs.lustre -G option ========================================================== 05:29:20 (1734258560) SKIP: conf-sanity-1 test_86 LU-6442: no such mkfs params for ZFS OSTs SKIP 86 (1s) == conf-sanity-1 test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 05:29:21 (1734258561) SKIP: conf-sanity-1 test_87 ldiskfs only test SKIP 87 (1s) == conf-sanity-1 test 89: check tunefs --param and --erase-param{s} options ========================================================== 05:29:22 (1734258562) Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:) tunefs --param failover.node=192.0.2.254@tcp0 tunefs --param failover.node=192.0.2.255@tcp0 tunefs --erase-param failover.node tunefs --erase-params tunefs --param failover.node=192.0.2.254@tcp0 --erase-params Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server PASS 89 (82s) == conf-sanity-1 test 93: register mulitple MDT at the same time ========================================================== 05:30:44 (1734258644) SKIP: conf-sanity-1 test_93 needs >= 3 MDTs SKIP 93 (1s) == conf-sanity-1 test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 05:30:45 (1734258645) Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_size: failed to read coresize from /sys/module/pcc_cpufreq libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre setup single mount lustre success lustre-MDT0000: lustre-OST0000: umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 PASS 100 (86s) == conf-sanity-1 test 104a: Make sure user defined options are reflected in mount ========================================================== 05:32:11 (1734258731) mountfsopt: acl,user_xattr Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 Starting mds1: -o localrecov,noacl lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 Starting mds2: -o localrecov,noacl lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre setfacl: /mnt/lustre: Operation not supported PASS 104a (70s) == conf-sanity-1 test 105: check file creation for ro and rw bind mnt pt ========================================================== 05:33:21 (1734258801) umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local oleg317-server: rmmod: ERROR: Module lustre is in use pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 modules unloaded. Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-ost2 (opts:-f) on oleg317-server oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre touch: cannot touch '/tmp/d105.conf-sanity-1/f105.conf-sanity-1': Read-only file system umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 PASS 105 (121s) == conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:35:22 (1734258922) Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16131 291 15840 2% /mnt/lustre[OST:0] filesystem_summary: 16483 643 15840 4% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1682 (time 1734258996.69 total 10.01 last 168.07) - open/close 3457 (time 1734259006.69 total 20.01 last 177.49) - open/close 5237 (time 1734259016.69 total 30.01 last 177.98) - open/close 6801 (time 1734259026.69 total 40.01 last 156.35) - open/close 8263 (time 1734259036.69 total 50.01 last 146.17) - open/close 9713 (time 1734259046.70 total 60.02 last 144.92) - open/close 11211 (time 1734259056.70 total 70.02 last 149.72) - open/close 12786 (time 1734259066.71 total 80.03 last 157.40) - open/close 14224 (time 1734259076.72 total 90.04 last 143.69) - open/close 15645 (time 1734259086.72 total 100.04 last 142.10) - open/close 17060 (time 1734259096.72 total 110.04 last 141.49) - open/close 18426 (time 1734259106.73 total 120.05 last 136.46) - open/close 19984 (time 1734259116.73 total 130.05 last 155.78) - open/close 21662 (time 1734259126.73 total 140.05 last 167.77) - open/close 23078 (time 1734259136.73 total 150.05 last 141.59) - open/close 24476 (time 1734259146.74 total 160.06 last 139.76) - open/close 25945 (time 1734259156.74 total 170.06 last 146.82) - open/close 27553 (time 1734259166.74 total 180.06 last 160.76) - open/close 29172 (time 1734259176.75 total 190.07 last 161.85) - open/close 30000 (time 1734259181.91 total 195.23 last 160.46) - open/close 31437 (time 1734259191.91 total 205.23 last 143.70) - open/close 32995 (time 1734259201.91 total 215.23 last 155.75) - open/close 34470 (time 1734259211.92 total 225.24 last 147.42) - open/close 35898 (time 1734259221.92 total 235.24 last 142.80) - open/close 37468 (time 1734259231.92 total 245.24 last 156.96) - open/close 39068 (time 1734259241.92 total 255.24 last 159.98) - open/close 40000 (time 1734259246.84 total 260.16 last 189.28) - open/close 41981 (time 1734259256.85 total 270.17 last 198.09) - open/close 43813 (time 1734259266.85 total 280.17 last 183.14) - open/close 45312 (time 1734259276.85 total 290.17 last 149.86) - open/close 47041 (time 1734259286.85 total 300.17 last 172.87) - open/close 48641 (time 1734259296.86 total 310.18 last 159.96) - open/close 50000 (time 1734259305.56 total 318.88 last 156.07) - open/close 51585 (time 1734259315.56 total 328.88 last 158.49) - open/close 53264 (time 1734259325.57 total 338.89 last 167.78) - open/close 55142 (time 1734259335.58 total 348.90 last 187.70) - open/close 56945 (time 1734259345.58 total 358.90 last 180.29) - open/close 58484 (time 1734259355.58 total 368.90 last 153.85) - open/close 60000 (time 1734259365.22 total 378.54 last 157.28) - open/close 61584 (time 1734259375.22 total 388.54 last 158.32) - open/close 63126 (time 1734259385.23 total 398.55 last 154.14) total: 64768 open/close in 408.43 seconds: 158.58 ops/second - /unlink 2723 (time 1734259406.50 total 10.00 last 272.27) - /unlink 5387 (time 1734259416.50 total 20.00 last 266.35) - /unlink 8004 (time 1734259426.50 total 30.00 last 261.68) - /unlink 10000 (time 1734259434.01 total 37.51 last 265.80) - /unlink 12722 (time 1734259444.01 total 47.51 last 272.17) - /unlink 15447 (time 1734259454.02 total 57.52 last 272.37) - /unlink 18089 (time 1734259464.02 total 67.52 last 264.17) - /unlink 20000 (time 1734259471.16 total 74.66 last 267.71) - /unlink 22728 (time 1734259481.16 total 84.66 last 272.74) - /unlink 25800 (time 1734259491.16 total 94.66 last 307.10) - /unlink 28605 (time 1734259501.16 total 104.67 last 280.42) - /unlink 30000 (time 1734259506.81 total 110.31 last 246.95) - /unlink 32886 (time 1734259516.82 total 120.32 last 288.56) - /unlink 35998 (time 1734259526.82 total 130.32 last 311.16) - /unlink 39607 (time 1734259536.82 total 140.32 last 360.88) - /unlink 42543 (time 1734259546.82 total 150.32 last 293.54) - /unlink 45428 (time 1734259556.82 total 160.32 last 288.43) - /unlink 48417 (time 1734259566.82 total 170.33 last 298.80) - /unlink 50000 (time 1734259572.51 total 176.01 last 278.25) - /unlink 53052 (time 1734259582.52 total 186.02 last 305.13) - /unlink 56341 (time 1734259592.52 total 196.02 last 328.84) - /unlink 59834 (time 1734259602.52 total 206.02 last 349.29) - /unlink 63695 (time 1734259612.52 total 216.02 last 386.01) total: 64768 /unlink in 218.84 seconds: 295.96 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2075 (time 1734259641.10 total 10.00 last 207.42) - open/close 4114 (time 1734259651.10 total 20.01 last 203.83) - open/close 6100 (time 1734259661.10 total 30.01 last 198.60) - open/close 8097 (time 1734259671.10 total 40.01 last 199.66) - open/close 10000 (time 1734259680.97 total 49.88 last 192.81) - open/close 12014 (time 1734259690.97 total 59.88 last 201.38) - open/close 14186 (time 1734259700.98 total 69.88 last 217.16) - open/close 16383 (time 1734259710.98 total 79.88 last 219.65) - open/close 18344 (time 1734259720.98 total 89.89 last 196.05) - open/close 20000 (time 1734259729.42 total 98.33 last 196.26) - open/close 21803 (time 1734259739.42 total 108.33 last 180.28) - open/close 23664 (time 1734259749.43 total 118.33 last 186.00) - open/close 25445 (time 1734259759.43 total 128.33 last 178.06) - open/close 27382 (time 1734259769.43 total 138.34 last 193.61) - open/close 29370 (time 1734259779.43 total 148.34 last 198.78) - open/close 30000 (time 1734259782.52 total 151.43 last 204.11) - open/close 32074 (time 1734259792.52 total 161.43 last 207.30) - open/close 33864 (time 1734259802.53 total 171.43 last 178.93) - open/close 35569 (time 1734259812.53 total 181.44 last 170.42) - open/close 37486 (time 1734259822.54 total 191.44 last 191.66) - open/close 39686 (time 1734259832.54 total 201.44 last 219.92) - open/close 41853 (time 1734259842.54 total 211.45 last 216.66) - open/close 44063 (time 1734259852.54 total 221.45 last 220.96) - open/close 46158 (time 1734259862.55 total 231.45 last 209.40) - open/close 48037 (time 1734259872.55 total 241.46 last 187.81) - open/close 49898 (time 1734259882.56 total 251.46 last 186.05) - open/close 51649 (time 1734259892.56 total 261.46 last 175.04) - open/close 53355 (time 1734259902.56 total 271.47 last 170.53) - open/close 55249 (time 1734259912.56 total 281.47 last 189.38) - open/close 57245 (time 1734259922.56 total 291.47 last 199.60) - open/close 59230 (time 1734259932.57 total 301.47 last 198.42) - open/close 60000 (time 1734259936.45 total 305.36 last 198.17) - open/close 61834 (time 1734259946.45 total 315.36 last 183.39) - open/close 63739 (time 1734259956.46 total 325.36 last 190.41) total: 64768 open/close in 331.00 seconds: 195.67 ops/second - /unlink 3335 (time 1734259973.33 total 10.00 last 333.41) - /unlink 6741 (time 1734259983.33 total 20.00 last 340.58) - /unlink 10000 (time 1734259992.37 total 29.04 last 360.61) - /unlink 14193 (time 1734260002.37 total 39.04 last 419.23) - /unlink 18669 (time 1734260012.37 total 49.04 last 447.49) - /unlink 20000 (time 1734260015.72 total 52.40 last 397.25) - /unlink 23949 (time 1734260025.72 total 62.40 last 394.85) - /unlink 28067 (time 1734260035.72 total 72.40 last 411.79) - /unlink 30000 (time 1734260040.68 total 77.36 last 389.77) - /unlink 33972 (time 1734260050.68 total 87.36 last 397.12) - /unlink 38968 (time 1734260060.68 total 97.36 last 499.59) - /unlink 43850 (time 1734260070.69 total 107.36 last 488.12) - /unlink 48191 (time 1734260080.69 total 117.36 last 434.07) - /unlink 50000 (time 1734260084.78 total 121.45 last 442.17) - /unlink 54293 (time 1734260094.78 total 131.45 last 429.22) - /unlink 58746 (time 1734260104.78 total 141.46 last 445.22) - /unlink 60000 (time 1734260107.44 total 144.11 last 472.09) - /unlink 64525 (time 1734260117.44 total 154.11 last 452.44) total: 64768 /unlink in 154.67 seconds: 418.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2556 (time 1734260146.37 total 10.00 last 255.59) - open/close 5278 (time 1734260156.38 total 20.00 last 272.15) - open/close 8248 (time 1734260166.38 total 30.00 last 296.91) - open/close 10000 (time 1734260172.85 total 36.48 last 270.75) - open/close 12703 (time 1734260182.85 total 46.48 last 270.25) - open/close 15590 (time 1734260192.85 total 56.48 last 288.68) - open/close 18103 (time 1734260202.85 total 66.48 last 251.28) - open/close 20000 (time 1734260210.44 total 74.07 last 249.91) - open/close 22447 (time 1734260220.45 total 84.07 last 244.65) - open/close 24822 (time 1734260230.45 total 94.07 last 237.46) - open/close 27220 (time 1734260240.45 total 104.08 last 239.73) - open/close 29632 (time 1734260250.45 total 114.08 last 241.12) - open/close 32383 (time 1734260260.45 total 124.08 last 275.08) - open/close 34905 (time 1734260270.46 total 134.08 last 252.18) - open/close 37168 (time 1734260280.46 total 144.08 last 226.26) - open/close 39600 (time 1734260290.46 total 154.09 last 243.13) - open/close 41638 (time 1734260300.46 total 164.09 last 203.76) - open/close 43807 (time 1734260310.47 total 174.09 last 216.78) - open/close 45979 (time 1734260320.47 total 184.10 last 217.16) - open/close 48376 (time 1734260330.47 total 194.10 last 239.64) - open/close 50000 (time 1734260337.49 total 201.12 last 231.32) - open/close 52351 (time 1734260347.49 total 211.12 last 235.04) - open/close 54783 (time 1734260357.50 total 221.13 last 243.08) - open/close 57031 (time 1734260367.50 total 231.13 last 224.71) - open/close 59258 (time 1734260377.51 total 241.13 last 222.66) - open/close 60000 (time 1734260381.16 total 244.78 last 203.20) - open/close 62134 (time 1734260391.16 total 254.78 last 213.37) - open/close 64149 (time 1734260401.16 total 264.79 last 201.47) total: 64768 open/close in 268.08 seconds: 241.60 ops/second - /unlink 3523 (time 1734260415.72 total 10.00 last 352.24) - /unlink 7337 (time 1734260425.72 total 20.00 last 381.35) - /unlink 10000 (time 1734260432.75 total 27.03 last 379.00) - /unlink 13471 (time 1734260442.75 total 37.03 last 347.02) - /unlink 17080 (time 1734260452.75 total 47.03 last 360.85) - /unlink 20000 (time 1734260460.60 total 54.88 last 372.01) - /unlink 24096 (time 1734260470.60 total 64.88 last 409.57) - /unlink 28685 (time 1734260480.60 total 74.88 last 458.84) - /unlink 30000 (time 1734260483.43 total 77.71 last 464.87) - /unlink 34318 (time 1734260493.43 total 87.72 last 431.64) - /unlink 38937 (time 1734260503.43 total 97.72 last 461.83) - /unlink 40000 (time 1734260505.80 total 100.08 last 449.52) - /unlink 44479 (time 1734260515.80 total 110.08 last 447.88) - /unlink 49191 (time 1734260525.80 total 120.09 last 471.08) - /unlink 54551 (time 1734260535.80 total 130.09 last 535.93) - /unlink 58858 (time 1734260545.81 total 140.09 last 430.55) - /unlink 60000 (time 1734260549.16 total 143.45 last 340.39) - /unlink 64263 (time 1734260559.16 total 153.45 last 426.27) total: 64768 /unlink in 154.61 seconds: 418.90 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2720 (time 1734260587.09 total 10.00 last 271.94) - open/close 5497 (time 1734260597.10 total 20.01 last 277.61) - open/close 7918 (time 1734260607.10 total 30.01 last 242.01) - open/close 10000 (time 1734260615.48 total 38.39 last 248.57) - open/close 12689 (time 1734260625.48 total 48.39 last 268.81) - open/close 15091 (time 1734260635.48 total 58.39 last 240.18) - open/close 17643 (time 1734260645.48 total 68.39 last 255.20) - open/close 20000 (time 1734260654.15 total 77.05 last 272.03) - open/close 23016 (time 1734260664.15 total 87.06 last 301.56) - open/close 25555 (time 1734260674.15 total 97.06 last 253.82) - open/close 28086 (time 1734260684.15 total 107.06 last 253.03) - open/close 30000 (time 1734260691.10 total 114.01 last 275.46) - open/close 32725 (time 1734260701.10 total 124.01 last 272.50) - open/close 35448 (time 1734260711.10 total 134.01 last 272.28) - open/close 38510 (time 1734260721.10 total 144.01 last 306.20) - open/close 40000 (time 1734260725.51 total 148.41 last 338.34) - open/close 43504 (time 1734260735.51 total 158.42 last 350.37) - open/close 47194 (time 1734260745.51 total 168.42 last 368.98) - open/close 50000 (time 1734260753.40 total 176.30 last 355.72) - open/close 53611 (time 1734260763.40 total 186.31 last 361.02) - open/close 57796 (time 1734260773.40 total 196.31 last 418.44) - open/close 60000 (time 1734260777.58 total 200.49 last 526.68) - open/close 64468 (time 1734260787.59 total 210.50 last 446.68) total: 64768 open/close in 211.19 seconds: 306.68 ops/second - /unlink 9610 (time 1734260799.13 total 10.00 last 960.93) - /unlink 18343 (time 1734260809.13 total 20.00 last 873.16) - /unlink 20000 (time 1734260811.45 total 22.32 last 716.10) - /unlink 27781 (time 1734260821.45 total 32.32 last 778.02) - /unlink 30000 (time 1734260824.08 total 34.95 last 843.33) - /unlink 38189 (time 1734260834.08 total 44.95 last 818.86) - /unlink 40000 (time 1734260836.33 total 47.20 last 805.97) - /unlink 47955 (time 1734260846.33 total 57.20 last 795.48) - /unlink 50000 (time 1734260849.10 total 59.97 last 736.72) - /unlink 57552 (time 1734260859.11 total 69.97 last 755.13) - /unlink 60000 (time 1734260862.10 total 72.97 last 817.85) total: 64768 /unlink in 78.02 seconds: 830.19 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4784 (time 1734260897.94 total 10.00 last 478.39) - open/close 9757 (time 1734260907.94 total 20.00 last 497.29) - open/close 14533 (time 1734260917.94 total 30.00 last 477.56) - open/close 19320 (time 1734260927.94 total 40.00 last 478.69) - open/close 24770 (time 1734260937.94 total 50.00 last 544.91) - open/close 30000 (time 1734260947.30 total 59.36 last 558.97) - open/close 35419 (time 1734260957.30 total 69.36 last 541.87) - open/close 39928 (time 1734260967.30 total 79.36 last 450.89) - open/close 44692 (time 1734260977.30 total 89.36 last 476.35) - open/close 49553 (time 1734260987.30 total 99.36 last 486.05) - open/close 54170 (time 1734260997.30 total 109.36 last 461.69) - open/close 58875 (time 1734261007.30 total 119.36 last 470.42) - open/close 60000 (time 1734261009.59 total 121.65 last 491.22) total: 64768 open/close in 131.40 seconds: 492.89 ops/second - /unlink 7863 (time 1734261030.20 total 10.00 last 786.24) - /unlink 10000 (time 1734261032.77 total 12.57 last 833.36) - /unlink 18662 (time 1734261042.77 total 22.57 last 866.16) - /unlink 27394 (time 1734261052.77 total 32.57 last 873.13) - /unlink 30000 (time 1734261055.80 total 35.60 last 858.72) - /unlink 38513 (time 1734261065.80 total 45.60 last 851.29) - /unlink 47187 (time 1734261075.80 total 55.60 last 867.39) - /unlink 50000 (time 1734261079.14 total 58.94 last 842.73) - /unlink 58809 (time 1734261089.14 total 68.94 last 880.82) total: 64768 /unlink in 75.57 seconds: 857.03 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 PASS 106 (2218s) == conf-sanity-1 test 109a: test lctl clear_conf fsname == 06:12:20 (1734261140) SKIP: conf-sanity-1 test_109a LU-8727: no implementation for ZFS SKIP 109a (1s) == conf-sanity-1 test 109b: test lctl clear_conf one config ========================================================== 06:12:21 (1734261141) SKIP: conf-sanity-1 test_109b LU-8727: no implementation for ZFS SKIP 109b (0s) == conf-sanity-1 test 116: big size MDT support ========== 06:12:21 (1734261141) SKIP: conf-sanity-1 test_116 ldiskfs only test SKIP 116 (1s) == conf-sanity-1 test 122a: Check OST sequence update ==== 06:12:22 (1734261142) Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre fail_loc=0 total: 1000 open/close in 2.10 seconds: 476.29 ops/second umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 PASS 122a (70s) == conf-sanity-1 test 123ah: del_ost cancels config log entries correctly ========================================================== 06:13:32 (1734261212) Checking servers environments Checking clients oleg317-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' Setup mgs, mdt, osts Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Started clients oleg317-client.virtnet: 192.168.203.117@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012dba6800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012dba6800.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 6s: want 'procname_uid' got 'procname_uid' disable quota as required del_ost: dry run for target lustre-OST0000 config_log: lustre-MDT0000 [DRY RUN] cancel catalog 'lustre-MDT0000:29':"- { index: 29, event: add_osc, device: lustre-MDT0000-mdtlov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }" [DRY RUN] cancel catalog 'lustre-MDT0000:28':"- { index: 28, event: setup, device: lustre-OST0000-osc-MDT0000, UUID: lustre-OST0000_UUID, node: 192.168.203.117@tcp }" [DRY RUN] cancel catalog 'lustre-MDT0000:27':"- { index: 27, event: attach, device: lustre-OST0000-osc-MDT0000, type: osc, UUID: lustre-MDT0000-mdtlov_UUID }" del_ost: no catalog entry deleted config_log: lustre-MDT0001 [DRY RUN] cancel catalog 'lustre-MDT0001:32':"- { index: 32, event: add_osc, device: lustre-MDT0001-mdtlov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }" [DRY RUN] cancel catalog 'lustre-MDT0001:31':"- { index: 31, event: setup, device: lustre-OST0000-osc-MDT0001, UUID: lustre-OST0000_UUID, node: 192.168.203.117@tcp }" [DRY RUN] cancel catalog 'lustre-MDT0001:30':"- { index: 30, event: attach, device: lustre-OST0000-osc-MDT0001, type: osc, UUID: lustre-MDT0001-mdtlov_UUID }" del_ost: no catalog entry deleted config_log: lustre-client [DRY RUN] cancel catalog 'lustre-client:34':"- { index: 34, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }" [DRY RUN] cancel catalog 'lustre-client:33':"- { index: 33, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.203.117@tcp }" [DRY RUN] cancel catalog 'lustre-client:32':"- { index: 32, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID }" del_ost: no catalog entry deleted config_log: lustre-MDT0000 cancel catalog lustre-MDT0000 log_idx 29: done cancel catalog lustre-MDT0000 log_idx 28: done cancel catalog lustre-MDT0000 log_idx 27: done del_ost: cancelled 3 catalog entries config_log: lustre-MDT0001 cancel catalog lustre-MDT0001 log_idx 32: done cancel catalog lustre-MDT0001 log_idx 31: done cancel catalog lustre-MDT0001 log_idx 30: done del_ost: cancelled 3 catalog entries config_log: lustre-client cancel catalog lustre-client log_idx 34: done cancel catalog lustre-client log_idx 33: done cancel catalog lustre-client log_idx 32: done del_ost: cancelled 3 catalog entries umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local oleg317-server: rmmod: ERROR: Module lustre is in use pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 modules unloaded. Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-ost2 (opts:-f) on oleg317-server oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server fail_loc=0 PASS 123ah (121s) == conf-sanity-1 test 132: hsm_actions processed after failover ========================================================== 06:15:33 (1734261333) Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x5 (MDT MGS ) Persistent mount opts: Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x45 (MDT MGS update ) Persistent mount opts: Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mdt.hsm_control=enabled Writing lustre-mdt1/mdt1 properties lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:mdt.hsm_control=enabled lustre:version=1 lustre:flags=69 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-MDT0000 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server PASS 132 (27s) == conf-sanity-1 test 133: stripe QOS: free space balance in a pool ========================================================== 06:16:00 (1734261360) SKIP: conf-sanity-1 test_133 needs >= 4 OSTs SKIP 133 (1s) == conf-sanity-1 test 135: check the behavior when changelog is wrapped around ========================================================== 06:16:01 (1734261361) Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre oleg317-client: fail_loc=0x1312 oleg317-client: fail_val=5 oleg317-server: fail_loc=0x1312 oleg317-server: fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity-1 mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog - open/close 4209 (time 1734261406.50 total 10.00 last 420.77) total: 4500 open/close in 10.70 seconds: 420.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #12998 total: 4500 /unlink in 5.48 seconds: 821.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 4395 (time 1734261424.49 total 10.00 last 439.42) total: 4500 open/close in 10.22 seconds: 440.26 ops/second total: 4500 /unlink in 5.16 seconds: 872.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 total: 4500 open/close in 9.83 seconds: 457.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 total: 4500 /unlink in 5.06 seconds: 888.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 total: 4500 open/close in 9.79 seconds: 459.57 ops/second total: 4500 /unlink in 5.51 seconds: 817.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 total: 4500 open/close in 9.85 seconds: 456.71 ops/second total: 4500 /unlink in 5.13 seconds: 877.30 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 10973: 12973 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local modules unloaded. PASS 135 (150s) == conf-sanity-1 test 136: don't panic with bad obdecho setup ========================================================== 06:18:31 (1734261511) Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg317-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre oleg317-server: oleg317-server.virtnet: executing load_module obdecho/obdecho oleg317-server: error: setup: Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: test_mkdir: No such file or directory pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre opts: Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server PASS 136 (112s) == conf-sanity-1 test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:20:23 (1734261623) SKIP: conf-sanity-1 test_137 needs >= 3 MDTs SKIP 137 (1s) == conf-sanity-1 test 140: remove_updatelog script actions ========================================================== 06:20:24 (1734261624) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d140.conf-sanity-1 stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server Dry run was requested, no changes will be applied Scan update_log at '/mnt/lustre-mds2': Selected MDTS: 0 1 Processing MDT0 llog catalog [0x240000401:0x1:0x0] ... rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x2:0x0] rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x3:0x0] > /mnt/lustre-mds2/update_log_dir/[0x240000401:0x1:0x0] Processing MDT1 llog catalog [0x240000400:0x1:0x0] ... remove_updatelog: /mnt/lustre-mds2/update_log_dir/[0x240000400:0x1:0x0] is too small. > /mnt/lustre-mds2/update_log_dir/[0x240000400:0x1:0x0] Dry run was requested, no changes will be applied Scan update_log at '/mnt/lustre-mds2': Selected MDTS: 1 0 Processing MDT1 llog catalog [0x240000400:0x1:0x0] ... remove_updatelog: /mnt/lustre-mds2/update_log_dir/[0x240000400:0x1:0x0] is too small. > /mnt/lustre-mds2/update_log_dir/[0x240000400:0x1:0x0] Processing MDT0 llog catalog [0x240000401:0x1:0x0] ... rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x2:0x0] rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x3:0x0] > /mnt/lustre-mds2/update_log_dir/[0x240000401:0x1:0x0] Scan update_log at '/mnt/lustre-mds2': Selected MDTS: 0 Processing MDT0 llog catalog [0x240000401:0x1:0x0] ... rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x2:0x0] rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x3:0x0] > /mnt/lustre-mds2/update_log_dir/[0x240000401:0x1:0x0] start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 4 sec Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre opts: Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server PASS 140 (236s) == conf-sanity-1 test 151a: damaged local config doesn't prevent mounting ========================================================== 06:24:20 (1734261860) SKIP: conf-sanity-1 test_151a ldiskfs only test SKIP 151a (0s) == conf-sanity-1 test 151b: -ENOSPC doesn't affect mount ========================================================== 06:24:20 (1734261860) SKIP: conf-sanity-1 test_151b ldiskfs only test SKIP 151b (1s) == conf-sanity-1 test 153a: bypass invalid NIDs quickly == 06:24:21 (1734261861) Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid waiting for mount ... "192.168.203.117@tcp": { connects: 1, replied: 1, uptodate: uptodate, sec_ago: 5 } "192.168.252.112@tcp": { connects: 0, replied: 0, uptodate: discovering, sec_ago: never } "10.252.252.113@tcp": { connects: 0, replied: 0, uptodate: discovering, sec_ago: never } "192.168.203.117@tcp": { connects: 0, replied: 0, uptodate: unknown, sec_ago: never } setup single mount lustre success umount lustre on /mnt/lustre..... stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 PASS 153a (195s) == conf-sanity-1 test 153b: added IPv6 NID support ======= 06:27:36 (1734262056) Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server Checking servers environments Checking clients oleg317-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Started clients oleg317-client.virtnet: 192.168.203.117@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012dba2000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012dba2000.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' disable quota as required PASS 153b (84s) == conf-sanity-1 test 153c: don't stuck on unreached NID ========================================================== 06:29:00 (1734262140) Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg317-server oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid umount lustre on /mnt/lustre..... waiting for mount ... waiting for mount ... waiting for mount ... waiting for mount ... waiting for mount ... waiting for mount ... "192.168.252.112@tcp": { connects: 3, replied: 0, uptodate: discovering, sec_ago: 0 } "192.168.252.113@tcp": { connects: 2, replied: 0, uptodate: discovering, sec_ago: 10 } "192.168.252.115@tcp": { connects: 2, replied: 0, uptodate: discovering, sec_ago: 5 } connection attempts: 7 Waiting for mount to fail mount.lustre: mount 192.168.252.112@tcp:192.168.252.113@tcp:192.168.252.115@tcp:/lustre at /mnt/lustre failed: Input/output error Is the MGS running? umount lustre on /mnt/lustre..... stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 PASS 153c (341s) == conf-sanity-1 test 154: expand .. on rename after MDT backup restore ========================================================== 06:34:41 (1734262481) SKIP: conf-sanity-1 test_154 ldiskfs only test SKIP 154 (1s) Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity-1 test complete, duration 7421 sec ======== 06:35:08 (1734262508) === conf-sanity-1: start cleanup 06:35:08 (1734262508) === === conf-sanity-1: finish cleanup 06:35:09 (1734262509) ===