-----============= acceptance-small: conf-sanity-1 ============----- Sun Dec 15 04:31:34 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg113-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity-1.*ex: No such file or directory excepting tests: skipping tests SLOW=no: 106 Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg113-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg113-server oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server == conf-sanity-1 test 17: Verify failed mds_postsetup won't fail assertion (2936) (should return errs) ========================================================== 04:32:35 (1734255155) SKIP: conf-sanity-1 test_17 ldiskfs only test SKIP 17 (1s) == conf-sanity-1 test 18: check mkfs creates large journals ========================================================== 04:32:36 (1734255156) SKIP: conf-sanity-1 test_18 ldiskfs only test SKIP 18 (2s) == conf-sanity-1 test 21c: start mds between two osts, stop mds last ========================================================== 04:32:38 (1734255158) start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost2 service on oleg113-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop ost2 service on oleg113-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server PASS 21c (114s) == conf-sanity-1 test 29: permanently remove an OST ====== 04:34:32 (1734255272) start ost2 service on oleg113-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Setting lustre-OST0001.osc.active from 1 to 0 Waiting 90s for '0' Updated after 2s: want '0' got '0' Live client success: got lustre-OST0001_UUID FULL DEACTIVATED osc.lustre-OST0001-osc-MDT0000.active updated after 0 sec (got 0) umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre New client success: got 'lustre-OST0001_UUID NEW DEACTIVATED' Setting lustre-OST0001.osc.active from 0 to 1 Waiting 90s for '1' Updated after 2s: want '1' got '1' umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg113-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg113-server stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server unloading modules via unload_modules_local on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local modules unloaded. oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server PASS 29 (98s) == conf-sanity-1 test 35a: Reconnect to the last active server first ========================================================== 04:36:10 (1734255370) start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre debug=ha Set up a fake failnode for the MDS Wait for RECONNECT_INTERVAL seconds (10s) conf-sanity.sh test_35a 2024-12-15 4h36m36s Stopping the MDT: lustre-MDT0000 stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server Restarting the MDT: lustre-MDT0000 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 Wait for df (23788) ... done debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout Debug log: 101 lines, 101 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server unloading modules via unload_modules_local on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local modules unloaded. oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server PASS 35a (95s) == conf-sanity-1 test 35b: Continue reconnection retries, if the active server is busy ========================================================== 04:37:45 (1734255465) start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-12-15 4h38m01s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800a9cd2800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity-1/f35b.conf-sanity-1' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-15 04:38:45.000000000 -0500 Modify: 2024-12-15 04:38:45.000000000 -0500 Change: 2024-12-15 04:38:45.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 48 lines, 48 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server unloading modules via unload_modules_local on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local modules unloaded. oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server PASS 35b (116s) == conf-sanity-1 test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 04:39:41 (1734255581) Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:) Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov -o nosvc -n lustre-mdt1/mdt1 /mnt/lustre-mds1 nomtab: 1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all Start lustre-mdt1/mdt1 without service Commit the device label on lustre-mdt1/mdt1 Started lustre:MDT0000 start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg113-server Starting mds1: -o localrecov -o nomgs,force lustre-mdt1/mdt1 /mnt/lustre-mds1 force: 1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre blah blah umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server PASS 41b (69s) == conf-sanity-1 test 46a: handle ost additional - wide striped file ========================================================== 04:40:50 (1734255650) Testing with 2 OSTs start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg113-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800ac6a9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800ac6a9000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre2..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre2 /mnt/lustre2 stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre2/f41b.conf-sanity-1 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2 0x2 0x240000400 /mnt/lustre2/widestripe lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2 0x2 0x280000400 0 66 0x42 0x240000400 File: '/mnt/lustre/widestripe' Size: 3 Blocks: 2 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826934273 Links: 1 Access: (0674/-rw-rwxr--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-15 04:41:16.000000000 -0500 Modify: 2024-12-15 04:41:16.000000000 -0500 Change: 2024-12-15 04:41:17.000000000 -0500 Birth: - umount lustre on /mnt/lustre2..... Stopping client oleg113-client.virtnet /mnt/lustre2 (opts:) umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost2 (opts:-f) on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server stop ost1 service on oleg113-server stop mds service on oleg113-server unloading modules via unload_modules_local on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local modules unloaded. oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server PASS 46a (124s) == conf-sanity-1 test 47a: server restart does not lose lru_resize ========================================================== 04:42:54 (1734255774) Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre setup single mount lustre success ldlm.namespaces.lustre-MDT0000-mdc-ffff8800ac7a1000.lru_size=100 ldlm.namespaces.lustre-OST0000-osc-ffff8800ac7a1000.lru_size=100 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800ac7a1000.lru_size=100 Failing ost1 on oleg113-server Stopping /mnt/lustre-ost1 (opts:) on oleg113-server 04:43:35 (1734255815) shut down facet: ost1 facet_host: oleg113-server facet_failover_host: oleg113-server Failover ost1 to oleg113-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 04:43:48 (1734255828) targets are mounted 04:43:48 (1734255828) facet_failover done Failing mds1 on oleg113-server Stopping /mnt/lustre-mds1 (opts:) on oleg113-server 04:43:50 (1734255830) shut down facet: mds1 facet_host: oleg113-server facet_failover_host: oleg113-server Failover mds1 to oleg113-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 04:44:02 (1734255842) targets are mounted 04:44:02 (1734255842) facet_failover done ldlm.namespaces.lustre-MDT0000-mdc-ffff8800ac7a1000.lru_size=100 umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server unloading modules via unload_modules_local on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 PASS 47a (88s) == conf-sanity-1 test 47b: client restart does not lose lru_resize ========================================================== 04:44:22 (1734255862) Failing mds1 on oleg113-server 04:44:24 (1734255864) shut down facet: mds1 facet_host: oleg113-server facet_failover_host: oleg113-server Failover mds1 to oleg113-server mount facets: mds1 Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 04:44:48 (1734255888) targets are mounted 04:44:48 (1734255888) facet_failover done pdsh@oleg113-client: oleg113-client: ssh exited with exit code 95 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid pdsh@oleg113-client: oleg113-client: ssh exited with exit code 95 Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre setup single mount lustre success ln: failed to create symbolic link '/home/green/git/lustre-release/lustre/utils/lctl': File exists SKIP: conf-sanity-1 test_47b Can not create symlink /usr/sbin/lctl SKIP 47b (75s) == conf-sanity-1 test 48: too many acls on file ========== 04:45:37 (1734255937) start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: mount.lustre: according to /etc/mtab lustre-mdt1/mdt1 is already mounted on /mnt/lustre-mds1 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 17 Start of lustre-mdt1/mdt1 on mds1 failed 17 start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg113-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 17 seq.cli-lustre-OST0000-super.width=65536 Start of lustre-ost1/ost1 on ost1 failed 17 mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre mount.lustre: according to /etc/mtab 192.168.201.113@tcp:/lustre is already mounted on /mnt/lustre setup single mount lustre success debug=0 debug=0 /mnt/lustre stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre/widestripe lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 4 0x4 0x240000400 It is expected to hold at least 4500 ACL entries File: '/mnt/lustre/widestripe' Size: 3 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272502275 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-15 04:45:43.000000000 -0500 Modify: 2024-12-15 04:45:43.000000000 -0500 Change: 2024-12-15 04:47:01.000000000 -0500 Birth: - getfacl: Removing leading '/' from absolute path names Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg113-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server PASS 48 (143s) == conf-sanity-1 test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:48:00 (1734256080) Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre setup single mount lustre success check ldlm_timout... umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server PASS 49a (44s) == conf-sanity-1 test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:48:44 (1734256124) Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server unloading modules via unload_modules_local on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 PASS 49b (42s) == conf-sanity-1 test 50c: lazystatfs one server down ==== 04:49:26 (1734256166) start mds service on oleg113-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg113-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre llite.lustre-ffff8800ac6ad000.lazystatfs=1 stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server oleg113-server: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.7015 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3328 378240 1% /mnt/lustre[MDT:0] lustre-OST0001_UUID 382976 3072 377856 1% /mnt/lustre[OST:1] filesystem_summary: 382976 3072 377856 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity-framework.sh: line 241: kill: (5722) - No such process umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg113-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server PASS 50c (97s) == conf-sanity-1 test 50d: lazystatfs client/server conn race ========================================================== 04:51:03 (1734256263) start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg113-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre llite.lustre-ffff8800b6016000.lazystatfs=1 stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.7015 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3328 378240 1% /mnt/lustre[MDT:0] lustre-OST0001_UUID 382976 3072 377856 1% /mnt/lustre[OST:1] filesystem_summary: 382976 3072 377856 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity-framework.sh: line 241: kill: (10670) - No such process umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg113-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server PASS 50d (92s) == conf-sanity-1 test 50f: normal statfs one server in down ========================================================== 04:52:35 (1734256355) start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec start ost2 service on oleg113-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg113-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg113-server oleg113-server: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre llite.lustre-ffff8801305e0800.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.7015 start ost2 service on oleg113-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0001 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg113-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg113-server umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server PASS 50f (121s) == conf-sanity-1 test 50g: deactivated OST should not cause panic ========================================================== 04:54:36 (1734256476) start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre start ost2 service on oleg113-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg113-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012d536800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012d536800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.113@tcp:/lustre 382976 3072 377856 1% /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg113-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg113-server stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server PASS 50g (80s) == conf-sanity-1 test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 04:55:56 (1734256556) Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg113-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg113-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg113-server umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server unloading modules via unload_modules_local on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local modules unloaded. oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server PASS 51 (134s) == conf-sanity-1 test 54a: test llverdev and partial verify of device ========================================================== 04:58:10 (1734256690) SKIP: conf-sanity-1 test_54a ldiskfs only test SKIP 54a (1s) == conf-sanity-1 test 55: check lov_objid size =========== 04:58:11 (1734256691) SKIP: conf-sanity-1 test_55 ldiskfs only test SKIP 55 (1s) == conf-sanity-1 test 56a: check big OST indexes and out-of-index-order start ========================================================== 04:58:13 (1734256693) Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 Permanent disk data: Target: lustre:OST2710 Index: 10000 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.201.113@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.113@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=10000 lustre:fsname=lustre lustre:svname=lustre:OST2710 Permanent disk data: Target: lustre:OST03e8 Index: 1000 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.201.113@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost2 /dev/vdf mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2/ost2 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost2/ost2 properties lustre:mgsnode=192.168.201.113@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=1000 lustre:fsname=lustre lustre:svname=lustre:OST03e8 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST2710-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST2710 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg113-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST03e8-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST03e8 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre ok OBDS: 1000: lustre-OST03e8_UUID ACTIVE 10000: lustre-OST2710_UUID ACTIVE 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00414165 s, 989 kB/s oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:) Stopping client oleg113-client.virtnet /mnt/lustre opts: Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg113-server Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server PASS 56a (119s) == conf-sanity-1 test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 05:00:13 (1734256813) SKIP: conf-sanity-1 test_56b needs >= 3 MDTs SKIP 56b (2s) == conf-sanity-1 test 57a: initial registration from failnode should fail (should return errs) ========================================================== 05:00:14 (1734256814) oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: mgsnode=192.168.201.113@tcp sys.timeout=20 autodegrade=on Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.113@tcp sys.timeout=20 autodegrade=on failover.node=192.168.201.113@tcp autodegrade=on Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.113@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:failover.node=192.168.201.113@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg113-server: mount.lustre: mount lustre-ost1/ost1 at /mnt/lustre-ost1 failed: Cannot assign requested address pdsh@oleg113-client: oleg113-server: ssh exited with exit code 99 oleg113-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory oleg113-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 Start of lustre-ost1/ost1 on ost1 failed 99 umount lustre on /mnt/lustre..... stop ost1 service on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server unloading modules via unload_modules_local on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 PASS 57a (75s) == conf-sanity-1 test 57b: initial registration from servicenode should not fail ========================================================== 05:01:29 (1734256889) oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: mgsnode=192.168.201.113@tcp autodegrade=on sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x1042 (OST update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.201.113@tcp autodegrade=on sys.timeout=20 failover.node=192.168.201.113@tcp autodegrade=on Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.113@tcp lustre:autodegrade=on lustre:sys.timeout=20 lustre:failover.node=192.168.201.113@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4162 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid umount lustre on /mnt/lustre..... stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server unloading modules via unload_modules_local on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 PASS 57b (93s) == conf-sanity-1 test 59: writeconf mount option ========= 05:03:02 (1734256982) oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server original ost count: 1 (expect > 0) oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid after mdt writeconf count: 0 (expect 0) OST start without writeconf should fail: oleg113-server: mount.lustre: mount lustre-ost1/ost1 at /mnt/lustre-ost1 failed: No such file or directory oleg113-server: Is the MGS specification correct? oleg113-server: Is the filesystem name correct? oleg113-server: If upgrading, is the copied client log valid? (see upgrade docs) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 oleg113-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory oleg113-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 OST start with writeconf should succeed: oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid after ost writeconf count: 1 (expect 1) oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid after ost2 writeconf count: 2 (expect 2) oleg113-server: oleg113-server.virtnet: executing unload_modules_local oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server PASS 59 (121s) == conf-sanity-1 test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 05:05:03 (1734257103) SKIP: conf-sanity-1 test_60a ldiskfs only test SKIP 60a (2s) == conf-sanity-1 test 62: start with disabled journal ==== 05:05:05 (1734257105) SKIP: conf-sanity-1 test_62 ldiskfs only test SKIP 62 (1s) == conf-sanity-1 test 64: check lfs df --lazy ============ 05:05:06 (1734257106) start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg113-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre stop ost2 service on oleg113-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg113-server /home/green/git/lustre-release/lustre/utils/lfs df UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3328 378240 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server unloading modules via unload_modules_local on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local modules unloaded. oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg113-server: oleg113-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg113-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server PASS 64 (127s) == conf-sanity-1 test 66: replace nids =================== 05:07:13 (1734257233) client=34615898 MDS=34615898 OSS=34615898 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre Setting lustre-OST0000.osc.active from 1 to 0 Waiting 90s for '0' Updated after 10s: want '0' got '0' replace_nids should fail if MDS, OSTs and clients are UP oleg113-server: error: replace_nids: Operation now in progress pdsh@oleg113-client: oleg113-server: ssh exited with exit code 115 umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) replace_nids should fail if MDS and OSTs are UP oleg113-server: error: replace_nids: Operation now in progress pdsh@oleg113-client: oleg113-server: ssh exited with exit code 115 stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server replace_nids should fail if MDS is UP oleg113-server: error: replace_nids: Operation now in progress pdsh@oleg113-client: oleg113-server: ssh exited with exit code 115 stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server start mds service on oleg113-server Starting mds1: -o localrecov -o nosvc lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all Start lustre-mdt1/mdt1 without service Started lustre-MDT0000 command should accept two parameters pdsh@oleg113-client: oleg113-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] correct device name should be passed oleg113-server: error: replace_nids: No such device or address pdsh@oleg113-client: oleg113-server: ssh exited with exit code 6 wrong nids list should not destroy the system pdsh@oleg113-client: oleg113-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] pdsh@oleg113-client: oleg113-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] replace OST nid command should accept two parameters pdsh@oleg113-client: oleg113-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] wrong nids list should not destroy the system pdsh@oleg113-client: oleg113-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] set NIDs with failover replace MDS nid stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Setting lustre-OST0000.osc.active from 0 to 1 Waiting 90s for '1' Updated after 5s: want '1' got '1' start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server unloading modules via unload_modules_local on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local modules unloaded. Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server PASS 66 (152s) == conf-sanity-1 test 70e: Sync-on-Cancel will be enabled by default on DNE ========================================================== 05:09:46 (1734257386) SKIP: conf-sanity-1 test_70e needs >= 2 MDTs SKIP 70e (1s) == conf-sanity-1 test 75: The order of --index should be irrelevant ========================================================== 05:09:47 (1734257387) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions Permanent disk data: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x45 (MDT MGS update ) Persistent mount opts: Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mkfs_cmd = zpool create -f -O canmount=off lustre-mdt1 /dev/vdc mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt1/mdt1 xattr=sa dnodesize=auto Writing lustre-mdt1/mdt1 properties lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=69 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-MDT0000 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.113@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.113@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 Permanent disk data: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x45 (MDT MGS update ) Persistent mount opts: Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mkfs_cmd = zpool create -f -O canmount=off lustre-mdt1 /dev/vdc mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt1/mdt1 xattr=sa dnodesize=auto Writing lustre-mdt1/mdt1 properties lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=69 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-MDT0000 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.113@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.113@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server PASS 75 (63s) == conf-sanity-1 test 78: run resize2fs on MDT and OST filesystems ========================================================== 05:10:50 (1734257450) SKIP: conf-sanity-1 test_78 ldiskfs only test SKIP 78 (2s) == conf-sanity-1 test 79: format MDT/OST without mgs option (should return errors) ========================================================== 05:10:52 (1734257452) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: oleg113-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode oleg113-server: mkfs.lustre: exiting with 22 (Invalid argument) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: oleg113-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode oleg113-server: mkfs.lustre: exiting with 22 (Invalid argument) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: oleg113-server: mkfs.lustre FATAL: Must specify --mgsnode oleg113-server: mkfs.lustre: exiting with 22 (Invalid argument) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server PASS 79 (58s) == conf-sanity-1 test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 05:11:50 (1734257510) SKIP: conf-sanity-1 test_83 ldiskfs only test SKIP 83 (1s) == conf-sanity-1 test 86: Replacing mkfs.lustre -G option ========================================================== 05:11:51 (1734257511) SKIP: conf-sanity-1 test_86 LU-6442: no such mkfs params for ZFS OSTs SKIP 86 (1s) == conf-sanity-1 test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 05:11:53 (1734257513) SKIP: conf-sanity-1 test_87 ldiskfs only test SKIP 87 (1s) == conf-sanity-1 test 89: check tunefs --param and --erase-param{s} options ========================================================== 05:11:54 (1734257514) Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:) tunefs --param failover.node=192.0.2.254@tcp0 tunefs --param failover.node=192.0.2.255@tcp0 tunefs --erase-param failover.node tunefs --erase-params tunefs --param failover.node=192.0.2.254@tcp0 --erase-params Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server PASS 89 (48s) == conf-sanity-1 test 93: register mulitple MDT at the same time ========================================================== 05:12:42 (1734257562) SKIP: conf-sanity-1 test_93 needs >= 3 MDTs SKIP 93 (1s) == conf-sanity-1 test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 05:12:43 (1734257563) Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre setup single mount lustre success lustre-MDT0000: lustre-OST0000: umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server unloading modules via unload_modules_local on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 PASS 100 (55s) == conf-sanity-1 test 104a: Make sure user defined options are reflected in mount ========================================================== 05:13:38 (1734257618) mountfsopt: acl,user_xattr Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 Starting mds1: -o localrecov,noacl lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre setfacl: /mnt/lustre: Operation not supported PASS 104a (44s) == conf-sanity-1 test 105: check file creation for ro and rw bind mnt pt ========================================================== 05:14:22 (1734257662) umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server unloading modules via unload_modules_local on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local oleg113-server: rmmod: ERROR: Module lustre is in use pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 modules unloaded. Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-ost2 (opts:-f) on oleg113-server oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre touch: cannot touch '/tmp/d105.conf-sanity-1/f105.conf-sanity-1': Read-only file system umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server unloading modules via unload_modules_local on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 PASS 105 (93s) == conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:15:55 (1734257755) Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2218 (time 1734257805.87 total 10.00 last 221.74) - open/close 4403 (time 1734257815.87 total 20.00 last 218.48) - open/close 6820 (time 1734257825.88 total 30.01 last 241.63) - open/close 9071 (time 1734257835.88 total 40.01 last 225.05) - open/close 10000 (time 1734257840.18 total 44.30 last 216.26) - open/close 12057 (time 1734257850.18 total 54.31 last 205.60) - open/close 13819 (time 1734257860.18 total 64.31 last 176.17) - open/close 15882 (time 1734257870.18 total 74.31 last 206.25) - open/close 17678 (time 1734257880.18 total 84.31 last 179.60) - open/close 19755 (time 1734257890.19 total 94.32 last 207.65) - open/close 22683 (time 1734257900.19 total 104.32 last 292.73) - open/close 24874 (time 1734257910.19 total 114.32 last 219.04) - open/close 26672 (time 1734257920.20 total 124.33 last 179.70) - open/close 28317 (time 1734257930.20 total 134.33 last 164.43) - open/close 30000 (time 1734257939.53 total 143.66 last 180.43) - open/close 32424 (time 1734257949.53 total 153.66 last 242.35) - open/close 34381 (time 1734257959.54 total 163.67 last 195.60) - open/close 36359 (time 1734257969.54 total 173.67 last 197.72) - open/close 38069 (time 1734257979.54 total 183.67 last 170.95) - open/close 39721 (time 1734257989.55 total 193.68 last 165.14) - open/close 41192 (time 1734257999.55 total 203.68 last 147.08) - open/close 42716 (time 1734258009.55 total 213.68 last 152.36) - open/close 44180 (time 1734258019.55 total 223.68 last 146.38) - open/close 45869 (time 1734258029.56 total 233.69 last 168.71) - open/close 47265 (time 1734258039.57 total 243.70 last 139.55) - open/close 48510 (time 1734258049.57 total 253.70 last 124.42) - open/close 49750 (time 1734258059.57 total 263.70 last 123.99) - open/close 51073 (time 1734258069.58 total 273.71 last 132.20) - open/close 52253 (time 1734258079.58 total 283.71 last 117.99) - open/close 53600 (time 1734258089.58 total 293.71 last 134.69) - open/close 55193 (time 1734258099.59 total 303.72 last 159.24) - open/close 56619 (time 1734258109.59 total 313.72 last 142.49) - open/close 58038 (time 1734258119.61 total 323.74 last 141.72) - open/close 59545 (time 1734258129.61 total 333.74 last 150.63) - open/close 60000 (time 1734258132.76 total 336.89 last 144.32) - open/close 61539 (time 1734258142.77 total 346.90 last 153.86) - open/close 63000 (time 1734258152.77 total 356.90 last 146.08) - open/close 64457 (time 1734258162.77 total 366.90 last 145.67) total: 64768 open/close in 368.95 seconds: 175.55 ops/second - /unlink 3005 (time 1734258176.56 total 10.00 last 300.49) - /unlink 5827 (time 1734258186.56 total 20.00 last 282.15) - /unlink 8546 (time 1734258196.56 total 30.00 last 271.85) - /unlink 10000 (time 1734258201.07 total 34.51 last 322.88) - /unlink 12976 (time 1734258211.07 total 44.51 last 297.50) - /unlink 15612 (time 1734258221.07 total 54.51 last 263.49) - /unlink 17775 (time 1734258231.07 total 64.52 last 216.26) - /unlink 19924 (time 1734258241.08 total 74.52 last 214.87) - /unlink 21910 (time 1734258251.08 total 84.52 last 198.54) - /unlink 23929 (time 1734258261.08 total 94.52 last 201.88) - /unlink 26079 (time 1734258271.09 total 104.53 last 214.77) - /unlink 28171 (time 1734258281.09 total 114.53 last 209.16) - /unlink 29999 (time 1734258291.10 total 124.54 last 182.70) - /unlink 32047 (time 1734258301.10 total 134.54 last 204.78) - /unlink 34111 (time 1734258311.10 total 144.54 last 206.32) - /unlink 36046 (time 1734258321.11 total 154.55 last 193.43) - /unlink 37934 (time 1734258331.11 total 164.55 last 188.69) - /unlink 39928 (time 1734258341.11 total 174.56 last 199.39) - /unlink 42257 (time 1734258351.12 total 184.56 last 232.84) - /unlink 44889 (time 1734258361.12 total 194.56 last 263.15) - /unlink 47496 (time 1734258371.12 total 204.56 last 260.64) - /unlink 49807 (time 1734258381.12 total 214.57 last 231.01) - /unlink 52138 (time 1734258391.13 total 224.57 last 233.05) - /unlink 54437 (time 1734258401.13 total 234.57 last 229.84) - /unlink 56697 (time 1734258411.13 total 244.57 last 225.96) - /unlink 58970 (time 1734258421.13 total 254.57 last 227.26) - /unlink 60000 (time 1734258425.36 total 258.80 last 243.73) - /unlink 62408 (time 1734258435.36 total 268.80 last 240.80) total: 64768 /unlink in 277.40 seconds: 233.49 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1734 (time 1734258470.56 total 10.00 last 173.37) - open/close 3658 (time 1734258480.56 total 20.01 last 192.34) - open/close 5673 (time 1734258490.57 total 30.01 last 201.42) - open/close 7727 (time 1734258500.59 total 40.03 last 205.00) - open/close 9459 (time 1734258510.59 total 50.03 last 173.11) - open/close 10000 (time 1734258513.74 total 53.18 last 172.09) - open/close 11642 (time 1734258523.74 total 63.18 last 164.16) - open/close 13239 (time 1734258533.75 total 73.19 last 159.59) - open/close 14646 (time 1734258543.75 total 83.19 last 140.68) - open/close 16131 (time 1734258553.75 total 93.19 last 148.44) - open/close 17567 (time 1734258563.76 total 103.20 last 143.48) - open/close 19035 (time 1734258573.76 total 113.20 last 146.76) - open/close 20000 (time 1734258580.71 total 120.15 last 138.88) - open/close 21349 (time 1734258590.72 total 130.16 last 134.83) - open/close 22860 (time 1734258600.72 total 140.16 last 151.03) - open/close 24290 (time 1734258610.73 total 150.17 last 142.90) - open/close 25742 (time 1734258620.73 total 160.17 last 145.20) - open/close 27203 (time 1734258630.73 total 170.17 last 146.08) - open/close 28584 (time 1734258640.73 total 180.17 last 138.10) - open/close 30000 (time 1734258650.37 total 189.81 last 146.94) - open/close 31442 (time 1734258660.38 total 199.82 last 144.06) - open/close 32995 (time 1734258670.38 total 209.82 last 155.30) - open/close 34467 (time 1734258680.38 total 219.82 last 147.12) - open/close 36181 (time 1734258690.38 total 229.82 last 171.37) - open/close 37793 (time 1734258700.38 total 239.83 last 161.20) - open/close 39359 (time 1734258710.39 total 249.83 last 156.49) - open/close 40000 (time 1734258714.77 total 254.21 last 146.57) - open/close 41393 (time 1734258724.77 total 264.21 last 139.23) - open/close 42736 (time 1734258734.77 total 274.21 last 134.25) - open/close 44130 (time 1734258744.78 total 284.22 last 139.29) - open/close 45585 (time 1734258754.79 total 294.23 last 145.43) - open/close 46997 (time 1734258764.79 total 304.23 last 141.14) - open/close 48456 (time 1734258774.79 total 314.23 last 145.87) - open/close 50000 (time 1734258784.54 total 323.98 last 158.45) - open/close 51722 (time 1734258794.54 total 333.98 last 172.12) - open/close 53469 (time 1734258804.55 total 343.99 last 174.65) - open/close 55016 (time 1734258814.55 total 353.99 last 154.66) - open/close 56717 (time 1734258824.56 total 364.00 last 169.99) - open/close 58486 (time 1734258834.56 total 374.00 last 176.81) - open/close 60000 (time 1734258843.15 total 382.59 last 176.24) - open/close 61667 (time 1734258853.15 total 392.59 last 166.64) - open/close 63209 (time 1734258863.16 total 402.60 last 154.08) total: 64768 open/close in 411.97 seconds: 157.21 ops/second - /unlink 2901 (time 1734258883.95 total 10.00 last 290.07) - /unlink 5730 (time 1734258893.95 total 20.00 last 282.87) - /unlink 8606 (time 1734258903.96 total 30.00 last 287.52) - /unlink 10000 (time 1734258908.92 total 34.97 last 280.67) - /unlink 12882 (time 1734258918.92 total 44.97 last 288.18) - /unlink 15992 (time 1734258928.93 total 54.97 last 310.97) - /unlink 19389 (time 1734258938.93 total 64.97 last 339.64) - /unlink 22845 (time 1734258948.93 total 74.98 last 345.56) - /unlink 26147 (time 1734258958.93 total 84.98 last 330.13) - /unlink 28935 (time 1734258968.93 total 94.98 last 278.79) - /unlink 30000 (time 1734258972.77 total 98.82 last 277.32) - /unlink 32752 (time 1734258982.77 total 108.82 last 275.15) - /unlink 35572 (time 1734258992.78 total 118.82 last 281.94) - /unlink 38389 (time 1734259002.78 total 128.82 last 281.69) - /unlink 40000 (time 1734259008.60 total 134.65 last 276.53) - /unlink 43013 (time 1734259018.60 total 144.65 last 301.22) - /unlink 46130 (time 1734259028.61 total 154.66 last 311.55) - /unlink 48788 (time 1734259038.61 total 164.66 last 265.73) - /unlink 50000 (time 1734259042.77 total 168.81 last 291.64) - /unlink 52639 (time 1734259052.77 total 178.81 last 263.90) - /unlink 55478 (time 1734259062.77 total 188.82 last 283.87) - /unlink 58062 (time 1734259072.77 total 198.82 last 258.38) - /unlink 60000 (time 1734259079.76 total 205.81 last 277.18) - /unlink 63118 (time 1734259089.76 total 215.81 last 311.79) total: 64768 /unlink in 221.85 seconds: 291.95 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1475 (time 1734259125.83 total 10.01 last 147.40) - open/close 2885 (time 1734259135.83 total 20.01 last 140.90) - open/close 4400 (time 1734259145.84 total 30.02 last 151.35) - open/close 5789 (time 1734259155.84 total 40.02 last 138.89) - open/close 7208 (time 1734259165.85 total 50.03 last 141.79) - open/close 8723 (time 1734259175.86 total 60.04 last 151.45) - open/close 10000 (time 1734259184.54 total 68.72 last 147.10) - open/close 11513 (time 1734259194.54 total 78.72 last 151.27) - open/close 13001 (time 1734259204.54 total 88.72 last 148.72) - open/close 14469 (time 1734259214.55 total 98.73 last 146.70) - open/close 15908 (time 1734259224.56 total 108.74 last 143.80) - open/close 17411 (time 1734259234.56 total 118.74 last 150.22) - open/close 18865 (time 1734259244.57 total 128.75 last 145.36) - open/close 20000 (time 1734259251.67 total 135.85 last 159.80) - open/close 21637 (time 1734259261.67 total 145.85 last 163.64) - open/close 23245 (time 1734259271.68 total 155.86 last 160.72) - open/close 25100 (time 1734259281.68 total 165.86 last 185.41) - open/close 26795 (time 1734259291.68 total 175.86 last 169.49) - open/close 28355 (time 1734259301.68 total 185.86 last 155.98) - open/close 29987 (time 1734259311.68 total 195.86 last 163.20) - open/close 31506 (time 1734259321.69 total 205.87 last 151.84) - open/close 33037 (time 1734259331.69 total 215.87 last 153.04) - open/close 34336 (time 1734259341.70 total 225.88 last 129.85) - open/close 35932 (time 1734259351.70 total 235.88 last 159.54) - open/close 37487 (time 1734259361.70 total 245.88 last 155.46) - open/close 38884 (time 1734259371.71 total 255.89 last 139.63) - open/close 40000 (time 1734259378.74 total 262.92 last 158.72) - open/close 41815 (time 1734259388.74 total 272.92 last 181.42) - open/close 43551 (time 1734259398.75 total 282.93 last 173.53) - open/close 45285 (time 1734259408.75 total 292.93 last 173.40) - open/close 47099 (time 1734259418.75 total 302.93 last 181.30) - open/close 48984 (time 1734259428.76 total 312.94 last 188.43) - open/close 50000 (time 1734259434.34 total 318.52 last 182.11) - open/close 51729 (time 1734259444.34 total 328.52 last 172.84) - open/close 53464 (time 1734259454.34 total 338.52 last 173.46) - open/close 55411 (time 1734259464.34 total 348.52 last 194.70) - open/close 57467 (time 1734259474.34 total 358.52 last 205.56) - open/close 59538 (time 1734259484.35 total 368.53 last 207.04) - open/close 60000 (time 1734259486.66 total 370.84 last 199.72) - open/close 62235 (time 1734259496.66 total 380.84 last 223.39) - open/close 64327 (time 1734259506.67 total 390.85 last 209.16) total: 64768 open/close in 393.14 seconds: 164.74 ops/second - /unlink 3821 (time 1734259520.03 total 10.00 last 382.07) - /unlink 7731 (time 1734259530.03 total 20.00 last 390.95) - /unlink 10000 (time 1734259536.63 total 26.60 last 344.06) - /unlink 13088 (time 1734259546.63 total 36.60 last 308.63) - /unlink 15751 (time 1734259556.63 total 46.61 last 266.22) - /unlink 18673 (time 1734259566.64 total 56.61 last 292.08) - /unlink 20000 (time 1734259571.15 total 61.12 last 293.93) - /unlink 23110 (time 1734259581.16 total 71.13 last 310.88) - /unlink 25938 (time 1734259591.16 total 81.13 last 282.79) - /unlink 28982 (time 1734259601.16 total 91.13 last 304.31) - /unlink 30000 (time 1734259604.34 total 94.31 last 320.57) - /unlink 33491 (time 1734259614.34 total 104.31 last 349.10) - /unlink 36757 (time 1734259624.34 total 114.31 last 326.59) - /unlink 39820 (time 1734259634.34 total 124.31 last 306.13) - /unlink 43130 (time 1734259644.34 total 134.31 last 330.94) - /unlink 46451 (time 1734259654.34 total 144.32 last 332.07) - /unlink 49629 (time 1734259664.35 total 154.32 last 317.77) - /unlink 52955 (time 1734259674.35 total 164.32 last 332.52) - /unlink 56284 (time 1734259684.35 total 174.32 last 332.72) - /unlink 60000 (time 1734259694.23 total 184.20 last 376.34) - /unlink 63258 (time 1734259704.23 total 194.20 last 325.79) total: 64768 /unlink in 198.58 seconds: 326.16 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1741 (time 1734259736.52 total 10.00 last 174.06) - open/close 3752 (time 1734259746.53 total 20.00 last 201.07) - open/close 5644 (time 1734259756.53 total 30.01 last 189.14) - open/close 7453 (time 1734259766.53 total 40.01 last 180.79) - open/close 9521 (time 1734259776.54 total 50.01 last 206.79) - open/close 11693 (time 1734259786.54 total 60.02 last 217.07) - open/close 13991 (time 1734259796.54 total 70.02 last 229.78) - open/close 16007 (time 1734259806.54 total 80.02 last 201.58) - open/close 18245 (time 1734259816.54 total 90.02 last 223.80) - open/close 20000 (time 1734259825.40 total 98.87 last 198.24) - open/close 21830 (time 1734259835.40 total 108.88 last 182.93) - open/close 23615 (time 1734259845.40 total 118.88 last 178.43) - open/close 25583 (time 1734259855.41 total 128.88 last 196.76) - open/close 27338 (time 1734259865.41 total 138.89 last 175.44) - open/close 29093 (time 1734259875.41 total 148.89 last 175.42) - open/close 30000 (time 1734259880.53 total 154.01 last 177.26) - open/close 31894 (time 1734259890.54 total 164.01 last 189.30) - open/close 33797 (time 1734259900.54 total 174.02 last 190.16) - open/close 35929 (time 1734259910.54 total 184.02 last 213.19) - open/close 38171 (time 1734259920.54 total 194.02 last 224.19) - open/close 40000 (time 1734259928.95 total 202.43 last 217.66) - open/close 42197 (time 1734259938.96 total 212.44 last 219.46) - open/close 44443 (time 1734259948.96 total 222.44 last 224.55) - open/close 46824 (time 1734259958.96 total 232.44 last 238.07) - open/close 49126 (time 1734259968.96 total 242.44 last 230.14) - open/close 50000 (time 1734259973.12 total 246.60 last 210.06) - open/close 52275 (time 1734259983.13 total 256.60 last 227.49) - open/close 54584 (time 1734259993.13 total 266.60 last 230.90) - open/close 56978 (time 1734260003.13 total 276.60 last 239.37) - open/close 59379 (time 1734260013.13 total 286.61 last 240.02) - open/close 60000 (time 1734260015.64 total 289.12 last 247.64) - open/close 62339 (time 1734260025.64 total 299.12 last 233.83) - open/close 64687 (time 1734260035.64 total 309.12 last 234.75) total: 64768 open/close in 309.43 seconds: 209.32 ops/second - /unlink 4123 (time 1734260047.10 total 10.00 last 412.21) - /unlink 8166 (time 1734260057.10 total 20.00 last 404.23) - /unlink 10000 (time 1734260061.56 total 24.47 last 410.67) - /unlink 13387 (time 1734260071.56 total 34.47 last 338.70) - /unlink 16911 (time 1734260081.57 total 44.47 last 352.31) - /unlink 20000 (time 1734260089.65 total 52.55 last 382.20) - /unlink 23754 (time 1734260099.65 total 62.56 last 375.35) - /unlink 27568 (time 1734260109.65 total 72.56 last 381.40) - /unlink 30000 (time 1734260115.94 total 78.85 last 386.61) - /unlink 33948 (time 1734260125.94 total 88.85 last 394.78) - /unlink 38129 (time 1734260135.94 total 98.85 last 418.08) - /unlink 40000 (time 1734260140.05 total 102.95 last 455.93) - /unlink 44220 (time 1734260150.05 total 112.95 last 421.97) - /unlink 49420 (time 1734260160.05 total 122.95 last 519.98) - /unlink 53993 (time 1734260170.05 total 132.95 last 457.25) - /unlink 59428 (time 1734260180.05 total 142.95 last 543.45) total: 64768 /unlink in 151.76 seconds: 426.77 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3288 (time 1734260217.15 total 10.00 last 328.76) - open/close 6719 (time 1734260227.15 total 20.00 last 343.09) - open/close 10000 (time 1734260236.89 total 29.74 last 336.84) - open/close 12966 (time 1734260246.90 total 39.75 last 296.50) - open/close 16140 (time 1734260256.90 total 49.75 last 317.37) - open/close 19384 (time 1734260266.90 total 59.75 last 324.36) - open/close 22702 (time 1734260276.90 total 69.75 last 331.76) - open/close 26367 (time 1734260286.90 total 79.75 last 366.43) - open/close 29716 (time 1734260296.90 total 89.75 last 334.88) - open/close 32626 (time 1734260306.90 total 99.75 last 290.98) - open/close 35325 (time 1734260316.91 total 109.76 last 269.81) - open/close 37881 (time 1734260326.91 total 119.76 last 255.55) - open/close 40000 (time 1734260334.01 total 126.86 last 298.18) - open/close 42578 (time 1734260344.02 total 136.87 last 257.75) - open/close 45592 (time 1734260354.02 total 146.87 last 301.33) - open/close 49296 (time 1734260364.02 total 156.87 last 370.34) - open/close 50000 (time 1734260366.15 total 159.00 last 330.98) - open/close 53027 (time 1734260376.15 total 169.00 last 302.65) - open/close 56114 (time 1734260386.15 total 179.00 last 308.64) - open/close 59250 (time 1734260396.15 total 189.00 last 313.59) - open/close 60000 (time 1734260398.31 total 191.16 last 347.66) - open/close 63380 (time 1734260408.31 total 201.16 last 337.93) total: 64768 open/close in 205.34 seconds: 315.42 ops/second - /unlink 4950 (time 1734260423.47 total 10.00 last 494.92) - /unlink 10000 (time 1734260432.48 total 19.00 last 560.93) - /unlink 14695 (time 1734260442.48 total 29.00 last 469.49) - /unlink 19120 (time 1734260452.48 total 39.00 last 442.50) - /unlink 24017 (time 1734260462.48 total 49.01 last 489.69) - /unlink 28694 (time 1734260472.48 total 59.01 last 467.60) - /unlink 30000 (time 1734260475.97 total 62.50 last 374.32) - /unlink 34812 (time 1734260485.97 total 72.50 last 481.10) - /unlink 39216 (time 1734260495.97 total 82.50 last 440.37) - /unlink 44182 (time 1734260505.97 total 92.50 last 496.57) - /unlink 49796 (time 1734260515.97 total 102.50 last 561.31) - /unlink 56014 (time 1734260525.98 total 112.50 last 621.72) - /unlink 60000 (time 1734260532.16 total 118.69 last 644.37) total: 64768 /unlink in 126.51 seconds: 511.97 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server unloading modules via unload_modules_local on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 PASS 106 (2817s) == conf-sanity-1 test 109a: test lctl clear_conf fsname == 06:02:52 (1734260572) SKIP: conf-sanity-1 test_109a LU-8727: no implementation for ZFS SKIP 109a (1s) == conf-sanity-1 test 109b: test lctl clear_conf one config ========================================================== 06:02:53 (1734260573) SKIP: conf-sanity-1 test_109b LU-8727: no implementation for ZFS SKIP 109b (0s) == conf-sanity-1 test 116: big size MDT support ========== 06:02:53 (1734260573) SKIP: conf-sanity-1 test_116 ldiskfs only test SKIP 116 (1s) == conf-sanity-1 test 122a: Check OST sequence update ==== 06:02:54 (1734260574) SKIP: conf-sanity-1 test_122a needs >= 2 MDTs SKIP 122a (1s) == conf-sanity-1 test 123ah: del_ost cancels config log entries correctly ========================================================== 06:02:55 (1734260575) Checking servers environments Checking clients oleg113-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' Setup mgs, mdt, osts Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre Starting client oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre Started clients oleg113-client.virtnet: 192.168.201.113@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012ab1d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012ab1d000.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 4s: want 'procname_uid' got 'procname_uid' disable quota as required del_ost: dry run for target lustre-OST0000 config_log: lustre-client [DRY RUN] cancel catalog 'lustre-client:25':"- { index: 25, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }" [DRY RUN] cancel catalog 'lustre-client:24':"- { index: 24, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.201.113@tcp }" [DRY RUN] cancel catalog 'lustre-client:23':"- { index: 23, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID }" del_ost: no catalog entry deleted config_log: lustre-MDT0000 [DRY RUN] cancel catalog 'lustre-MDT0000:20':"- { index: 20, event: add_osc, device: lustre-MDT0000-mdtlov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }" [DRY RUN] cancel catalog 'lustre-MDT0000:19':"- { index: 19, event: setup, device: lustre-OST0000-osc-MDT0000, UUID: lustre-OST0000_UUID, node: 192.168.201.113@tcp }" [DRY RUN] cancel catalog 'lustre-MDT0000:18':"- { index: 18, event: attach, device: lustre-OST0000-osc-MDT0000, type: osc, UUID: lustre-MDT0000-mdtlov_UUID }" del_ost: no catalog entry deleted config_log: lustre-client cancel catalog lustre-client log_idx 25: done cancel catalog lustre-client log_idx 24: done cancel catalog lustre-client log_idx 23: done del_ost: cancelled 3 catalog entries config_log: lustre-MDT0000 cancel catalog lustre-MDT0000 log_idx 20: done cancel catalog lustre-MDT0000 log_idx 19: done cancel catalog lustre-MDT0000 log_idx 18: done del_ost: cancelled 3 catalog entries umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server unloading modules via unload_modules_local on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local oleg113-server: rmmod: ERROR: Module lustre is in use pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 modules unloaded. Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-ost2 (opts:-f) on oleg113-server oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server fail_loc=0 PASS 123ah (100s) == conf-sanity-1 test 132: hsm_actions processed after failover ========================================================== 06:04:35 (1734260675) Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x5 (MDT MGS ) Persistent mount opts: Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20 Permanent disk data: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x45 (MDT MGS update ) Persistent mount opts: Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20 mdt.hsm_control=enabled Writing lustre-mdt1/mdt1 properties lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:sys.timeout=20 lustre:mdt.hsm_control=enabled lustre:version=1 lustre:flags=69 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-MDT0000 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server PASS 132 (26s) == conf-sanity-1 test 133: stripe QOS: free space balance in a pool ========================================================== 06:05:01 (1734260701) SKIP: conf-sanity-1 test_133 needs >= 4 OSTs SKIP 133 (1s) == conf-sanity-1 test 135: check the behavior when changelog is wrapped around ========================================================== 06:05:02 (1734260702) Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre oleg113-client: fail_loc=0x1312 oleg113-client: fail_val=5 oleg113-server: fail_loc=0x1312 oleg113-server: fail_val=5 mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl1' Wrap arround changelog catalog - open/close 3357 (time 1734260741.26 total 10.00 last 335.62) total: 4500 open/close in 13.40 seconds: 335.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #12998 total: 4500 /unlink in 6.31 seconds: 713.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 4055 (time 1734260763.29 total 10.00 last 405.46) total: 4500 open/close in 11.05 seconds: 407.36 ops/second total: 4500 /unlink in 5.12 seconds: 878.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 3903 (time 1734260781.21 total 10.00 last 390.23) total: 4500 open/close in 11.44 seconds: 393.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 total: 4500 /unlink in 5.54 seconds: 811.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 total: 4500 open/close in 9.74 seconds: 462.12 ops/second total: 4500 /unlink in 4.19 seconds: 1073.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 total: 4500 open/close in 8.70 seconds: 516.97 ops/second total: 4500 /unlink in 4.65 seconds: 968.03 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 10973: 22700 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server unloading modules via unload_modules_local on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local modules unloaded. PASS 135 (141s) == conf-sanity-1 test 136: don't panic with bad obdecho setup ========================================================== 06:07:23 (1734260843) SKIP: conf-sanity-1 test_136 needs >= 2 MDTs SKIP 136 (1s) == conf-sanity-1 test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:07:24 (1734260844) SKIP: conf-sanity-1 test_137 needs >= 3 MDTs SKIP 137 (1s) == conf-sanity-1 test 140: remove_updatelog script actions ========================================================== 06:07:25 (1734260845) SKIP: conf-sanity-1 test_140 needs >= 2 MDTs SKIP 140 (1s) == conf-sanity-1 test 151a: damaged local config doesn't prevent mounting ========================================================== 06:07:26 (1734260846) SKIP: conf-sanity-1 test_151a ldiskfs only test SKIP 151a (0s) == conf-sanity-1 test 151b: -ENOSPC doesn't affect mount ========================================================== 06:07:26 (1734260846) SKIP: conf-sanity-1 test_151b ldiskfs only test SKIP 151b (1s) == conf-sanity-1 test 153a: bypass invalid NIDs quickly == 06:07:27 (1734260847) Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid waiting for mount ... waiting for mount ... "192.168.201.113@tcp": { connects: 1, replied: 1, uptodate: uptodate, sec_ago: 5 } "192.168.252.112@tcp": { connects: 1, replied: 0, uptodate: discovering, sec_ago: 10 } "10.252.252.113@tcp": { connects: 0, replied: 0, uptodate: discovering, sec_ago: never } "192.168.201.113@tcp": { connects: 0, replied: 0, uptodate: discovering, sec_ago: never } setup single mount lustre success umount lustre on /mnt/lustre..... stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server unloading modules via unload_modules_local on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 PASS 153a (178s) == conf-sanity-1 test 153b: added IPv6 NID support ======= 06:10:25 (1734261025) Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server Checking servers environments Checking clients oleg113-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Setup mgs, mdt, osts Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre Starting client oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre Started clients oleg113-client.virtnet: 192.168.201.113@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b604a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b604a000.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 3s: want 'procname_uid' got 'procname_uid' disable quota as required PASS 153b (72s) == conf-sanity-1 test 153c: don't stuck on unreached NID ========================================================== 06:11:38 (1734261098) Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg113-server oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server start mds service on oleg113-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid umount lustre on /mnt/lustre..... waiting for mount ... waiting for mount ... waiting for mount ... waiting for mount ... waiting for mount ... waiting for mount ... "192.168.252.112@tcp": { connects: 3, replied: 0, uptodate: discovering, sec_ago: 0 } "192.168.252.113@tcp": { connects: 2, replied: 0, uptodate: discovering, sec_ago: 10 } "192.168.252.115@tcp": { connects: 2, replied: 0, uptodate: discovering, sec_ago: 5 } connection attempts: 7 Waiting for mount to fail mount.lustre: mount 192.168.252.112@tcp:192.168.252.113@tcp:192.168.252.115@tcp:/lustre at /mnt/lustre failed: Input/output error Is the MGS running? umount lustre on /mnt/lustre..... stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server unloading modules via unload_modules_local on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 PASS 153c (318s) == conf-sanity-1 test 154: expand .. on rename after MDT backup restore ========================================================== 06:16:56 (1734261416) SKIP: conf-sanity-1 test_154 ldiskfs only test SKIP 154 (0s) Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity-1 test complete, duration 6339 sec ======== 06:17:14 (1734261434) === conf-sanity-1: start cleanup 06:17:14 (1734261434) === === conf-sanity-1: finish cleanup 06:17:15 (1734261435) ===