-----============= acceptance-small: conf-sanity-1 ============----- Sun Dec 15 04:31:36 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg245-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity-1.*ex: No such file or directory excepting tests: skipping tests SLOW=no: 106 Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg245-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg245-server oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_size: failed to read coresize from /sys/module/acpi_cpufreq libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server == conf-sanity-1 test 17: Verify failed mds_postsetup won't fail assertion (2936) (should return errs) ========================================================== 04:32:52 (1734255172) SKIP: conf-sanity-1 test_17 ldiskfs only test SKIP 17 (1s) == conf-sanity-1 test 18: check mkfs creates large journals ========================================================== 04:32:53 (1734255173) SKIP: conf-sanity-1 test_18 ldiskfs only test SKIP 18 (2s) == conf-sanity-1 test 21c: start mds between two osts, stop mds last ========================================================== 04:32:55 (1734255175) start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost2 service on oleg245-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop ost2 service on oleg245-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server PASS 21c (132s) == conf-sanity-1 test 29: permanently remove an OST ====== 04:35:07 (1734255307) start ost2 service on oleg245-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Setting lustre-OST0001.osc.active from 1 to 0 Waiting 90s for '0' Updated after 2s: want '0' got '0' Live client success: got lustre-OST0001_UUID FULL DEACTIVATED osc.lustre-OST0001-osc-MDT0000.active updated after 0 sec (got 0) umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre New client success: got 'lustre-OST0001_UUID NEW DEACTIVATED' Setting lustre-OST0001.osc.active from 0 to 1 Waiting 90s for '1' Updated after 2s: want '1' got '1' umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg245-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg245-server stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local modules unloaded. oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server PASS 29 (100s) == conf-sanity-1 test 35a: Reconnect to the last active server first ========================================================== 04:36:47 (1734255407) start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre debug=ha Set up a fake failnode for the MDS Wait for RECONNECT_INTERVAL seconds (10s) conf-sanity.sh test_35a 2024-12-15 4h37m15s Stopping the MDT: lustre-MDT0000 stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server Restarting the MDT: lustre-MDT0000 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 Wait for df (23834) ... done debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout Debug log: 72 lines, 72 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local modules unloaded. oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server PASS 35a (99s) == conf-sanity-1 test 35b: Continue reconnection retries, if the active server is busy ========================================================== 04:38:26 (1734255506) start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-12-15 4h38m44s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff88012b2f0800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity-1/f35b.conf-sanity-1' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-15 04:39:27.000000000 -0500 Modify: 2024-12-15 04:39:27.000000000 -0500 Change: 2024-12-15 04:39:27.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 48 lines, 48 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local modules unloaded. oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server PASS 35b (135s) == conf-sanity-1 test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 04:40:41 (1734255641) Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:) Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov -o nosvc -n lustre-mdt1/mdt1 /mnt/lustre-mds1 nomtab: 1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all Start lustre-mdt1/mdt1 without service Commit the device label on lustre-mdt1/mdt1 Started lustre:MDT0000 start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg245-server Starting mds1: -o localrecov -o nomgs,force lustre-mdt1/mdt1 /mnt/lustre-mds1 force: 1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre blah blah umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server PASS 41b (75s) == conf-sanity-1 test 46a: handle ost additional - wide striped file ========================================================== 04:41:56 (1734255716) Testing with 2 OSTs start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg245-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800aa4be000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800aa4be000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre2..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre2 /mnt/lustre2 stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre2/f41b.conf-sanity-1 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2 0x2 0x240000400 /mnt/lustre2/widestripe lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2 0x2 0x280000400 0 66 0x42 0x240000400 File: '/mnt/lustre/widestripe' Size: 3 Blocks: 2 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826934273 Links: 1 Access: (0674/-rw-rwxr--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-15 04:42:26.000000000 -0500 Modify: 2024-12-15 04:42:26.000000000 -0500 Change: 2024-12-15 04:42:26.000000000 -0500 Birth: - umount lustre on /mnt/lustre2..... Stopping client oleg245-client.virtnet /mnt/lustre2 (opts:) umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost2 (opts:-f) on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server stop ost1 service on oleg245-server stop mds service on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local modules unloaded. oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server PASS 46a (126s) == conf-sanity-1 test 47a: server restart does not lose lru_resize ========================================================== 04:44:02 (1734255842) Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre setup single mount lustre success ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b589c000.lru_size=100 ldlm.namespaces.lustre-OST0000-osc-ffff8800b589c000.lru_size=100 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b589c000.lru_size=100 Failing ost1 on oleg245-server Stopping /mnt/lustre-ost1 (opts:) on oleg245-server 04:44:42 (1734255882) shut down facet: ost1 facet_host: oleg245-server facet_failover_host: oleg245-server Failover ost1 to oleg245-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 04:44:56 (1734255896) targets are mounted 04:44:56 (1734255896) facet_failover done Failing mds1 on oleg245-server Stopping /mnt/lustre-mds1 (opts:) on oleg245-server 04:44:58 (1734255898) shut down facet: mds1 facet_host: oleg245-server facet_failover_host: oleg245-server Failover mds1 to oleg245-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 04:45:11 (1734255911) targets are mounted 04:45:11 (1734255911) facet_failover done ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b589c000.lru_size=100 umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 PASS 47a (90s) == conf-sanity-1 test 47b: client restart does not lose lru_resize ========================================================== 04:45:32 (1734255932) Failing mds1 on oleg245-server 04:45:33 (1734255933) shut down facet: mds1 facet_host: oleg245-server facet_failover_host: oleg245-server Failover mds1 to oleg245-server mount facets: mds1 Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 04:45:56 (1734255956) targets are mounted 04:45:56 (1734255956) facet_failover done pdsh@oleg245-client: oleg245-client: ssh exited with exit code 95 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid pdsh@oleg245-client: oleg245-client: ssh exited with exit code 95 Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre setup single mount lustre success ln: failed to create symbolic link '/home/green/git/lustre-release/lustre/utils/lctl': File exists SKIP: conf-sanity-1 test_47b Can not create symlink /usr/sbin/lctl SKIP 47b (59s) == conf-sanity-1 test 48: too many acls on file ========== 04:46:31 (1734255991) start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: mount.lustre: according to /etc/mtab lustre-mdt1/mdt1 is already mounted on /mnt/lustre-mds1 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 17 Start of lustre-mdt1/mdt1 on mds1 failed 17 start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg245-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 17 seq.cli-lustre-OST0000-super.width=65536 Start of lustre-ost1/ost1 on ost1 failed 17 mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre mount.lustre: according to /etc/mtab 192.168.202.145@tcp:/lustre is already mounted on /mnt/lustre setup single mount lustre success debug=0 debug=0 /mnt/lustre stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre/widestripe lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 4 0x4 0x240000400 It is expected to hold at least 4500 ACL entries File: '/mnt/lustre/widestripe' Size: 3 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272502275 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-15 04:46:37.000000000 -0500 Modify: 2024-12-15 04:46:37.000000000 -0500 Change: 2024-12-15 04:48:12.000000000 -0500 Birth: - getfacl: Removing leading '/' from absolute path names Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg245-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server PASS 48 (167s) == conf-sanity-1 test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:49:18 (1734256158) Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre setup single mount lustre success check ldlm_timout... umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server PASS 49a (45s) == conf-sanity-1 test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:50:03 (1734256203) Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 PASS 49b (47s) == conf-sanity-1 test 50c: lazystatfs one server down ==== 04:50:50 (1734256250) start mds service on oleg245-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg245-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre llite.lustre-ffff8800b6385800.lazystatfs=1 stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server oleg245-server: oleg245-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.7034 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3328 378240 1% /mnt/lustre[MDT:0] lustre-OST0001_UUID 382976 3072 377856 1% /mnt/lustre[OST:1] filesystem_summary: 382976 3072 377856 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity-framework.sh: line 241: kill: (5817) - No such process umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg245-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server PASS 50c (96s) == conf-sanity-1 test 50d: lazystatfs client/server conn race ========================================================== 04:52:26 (1734256346) start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg245-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre llite.lustre-ffff8800b6353800.lazystatfs=1 stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.7034 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3328 378240 1% /mnt/lustre[MDT:0] lustre-OST0001_UUID 382976 3072 377856 1% /mnt/lustre[OST:1] filesystem_summary: 382976 3072 377856 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity-framework.sh: line 241: kill: (10756) - No such process umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg245-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server PASS 50d (75s) == conf-sanity-1 test 50f: normal statfs one server in down ========================================================== 04:53:41 (1734256421) start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec start ost2 service on oleg245-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg245-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg245-server oleg245-server: oleg245-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre llite.lustre-ffff8800b6364000.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.7034 start ost2 service on oleg245-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0001 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg245-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg245-server umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg245-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server PASS 50f (124s) == conf-sanity-1 test 50g: deactivated OST should not cause panic ========================================================== 04:55:45 (1734256545) start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre start ost2 service on oleg245-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg245-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b636a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b636a000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.145@tcp:/lustre 382976 3072 377856 1% /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg245-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg245-server stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg245-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server PASS 50g (86s) == conf-sanity-1 test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 04:57:11 (1734256631) Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg245-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec stop ost2 service on oleg245-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg245-server umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local modules unloaded. oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server PASS 51 (154s) == conf-sanity-1 test 54a: test llverdev and partial verify of device ========================================================== 04:59:45 (1734256785) SKIP: conf-sanity-1 test_54a ldiskfs only test SKIP 54a (1s) == conf-sanity-1 test 55: check lov_objid size =========== 04:59:46 (1734256786) SKIP: conf-sanity-1 test_55 ldiskfs only test SKIP 55 (1s) == conf-sanity-1 test 56a: check big OST indexes and out-of-index-order start ========================================================== 04:59:47 (1734256787) Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 Permanent disk data: Target: lustre:OST2710 Index: 10000 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.145@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.145@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=10000 lustre:fsname=lustre lustre:svname=lustre:OST2710 Permanent disk data: Target: lustre:OST03e8 Index: 1000 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.145@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost2 /dev/vdf mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2/ost2 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost2/ost2 properties lustre:mgsnode=192.168.202.145@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=1000 lustre:fsname=lustre lustre:svname=lustre:OST03e8 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST2710-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST2710 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg245-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST03e8-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST03e8 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre ok OBDS: 1000: lustre-OST03e8_UUID ACTIVE 10000: lustre-OST2710_UUID ACTIVE 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00875487 s, 468 kB/s oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:) Stopping client oleg245-client.virtnet /mnt/lustre opts: Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg245-server Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server PASS 56a (120s) == conf-sanity-1 test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 05:01:47 (1734256907) SKIP: conf-sanity-1 test_56b needs >= 3 MDTs SKIP 56b (1s) == conf-sanity-1 test 57a: initial registration from failnode should fail (should return errs) ========================================================== 05:01:48 (1734256908) oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg245-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.145@tcp Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.145@tcp failover.node=192.168.202.145@tcp autodegrade=on Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.202.145@tcp lustre:failover.node=192.168.202.145@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg245-server: mount.lustre: mount lustre-ost1/ost1 at /mnt/lustre-ost1 failed: Cannot assign requested address pdsh@oleg245-client: oleg245-server: ssh exited with exit code 99 oleg245-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory oleg245-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 Start of lustre-ost1/ost1 on ost1 failed 99 umount lustre on /mnt/lustre..... stop ost1 service on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 PASS 57a (67s) == conf-sanity-1 test 57b: initial registration from servicenode should not fail ========================================================== 05:02:55 (1734256975) oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: mgsnode=192.168.202.145@tcp autodegrade=on sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x1042 (OST update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.145@tcp autodegrade=on sys.timeout=20 failover.node=192.168.202.145@tcp autodegrade=on Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.145@tcp lustre:autodegrade=on lustre:sys.timeout=20 lustre:failover.node=192.168.202.145@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4162 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid umount lustre on /mnt/lustre..... stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 PASS 57b (87s) == conf-sanity-1 test 59: writeconf mount option ========= 05:04:23 (1734257063) libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server original ost count: 1 (expect > 0) oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid after mdt writeconf count: 0 (expect 0) OST start without writeconf should fail: oleg245-server: mount.lustre: mount lustre-ost1/ost1 at /mnt/lustre-ost1 failed: No such file or directory oleg245-server: Is the MGS specification correct? oleg245-server: Is the filesystem name correct? oleg245-server: If upgrading, is the copied client log valid? (see upgrade docs) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 oleg245-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory oleg245-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 OST start with writeconf should succeed: oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid after ost writeconf count: 1 (expect 1) oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid after ost2 writeconf count: 2 (expect 2) oleg245-server: oleg245-server.virtnet: executing unload_modules_local oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server PASS 59 (126s) == conf-sanity-1 test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 05:06:28 (1734257188) SKIP: conf-sanity-1 test_60a ldiskfs only test SKIP 60a (0s) == conf-sanity-1 test 62: start with disabled journal ==== 05:06:29 (1734257189) SKIP: conf-sanity-1 test_62 ldiskfs only test SKIP 62 (0s) == conf-sanity-1 test 64: check lfs df --lazy ============ 05:06:29 (1734257189) start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg245-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre stop ost2 service on oleg245-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg245-server /home/green/git/lustre-release/lustre/utils/lfs df UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3200 378368 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local modules unloaded. oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg245-server: oleg245-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg245-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server PASS 64 (112s) == conf-sanity-1 test 66: replace nids =================== 05:08:21 (1734257301) client=34615898 MDS=34615898 OSS=34615898 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre Setting lustre-OST0000.osc.active from 1 to 0 Waiting 90s for '0' Updated after 9s: want '0' got '0' replace_nids should fail if MDS, OSTs and clients are UP oleg245-server: error: replace_nids: Operation now in progress pdsh@oleg245-client: oleg245-server: ssh exited with exit code 115 umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) replace_nids should fail if MDS and OSTs are UP oleg245-server: error: replace_nids: Operation now in progress pdsh@oleg245-client: oleg245-server: ssh exited with exit code 115 stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server replace_nids should fail if MDS is UP oleg245-server: error: replace_nids: Operation now in progress pdsh@oleg245-client: oleg245-server: ssh exited with exit code 115 stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server start mds service on oleg245-server Starting mds1: -o localrecov -o nosvc lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all Start lustre-mdt1/mdt1 without service Started lustre-MDT0000 command should accept two parameters pdsh@oleg245-client: oleg245-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] correct device name should be passed oleg245-server: error: replace_nids: No such device or address pdsh@oleg245-client: oleg245-server: ssh exited with exit code 6 wrong nids list should not destroy the system pdsh@oleg245-client: oleg245-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] pdsh@oleg245-client: oleg245-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] replace OST nid command should accept two parameters pdsh@oleg245-client: oleg245-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] wrong nids list should not destroy the system pdsh@oleg245-client: oleg245-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] set NIDs with failover replace MDS nid stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Setting lustre-OST0000.osc.active from 0 to 1 Waiting 90s for '1' Updated after 6s: want '1' got '1' start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local modules unloaded. Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server PASS 66 (156s) == conf-sanity-1 test 70e: Sync-on-Cancel will be enabled by default on DNE ========================================================== 05:10:58 (1734257458) SKIP: conf-sanity-1 test_70e needs >= 2 MDTs SKIP 70e (1s) == conf-sanity-1 test 75: The order of --index should be irrelevant ========================================================== 05:10:59 (1734257459) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions Permanent disk data: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x45 (MDT MGS update ) Persistent mount opts: Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mkfs_cmd = zpool create -f -O canmount=off lustre-mdt1 /dev/vdc mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt1/mdt1 xattr=sa dnodesize=auto Writing lustre-mdt1/mdt1 properties lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=69 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-MDT0000 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.145@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.145@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 Permanent disk data: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x45 (MDT MGS update ) Persistent mount opts: Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mkfs_cmd = zpool create -f -O canmount=off lustre-mdt1 /dev/vdc mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt1/mdt1 xattr=sa dnodesize=auto Writing lustre-mdt1/mdt1 properties lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=69 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-MDT0000 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.145@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.145@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server PASS 75 (66s) == conf-sanity-1 test 78: run resize2fs on MDT and OST filesystems ========================================================== 05:12:05 (1734257525) SKIP: conf-sanity-1 test_78 ldiskfs only test SKIP 78 (1s) == conf-sanity-1 test 79: format MDT/OST without mgs option (should return errors) ========================================================== 05:12:06 (1734257526) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: oleg245-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode oleg245-server: mkfs.lustre: exiting with 22 (Invalid argument) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 22 oleg245-server: oleg245-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode oleg245-server: mkfs.lustre: exiting with 22 (Invalid argument) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 22 oleg245-server: oleg245-server: mkfs.lustre FATAL: Must specify --mgsnode oleg245-server: mkfs.lustre: exiting with 22 (Invalid argument) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 22 Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server PASS 79 (68s) == conf-sanity-1 test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 05:13:14 (1734257594) SKIP: conf-sanity-1 test_83 ldiskfs only test SKIP 83 (2s) == conf-sanity-1 test 86: Replacing mkfs.lustre -G option ========================================================== 05:13:16 (1734257596) SKIP: conf-sanity-1 test_86 LU-6442: no such mkfs params for ZFS OSTs SKIP 86 (1s) == conf-sanity-1 test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 05:13:17 (1734257597) SKIP: conf-sanity-1 test_87 ldiskfs only test SKIP 87 (1s) == conf-sanity-1 test 89: check tunefs --param and --erase-param{s} options ========================================================== 05:13:18 (1734257598) Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:) tunefs --param failover.node=192.0.2.254@tcp0 tunefs --param failover.node=192.0.2.255@tcp0 tunefs --erase-param failover.node tunefs --erase-params tunefs --param failover.node=192.0.2.254@tcp0 --erase-params Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg245-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server PASS 89 (62s) == conf-sanity-1 test 93: register mulitple MDT at the same time ========================================================== 05:14:20 (1734257660) SKIP: conf-sanity-1 test_93 needs >= 3 MDTs SKIP 93 (1s) == conf-sanity-1 test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 05:14:22 (1734257662) Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre setup single mount lustre success lustre-MDT0000: lustre-OST0000: umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 PASS 100 (64s) == conf-sanity-1 test 104a: Make sure user defined options are reflected in mount ========================================================== 05:15:26 (1734257726) mountfsopt: acl,user_xattr Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 Starting mds1: -o localrecov,noacl lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre setfacl: /mnt/lustre: Operation not supported PASS 104a (57s) == conf-sanity-1 test 105: check file creation for ro and rw bind mnt pt ========================================================== 05:16:23 (1734257783) umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local oleg245-server: rmmod: ERROR: Module lustre is in use pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 modules unloaded. Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-ost2 (opts:-f) on oleg245-server oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre touch: cannot touch '/tmp/d105.conf-sanity-1/f105.conf-sanity-1': Read-only file system umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 PASS 105 (95s) == conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:17:58 (1734257878) Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16200 392 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1743 (time 1734257949.42 total 10.00 last 174.29) - open/close 3440 (time 1734257959.42 total 20.00 last 169.69) - open/close 5193 (time 1734257969.43 total 30.01 last 175.19) - open/close 7056 (time 1734257979.44 total 40.01 last 186.19) - open/close 9003 (time 1734257989.44 total 50.02 last 194.65) - open/close 10000 (time 1734257995.52 total 56.10 last 163.99) - open/close 11546 (time 1734258005.52 total 66.10 last 154.54) - open/close 12750 (time 1734258015.52 total 76.10 last 120.40) - open/close 14037 (time 1734258025.53 total 86.10 last 128.64) - open/close 15404 (time 1734258035.53 total 96.11 last 136.60) - open/close 16678 (time 1734258045.54 total 106.12 last 127.32) - open/close 18016 (time 1734258055.55 total 116.12 last 133.72) - open/close 19383 (time 1734258065.55 total 126.13 last 136.63) - open/close 20000 (time 1734258069.88 total 130.46 last 142.40) - open/close 21296 (time 1734258079.89 total 140.47 last 129.49) - open/close 22685 (time 1734258089.90 total 150.48 last 138.82) - open/close 24035 (time 1734258099.90 total 160.48 last 134.98) - open/close 25311 (time 1734258109.91 total 170.48 last 127.52) - open/close 26597 (time 1734258119.91 total 180.49 last 128.52) - open/close 27979 (time 1734258129.91 total 190.49 last 138.20) - open/close 29451 (time 1734258139.91 total 200.49 last 147.16) - open/close 30000 (time 1734258143.58 total 204.15 last 149.91) - open/close 31522 (time 1734258153.59 total 214.16 last 152.06) - open/close 33090 (time 1734258163.59 total 224.17 last 156.76) - open/close 34531 (time 1734258173.59 total 234.17 last 144.03) - open/close 35989 (time 1734258183.59 total 244.17 last 145.80) - open/close 37760 (time 1734258193.59 total 254.17 last 177.09) - open/close 39319 (time 1734258203.59 total 264.17 last 155.90) - open/close 40000 (time 1734258207.69 total 268.27 last 166.24) - open/close 41455 (time 1734258217.69 total 278.27 last 145.46) - open/close 43170 (time 1734258227.70 total 288.27 last 171.46) - open/close 44861 (time 1734258237.70 total 298.28 last 169.01) - open/close 46480 (time 1734258247.70 total 308.28 last 161.87) - open/close 48053 (time 1734258257.71 total 318.28 last 157.24) - open/close 49673 (time 1734258267.72 total 328.30 last 161.82) - open/close 50000 (time 1734258269.90 total 330.48 last 149.78) - open/close 51575 (time 1734258279.90 total 340.48 last 157.50) - open/close 53205 (time 1734258289.91 total 350.49 last 162.89) - open/close 54890 (time 1734258299.91 total 360.49 last 168.46) - open/close 56608 (time 1734258309.91 total 370.49 last 171.74) - open/close 58164 (time 1734258319.92 total 380.50 last 155.53) - open/close 59698 (time 1734258329.92 total 390.50 last 153.34) - open/close 60000 (time 1734258332.08 total 392.66 last 139.82) - open/close 61517 (time 1734258342.09 total 402.66 last 151.66) - open/close 62998 (time 1734258352.09 total 412.67 last 148.06) - open/close 64639 (time 1734258362.09 total 422.67 last 164.08) total: 64768 open/close in 423.46 seconds: 152.95 ops/second - /unlink 2701 (time 1734258374.33 total 10.00 last 270.05) - /unlink 5833 (time 1734258384.33 total 20.00 last 313.11) - /unlink 9147 (time 1734258394.33 total 30.01 last 331.37) - /unlink 10000 (time 1734258396.65 total 32.33 last 367.34) - /unlink 13665 (time 1734258406.65 total 42.33 last 366.50) - /unlink 17168 (time 1734258416.66 total 52.33 last 350.21) - /unlink 20000 (time 1734258423.77 total 59.45 last 398.01) - /unlink 23223 (time 1734258433.77 total 69.45 last 322.25) - /unlink 26747 (time 1734258443.77 total 79.45 last 352.33) - /unlink 29668 (time 1734258453.78 total 89.45 last 292.05) - /unlink 32919 (time 1734258463.78 total 99.45 last 325.07) - /unlink 35677 (time 1734258473.78 total 109.45 last 275.74) - /unlink 38202 (time 1734258483.78 total 119.45 last 252.48) - /unlink 40000 (time 1734258490.53 total 126.21 last 266.25) - /unlink 42942 (time 1734258500.54 total 136.21 last 294.06) - /unlink 45412 (time 1734258510.54 total 146.21 last 246.99) - /unlink 48064 (time 1734258520.54 total 156.22 last 265.11) - /unlink 50000 (time 1734258527.76 total 163.43 last 268.39) - /unlink 52167 (time 1734258537.76 total 173.43 last 216.66) - /unlink 54745 (time 1734258547.76 total 183.44 last 257.66) - /unlink 57432 (time 1734258557.76 total 193.44 last 268.66) - /unlink 60000 (time 1734258567.70 total 203.37 last 258.45) - /unlink 62707 (time 1734258577.70 total 213.38 last 270.61) total: 64768 /unlink in 219.88 seconds: 294.55 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1572 (time 1734258613.26 total 10.00 last 157.18) - open/close 3155 (time 1734258623.27 total 20.01 last 158.22) - open/close 4524 (time 1734258633.27 total 30.01 last 136.85) - open/close 5850 (time 1734258643.27 total 40.01 last 132.60) - open/close 7266 (time 1734258653.28 total 50.02 last 141.52) - open/close 8751 (time 1734258663.28 total 60.02 last 148.43) - open/close 10000 (time 1734258672.16 total 68.90 last 140.71) - open/close 11774 (time 1734258682.16 total 78.90 last 177.37) - open/close 13525 (time 1734258692.16 total 88.90 last 175.02) - open/close 15374 (time 1734258702.17 total 98.91 last 184.85) - open/close 17339 (time 1734258712.17 total 108.91 last 196.49) - open/close 19065 (time 1734258722.17 total 118.91 last 172.49) - open/close 20000 (time 1734258728.15 total 124.88 last 156.58) - open/close 21695 (time 1734258738.15 total 134.89 last 169.42) - open/close 23378 (time 1734258748.15 total 144.89 last 168.23) - open/close 25176 (time 1734258758.16 total 154.89 last 179.78) - open/close 27399 (time 1734258768.16 total 164.90 last 222.26) - open/close 29324 (time 1734258778.16 total 174.90 last 192.42) - open/close 30000 (time 1734258782.11 total 178.85 last 171.25) - open/close 32263 (time 1734258792.11 total 188.85 last 226.29) - open/close 34352 (time 1734258802.11 total 198.85 last 208.88) - open/close 35973 (time 1734258812.11 total 208.85 last 162.05) - open/close 37652 (time 1734258822.12 total 218.86 last 167.75) - open/close 39563 (time 1734258832.12 total 228.86 last 191.09) - open/close 40000 (time 1734258834.57 total 231.31 last 178.46) - open/close 41731 (time 1734258844.57 total 241.31 last 173.06) - open/close 43830 (time 1734258854.58 total 251.32 last 209.80) - open/close 46302 (time 1734258864.58 total 261.32 last 247.20) - open/close 49131 (time 1734258874.58 total 271.32 last 282.84) - open/close 50000 (time 1734258878.26 total 275.00 last 236.30) - open/close 52064 (time 1734258888.26 total 285.00 last 206.32) - open/close 54395 (time 1734258898.26 total 295.00 last 233.06) - open/close 56279 (time 1734258908.27 total 305.00 last 188.38) - open/close 58008 (time 1734258918.27 total 315.00 last 172.89) - open/close 59984 (time 1734258928.27 total 325.01 last 197.54) - open/close 62143 (time 1734258938.27 total 335.01 last 215.86) - open/close 64624 (time 1734258948.27 total 345.01 last 248.05) total: 64768 open/close in 345.71 seconds: 187.35 ops/second - /unlink 4553 (time 1734258960.10 total 10.00 last 455.29) - /unlink 8630 (time 1734258970.10 total 20.00 last 407.69) - /unlink 10000 (time 1734258973.50 total 23.40 last 402.73) - /unlink 13937 (time 1734258983.51 total 33.41 last 393.58) - /unlink 18222 (time 1734258993.51 total 43.41 last 428.49) - /unlink 20000 (time 1734258997.21 total 47.11 last 479.43) - /unlink 24828 (time 1734259007.21 total 57.11 last 482.80) - /unlink 29353 (time 1734259017.22 total 67.12 last 452.43) - /unlink 32891 (time 1734259027.22 total 77.12 last 353.73) - /unlink 36387 (time 1734259037.22 total 87.12 last 349.59) - /unlink 40000 (time 1734259046.57 total 96.47 last 386.27) - /unlink 43770 (time 1734259056.57 total 106.47 last 376.96) - /unlink 48001 (time 1734259066.57 total 116.47 last 423.07) - /unlink 50000 (time 1734259071.39 total 121.29 last 414.83) - /unlink 54156 (time 1734259081.40 total 131.29 last 415.50) - /unlink 58100 (time 1734259091.40 total 141.29 last 394.39) - /unlink 60000 (time 1734259096.16 total 146.06 last 398.51) - /unlink 63832 (time 1734259106.16 total 156.06 last 383.18) total: 64768 /unlink in 158.35 seconds: 409.02 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2314 (time 1734259134.11 total 10.00 last 231.34) - open/close 4603 (time 1734259144.11 total 20.01 last 228.83) - open/close 6783 (time 1734259154.12 total 30.01 last 217.93) - open/close 9142 (time 1734259164.12 total 40.01 last 235.86) - open/close 10000 (time 1734259167.85 total 43.74 last 229.89) - open/close 12197 (time 1734259177.85 total 53.75 last 219.63) - open/close 14327 (time 1734259187.86 total 63.75 last 212.94) - open/close 16718 (time 1734259197.86 total 73.75 last 239.07) - open/close 18815 (time 1734259207.86 total 83.75 last 209.64) - open/close 20000 (time 1734259212.57 total 88.46 last 251.50) - open/close 22317 (time 1734259222.57 total 98.47 last 231.67) - open/close 24272 (time 1734259232.58 total 108.47 last 195.46) - open/close 26290 (time 1734259242.58 total 118.47 last 201.79) - open/close 28547 (time 1734259252.58 total 128.47 last 225.67) - open/close 30000 (time 1734259259.30 total 135.19 last 216.19) - open/close 32141 (time 1734259269.30 total 145.19 last 214.09) - open/close 34764 (time 1734259279.30 total 155.19 last 262.26) - open/close 37458 (time 1734259289.30 total 165.19 last 269.37) - open/close 39898 (time 1734259299.30 total 175.20 last 243.94) - open/close 42020 (time 1734259309.31 total 185.20 last 212.18) - open/close 43852 (time 1734259319.31 total 195.20 last 183.16) - open/close 45944 (time 1734259329.31 total 205.20 last 209.13) - open/close 48118 (time 1734259339.31 total 215.20 last 217.37) - open/close 50000 (time 1734259346.65 total 222.54 last 256.65) - open/close 52462 (time 1734259356.65 total 232.54 last 246.18) - open/close 54874 (time 1734259366.65 total 242.54 last 241.12) - open/close 57044 (time 1734259376.65 total 252.54 last 216.93) - open/close 59445 (time 1734259386.66 total 262.55 last 240.01) - open/close 60000 (time 1734259388.98 total 264.88 last 238.48) - open/close 62326 (time 1734259398.99 total 274.88 last 232.48) - open/close 64587 (time 1734259408.99 total 284.88 last 226.00) total: 64768 open/close in 285.63 seconds: 226.75 ops/second - /unlink 4119 (time 1734259420.94 total 10.00 last 411.84) - /unlink 7749 (time 1734259430.95 total 20.00 last 362.96) - /unlink 10000 (time 1734259436.55 total 25.60 last 401.89) - /unlink 13586 (time 1734259446.55 total 35.61 last 358.54) - /unlink 17328 (time 1734259456.55 total 45.61 last 374.14) - /unlink 20000 (time 1734259463.31 total 52.37 last 395.06) - /unlink 24546 (time 1734259473.31 total 62.37 last 454.59) - /unlink 28633 (time 1734259483.31 total 72.37 last 408.69) - /unlink 30000 (time 1734259486.83 total 75.89 last 388.64) - /unlink 34275 (time 1734259496.83 total 85.89 last 427.47) - /unlink 37941 (time 1734259506.83 total 95.89 last 366.57) - /unlink 40000 (time 1734259511.41 total 100.47 last 449.79) - /unlink 43949 (time 1734259521.41 total 110.47 last 394.83) - /unlink 47677 (time 1734259531.41 total 120.47 last 372.80) - /unlink 50000 (time 1734259537.35 total 126.41 last 391.01) - /unlink 53597 (time 1734259547.35 total 136.41 last 359.69) - /unlink 57514 (time 1734259557.35 total 146.41 last 391.65) - /unlink 60000 (time 1734259563.31 total 152.37 last 417.43) - /unlink 64204 (time 1734259573.31 total 162.37 last 420.35) total: 64768 /unlink in 164.10 seconds: 394.69 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2643 (time 1734259604.90 total 10.00 last 264.24) - open/close 5129 (time 1734259614.90 total 20.01 last 248.51) - open/close 7779 (time 1734259624.90 total 30.01 last 264.98) - open/close 10000 (time 1734259632.06 total 37.17 last 310.24) - open/close 12915 (time 1734259642.06 total 47.17 last 291.41) - open/close 15805 (time 1734259652.06 total 57.17 last 288.98) - open/close 18441 (time 1734259662.07 total 67.17 last 263.49) - open/close 20000 (time 1734259668.69 total 73.79 last 235.55) - open/close 22585 (time 1734259678.69 total 83.79 last 258.48) - open/close 25234 (time 1734259688.69 total 93.79 last 264.84) - open/close 28281 (time 1734259698.69 total 103.79 last 304.70) - open/close 30000 (time 1734259704.24 total 109.34 last 309.76) - open/close 32683 (time 1734259714.24 total 119.35 last 268.22) - open/close 35143 (time 1734259724.25 total 129.35 last 245.93) - open/close 37662 (time 1734259734.25 total 139.35 last 251.89) - open/close 40000 (time 1734259744.18 total 149.29 last 235.28) - open/close 42549 (time 1734259754.18 total 159.29 last 254.90) - open/close 44961 (time 1734259764.18 total 169.29 last 241.17) - open/close 46991 (time 1734259774.19 total 179.29 last 202.93) - open/close 49209 (time 1734259784.19 total 189.30 last 221.73) - open/close 50000 (time 1734259787.54 total 192.64 last 236.51) - open/close 52355 (time 1734259797.54 total 202.64 last 235.49) - open/close 55347 (time 1734259807.54 total 212.64 last 299.15) - open/close 58220 (time 1734259817.54 total 222.64 last 287.23) - open/close 60000 (time 1734259824.24 total 229.34 last 265.75) - open/close 62843 (time 1734259834.24 total 239.34 last 284.28) total: 64768 open/close in 246.92 seconds: 262.30 ops/second - /unlink 4747 (time 1734259852.85 total 10.00 last 474.66) - /unlink 9042 (time 1734259862.85 total 20.00 last 429.44) - /unlink 13580 (time 1734259872.85 total 30.00 last 453.76) - /unlink 17925 (time 1734259882.85 total 40.00 last 434.49) - /unlink 20000 (time 1734259886.81 total 43.96 last 524.22) - /unlink 24965 (time 1734259896.81 total 53.96 last 496.43) - /unlink 29731 (time 1734259906.81 total 63.96 last 476.54) - /unlink 34716 (time 1734259916.81 total 73.96 last 498.49) - /unlink 40000 (time 1734259926.69 total 83.84 last 535.09) - /unlink 45599 (time 1734259936.69 total 93.84 last 559.86) - /unlink 50000 (time 1734259944.83 total 101.98 last 540.65) - /unlink 55824 (time 1734259954.83 total 111.98 last 582.31) - /unlink 60000 (time 1734259962.70 total 119.85 last 530.75) total: 64768 /unlink in 127.73 seconds: 507.06 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2898 (time 1734260000.00 total 10.00 last 289.70) - open/close 6167 (time 1734260010.00 total 20.00 last 326.90) - open/close 9074 (time 1734260020.00 total 30.01 last 290.64) - open/close 10000 (time 1734260023.11 total 33.11 last 298.18) - open/close 13196 (time 1734260033.11 total 43.11 last 319.56) - open/close 16185 (time 1734260043.11 total 53.11 last 298.89) - open/close 18683 (time 1734260053.11 total 63.11 last 249.79) - open/close 20000 (time 1734260057.64 total 67.65 last 290.60) - open/close 23275 (time 1734260067.64 total 77.65 last 327.48) - open/close 26366 (time 1734260077.64 total 87.65 last 309.06) - open/close 29119 (time 1734260087.65 total 97.65 last 275.24) - open/close 30000 (time 1734260090.73 total 100.73 last 285.65) - open/close 32746 (time 1734260100.73 total 110.73 last 274.59) - open/close 35620 (time 1734260110.73 total 120.74 last 287.37) - open/close 38599 (time 1734260120.74 total 130.74 last 297.83) - open/close 40000 (time 1734260125.74 total 135.74 last 280.20) - open/close 42739 (time 1734260135.74 total 145.74 last 273.82) - open/close 45704 (time 1734260145.74 total 155.74 last 296.48) - open/close 48508 (time 1734260155.74 total 165.74 last 280.37) - open/close 50000 (time 1734260160.37 total 170.37 last 322.34) - open/close 52940 (time 1734260170.37 total 180.37 last 293.91) - open/close 55691 (time 1734260180.37 total 190.37 last 275.10) - open/close 58044 (time 1734260190.37 total 200.38 last 235.27) - open/close 60000 (time 1734260198.16 total 208.17 last 251.03) - open/close 62916 (time 1734260208.17 total 218.17 last 291.57) total: 64768 open/close in 225.42 seconds: 287.33 ops/second - /unlink 4473 (time 1734260226.43 total 10.00 last 447.23) - /unlink 8939 (time 1734260236.43 total 20.00 last 446.57) - /unlink 14057 (time 1734260246.43 total 30.00 last 511.75) - /unlink 19090 (time 1734260256.43 total 40.00 last 503.26) - /unlink 24325 (time 1734260266.43 total 50.00 last 523.46) - /unlink 29850 (time 1734260276.43 total 60.01 last 552.44) - /unlink 35313 (time 1734260286.43 total 70.01 last 546.30) - /unlink 40000 (time 1734260294.25 total 77.82 last 599.67) - /unlink 44772 (time 1734260304.25 total 87.82 last 477.15) - /unlink 49142 (time 1734260314.25 total 97.82 last 436.97) - /unlink 53991 (time 1734260324.25 total 107.83 last 484.79) - /unlink 59043 (time 1734260334.25 total 117.83 last 505.17) - /unlink 60000 (time 1734260336.44 total 120.02 last 436.54) - /unlink 64478 (time 1734260346.44 total 130.02 last 447.77) total: 64768 /unlink in 130.54 seconds: 496.14 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 PASS 106 (2502s) == conf-sanity-1 test 109a: test lctl clear_conf fsname == 05:59:40 (1734260380) SKIP: conf-sanity-1 test_109a LU-8727: no implementation for ZFS SKIP 109a (1s) == conf-sanity-1 test 109b: test lctl clear_conf one config ========================================================== 05:59:41 (1734260381) SKIP: conf-sanity-1 test_109b LU-8727: no implementation for ZFS SKIP 109b (1s) == conf-sanity-1 test 116: big size MDT support ========== 05:59:42 (1734260382) SKIP: conf-sanity-1 test_116 ldiskfs only test SKIP 116 (1s) == conf-sanity-1 test 122a: Check OST sequence update ==== 05:59:43 (1734260383) SKIP: conf-sanity-1 test_122a needs >= 2 MDTs SKIP 122a (1s) == conf-sanity-1 test 123ah: del_ost cancels config log entries correctly ========================================================== 05:59:44 (1734260384) Checking servers environments Checking clients oleg245-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' Setup mgs, mdt, osts Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre Starting client oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre Started clients oleg245-client.virtnet: 192.168.202.145@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012aaa3800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012aaa3800.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 2s: want 'procname_uid' got 'procname_uid' disable quota as required del_ost: dry run for target lustre-OST0000 config_log: lustre-MDT0000 [DRY RUN] cancel catalog 'lustre-MDT0000:20':"- { index: 20, event: add_osc, device: lustre-MDT0000-mdtlov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }" [DRY RUN] cancel catalog 'lustre-MDT0000:19':"- { index: 19, event: setup, device: lustre-OST0000-osc-MDT0000, UUID: lustre-OST0000_UUID, node: 192.168.202.145@tcp }" [DRY RUN] cancel catalog 'lustre-MDT0000:18':"- { index: 18, event: attach, device: lustre-OST0000-osc-MDT0000, type: osc, UUID: lustre-MDT0000-mdtlov_UUID }" del_ost: no catalog entry deleted config_log: lustre-client [DRY RUN] cancel catalog 'lustre-client:25':"- { index: 25, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }" [DRY RUN] cancel catalog 'lustre-client:24':"- { index: 24, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.202.145@tcp }" [DRY RUN] cancel catalog 'lustre-client:23':"- { index: 23, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID }" del_ost: no catalog entry deleted config_log: lustre-MDT0000 cancel catalog lustre-MDT0000 log_idx 20: done cancel catalog lustre-MDT0000 log_idx 19: done cancel catalog lustre-MDT0000 log_idx 18: done del_ost: cancelled 3 catalog entries config_log: lustre-client cancel catalog lustre-client log_idx 25: done cancel catalog lustre-client log_idx 24: done cancel catalog lustre-client log_idx 23: done del_ost: cancelled 3 catalog entries umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local oleg245-server: rmmod: ERROR: Module lustre is in use pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 modules unloaded. Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-ost2 (opts:-f) on oleg245-server oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_refcnt: could not read integer from '/sys/module/acpi_cpufreq/refcnt': 'No such device' libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server fail_loc=0 PASS 123ah (100s) == conf-sanity-1 test 132: hsm_actions processed after failover ========================================================== 06:01:24 (1734260484) Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x5 (MDT MGS ) Persistent mount opts: Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20 Permanent disk data: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x45 (MDT MGS update ) Persistent mount opts: Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20 mdt.hsm_control=enabled Writing lustre-mdt1/mdt1 properties lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:sys.timeout=20 lustre:mdt.hsm_control=enabled lustre:version=1 lustre:flags=69 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-MDT0000 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server PASS 132 (28s) == conf-sanity-1 test 133: stripe QOS: free space balance in a pool ========================================================== 06:01:52 (1734260512) SKIP: conf-sanity-1 test_133 needs >= 4 OSTs SKIP 133 (1s) == conf-sanity-1 test 135: check the behavior when changelog is wrapped around ========================================================== 06:01:53 (1734260513) Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre oleg245-client: fail_loc=0x1312 oleg245-client: fail_val=5 oleg245-server: fail_loc=0x1312 oleg245-server: fail_val=5 mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl1' Wrap arround changelog catalog - open/close 2397 (time 1734260554.93 total 10.00 last 239.67) - open/close 4332 (time 1734260564.93 total 20.00 last 193.48) total: 4500 open/close in 20.81 seconds: 216.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #12998 - /unlink 4424 (time 1734260576.84 total 10.00 last 442.25) total: 4500 /unlink in 10.16 seconds: 442.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 1842 (time 1734260588.00 total 10.00 last 184.14) - open/close 4413 (time 1734260598.00 total 20.00 last 257.10) total: 4500 open/close in 20.32 seconds: 221.45 ops/second - /unlink 4494 (time 1734260609.32 total 10.00 last 449.37) total: 4500 /unlink in 10.01 seconds: 449.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 2270 (time 1734260620.44 total 10.00 last 226.94) total: 4500 open/close in 18.99 seconds: 236.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 total: 4500 /unlink in 8.43 seconds: 533.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 2276 (time 1734260649.78 total 10.00 last 227.58) total: 4500 open/close in 18.75 seconds: 239.96 ops/second total: 4500 /unlink in 7.95 seconds: 566.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 2627 (time 1734260678.47 total 10.00 last 262.64) total: 4500 open/close in 17.55 seconds: 256.47 ops/second total: 4500 /unlink in 9.85 seconds: 456.79 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 10973: 22624 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local modules unloaded. PASS 135 (210s) == conf-sanity-1 test 136: don't panic with bad obdecho setup ========================================================== 06:05:23 (1734260723) SKIP: conf-sanity-1 test_136 needs >= 2 MDTs SKIP 136 (1s) == conf-sanity-1 test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:05:24 (1734260724) SKIP: conf-sanity-1 test_137 needs >= 3 MDTs SKIP 137 (1s) == conf-sanity-1 test 140: remove_updatelog script actions ========================================================== 06:05:25 (1734260725) SKIP: conf-sanity-1 test_140 needs >= 2 MDTs SKIP 140 (1s) == conf-sanity-1 test 151a: damaged local config doesn't prevent mounting ========================================================== 06:05:26 (1734260726) SKIP: conf-sanity-1 test_151a ldiskfs only test SKIP 151a (0s) == conf-sanity-1 test 151b: -ENOSPC doesn't affect mount ========================================================== 06:05:26 (1734260726) SKIP: conf-sanity-1 test_151b ldiskfs only test SKIP 151b (1s) == conf-sanity-1 test 153a: bypass invalid NIDs quickly == 06:05:27 (1734260727) Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid waiting for mount ... waiting for mount ... "192.168.202.145@tcp": { connects: 1, replied: 1, uptodate: uptodate, sec_ago: 5 } "192.168.252.112@tcp": { connects: 1, replied: 0, uptodate: discovering, sec_ago: 10 } "10.252.252.113@tcp": { connects: 0, replied: 0, uptodate: discovering, sec_ago: never } "192.168.202.145@tcp": { connects: 0, replied: 0, uptodate: discovering, sec_ago: never } setup single mount lustre success umount lustre on /mnt/lustre..... stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 PASS 153a (184s) == conf-sanity-1 test 153b: added IPv6 NID support ======= 06:08:31 (1734260911) Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server Checking servers environments Checking clients oleg245-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre Starting client oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre Started clients oleg245-client.virtnet: 192.168.202.145@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012b765800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012b765800.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 5s: want 'procname_uid' got 'procname_uid' disable quota as required PASS 153b (74s) == conf-sanity-1 test 153c: don't stuck on unreached NID ========================================================== 06:09:45 (1734260985) Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg245-server oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid umount lustre on /mnt/lustre..... waiting for mount ... waiting for mount ... waiting for mount ... waiting for mount ... waiting for mount ... waiting for mount ... "192.168.252.112@tcp": { connects: 3, replied: 0, uptodate: discovering, sec_ago: 0 } "192.168.252.113@tcp": { connects: 2, replied: 0, uptodate: discovering, sec_ago: 10 } "192.168.252.115@tcp": { connects: 2, replied: 0, uptodate: discovering, sec_ago: 5 } connection attempts: 7 Waiting for mount to fail mount.lustre: mount 192.168.252.112@tcp:192.168.252.113@tcp:192.168.252.115@tcp:/lustre at /mnt/lustre failed: Input/output error Is the MGS running? umount lustre on /mnt/lustre..... stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 PASS 153c (307s) == conf-sanity-1 test 154: expand .. on rename after MDT backup restore ========================================================== 06:14:52 (1734261292) SKIP: conf-sanity-1 test_154 ldiskfs only test SKIP 154 (1s) Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity-1 test complete, duration 6214 sec ======== 06:15:11 (1734261311) === conf-sanity-1: start cleanup 06:15:12 (1734261312) === === conf-sanity-1: finish cleanup 06:15:12 (1734261312) ===