-----============= acceptance-small: conf-sanity-1 ============----- Sun Dec 15 04:31:32 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg112-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity-1.*ex: No such file or directory excepting tests: skipping tests SLOW=no: 106 Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg112-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg112-server oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server == conf-sanity-1 test 17: Verify failed mds_postsetup won't fail assertion (2936) (should return errs) ========================================================== 04:32:37 (1734255157) SKIP: conf-sanity-1 test_17 ldiskfs only test SKIP 17 (1s) == conf-sanity-1 test 18: check mkfs creates large journals ========================================================== 04:32:38 (1734255158) SKIP: conf-sanity-1 test_18 ldiskfs only test SKIP 18 (1s) == conf-sanity-1 test 21c: start mds between two osts, stop mds last ========================================================== 04:32:39 (1734255159) start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost2 service on oleg112-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop ost2 service on oleg112-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server PASS 21c (115s) == conf-sanity-1 test 29: permanently remove an OST ====== 04:34:34 (1734255274) start ost2 service on oleg112-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Setting lustre-OST0001.osc.active from 1 to 0 Waiting 90s for '0' Updated after 2s: want '0' got '0' Live client success: got lustre-OST0001_UUID FULL DEACTIVATED osc.lustre-OST0001-osc-MDT0000.active updated after 0 sec (got 0) umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre New client success: got 'lustre-OST0001_UUID NEW DEACTIVATED' Setting lustre-OST0001.osc.active from 0 to 1 Waiting 90s for '1' Updated after 3s: want '1' got '1' umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg112-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg112-server stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local modules unloaded. oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server PASS 29 (98s) == conf-sanity-1 test 35a: Reconnect to the last active server first ========================================================== 04:36:12 (1734255372) start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre debug=ha Set up a fake failnode for the MDS Wait for RECONNECT_INTERVAL seconds (10s) conf-sanity.sh test_35a 2024-12-15 4h36m38s Stopping the MDT: lustre-MDT0000 stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server Restarting the MDT: lustre-MDT0000 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 Wait for df (23775) ... done debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout Debug log: 72 lines, 72 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local modules unloaded. oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server PASS 35a (93s) == conf-sanity-1 test 35b: Continue reconnection retries, if the active server is busy ========================================================== 04:37:45 (1734255465) start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-12-15 4h38m01s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800b7b5f800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity-1/f35b.conf-sanity-1' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-15 04:38:45.000000000 -0500 Modify: 2024-12-15 04:38:45.000000000 -0500 Change: 2024-12-15 04:38:45.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 48 lines, 48 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local modules unloaded. oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server PASS 35b (112s) == conf-sanity-1 test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 04:39:37 (1734255577) Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:) Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov -o nosvc -n lustre-mdt1/mdt1 /mnt/lustre-mds1 nomtab: 1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all Start lustre-mdt1/mdt1 without service Commit the device label on lustre-mdt1/mdt1 Started lustre:MDT0000 start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg112-server Starting mds1: -o localrecov -o nomgs,force lustre-mdt1/mdt1 /mnt/lustre-mds1 force: 1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre blah blah umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server PASS 41b (68s) == conf-sanity-1 test 46a: handle ost additional - wide striped file ========================================================== 04:40:45 (1734255645) Testing with 2 OSTs start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg112-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800a9fd8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800a9fd8000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre2..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre2 /mnt/lustre2 stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre2/f41b.conf-sanity-1 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2 0x2 0x240000400 /mnt/lustre2/widestripe lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 34 0x22 0x240000400 1 2 0x2 0x280000400 File: '/mnt/lustre/widestripe' Size: 3 Blocks: 2 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826934273 Links: 1 Access: (0674/-rw-rwxr--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-15 04:41:12.000000000 -0500 Modify: 2024-12-15 04:41:12.000000000 -0500 Change: 2024-12-15 04:41:13.000000000 -0500 Birth: - umount lustre on /mnt/lustre2..... Stopping client oleg112-client.virtnet /mnt/lustre2 (opts:) umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost2 (opts:-f) on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server stop ost1 service on oleg112-server stop mds service on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local modules unloaded. oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server PASS 46a (126s) == conf-sanity-1 test 47a: server restart does not lose lru_resize ========================================================== 04:42:51 (1734255771) Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre setup single mount lustre success ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b5dc0000.lru_size=100 ldlm.namespaces.lustre-OST0000-osc-ffff8800b5dc0000.lru_size=100 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b5dc0000.lru_size=100 Failing ost1 on oleg112-server Stopping /mnt/lustre-ost1 (opts:) on oleg112-server 04:43:38 (1734255818) shut down facet: ost1 facet_host: oleg112-server facet_failover_host: oleg112-server Failover ost1 to oleg112-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 04:43:51 (1734255831) targets are mounted 04:43:51 (1734255831) facet_failover done Failing mds1 on oleg112-server Stopping /mnt/lustre-mds1 (opts:) on oleg112-server 04:43:52 (1734255832) shut down facet: mds1 facet_host: oleg112-server facet_failover_host: oleg112-server Failover mds1 to oleg112-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 04:44:05 (1734255845) targets are mounted 04:44:05 (1734255845) facet_failover done ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b5dc0000.lru_size=100 umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 PASS 47a (94s) == conf-sanity-1 test 47b: client restart does not lose lru_resize ========================================================== 04:44:25 (1734255865) Failing mds1 on oleg112-server 04:44:27 (1734255867) shut down facet: mds1 facet_host: oleg112-server facet_failover_host: oleg112-server Failover mds1 to oleg112-server mount facets: mds1 Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 04:44:52 (1734255892) targets are mounted 04:44:52 (1734255892) facet_failover done pdsh@oleg112-client: oleg112-client: ssh exited with exit code 95 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid pdsh@oleg112-client: oleg112-client: ssh exited with exit code 95 Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre setup single mount lustre success ln: failed to create symbolic link '/home/green/git/lustre-release/lustre/utils/lctl': File exists SKIP: conf-sanity-1 test_47b Can not create symlink /usr/sbin/lctl SKIP 47b (77s) == conf-sanity-1 test 48: too many acls on file ========== 04:45:42 (1734255942) start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: mount.lustre: according to /etc/mtab lustre-mdt1/mdt1 is already mounted on /mnt/lustre-mds1 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 17 Start of lustre-mdt1/mdt1 on mds1 failed 17 start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg112-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 17 seq.cli-lustre-OST0000-super.width=65536 Start of lustre-ost1/ost1 on ost1 failed 17 mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre mount.lustre: according to /etc/mtab 192.168.201.112@tcp:/lustre is already mounted on /mnt/lustre setup single mount lustre success debug=0 debug=0 /mnt/lustre stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre/widestripe lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 4 0x4 0x240000400 It is expected to hold at least 4500 ACL entries File: '/mnt/lustre/widestripe' Size: 3 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272502275 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-15 04:45:48.000000000 -0500 Modify: 2024-12-15 04:45:48.000000000 -0500 Change: 2024-12-15 04:47:09.000000000 -0500 Birth: - getfacl: Removing leading '/' from absolute path names Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg112-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server PASS 48 (151s) == conf-sanity-1 test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:48:13 (1734256093) Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre setup single mount lustre success check ldlm_timout... umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server PASS 49a (47s) == conf-sanity-1 test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:49:00 (1734256140) Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 PASS 49b (46s) == conf-sanity-1 test 50c: lazystatfs one server down ==== 04:49:46 (1734256186) start mds service on oleg112-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg112-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre llite.lustre-ffff8800ab24e800.lazystatfs=1 stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server oleg112-server: oleg112-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.7000 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3328 378240 1% /mnt/lustre[MDT:0] lustre-OST0001_UUID 382976 3072 377856 1% /mnt/lustre[OST:1] filesystem_summary: 382976 3072 377856 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity-framework.sh: line 241: kill: (5729) - No such process umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg112-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server PASS 50c (97s) == conf-sanity-1 test 50d: lazystatfs client/server conn race ========================================================== 04:51:23 (1734256283) start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg112-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre llite.lustre-ffff8800b5dc7000.lazystatfs=1 stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.7000 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3328 378240 1% /mnt/lustre[MDT:0] lustre-OST0001_UUID 382976 3072 377856 1% /mnt/lustre[OST:1] filesystem_summary: 382976 3072 377856 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity-framework.sh: line 241: kill: (10673) - No such process umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg112-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server PASS 50d (88s) == conf-sanity-1 test 50f: normal statfs one server in down ========================================================== 04:52:51 (1734256371) start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec start ost2 service on oleg112-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec stop ost2 service on oleg112-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg112-server oleg112-server: oleg112-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre llite.lustre-ffff88012abbe000.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.7000 start ost2 service on oleg112-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0001 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg112-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg112-server umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server PASS 50f (122s) == conf-sanity-1 test 50g: deactivated OST should not cause panic ========================================================== 04:54:53 (1734256493) start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre start ost2 service on oleg112-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg112-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800a9496800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800a9496800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.112@tcp:/lustre 382976 3072 377856 1% /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg112-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg112-server stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server PASS 50g (73s) == conf-sanity-1 test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 04:56:06 (1734256566) Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg112-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec stop ost2 service on oleg112-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg112-server umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local modules unloaded. oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server PASS 51 (136s) == conf-sanity-1 test 54a: test llverdev and partial verify of device ========================================================== 04:58:22 (1734256702) SKIP: conf-sanity-1 test_54a ldiskfs only test SKIP 54a (1s) == conf-sanity-1 test 55: check lov_objid size =========== 04:58:23 (1734256703) SKIP: conf-sanity-1 test_55 ldiskfs only test SKIP 55 (1s) == conf-sanity-1 test 56a: check big OST indexes and out-of-index-order start ========================================================== 04:58:24 (1734256704) Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 Permanent disk data: Target: lustre:OST2710 Index: 10000 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.201.112@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.112@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=10000 lustre:fsname=lustre lustre:svname=lustre:OST2710 Permanent disk data: Target: lustre:OST03e8 Index: 1000 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.201.112@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost2 /dev/vdf mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2/ost2 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost2/ost2 properties lustre:mgsnode=192.168.201.112@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=1000 lustre:fsname=lustre lustre:svname=lustre:OST03e8 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST2710-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST2710 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg112-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST03e8-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST03e8 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre ok OBDS: 1000: lustre-OST03e8_UUID ACTIVE 10000: lustre-OST2710_UUID ACTIVE 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00450822 s, 909 kB/s oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 1 sec Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:) Stopping client oleg112-client.virtnet /mnt/lustre opts: Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg112-server Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server PASS 56a (105s) == conf-sanity-1 test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 05:00:09 (1734256809) SKIP: conf-sanity-1 test_56b needs >= 3 MDTs SKIP 56b (1s) == conf-sanity-1 test 57a: initial registration from failnode should fail (should return errs) ========================================================== 05:00:10 (1734256810) oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: mgsnode=192.168.201.112@tcp sys.timeout=20 autodegrade=on Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.112@tcp sys.timeout=20 autodegrade=on failover.node=192.168.201.112@tcp autodegrade=on Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.112@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:failover.node=192.168.201.112@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg112-server: mount.lustre: mount lustre-ost1/ost1 at /mnt/lustre-ost1 failed: Cannot assign requested address pdsh@oleg112-client: oleg112-server: ssh exited with exit code 99 oleg112-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory oleg112-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 Start of lustre-ost1/ost1 on ost1 failed 99 umount lustre on /mnt/lustre..... stop ost1 service on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 PASS 57a (78s) == conf-sanity-1 test 57b: initial registration from servicenode should not fail ========================================================== 05:01:28 (1734256888) oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: mgsnode=192.168.201.112@tcp sys.timeout=20 autodegrade=on Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x1042 (OST update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.201.112@tcp sys.timeout=20 autodegrade=on failover.node=192.168.201.112@tcp autodegrade=on Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.112@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:failover.node=192.168.201.112@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4162 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid umount lustre on /mnt/lustre..... stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 PASS 57b (95s) == conf-sanity-1 test 59: writeconf mount option ========= 05:03:03 (1734256983) oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server original ost count: 1 (expect > 0) oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid after mdt writeconf count: 0 (expect 0) OST start without writeconf should fail: oleg112-server: mount.lustre: mount lustre-ost1/ost1 at /mnt/lustre-ost1 failed: No such file or directory oleg112-server: Is the MGS specification correct? oleg112-server: Is the filesystem name correct? oleg112-server: If upgrading, is the copied client log valid? (see upgrade docs) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 oleg112-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory oleg112-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 OST start with writeconf should succeed: oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid after ost writeconf count: 1 (expect 1) oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid after ost2 writeconf count: 2 (expect 2) oleg112-server: oleg112-server.virtnet: executing unload_modules_local oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server PASS 59 (126s) == conf-sanity-1 test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 05:05:09 (1734257109) SKIP: conf-sanity-1 test_60a ldiskfs only test SKIP 60a (1s) == conf-sanity-1 test 62: start with disabled journal ==== 05:05:11 (1734257111) SKIP: conf-sanity-1 test_62 ldiskfs only test SKIP 62 (2s) == conf-sanity-1 test 64: check lfs df --lazy ============ 05:05:12 (1734257112) start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg112-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre stop ost2 service on oleg112-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg112-server /home/green/git/lustre-release/lustre/utils/lfs df UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3328 378240 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local modules unloaded. oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg112-server: oleg112-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg112-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server PASS 64 (121s) == conf-sanity-1 test 66: replace nids =================== 05:07:13 (1734257233) client=34615898 MDS=34615898 OSS=34615898 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre Setting lustre-OST0000.osc.active from 1 to 0 Waiting 90s for '0' Updated after 10s: want '0' got '0' replace_nids should fail if MDS, OSTs and clients are UP oleg112-server: error: replace_nids: Operation now in progress pdsh@oleg112-client: oleg112-server: ssh exited with exit code 115 umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) replace_nids should fail if MDS and OSTs are UP oleg112-server: error: replace_nids: Operation now in progress pdsh@oleg112-client: oleg112-server: ssh exited with exit code 115 stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server replace_nids should fail if MDS is UP oleg112-server: error: replace_nids: Operation now in progress pdsh@oleg112-client: oleg112-server: ssh exited with exit code 115 stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server start mds service on oleg112-server Starting mds1: -o localrecov -o nosvc lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all Start lustre-mdt1/mdt1 without service Started lustre-MDT0000 command should accept two parameters pdsh@oleg112-client: oleg112-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] correct device name should be passed oleg112-server: error: replace_nids: No such device or address pdsh@oleg112-client: oleg112-server: ssh exited with exit code 6 wrong nids list should not destroy the system pdsh@oleg112-client: oleg112-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] pdsh@oleg112-client: oleg112-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] replace OST nid command should accept two parameters pdsh@oleg112-client: oleg112-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] wrong nids list should not destroy the system pdsh@oleg112-client: oleg112-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] set NIDs with failover replace MDS nid stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Setting lustre-OST0000.osc.active from 0 to 1 Waiting 90s for '1' Updated after 4s: want '1' got '1' start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local modules unloaded. Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server PASS 66 (148s) == conf-sanity-1 test 70e: Sync-on-Cancel will be enabled by default on DNE ========================================================== 05:09:41 (1734257381) SKIP: conf-sanity-1 test_70e needs >= 2 MDTs SKIP 70e (1s) == conf-sanity-1 test 75: The order of --index should be irrelevant ========================================================== 05:09:42 (1734257382) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Permanent disk data: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x45 (MDT MGS update ) Persistent mount opts: Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mkfs_cmd = zpool create -f -O canmount=off lustre-mdt1 /dev/vdc mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt1/mdt1 xattr=sa dnodesize=auto Writing lustre-mdt1/mdt1 properties lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=69 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-MDT0000 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.112@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.112@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 Permanent disk data: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x45 (MDT MGS update ) Persistent mount opts: Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mkfs_cmd = zpool create -f -O canmount=off lustre-mdt1 /dev/vdc mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt1/mdt1 xattr=sa dnodesize=auto Writing lustre-mdt1/mdt1 properties lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=69 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-MDT0000 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.112@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.112@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server PASS 75 (57s) == conf-sanity-1 test 78: run resize2fs on MDT and OST filesystems ========================================================== 05:10:39 (1734257439) SKIP: conf-sanity-1 test_78 ldiskfs only test SKIP 78 (1s) == conf-sanity-1 test 79: format MDT/OST without mgs option (should return errors) ========================================================== 05:10:40 (1734257440) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg112-server: oleg112-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode oleg112-server: mkfs.lustre: exiting with 22 (Invalid argument) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 22 oleg112-server: oleg112-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode oleg112-server: mkfs.lustre: exiting with 22 (Invalid argument) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 22 oleg112-server: oleg112-server: mkfs.lustre FATAL: Must specify --mgsnode oleg112-server: mkfs.lustre: exiting with 22 (Invalid argument) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 22 Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server PASS 79 (62s) == conf-sanity-1 test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 05:11:42 (1734257502) SKIP: conf-sanity-1 test_83 ldiskfs only test SKIP 83 (2s) == conf-sanity-1 test 86: Replacing mkfs.lustre -G option ========================================================== 05:11:44 (1734257504) SKIP: conf-sanity-1 test_86 LU-6442: no such mkfs params for ZFS OSTs SKIP 86 (1s) == conf-sanity-1 test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 05:11:45 (1734257505) SKIP: conf-sanity-1 test_87 ldiskfs only test SKIP 87 (1s) == conf-sanity-1 test 89: check tunefs --param and --erase-param{s} options ========================================================== 05:11:46 (1734257506) Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:) tunefs --param failover.node=192.0.2.254@tcp0 tunefs --param failover.node=192.0.2.255@tcp0 tunefs --erase-param failover.node tunefs --erase-params tunefs --param failover.node=192.0.2.254@tcp0 --erase-params Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server PASS 89 (54s) == conf-sanity-1 test 93: register mulitple MDT at the same time ========================================================== 05:12:40 (1734257560) SKIP: conf-sanity-1 test_93 needs >= 3 MDTs SKIP 93 (1s) == conf-sanity-1 test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 05:12:41 (1734257561) Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre setup single mount lustre success lustre-MDT0000: lustre-OST0000: umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 PASS 100 (55s) == conf-sanity-1 test 104a: Make sure user defined options are reflected in mount ========================================================== 05:13:36 (1734257616) mountfsopt: acl,user_xattr Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 Starting mds1: -o localrecov,noacl lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre setfacl: /mnt/lustre: Operation not supported PASS 104a (48s) == conf-sanity-1 test 105: check file creation for ro and rw bind mnt pt ========================================================== 05:14:24 (1734257664) umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local oleg112-server: rmmod: ERROR: Module lustre is in use pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 modules unloaded. Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-ost2 (opts:-f) on oleg112-server oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre touch: cannot touch '/tmp/d105.conf-sanity-1/f105.conf-sanity-1': Read-only file system umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 PASS 105 (87s) == conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:15:51 (1734257751) Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2045 (time 1734257808.36 total 10.00 last 204.45) - open/close 4134 (time 1734257818.36 total 20.01 last 208.84) - open/close 6354 (time 1734257828.36 total 30.01 last 221.97) - open/close 8405 (time 1734257838.37 total 40.01 last 205.02) - open/close 10000 (time 1734257846.60 total 48.24 last 193.71) - open/close 11595 (time 1734257856.60 total 58.25 last 159.43) - open/close 13394 (time 1734257866.60 total 68.25 last 179.89) - open/close 15252 (time 1734257876.60 total 78.25 last 185.80) - open/close 17165 (time 1734257886.61 total 88.25 last 191.27) - open/close 19534 (time 1734257896.61 total 98.25 last 236.87) - open/close 22083 (time 1734257906.61 total 108.25 last 254.89) - open/close 23728 (time 1734257916.61 total 118.25 last 164.47) - open/close 25255 (time 1734257926.61 total 128.26 last 152.62) - open/close 26830 (time 1734257936.62 total 138.26 last 157.42) - open/close 28716 (time 1734257946.62 total 148.27 last 188.53) - open/close 30000 (time 1734257952.23 total 153.88 last 228.99) - open/close 31682 (time 1734257962.24 total 163.88 last 168.12) - open/close 33457 (time 1734257972.24 total 173.88 last 177.50) - open/close 34865 (time 1734257982.24 total 183.88 last 140.76) - open/close 36208 (time 1734257992.24 total 193.89 last 134.23) - open/close 37458 (time 1734258002.25 total 203.89 last 124.95) - open/close 38707 (time 1734258012.25 total 213.90 last 124.82) - open/close 39977 (time 1734258022.26 total 223.90 last 126.96) - open/close 41416 (time 1734258032.27 total 233.91 last 143.78) - open/close 42564 (time 1734258042.28 total 243.92 last 114.70) - open/close 43542 (time 1734258052.28 total 253.93 last 97.74) - open/close 44649 (time 1734258062.28 total 263.93 last 110.69) - open/close 45737 (time 1734258072.28 total 273.93 last 108.78) - open/close 46768 (time 1734258082.29 total 283.93 last 103.05) - open/close 48034 (time 1734258092.29 total 293.94 last 126.56) - open/close 49374 (time 1734258102.30 total 303.94 last 133.89) - open/close 50000 (time 1734258107.46 total 309.10 last 121.37) - open/close 51293 (time 1734258117.46 total 319.11 last 129.22) - open/close 52557 (time 1734258127.47 total 329.11 last 126.37) - open/close 53890 (time 1734258137.47 total 339.12 last 133.22) - open/close 55324 (time 1734258147.48 total 349.12 last 143.33) - open/close 56612 (time 1734258157.48 total 359.12 last 128.77) - open/close 57905 (time 1734258167.48 total 369.12 last 129.30) - open/close 59424 (time 1734258177.48 total 379.13 last 151.88) - open/close 60000 (time 1734258181.41 total 383.05 last 146.76) - open/close 61404 (time 1734258191.41 total 393.05 last 140.35) - open/close 62887 (time 1734258201.42 total 403.06 last 148.21) - open/close 64416 (time 1734258211.42 total 413.06 last 152.88) total: 64768 open/close in 415.38 seconds: 155.92 ops/second - /unlink 2077 (time 1734258225.26 total 10.00 last 207.63) - /unlink 3953 (time 1734258235.26 total 20.00 last 187.59) - /unlink 5775 (time 1734258245.26 total 30.01 last 182.16) - /unlink 7615 (time 1734258255.27 total 40.01 last 183.92) - /unlink 9497 (time 1734258265.27 total 50.01 last 188.13) - /unlink 10000 (time 1734258267.77 total 52.51 last 201.16) - /unlink 11807 (time 1734258277.77 total 62.52 last 180.70) - /unlink 13634 (time 1734258287.78 total 72.52 last 182.64) - /unlink 15407 (time 1734258297.78 total 82.52 last 177.29) - /unlink 17402 (time 1734258307.78 total 92.52 last 199.46) - /unlink 19168 (time 1734258317.78 total 102.53 last 176.53) - /unlink 20000 (time 1734258322.75 total 107.50 last 167.34) - /unlink 21697 (time 1734258332.76 total 117.50 last 169.67) - /unlink 23465 (time 1734258342.76 total 127.50 last 176.73) - /unlink 25618 (time 1734258352.76 total 137.50 last 215.26) - /unlink 27976 (time 1734258362.77 total 147.51 last 235.71) - /unlink 30000 (time 1734258371.62 total 156.37 last 228.50) - /unlink 32007 (time 1734258381.62 total 166.37 last 200.67) - /unlink 34081 (time 1734258391.63 total 176.37 last 207.29) - /unlink 36160 (time 1734258401.63 total 186.37 last 207.89) - /unlink 38263 (time 1734258411.63 total 196.38 last 210.24) - /unlink 40000 (time 1734258420.07 total 204.81 last 205.89) - /unlink 42230 (time 1734258430.07 total 214.82 last 222.92) - /unlink 44405 (time 1734258440.07 total 224.82 last 217.48) - /unlink 46909 (time 1734258450.08 total 234.82 last 250.37) - /unlink 49416 (time 1734258460.08 total 244.82 last 250.69) - /unlink 50000 (time 1734258462.52 total 247.26 last 238.99) - /unlink 52734 (time 1734258472.52 total 257.26 last 273.37) - /unlink 55632 (time 1734258482.52 total 267.27 last 289.74) - /unlink 58790 (time 1734258492.52 total 277.27 last 315.74) - /unlink 60000 (time 1734258496.35 total 281.09 last 316.21) - /unlink 63221 (time 1734258506.35 total 291.09 last 322.08) total: 64768 /unlink in 296.42 seconds: 218.50 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1335 (time 1734258540.08 total 10.00 last 133.49) - open/close 2661 (time 1734258550.08 total 20.01 last 132.52) - open/close 3950 (time 1734258560.09 total 30.01 last 128.84) - open/close 5290 (time 1734258570.09 total 40.01 last 133.98) - open/close 6560 (time 1734258580.09 total 50.02 last 126.97) - open/close 7800 (time 1734258590.09 total 60.02 last 123.94) - open/close 9014 (time 1734258600.10 total 70.03 last 121.33) - open/close 10000 (time 1734258606.83 total 76.76 last 146.45) - open/close 11249 (time 1734258616.84 total 86.77 last 124.81) - open/close 12553 (time 1734258626.84 total 96.77 last 130.37) - open/close 13865 (time 1734258636.85 total 106.77 last 131.17) - open/close 15105 (time 1734258646.85 total 116.78 last 123.89) - open/close 16413 (time 1734258656.86 total 126.79 last 130.71) - open/close 17765 (time 1734258666.86 total 136.79 last 135.15) - open/close 19115 (time 1734258676.87 total 146.79 last 134.93) - open/close 20000 (time 1734258683.66 total 153.58 last 130.32) - open/close 21547 (time 1734258693.66 total 163.59 last 154.64) - open/close 22949 (time 1734258703.67 total 173.60 last 140.11) - open/close 24375 (time 1734258713.67 total 183.60 last 142.55) - open/close 25590 (time 1734258723.68 total 193.60 last 121.47) - open/close 26803 (time 1734258733.68 total 203.60 last 121.28) - open/close 28011 (time 1734258743.69 total 213.61 last 120.67) - open/close 29255 (time 1734258753.69 total 223.62 last 124.38) - open/close 30000 (time 1734258759.58 total 229.50 last 126.58) - open/close 31229 (time 1734258769.58 total 239.51 last 122.84) - open/close 32583 (time 1734258779.59 total 249.51 last 135.30) - open/close 33981 (time 1734258789.59 total 259.52 last 139.76) - open/close 35583 (time 1734258799.59 total 269.52 last 160.18) - open/close 37038 (time 1734258809.60 total 279.52 last 145.47) - open/close 38371 (time 1734258819.60 total 289.52 last 133.26) - open/close 40000 (time 1734258829.26 total 299.18 last 168.69) - open/close 41487 (time 1734258839.26 total 309.19 last 148.62) - open/close 42949 (time 1734258849.26 total 319.19 last 146.15) - open/close 44381 (time 1734258859.27 total 329.19 last 143.17) - open/close 45823 (time 1734258869.27 total 339.19 last 144.15) - open/close 47234 (time 1734258879.27 total 349.20 last 141.04) - open/close 48684 (time 1734258889.28 total 359.20 last 144.92) - open/close 50000 (time 1734258897.96 total 367.89 last 151.55) - open/close 51432 (time 1734258907.97 total 377.90 last 143.06) - open/close 52861 (time 1734258917.97 total 387.90 last 142.89) - open/close 54402 (time 1734258927.98 total 397.90 last 154.03) - open/close 56057 (time 1734258937.98 total 407.90 last 165.46) - open/close 57804 (time 1734258947.98 total 417.91 last 174.62) - open/close 59443 (time 1734258957.99 total 427.91 last 163.81) - open/close 60000 (time 1734258961.76 total 431.68 last 147.92) - open/close 61314 (time 1734258971.76 total 441.68 last 131.37) - open/close 62686 (time 1734258981.76 total 451.69 last 137.16) - open/close 64048 (time 1734258991.77 total 461.70 last 136.03) total: 64768 open/close in 467.11 seconds: 138.66 ops/second - /unlink 2518 (time 1734259008.67 total 10.00 last 251.70) - /unlink 5129 (time 1734259018.67 total 20.01 last 261.00) - /unlink 7892 (time 1734259028.67 total 30.01 last 276.24) - /unlink 10000 (time 1734259037.74 total 39.07 last 232.61) - /unlink 12482 (time 1734259047.74 total 49.07 last 248.19) - /unlink 15105 (time 1734259057.74 total 59.07 last 262.28) - /unlink 17504 (time 1734259067.74 total 69.08 last 239.80) - /unlink 19898 (time 1734259077.75 total 79.08 last 239.24) - /unlink 22474 (time 1734259087.75 total 89.09 last 257.55) - /unlink 24936 (time 1734259097.75 total 99.09 last 246.16) - /unlink 27409 (time 1734259107.76 total 109.09 last 247.21) - /unlink 29661 (time 1734259117.76 total 119.09 last 225.13) - /unlink 31925 (time 1734259127.76 total 129.10 last 226.27) - /unlink 34200 (time 1734259137.76 total 139.10 last 227.48) - /unlink 36494 (time 1734259147.77 total 149.10 last 229.39) - /unlink 38424 (time 1734259157.77 total 159.10 last 192.94) - /unlink 40000 (time 1734259165.56 total 166.89 last 202.35) - /unlink 42201 (time 1734259175.56 total 176.90 last 220.01) - /unlink 44343 (time 1734259185.56 total 186.90 last 214.12) - /unlink 46573 (time 1734259195.57 total 196.90 last 222.94) - /unlink 48780 (time 1734259205.57 total 206.90 last 220.67) - /unlink 50000 (time 1734259210.79 total 212.12 last 233.87) - /unlink 52192 (time 1734259220.79 total 222.12 last 219.14) - /unlink 54465 (time 1734259230.79 total 232.13 last 227.26) - /unlink 56743 (time 1734259240.79 total 242.13 last 227.75) - /unlink 59194 (time 1734259250.79 total 252.13 last 245.09) - /unlink 60000 (time 1734259253.97 total 255.30 last 253.94) - /unlink 62659 (time 1734259263.97 total 265.30 last 265.87) total: 64768 /unlink in 273.18 seconds: 237.09 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1387 (time 1734259301.76 total 10.00 last 138.66) - open/close 2867 (time 1734259311.76 total 20.01 last 147.94) - open/close 4284 (time 1734259321.77 total 30.01 last 141.67) - open/close 5673 (time 1734259331.77 total 40.02 last 138.81) - open/close 6822 (time 1734259341.78 total 50.02 last 114.86) - open/close 8298 (time 1734259351.78 total 60.03 last 147.51) - open/close 9717 (time 1734259361.78 total 70.03 last 141.86) - open/close 10000 (time 1734259363.86 total 72.10 last 136.55) - open/close 11297 (time 1734259373.86 total 82.10 last 129.69) - open/close 12868 (time 1734259383.86 total 92.10 last 157.06) - open/close 14465 (time 1734259393.87 total 102.11 last 159.63) - open/close 16049 (time 1734259403.87 total 112.11 last 158.36) - open/close 17627 (time 1734259413.87 total 122.12 last 157.69) - open/close 19321 (time 1734259423.88 total 132.12 last 169.35) - open/close 20000 (time 1734259427.91 total 136.15 last 168.43) - open/close 21626 (time 1734259437.91 total 146.16 last 162.50) - open/close 23187 (time 1734259447.92 total 156.16 last 156.02) - open/close 24732 (time 1734259457.92 total 166.17 last 154.47) - open/close 26549 (time 1734259467.92 total 176.17 last 181.66) - open/close 28477 (time 1734259477.93 total 186.17 last 192.75) - open/close 30000 (time 1734259486.27 total 194.51 last 182.53) - open/close 32064 (time 1734259496.27 total 204.52 last 206.36) - open/close 34019 (time 1734259506.27 total 214.52 last 195.47) - open/close 36003 (time 1734259516.27 total 224.52 last 198.38) - open/close 37967 (time 1734259526.28 total 234.52 last 196.33) - open/close 39763 (time 1734259536.28 total 244.52 last 179.58) - open/close 41369 (time 1734259546.28 total 254.53 last 160.53) - open/close 42757 (time 1734259556.29 total 264.53 last 138.70) - open/close 44278 (time 1734259566.30 total 274.54 last 152.02) - open/close 45853 (time 1734259576.30 total 284.54 last 157.45) - open/close 47362 (time 1734259586.30 total 294.55 last 150.87) - open/close 48915 (time 1734259596.31 total 304.55 last 155.22) - open/close 50000 (time 1734259603.48 total 311.73 last 151.22) - open/close 51741 (time 1734259613.48 total 321.73 last 174.06) - open/close 53426 (time 1734259623.49 total 331.73 last 168.41) - open/close 55035 (time 1734259633.49 total 341.74 last 160.84) - open/close 56680 (time 1734259643.49 total 351.74 last 164.47) - open/close 58349 (time 1734259653.50 total 361.74 last 166.87) - open/close 59940 (time 1734259663.50 total 371.74 last 159.08) - open/close 61651 (time 1734259673.50 total 381.74 last 171.04) - open/close 63253 (time 1734259683.50 total 391.75 last 160.19) total: 64768 open/close in 399.99 seconds: 161.92 ops/second - /unlink 2955 (time 1734259703.10 total 10.00 last 295.43) - /unlink 5895 (time 1734259713.10 total 20.00 last 293.92) - /unlink 8388 (time 1734259723.10 total 30.01 last 249.29) - /unlink 10000 (time 1734259729.41 total 36.31 last 255.58) - /unlink 12870 (time 1734259739.41 total 46.31 last 286.96) - /unlink 16117 (time 1734259749.41 total 56.32 last 324.59) - /unlink 19144 (time 1734259759.41 total 66.32 last 302.67) - /unlink 20000 (time 1734259762.16 total 69.07 last 311.29) - /unlink 22853 (time 1734259772.16 total 79.07 last 285.22) - /unlink 26502 (time 1734259782.17 total 89.07 last 364.84) - /unlink 30000 (time 1734259792.05 total 98.96 last 353.77) - /unlink 33423 (time 1734259802.05 total 108.96 last 342.27) - /unlink 36759 (time 1734259812.06 total 118.96 last 333.54) - /unlink 40000 (time 1734259821.31 total 128.21 last 350.38) - /unlink 43069 (time 1734259831.31 total 138.22 last 306.84) - /unlink 45944 (time 1734259841.31 total 148.22 last 287.37) - /unlink 48819 (time 1734259851.32 total 158.22 last 287.40) - /unlink 50000 (time 1734259854.93 total 161.83 last 327.11) - /unlink 52835 (time 1734259864.93 total 171.84 last 283.43) - /unlink 55539 (time 1734259874.93 total 181.84 last 270.38) - /unlink 58433 (time 1734259884.93 total 191.84 last 289.36) - /unlink 60000 (time 1734259889.69 total 196.60 last 329.22) - /unlink 63248 (time 1734259899.69 total 206.60 last 324.74) total: 64768 /unlink in 211.11 seconds: 306.80 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2026 (time 1734259932.66 total 10.01 last 202.49) - open/close 4106 (time 1734259942.66 total 20.01 last 207.96) - open/close 6125 (time 1734259952.66 total 30.01 last 201.89) - open/close 8364 (time 1734259962.66 total 40.01 last 223.84) - open/close 10000 (time 1734259970.51 total 47.86 last 208.45) - open/close 12110 (time 1734259980.51 total 57.86 last 210.99) - open/close 14184 (time 1734259990.51 total 67.86 last 207.38) - open/close 16342 (time 1734260000.51 total 77.86 last 215.79) - open/close 18633 (time 1734260010.51 total 87.86 last 229.09) - open/close 20000 (time 1734260016.81 total 94.16 last 217.17) - open/close 22201 (time 1734260026.81 total 104.16 last 220.08) - open/close 24448 (time 1734260036.81 total 114.16 last 224.62) - open/close 26593 (time 1734260046.81 total 124.16 last 214.45) - open/close 28724 (time 1734260056.82 total 134.17 last 213.05) - open/close 30000 (time 1734260062.95 total 140.30 last 208.03) - open/close 31841 (time 1734260072.95 total 150.30 last 184.06) - open/close 33718 (time 1734260082.96 total 160.31 last 187.62) - open/close 35810 (time 1734260092.96 total 170.31 last 209.18) - open/close 37732 (time 1734260102.96 total 180.31 last 192.20) - open/close 39789 (time 1734260112.96 total 190.31 last 205.59) - open/close 41866 (time 1734260122.96 total 200.31 last 207.68) - open/close 44079 (time 1734260132.97 total 210.32 last 221.26) - open/close 46357 (time 1734260142.97 total 220.32 last 227.73) - open/close 48615 (time 1734260152.97 total 230.32 last 225.75) - open/close 50000 (time 1734260158.24 total 235.59 last 262.90) - open/close 52414 (time 1734260168.24 total 245.59 last 241.38) - open/close 54964 (time 1734260178.24 total 255.59 last 254.97) - open/close 57954 (time 1734260188.24 total 265.59 last 298.91) - open/close 60000 (time 1734260194.65 total 272.00 last 319.21) - open/close 62902 (time 1734260204.66 total 282.01 last 290.13) total: 64768 open/close in 287.76 seconds: 225.08 ops/second - /unlink 5697 (time 1734260221.39 total 10.00 last 569.61) - /unlink 10000 (time 1734260228.36 total 16.97 last 617.42) - /unlink 15886 (time 1734260238.36 total 26.97 last 588.54) - /unlink 20000 (time 1734260246.33 total 34.94 last 516.30) - /unlink 25534 (time 1734260256.33 total 44.94 last 553.36) - /unlink 30000 (time 1734260263.95 total 52.56 last 585.92) - /unlink 35838 (time 1734260273.96 total 62.56 last 583.72) - /unlink 40000 (time 1734260280.95 total 69.56 last 594.85) - /unlink 45805 (time 1734260290.95 total 79.56 last 580.49) - /unlink 50000 (time 1734260297.95 total 86.56 last 599.51) - /unlink 54901 (time 1734260307.95 total 96.56 last 490.04) - /unlink 59416 (time 1734260317.95 total 106.56 last 451.48) - /unlink 63884 (time 1734260327.95 total 116.56 last 446.73) total: 64768 /unlink in 118.12 seconds: 548.34 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3115 (time 1734260357.52 total 10.00 last 311.46) - open/close 6604 (time 1734260367.52 total 20.00 last 348.86) - open/close 9563 (time 1734260377.52 total 30.00 last 295.84) - open/close 12503 (time 1734260387.52 total 40.01 last 293.94) - open/close 15536 (time 1734260397.52 total 50.01 last 303.26) - open/close 18882 (time 1734260407.53 total 60.01 last 334.57) - open/close 20000 (time 1734260411.11 total 63.59 last 311.79) - open/close 22818 (time 1734260421.11 total 73.60 last 281.76) - open/close 25814 (time 1734260431.12 total 83.60 last 299.49) - open/close 28385 (time 1734260441.12 total 93.60 last 257.04) - open/close 30000 (time 1734260448.72 total 101.21 last 212.38) - open/close 32246 (time 1734260458.73 total 111.21 last 224.40) - open/close 34565 (time 1734260468.73 total 121.22 last 231.86) - open/close 36737 (time 1734260478.74 total 131.22 last 217.03) - open/close 39151 (time 1734260488.74 total 141.23 last 241.33) - open/close 40000 (time 1734260492.13 total 144.62 last 250.57) - open/close 42538 (time 1734260502.14 total 154.62 last 253.71) - open/close 45446 (time 1734260512.14 total 164.62 last 290.69) - open/close 48678 (time 1734260522.14 total 174.62 last 323.14) - open/close 50000 (time 1734260525.85 total 178.33 last 356.67) - open/close 53337 (time 1734260535.85 total 188.33 last 333.70) - open/close 56311 (time 1734260545.85 total 198.33 last 297.35) - open/close 59368 (time 1734260555.85 total 208.33 last 305.69) - open/close 60000 (time 1734260558.47 total 210.95 last 241.24) - open/close 63471 (time 1734260568.47 total 220.95 last 347.09) total: 64768 open/close in 224.42 seconds: 288.60 ops/second - /unlink 6896 (time 1734260582.87 total 10.00 last 689.53) - /unlink 10000 (time 1734260587.17 total 14.30 last 722.72) - /unlink 16000 (time 1734260597.17 total 24.30 last 599.96) - /unlink 20000 (time 1734260603.64 total 30.77 last 618.19) - /unlink 27520 (time 1734260613.64 total 40.77 last 751.91) - /unlink 30000 (time 1734260616.73 total 43.85 last 803.56) - /unlink 36596 (time 1734260626.73 total 53.86 last 659.54) - /unlink 40000 (time 1734260631.05 total 58.18 last 787.41) - /unlink 48842 (time 1734260641.05 total 68.18 last 884.13) - /unlink 54706 (time 1734260651.05 total 78.18 last 586.36) - /unlink 60000 (time 1734260659.90 total 87.02 last 598.56) total: 64768 /unlink in 94.98 seconds: 681.94 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 PASS 106 (2947s) == conf-sanity-1 test 109a: test lctl clear_conf fsname == 06:04:58 (1734260698) SKIP: conf-sanity-1 test_109a LU-8727: no implementation for ZFS SKIP 109a (1s) == conf-sanity-1 test 109b: test lctl clear_conf one config ========================================================== 06:04:59 (1734260699) SKIP: conf-sanity-1 test_109b LU-8727: no implementation for ZFS SKIP 109b (0s) == conf-sanity-1 test 116: big size MDT support ========== 06:04:59 (1734260699) SKIP: conf-sanity-1 test_116 ldiskfs only test SKIP 116 (1s) == conf-sanity-1 test 122a: Check OST sequence update ==== 06:05:00 (1734260700) SKIP: conf-sanity-1 test_122a needs >= 2 MDTs SKIP 122a (1s) == conf-sanity-1 test 123ah: del_ost cancels config log entries correctly ========================================================== 06:05:01 (1734260701) Checking servers environments Checking clients oleg112-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' Setup mgs, mdt, osts Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre Starting client oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre Started clients oleg112-client.virtnet: 192.168.201.112@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800ab24f000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800ab24f000.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 2s: want 'procname_uid' got 'procname_uid' disable quota as required del_ost: dry run for target lustre-OST0000 config_log: lustre-MDT0000 [DRY RUN] cancel catalog 'lustre-MDT0000:20':"- { index: 20, event: add_osc, device: lustre-MDT0000-mdtlov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }" [DRY RUN] cancel catalog 'lustre-MDT0000:19':"- { index: 19, event: setup, device: lustre-OST0000-osc-MDT0000, UUID: lustre-OST0000_UUID, node: 192.168.201.112@tcp }" [DRY RUN] cancel catalog 'lustre-MDT0000:18':"- { index: 18, event: attach, device: lustre-OST0000-osc-MDT0000, type: osc, UUID: lustre-MDT0000-mdtlov_UUID }" del_ost: no catalog entry deleted config_log: lustre-client [DRY RUN] cancel catalog 'lustre-client:25':"- { index: 25, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }" [DRY RUN] cancel catalog 'lustre-client:24':"- { index: 24, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.201.112@tcp }" [DRY RUN] cancel catalog 'lustre-client:23':"- { index: 23, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID }" del_ost: no catalog entry deleted config_log: lustre-MDT0000 cancel catalog lustre-MDT0000 log_idx 20: done cancel catalog lustre-MDT0000 log_idx 19: done cancel catalog lustre-MDT0000 log_idx 18: done del_ost: cancelled 3 catalog entries config_log: lustre-client cancel catalog lustre-client log_idx 25: done cancel catalog lustre-client log_idx 24: done cancel catalog lustre-client log_idx 23: done del_ost: cancelled 3 catalog entries umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local oleg112-server: rmmod: ERROR: Module lustre is in use pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 modules unloaded. Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-ost2 (opts:-f) on oleg112-server oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server fail_loc=0 PASS 123ah (94s) == conf-sanity-1 test 132: hsm_actions processed after failover ========================================================== 06:06:36 (1734260796) Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x5 (MDT MGS ) Persistent mount opts: Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x45 (MDT MGS update ) Persistent mount opts: Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mdt.hsm_control=enabled Writing lustre-mdt1/mdt1 properties lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:mdt.hsm_control=enabled lustre:version=1 lustre:flags=69 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-MDT0000 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server PASS 132 (23s) == conf-sanity-1 test 133: stripe QOS: free space balance in a pool ========================================================== 06:06:58 (1734260818) SKIP: conf-sanity-1 test_133 needs >= 4 OSTs SKIP 133 (0s) == conf-sanity-1 test 135: check the behavior when changelog is wrapped around ========================================================== 06:06:58 (1734260818) Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre oleg112-client: fail_loc=0x1312 oleg112-client: fail_val=5 oleg112-server: fail_loc=0x1312 oleg112-server: fail_val=5 mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl1' Wrap arround changelog catalog - open/close 4185 (time 1734260854.24 total 10.00 last 418.50) total: 4500 open/close in 10.75 seconds: 418.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #12998 total: 4500 /unlink in 5.53 seconds: 813.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 4016 (time 1734260872.25 total 10.00 last 401.56) total: 4500 open/close in 11.34 seconds: 396.99 ops/second total: 4500 /unlink in 5.47 seconds: 823.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 4139 (time 1734260890.95 total 10.00 last 413.86) total: 4500 open/close in 10.82 seconds: 415.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 total: 4500 /unlink in 4.83 seconds: 931.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 total: 4500 open/close in 8.73 seconds: 515.64 ops/second total: 4500 /unlink in 4.74 seconds: 948.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 total: 4500 open/close in 8.31 seconds: 541.50 ops/second total: 4500 /unlink in 4.72 seconds: 952.49 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 10973: 22833 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local modules unloaded. PASS 135 (133s) == conf-sanity-1 test 136: don't panic with bad obdecho setup ========================================================== 06:09:11 (1734260951) SKIP: conf-sanity-1 test_136 needs >= 2 MDTs SKIP 136 (1s) == conf-sanity-1 test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:09:12 (1734260952) SKIP: conf-sanity-1 test_137 needs >= 3 MDTs SKIP 137 (1s) == conf-sanity-1 test 140: remove_updatelog script actions ========================================================== 06:09:13 (1734260953) SKIP: conf-sanity-1 test_140 needs >= 2 MDTs SKIP 140 (1s) == conf-sanity-1 test 151a: damaged local config doesn't prevent mounting ========================================================== 06:09:14 (1734260954) SKIP: conf-sanity-1 test_151a ldiskfs only test SKIP 151a (0s) == conf-sanity-1 test 151b: -ENOSPC doesn't affect mount ========================================================== 06:09:14 (1734260954) SKIP: conf-sanity-1 test_151b ldiskfs only test SKIP 151b (1s) == conf-sanity-1 test 153a: bypass invalid NIDs quickly == 06:09:15 (1734260955) Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid waiting for mount ... waiting for mount ... "192.168.201.112@tcp": { connects: 1, replied: 1, uptodate: uptodate, sec_ago: 5 } "192.168.252.112@tcp": { connects: 1, replied: 0, uptodate: discovering, sec_ago: 10 } "10.252.252.113@tcp": { connects: 0, replied: 0, uptodate: discovering, sec_ago: never } "192.168.201.112@tcp": { connects: 0, replied: 0, uptodate: discovering, sec_ago: never } setup single mount lustre success umount lustre on /mnt/lustre..... stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 PASS 153a (176s) == conf-sanity-1 test 153b: added IPv6 NID support ======= 06:12:12 (1734261132) Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server Checking servers environments Checking clients oleg112-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre Starting client oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre Started clients oleg112-client.virtnet: 192.168.201.112@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800abbde800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800abbde800.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 9s: want 'procname_uid' got 'procname_uid' disable quota as required PASS 153b (73s) == conf-sanity-1 test 153c: don't stuck on unreached NID ========================================================== 06:13:24 (1734261204) Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg112-server oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server start mds service on oleg112-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid umount lustre on /mnt/lustre..... waiting for mount ... waiting for mount ... waiting for mount ... waiting for mount ... waiting for mount ... waiting for mount ... "192.168.252.112@tcp": { connects: 3, replied: 0, uptodate: discovering, sec_ago: 0 } "192.168.252.113@tcp": { connects: 2, replied: 0, uptodate: discovering, sec_ago: 10 } "192.168.252.115@tcp": { connects: 2, replied: 0, uptodate: discovering, sec_ago: 5 } connection attempts: 7 Waiting for mount to fail mount.lustre: mount 192.168.252.112@tcp:192.168.252.113@tcp:192.168.252.115@tcp:/lustre at /mnt/lustre failed: Input/output error Is the MGS running? umount lustre on /mnt/lustre..... stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 PASS 153c (316s) == conf-sanity-1 test 154: expand .. on rename after MDT backup restore ========================================================== 06:18:40 (1734261520) SKIP: conf-sanity-1 test_154 ldiskfs only test SKIP 154 (1s) Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity-1 test complete, duration 6447 sec ======== 06:18:59 (1734261539) === conf-sanity-1: start cleanup 06:18:59 (1734261539) === === conf-sanity-1: finish cleanup 06:19:00 (1734261540) ===