-----============= acceptance-small: conf-sanity ============----- Tue Jun 3 18:16:07 EDT 2025 oleg243-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 21e 32b 32c 32newtarball 110 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a skipping tests SLOW=no: 45 69 106 111 Stopping clients: oleg243-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg243-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg243-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg243-server oleg243-server: oleg243-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg243-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg243-server: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg243-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server == conf-sanity test 0: single mount setup ================ 18:17:13 (1748989033) start mds service on oleg243-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 0 (23s) == conf-sanity test 1: start up ost twice (should return errors) ========================================================== 18:17:37 (1748989057) start mds service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost second time... start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 17 Start of lustre-ost1/ost1 on ost1 failed 17 mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 1 (35s) == conf-sanity test 2: start up mds twice (should return err) ========================================================== 18:18:12 (1748989092) start mds service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start mds second time.. start mds service on oleg243-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: mount.lustre: according to /etc/mtab lustre-mdt1/mdt1 is already mounted on /mnt/lustre-mds1 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 17 Start of lustre-mdt1/mdt1 on mds1 failed 17 start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 PASS 2 (36s) == conf-sanity test 3: mount client twice (should return err) ========================================================== 18:18:48 (1748989128) start mds service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre mount.lustre: according to /etc/mtab oleg243-server@tcp:/lustre is already mounted on /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 3 (33s) == conf-sanity test 4: force cleanup ost, then cleanup === 18:19:21 (1748989161) start mds service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 4 (55s) == conf-sanity test 5a: force cleanup mds, then cleanup == 18:20:16 (1748989216) start mds service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre USER PID ACCESS COMMAND /mnt/lustre: root kernel mount /mnt/lustre /mnt/lustre is in use by user space process. stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server killing umount waiting for umount to finish /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 367: 26683 Terminated $UMOUNT -f $MOUNT manual umount lustre on /mnt/lustre.... umount: /mnt/lustre: not mounted stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. manual umount lustre on /mnt/lustre.... umount: /mnt/lustre: not mounted /etc/mtab updated in 0 secs pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 5a (50s) == conf-sanity test 5b: Try to start a client with no MGS (should return errs) ========================================================== 18:21:06 (1748989266) start ost1 service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre mount.lustre: mount oleg243-server@tcp:/lustre at /mnt/lustre failed: Input/output error Is the MGS running? umount lustre on /mnt/lustre..... stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 5b (132s) == conf-sanity test 5c: cleanup after failed mount (bug 2712) (should return errs) ========================================================== 18:23:18 (1748989398) start mds service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount wrong.lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/wrong.lustre /mnt/lustre mount.lustre: mount oleg243-server@tcp:/wrong.lustre at /mnt/lustre failed: File name too long umount lustre on /mnt/lustre..... stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 5c (38s) == conf-sanity test 5d: mount with ost down ============== 18:23:56 (1748989436) start ost1 service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg243-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 5d (148s) == conf-sanity test 5e: delayed connect, don't crash (bug 10268) ========================================================== 18:26:24 (1748989584) start mds service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid fail_loc=0x80000506 mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. PASS pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 PASS 5e (47s) == conf-sanity test 5f: mds down, cleanup after failed mount (bug 2712) ========================================================== 18:27:11 (1748989631) SKIP: conf-sanity test_5f needs separate mgs and mds SKIP 5f (1s) == conf-sanity test 5g: handle missing debugfs =========== 18:27:12 (1748989632) modprobe: FATAL: Module lustre not found. error: get_param: param_path 'devices': No such file or directory none /sys/kernel/debug debugfs rw,relatime 0 0 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 5g (2s) == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 18:27:14 (1748989634) start mds service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg243-server fail_loc=0x80000135 start mds service on oleg243-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: mount.lustre: mount lustre-mdt1/mdt1 at /mnt/lustre-mds1 failed: No such file or directory oleg243-server: Is the MGS specification correct? oleg243-server: Is the filesystem name correct? oleg243-server: If upgrading, is the copied client log valid? (see upgrade docs) pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 Start of lustre-mdt1/mdt1 on mds1 failed 2 start mds service on oleg243-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 5h (44s) == conf-sanity test 5i: start mdt failure at mdt_quota_init() ========================================================== 18:27:58 (1748989678) start mds service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg243-server fail_loc=0x80000A05 start mds service on oleg243-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: mount.lustre: mount lustre-mdt1/mdt1 at /mnt/lustre-mds1 failed: Bad file descriptor pdsh@oleg243-client: oleg243-server: ssh exited with exit code 9 Start of lustre-mdt1/mdt1 on mds1 failed 9 start mds service on oleg243-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 5i (46s) == conf-sanity test 6: manual umount, then mount again === 18:28:44 (1748989724) start mds service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre manual umount lustre on /mnt/lustre.... mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 PASS 6 (39s) == conf-sanity test 7: manual umount, then cleanup ======= 18:29:23 (1748989763) start mds service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre manual umount lustre on /mnt/lustre.... stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 7 (42s) == conf-sanity test 8: double mount setup ================ 18:30:05 (1748989805) start mds service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre mount lustre on /mnt/lustre2..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre2 setup double mount lustre success umount lustre on /mnt/lustre2..... Stopping client oleg243-client.virtnet /mnt/lustre2 (opts:) umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 8 (52s) == conf-sanity test 9: test ptldebug and subsystem for mkfs ========================================================== 18:30:57 (1748989857) start ost1 service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid debug=inode trace subsystem_debug=mds ost lnet.debug success lnet.subsystem_debug success stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server PASS 9 (96s) == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 18:32:33 (1748989953) start mds service on oleg243-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 10a (41s) == conf-sanity test 17: Verify failed mds_postsetup won't fail assertion (2936) (should return errs) ========================================================== 18:33:14 (1748989994) SKIP: conf-sanity test_17 ldiskfs only test SKIP 17 (2s) == conf-sanity test 18: check mkfs creates large journals ========================================================== 18:33:16 (1748989996) SKIP: conf-sanity test_18 ldiskfs only test SKIP 18 (1s) == conf-sanity test 19a: start/stop MDS without OSTs ===== 18:33:17 (1748989997) start mds service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server PASS 19a (24s) == conf-sanity test 19b: start/stop OSTs without MDS ===== 18:33:41 (1748990021) start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server PASS 19b (66s) == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 18:34:47 (1748990087) start mds service on oleg243-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre setup single mount lustre success mount lustre with opts remount,ro on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o remount,ro oleg243-server@tcp:/lustre /mnt/lustre touch: cannot touch '/mnt/lustre/f20.conf-sanity': Read-only file system mount lustre with opts remount,rw on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o remount,rw oleg243-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server PASS 20 (33s) == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 18:35:20 (1748990120) start mds service on oleg243-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg243-server: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg243-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server PASS 21a (35s) == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 18:35:55 (1748990155) start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg243-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg243-server: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg243-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server PASS 21b (92s) == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 18:37:28 (1748990248) start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg243-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost2 service on oleg243-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg243-server: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 oleg243-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop ost2 service on oleg243-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server oleg243-server: oleg243-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg243-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg243-client: oleg243-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg243-server: oleg243-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg243-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg243-client: oleg243-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg243-server: oleg243-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg243-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg243-client: oleg243-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg243-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg243-client.virtnet /mnt/lustre2 (opts:-f) oleg243-server: oleg243-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg243-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg243-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg243-server: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg243-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server PASS 21c (165s) == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 18:40:12 (1748990412) SKIP: conf-sanity test_21d need separate mgs device SKIP 21d (1s) SKIP: conf-sanity test_21e skipping excluded test 21e == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 18:40:14 (1748990414) start mds service on oleg243-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Client mount with ost in logs, but none running start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg243-server: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg243-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:-f) PASS Client mount with a running ost start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre oleg243-server: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg243-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg243-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800acc85800.ost_server_uuid 40 osc.lustre-OST0000-osc-ffff8800acc85800.ost_server_uuid in FULL state after 0 sec setup single mount lustre success PASS umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 22 (55s) == conf-sanity test 23a: interrupt client during recovery mount delay ========================================================== 18:41:09 (1748990469) start mds service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg243-server Stopping client /mnt/lustre (opts: -f) Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre mount pid is 15790, mount.lustre pid is 15791 PID TTY TIME CMD 15791 pts/0 00:00:00 mount.lustre PID TTY TIME CMD waiting for mount to finish root 15790 15789 0 18:41 pts/0 00:00:00 mount -t lustre -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre root 15791 15790 0 18:41 pts/0 00:00:00 /sbin/mount.lustre oleg243-server@tcp:/lustre /mnt/lustre -o rw,user_xattr,flock root 15825 13598 0 18:42 pts/0 00:00:00 grep mount /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6009: 15790 Terminated LUSTRE="/home/green/git/lustre-release/lustre" bash -c "mount -t lustre -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre" PID1= PID2= umount lustre on /mnt/lustre..... stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 23a (82s) == conf-sanity test 23b: Simulate -EINTR during mount ==== 18:42:31 (1748990551) start mds service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid fail_loc=0x80000313 mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 23b (53s) == conf-sanity test 24a: Multiple MDTs on a single node == 18:43:24 (1748990604) pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Permanent disk data: Target: 969362ae:MDT0000 Index: 0 Lustre FS: 969362ae Mount type: zfs Flags: 0x61 (MDT first_time update ) Persistent mount opts: Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mgsnode=192.168.202.143@tcp:192.168.202.143@tcp mkfs_cmd = zpool create -f -O canmount=off lustre-mdt1_2 /tmp/lustre-mdt1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt1_2/mdt1_2 xattr=sa dnodesize=auto compression=off Writing lustre-mdt1_2/mdt1_2 properties lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:mgsnode=192.168.202.143@tcp:192.168.202.143@tcp lustre:version=1 lustre:flags=97 lustre:index=0 lustre:fsname=969362ae lustre:svname=969362ae:MDT0000 Permanent disk data: Target: 969362ae:OST0000 Index: 0 Lustre FS: 969362ae Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.143@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1_2 /tmp/lustre-ost1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.202.143@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=969362ae lustre:svname=969362ae:OST0000 start mds service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre Starting fs2mds: -o localrecov lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1_2/mdt1_2 Started 969362ae-MDT0000 Starting fs2ost: -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started 969362ae-OST0000 969362ae-MDT0000: Deregistered changelog user #1 setup single mount lustre success Failing fs2mds on oleg243-server Stopping /mnt/lustre-fs2mds (opts:) on oleg243-server reboot facets: fs2mds mount facets: fs2mds Starting fs2mds: -o localrecov lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started 969362ae-MDT0000 Failing fs2ost on oleg243-server Stopping /mnt/lustre-fs2ost (opts:) on oleg243-server reboot facets: fs2ost mount facets: fs2ost Starting fs2ost: -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started 969362ae-OST0000 Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1910156 0 1910156 0% /dev tmpfs 1910156 0 1910156 0% /dev/shm tmpfs 1910156 16868 1893288 1% /run tmpfs 1910156 0 1910156 0% /sys/fs/cgroup /dev/nbd0 30787492 12551972 16646272 43% / none 1910156 32080 1878076 2% /var/lib/stateless/writable /dev/vda 26880 26880 0 100% /home/green/git/lustre-release 192.168.200.253:/exports/state/oleg243-client.virtnet 123269120 104663040 12802048 90% /var/lib/stateless/state 192.168.200.253:/exports/state/oleg243-client.virtnet/boot 123269120 104663040 12802048 90% /boot 192.168.200.253:/exports/state/oleg243-client.virtnet/etc/kdump.conf 123269120 104663040 12802048 90% /etc/kdump.conf 192.168.200.253:/exports/testreports/52165/testresults/conf-sanity1-zfs-centos7_x86_64-centos7_x86_64 54446559232 30573986816 23872572416 57% /tmp/testlogs tmpfs 382032 0 382032 0% /run/user/0 192.168.202.143@tcp:/lustre 382976 3072 377856 1% /mnt/lustre 192.168.202.143@tcp:/969362ae 244736 3072 239616 2% /mnt/lustre2 umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server umount /mnt/lustre2 ... stopping fs2mds ... Stopping /mnt/lustre-fs2mds (opts:-f) on oleg243-server stopping fs2ost ... Stopping /mnt/lustre-fs2ost (opts:-f) on oleg243-server stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 24a (115s) == conf-sanity test 24b: Multiple MGSs on a single node (should return err) ========================================================== 18:45:19 (1748990719) pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Permanent disk data: Target: lustre2:MDT0000 Index: 0 Lustre FS: lustre2 Mount type: zfs Flags: 0x65 (MDT MGS first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.143@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mkfs_cmd = zpool create -f -O canmount=off lustre-mdt1_2 /tmp/lustre-mdt1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt1_2/mdt1_2 xattr=sa dnodesize=auto compression=off Writing lustre-mdt1_2/mdt1_2 properties lustre:mgsnode=192.168.202.143@tcp lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=101 lustre:index=0 lustre:fsname=lustre2 lustre:svname=lustre2:MDT0000 start mds service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_refcnt: could not read integer from '/sys/module/pcc_cpufreq/refcnt': 'No such device' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre Starting fs2mds: -o localrecov lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds oleg243-server: mount.lustre: mount lustre-mdt1_2/mdt1_2 at /mnt/lustre-fs2mds failed: Operation already in progress oleg243-server: The target service is already running. (lustre-mdt1_2/mdt1_2) pdsh@oleg243-client: oleg243-server: ssh exited with exit code 114 Start of lustre-mdt1_2/mdt1_2 on fs2mds failed 114 umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 24b (45s) == conf-sanity test 25: Verify modules are referenced ==== 18:46:04 (1748990764) start mds service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 PASS 25 (48s) == conf-sanity test 26: MDT startup failure cleans LOV (should return errs) ========================================================== 18:46:53 (1748990813) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' fail_loc=0x80000135 start mds service on oleg243-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: mount.lustre: mount lustre-mdt1/mdt1 at /mnt/lustre-mds1 failed: No such file or directory oleg243-server: Is the MGS specification correct? oleg243-server: Is the filesystem name correct? oleg243-server: If upgrading, is the copied client log valid? (see upgrade docs) pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 Start of lustre-mdt1/mdt1 on mds1 failed 2 start mds service on oleg243-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 26 (33s) == conf-sanity test 27a: Reacquire MGS lock if OST started first ========================================================== 18:47:26 (1748990846) umount lustre on /mnt/lustre..... stop ost1 service on oleg243-server stop mds service on oleg243-server unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local modules unloaded. start ost1 service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg243-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Requeue thread should have started: Setting lustre-OST0000.ost.client_cache_seconds from 110 to 115 Waiting 90s for '115' Updated after 2s: want '115' got '115' stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 27a (137s) == conf-sanity test 27b: Reacquire MGS lock after failover ========================================================== 18:49:43 (1748990983) start mds service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre Failing mds1 on oleg243-server Stopping /mnt/lustre-mds1 (opts:) on oleg243-server reboot facets: mds1 Failover mds1 to oleg243-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 Setting lustre-MDT0000.mdt.identity_acquire_expire from 30 to 35 Waiting 90s for '35' Updated after 6s: want '35' got '35' Setting lustre-MDT0000.mdc.max_rpcs_in_flight from 8 to 13 Waiting 90s for '13' Updated after 2s: want '13' got '13' setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 27b (63s) == conf-sanity test 28A: permanent parameter setting ===== 18:50:46 (1748991046) start mds service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre ORIG:29 MAX:29 ORIG:26 MAX:29 Setting lustre.llite.max_read_ahead_whole_mb from 29 to 27 Waiting 90s for '27' Updated after 9s: want '27' got '27' Setting lustre.llite.max_read_ahead_whole_mb from 27 to 28 Waiting 90s for '28' Updated after 6s: want '28' got '28' umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre New config success: got 28 Setting lustre.llite.max_read_ahead_whole_mb from 28 to 26 Waiting 90s for '26' Updated after 9s: want '26' got '26' umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 28A (59s) == conf-sanity test 28a: set symlink parameters permanently with lctl ========================================================== 18:51:45 (1748991105) SKIP: conf-sanity test_28a LU-4221: no such proc params for ZFS OSTs SKIP 28a (1s) == conf-sanity test 29: permanently remove an OST ======== 18:51:46 (1748991106) start ost2 service on oleg243-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec Setting lustre-OST0001.osc.active from 1 to 0 Waiting 90s for '0' Updated after 10s: want '0' got '0' Live client success: got lustre-OST0001_UUID IDLE DEACTIVATED check osc.lustre-OST0001-osc-MDT0000.active target updated after 0 sec (got 0) umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre New client success: got 'lustre-OST0001_UUID NEW DEACTIVATED' Setting lustre-OST0001.osc.active from 0 to 1 Waiting 90s for '1' Updated after 6s: want '1' got '1' umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg243-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg243-server stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. oleg243-server: oleg243-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg243-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg243-client: oleg243-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg243-server: oleg243-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg243-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg243-client: oleg243-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg243-server: oleg243-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg243-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg243-client: oleg243-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg243-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg243-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 oleg243-server: oleg243-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg243-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg243-server: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg243-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server PASS 29 (133s) == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 18:53:59 (1748991239) start mds service on oleg243-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre Big config llog Setting lustre.llite.max_read_ahead_whole_mb from 29 to 1 Waiting 90s for '1' Updated after 7s: want '1' got '1' Setting lustre.llite.max_read_ahead_whole_mb from 1 to 2 Waiting 90s for '2' Updated after 8s: want '2' got '2' Setting lustre.llite.max_read_ahead_whole_mb from 2 to 3 Waiting 90s for '3' Updated after 8s: want '3' got '3' Setting lustre.llite.max_read_ahead_whole_mb from 3 to 4 Waiting 90s for '4' Updated after 9s: want '4' got '4' Setting lustre.llite.max_read_ahead_whole_mb from 4 to 5 Waiting 90s for '5' Updated after 9s: want '5' got '5' Setting lustre.llite.max_read_ahead_whole_mb from 5 to 4 Waiting 90s for '4' Updated after 5s: want '4' got '4' Setting lustre.llite.max_read_ahead_whole_mb from 4 to 3 Waiting 90s for '3' Updated after 8s: want '3' got '3' Setting lustre.llite.max_read_ahead_whole_mb from 3 to 2 Waiting 90s for '2' Updated after 9s: want '2' got '2' Setting lustre.llite.max_read_ahead_whole_mb from 2 to 1 Waiting 90s for '1' Updated after 10s: want '1' got '1' Setting lustre.llite.max_read_ahead_whole_mb from 1 to 2 Waiting 90s for '2' Updated after 5s: want '2' got '2' Setting lustre.llite.max_read_ahead_whole_mb from 2 to 3 Waiting 90s for '3' Updated after 6s: want '3' got '3' Setting lustre.llite.max_read_ahead_whole_mb from 3 to 4 Waiting 90s for '4' Updated after 9s: want '4' got '4' Setting lustre.llite.max_read_ahead_whole_mb from 4 to 5 Waiting 90s for '5' Updated after 6s: want '5' got '5' Setting lustre.llite.max_read_ahead_whole_mb from 5 to 4 Waiting 90s for '4' Updated after 9s: want '4' got '4' Setting lustre.llite.max_read_ahead_whole_mb from 4 to 3 Waiting 90s for '3' Updated after 6s: want '3' got '3' Setting lustre.llite.max_read_ahead_whole_mb from 3 to 2 Waiting 90s for '2' Updated after 8s: want '2' got '2' Setting lustre.llite.max_read_ahead_whole_mb from 2 to 1 Waiting 90s for '1' Updated after 7s: want '1' got '1' Setting lustre.llite.max_read_ahead_whole_mb from 1 to 2 Waiting 90s for '2' Updated after 6s: want '2' got '2' Setting lustre.llite.max_read_ahead_whole_mb from 2 to 3 Waiting 90s for '3' Updated after 9s: want '3' got '3' Setting lustre.llite.max_read_ahead_whole_mb from 3 to 4 Waiting 90s for '4' Updated after 10s: want '4' got '4' Setting lustre.llite.max_read_ahead_whole_mb from 4 to 5 Waiting 90s for '5' Updated after 9s: want '5' got '5' umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre PASS Erase parameter setting umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre deleted (default) value=29, orig=29 umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 30a (199s) == conf-sanity test 30b: Remove failover nids ============ 18:57:18 (1748991438) start mds service on oleg243-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre Using fake nid 192.168.202.163@tcp Setting lustre-OST0000.failover.node from /home/green/git/lustre-release/lustre/utils/lctl get_param -n osc.lustre-OST0000-osc-[^M]*.import | grep failover_nids | sed -n 's/.*\(192.168.202.163@tcp\).*/\1/p' to 192.168.202.163@tcp Waiting 90s for '192.168.202.163@tcp' Updated after 5s: want '192.168.202.163@tcp' got '192.168.202.163@tcp' should have 5 entries in failover nids string, have 5 umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre only 4 final entries should remain in failover nids string, have 4 umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server LNET ready to unload unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: LNET ready to unload modules unloaded. pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 30b (47s) == conf-sanity test 31: Connect to non-existent node (shouldn't crash) ========================================================== 18:58:05 (1748991485) umount lustre on /mnt/lustre..... stop ost1 service on oleg243-server stop mds service on oleg243-server unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 PASS 31 (6s) Stopping clients: oleg243-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg243-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 oleg243-server: oleg243-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 2544 sec ========== 18:58:31 (1748991511)