-----============= acceptance-small: recovery-small ============----- Fri Jun 12 12:12:07 EDT 2026 oleg312-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/recovery-small.*ex': No such file or directory excepting tests: 136 oleg312-client.virtnet: executing check_config_client /mnt/lustre oleg312-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg312-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f72471fb800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f72471fb800.idle_timeout=debug disable quota as required oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 == recovery-small test 1: create, chmod, stat: drop req, drop rep ========================================================== 12:13:05 (1781280785) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x122 fail_loc=0 PASS 1 (70s) == recovery-small test 4: open: drop req, drop rep ======= 12:14:16 (1781280856) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x122 fail_loc=0 PASS 4 (26s) == recovery-small test 5: rename: drop req, drop rep ===== 12:14:42 (1781280882) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 PASS 5 (27s) == recovery-small test 6: link, unlink: drop req, drop rep ========================================================== 12:15:09 (1781280909) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 PASS 6 (34s) == recovery-small test 8: touch: drop rep (bug 1423) ===== 12:15:43 (1781280943) fail_loc=0x119 fail_loc=0 PASS 8 (16s) == recovery-small test 9: pause bulk on OST (bug 1420) === 12:15:59 (1781280959) timeout is 0/ fail_val=0 fail_loc=0x80000214 fail_loc=0 PASS 9 (16s) == recovery-small test 10a: finish request on server after client eviction (bug 1521) ========================================================== 12:16:15 (1781280975) ldlm.namespaces.MGC192.168.203.112@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9f72471fb800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9f72471fb800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9f72471fb800.early_lock_cancel=0 fail_loc=0x305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.203.112@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9f72471fb800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9f72471fb800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9f72471fb800.early_lock_cancel=1 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Input/output error Connected clients: oleg312-client.virtnet /mnt/lustre has perms 0777 OK PASS 10a (111s) == recovery-small test 10b: re-send BL AST =============== 12:18:06 (1781281086) ldlm.namespaces.MGC192.168.203.112@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9f72471fb800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9f72471fb800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9f72471fb800.early_lock_cancel=0 fail_loc=0x80000305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.203.112@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9f72471fb800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9f72471fb800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9f72471fb800.early_lock_cancel=1 Connected clients: oleg312-client.virtnet oleg312-client.virtnet /mnt/lustre has perms 0777 OK PASS 10b (21s) == recovery-small test 10c: re-send BL AST vs reconnect race (LU-5569) ========================================================== 12:18:27 (1781281107) ldlm.namespaces.MGC192.168.203.112@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9f72471fb800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9f72471fb800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9f72471fb800.early_lock_cancel=0 fail_loc=0x80000305 mdc.lustre-MDT0000-mdc-ffff9f72471fb800.import=connection=192.168.203.112@tcp fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.203.112@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9f72471fb800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9f72471fb800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9f72471fb800.early_lock_cancel=1 Connected clients: oleg312-client.virtnet oleg312-client.virtnet /mnt/lustre/d10c.recovery-small has perms 0777 OK PASS 10c (8s) == recovery-small test 10d: test failed blocking ast ===== 12:18:35 (1781281115) 7+0 records in 7+0 records out 7 bytes copied, 0.00017097 s, 40.9 kB/s Stopping client oleg312-client.virtnet /mnt/lustre (opts:) Starting client: oleg312-client.virtnet: -o user_xattr,flock oleg312-server@tcp:/lustre /mnt/lustre Starting client: oleg312-client.virtnet: -o user_xattr,flock oleg312-server@tcp:/lustre /mnt/lustre2 1+0 records in 1+0 records out 5 bytes copied, 0.0212647 s, 0.2 kB/s fail_err=71 ldlm.namespaces.MGC192.168.203.112@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9f724370f800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9f7250d24000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9f724370f800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9f7250d24000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9f724370f800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9f7250d24000.early_lock_cancel=0 fail_loc=0x305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.203.112@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9f724370f800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9f7250d24000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9f724370f800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9f7250d24000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9f724370f800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9f7250d24000.early_lock_cancel=1 Connected clients: oleg312-client.virtnet oleg312-client.virtnet fail_err=0 192.168.203.112@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg312-client.virtnet /mnt/lustre2 (opts:) PASS 10d (15s) == recovery-small test 10e: re-send BL AST vs reconnect race 2 ========================================================== 12:18:50 (1781281130) SKIP: recovery-small test_10e need two clients SKIP 10e (4s) == recovery-small test 11: wake up a thread waiting for completion after eviction (b=2460) ========================================================== 12:18:54 (1781281134) ldlm.namespaces.MGC192.168.203.112@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9f724370f800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9f724370f800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9f724370f800.early_lock_cancel=0 fail_loc=0x80000305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.203.112@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9f724370f800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9f724370f800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9f724370f800.early_lock_cancel=1 PASS 11 (21s) == recovery-small test 12: recover from timed out resend in ptlrpcd (b=2494) ========================================================== 12:19:16 (1781281156) fail_loc=0x115 multiop /mnt/lustre/f12.recovery-small vOS_c TMPPIPE=/tmp/multiop_open_wait_pipe.7155 waiting for multiop 16494 clearing fail_loc on mds1 fail_loc=0 PASS 12 (43s) == recovery-small test 13: mdc_readpage restart test (bug 1138) ========================================================== 12:20:00 (1781281200) fail_loc=0x80000104 newentry fail_loc=0 PASS 13 (16s) == recovery-small test 14: mdc_readpage resend test (bug 1138) ========================================================== 12:20:15 (1781281215) fail_loc=0x80000106 newentry fail_loc=0 PASS 14 (8s) == recovery-small test 15: failed open (-ENOMEM) ========= 12:20:23 (1781281223) fail_loc=0x80000128 touch: cannot touch '/mnt/lustre/f15.recovery-small': Cannot allocate memory PASS 15 (7s) == recovery-small test 16: timeout bulk put, don't evict client (2732) ========================================================== 12:20:31 (1781281231) fail_loc=0x80000504 fail_loc=0 PASS 16 (75s) == recovery-small test 17a: timeout bulk get, don't evict client (2732) ========================================================== 12:21:46 (1781281306) at_max=20 fail_loc=0x80000503 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.112@tcp:/lustre 7542784 6144 7532544 1% /mnt/lustre at_max=600 PASS 17a (54s) == recovery-small test 17b: timeout bulk get, dont evict client (3582) ========================================================== 12:22:40 (1781281360) SKIP: recovery-small test_17b Needs multiple clients SKIP 17b (4s) == recovery-small test 18a: manual ost invalidate clears page cache immediately ========================================================== 12:22:44 (1781281364) PASS 18a (7s) == recovery-small test 18b: eviction and reconnect clears page cache (2766) ========================================================== 12:22:51 (1781281371) PASS 18b (32s) == recovery-small test 18c: Dropped connect reply after eviction handing (14755) ========================================================== 12:23:24 (1781281404) fail_loc=0x80000225 PASS 18c (22s) == recovery-small test 19a: test expired_lock_main on mds (2867) ========================================================== 12:23:46 (1781281426) Starting client: oleg312-client.virtnet: -o user_xattr,flock oleg312-server@tcp:/lustre /mnt/lustre2 fail_loc=0x304 fail_loc=0 192.168.203.112@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg312-client.virtnet /mnt/lustre2 (opts:) Connected clients: oleg312-client.virtnet oleg312-client.virtnet PASS 19a (115s) == recovery-small test 19b: test expired_lock_main on ost (2867) ========================================================== 12:25:42 (1781281542) Starting client: oleg312-client.virtnet: -o user_xattr,flock oleg312-server@tcp:/lustre /mnt/lustre2 fail_loc=0x304 fail_loc=0 192.168.203.112@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg312-client.virtnet /mnt/lustre2 (opts:) Connected clients: oleg312-client.virtnet oleg312-client.virtnet PASS 19b (115s) == recovery-small test 19c: check reconnect and lock resend do not trigger expired_lock_main ========================================================== 12:27:36 (1781281656) Starting client: oleg312-client.virtnet: -o user_xattr,flock oleg312-server@tcp:/lustre /mnt/lustre2 ldlm.namespaces.MGC192.168.203.112@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9f724370f800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9f7247300000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9f724370f800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9f7247300000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9f724370f800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9f7247300000.early_lock_cancel=0 File: /mnt/lustre/f19c.recovery-small Size: 512 Blocks: 1 IO Block: 131072 directory Device: 2c54f966h/743766374d Inode: 144115205322833924 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-12 12:27:38.000000000 -0400 Modify: 2026-06-12 12:27:38.000000000 -0400 Change: 2026-06-12 12:27:38.000000000 -0400 Birth: 2026-06-12 12:27:38.000000000 -0400 fail_loc=0x80000516 ldlm.namespaces.MGC192.168.203.112@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9f724370f800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9f7247300000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9f724370f800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9f7247300000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9f724370f800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9f7247300000.early_lock_cancel=1 192.168.203.112@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg312-client.virtnet /mnt/lustre2 (opts:) PASS 19c (18s) == recovery-small test 20a: ldlm_handle_enqueue error (should return error) ========================================================== 12:27:54 (1781281674) multiop /mnt/lustre/d20a.recovery-small/f20a.recovery-small vO_wc TMPPIPE=/tmp/multiop_open_wait_pipe.7155 fail_loc=0x80000308 write: Cannot allocate memory PASS 20a (8s) == recovery-small test 20b: ldlm_handle_enqueue error (should return error) ========================================================== 12:28:02 (1781281682) fail_loc=0x80000308 dd: failed to open '/mnt/lustre/d20b.recovery-small/f20b.recovery-small': Cannot allocate memory PASS 20b (8s) == recovery-small test 21a: drop close request while close and open are both in flight ========================================================== 12:28:10 (1781281690) multiop /mnt/lustre/d21a.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7155 fail_loc=0x80000129 fail_loc=0 fail_loc=0x80000115 fail_loc=0 /mnt/lustre/d21a.recovery-small-1/f has type file OK /mnt/lustre/d21a.recovery-small-2/f has type file OK PASS 21a (17s) == recovery-small test 21b: drop open request while close and open are both in flight ========================================================== 12:28:28 (1781281708) multiop /mnt/lustre/d21b.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7155 fail_loc=0x80000107 fail_loc=0 /mnt/lustre/d21b.recovery-small-1/f has type file OK /mnt/lustre/d21b.recovery-small-2/f has type file OK PASS 21b (143s) == recovery-small test 21c: drop both request while close and open are both in flight ========================================================== 12:30:50 (1781281850) multiop /mnt/lustre/d21c.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7155 fail_loc=0x80000107 fail_loc=0 fail_loc=0x80000115 fail_loc=0 /mnt/lustre/d21c.recovery-small-1/f has type file OK /mnt/lustre/d21c.recovery-small-2/f has type file OK PASS 21c (21s) == recovery-small test 21d: drop close reply while close and open are both in flight ========================================================== 12:31:11 (1781281871) multiop /mnt/lustre/d21d.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7155 fail_loc=0x80000129 fail_loc=0 fail_loc=0x80000122 fail_loc=0 /mnt/lustre/d21d.recovery-small-1/f has type file OK /mnt/lustre/d21d.recovery-small-2/f has type file OK PASS 21d (19s) == recovery-small test 21e: drop open reply while close and open are both in flight ========================================================== 12:31:30 (1781281890) multiop /mnt/lustre/d21e.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7155 fail_loc=0x80000119 fail_loc=0 /mnt/lustre/d21e.recovery-small-1/f has type file OK /mnt/lustre/d21e.recovery-small-2/f has type file OK PASS 21e (135s) == recovery-small test 21f: drop both reply while close and open are both in flight ========================================================== 12:33:46 (1781282026) multiop /mnt/lustre/d21f.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7155 fail_loc=0x80000119 fail_loc=0 fail_loc=0x80000122 fail_loc=0 /mnt/lustre/d21f.recovery-small-1/f has type file OK /mnt/lustre/d21f.recovery-small-2/f has type file OK PASS 21f (17s) == recovery-small test 21g: drop open reply and close request while close and open are both in flight ========================================================== 12:34:03 (1781282043) multiop /mnt/lustre/d21g.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7155 fail_loc=0x80000119 fail_loc=0 fail_loc=0x80000115 fail_loc=0 /mnt/lustre/d21g.recovery-small-1/f has type file OK /mnt/lustre/d21g.recovery-small-2/f has type file OK PASS 21g (17s) == recovery-small test 21h: drop open request and close reply while close and open are both in flight ========================================================== 12:34:20 (1781282060) multiop /mnt/lustre/d21h.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7155 fail_loc=0x80000107 fail_loc=0 fail_loc=0x80000122 fail_loc=0 /mnt/lustre/d21h.recovery-small-1/f has type file OK /mnt/lustre/d21h.recovery-small-2/f has type file OK PASS 21h (16s) == recovery-small test 22: drop close request and do mknod ========================================================== 12:34:36 (1781282076) fail_loc=0x80000115 fail_loc=0 PASS 22 (13s) == recovery-small test 23: client hang when close a file after mds crash ========================================================== 12:34:49 (1781282089) multiop /mnt/lustre/f23.recovery-small vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7155 fail_val=0 fail_loc=0x123 fail_loc=0 Failing mds1 on oleg312-server Stopping /mnt/lustre-mds1 (opts:) on oleg312-server reboot facets: mds1 Failover mds1 to oleg312-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23 (61s) == recovery-small test 24a: fsync error (should return error) ========================================================== 12:35:50 (1781282150) multiop /mnt/lustre/d24a.recovery-small/f24a.recovery-small vOwy_wyc TMPPIPE=/tmp/multiop_open_wait_pipe.7155 fsync: Input/output error fail_loc=0x0 Connected clients: oleg312-client.virtnet oleg312-client.virtnet PASS 24a (8s) == recovery-small test 24b: test dirty page discard due to client eviction ========================================================== 12:35:59 (1781282159) multiop /mnt/lustre/d24b.recovery-small/f24b.recovery-small-1 vOw8192_yc TMPPIPE=/tmp/multiop_open_wait_pipe.7155 multiop /mnt/lustre/d24b.recovery-small/f24b.recovery-small-2 vOw8192_c TMPPIPE=/tmp/multiop_open_wait_pipe.7155 fsync: Input/output error fail_loc=0x0 Connected clients: oleg312-client.virtnet oleg312-client.virtnet recovery-small test_24b: @@@@@@ IGNORE (bz5494): multiop didn't fail fsync: 5 or close: 0 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6475:error_ignore() = /home/green/git/lustre-release/lustre/tests/recovery-small.sh:1052:test_24b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6817:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6864:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6705:run_test() = /home/green/git/lustre-release/lustre/tests/recovery-small.sh:1058:main() [ 1715.211431] Lustre: 2226:0:(llite_lib.c:3595:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.112@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108) PASS 24b (10s) == recovery-small test 26a: evict dead exports =========== 12:36:09 (1781282169) SKIP: recovery-small test_26a msg and ost1 are at the same node SKIP 26a (3s) == recovery-small test 26b: evict dead exports =========== 12:36:12 (1781282172) SKIP: recovery-small test_26b msg and ost1 are at the same node SKIP 26b (3s) == recovery-small test 27: fail LOV while using OSC's ==== 12:36:15 (1781282175) Failing mds1 on oleg312-server Stopping /mnt/lustre-mds1 (opts:) on oleg312-server reboot facets: mds1 Failover mds1 to oleg312-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0x80000407 waiting for fail_loc Waiting 90s for '-2147482617' Waiting 60s for '-2147482617' Waiting 50s for '-2147482617' Waiting 20s for '-2147482617' Waiting 10s for '-2147482617' Update not seen after 90s: want '-2147482617' got '2147484679' Failing mds1 on oleg312-server Stopping /mnt/lustre-mds1 (opts:) on oleg312-server reboot facets: mds1 Failover mds1 to oleg312-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 27 (180s) == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 12:39:15 (1781282355) ldlm.namespaces.MGC192.168.203.112@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9f724370f800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9f724370f800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9f724370f800.early_lock_cancel=0 fail_loc=0x80000305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.203.112@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9f724370f800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9f724370f800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9f724370f800.early_lock_cancel=1 fail_loc=0x8000012f Failing mds1 on oleg312-server Stopping /mnt/lustre-mds1 (opts:) on oleg312-server reboot facets: mds1 Failover mds1 to oleg312-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 28 (70s) == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 12:40:25 (1781282425) fail_loc=0x80000711 Stopping /mnt/lustre-mds1 (opts:) on oleg312-server Failover mds1 to oleg312-server Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec PASS 29a (34s) == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 12:40:59 (1781282459) fail_loc=0x80000711 Stopping /mnt/lustre-ost1 (opts:) on oleg312-server Failover ost1 to oleg312-server Starting ost1: -o localrecov -o abort_recovery lustre-ost1/ost1 /mnt/lustre-ost1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0000 PASS 29b (28s) == recovery-small test 50: failover MDS under load ======= 12:41:27 (1781282487) writemany pid 40090 Failing mds1 on oleg312-server Stopping /mnt/lustre-mds1 (opts:) on oleg312-server reboot facets: mds1 Failover mds1 to oleg312-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds1 on oleg312-server Stopping /mnt/lustre-mds1 (opts:) on oleg312-server reboot facets: mds1 Failover mds1 to oleg312-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds1 on oleg312-server Stopping /mnt/lustre-mds1 (opts:) on oleg312-server reboot facets: mds1 Failover mds1 to oleg312-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec writemany returned 0 PASS 50 (327s) == recovery-small test 51: failover MDS during recovery == 12:46:54 (1781282814) fail_loc=0x00001310 Failing mds1 on oleg312-server Stopping /mnt/lustre-mds1 (opts:) on oleg312-server reboot facets: mds1 Failover mds1 to oleg312-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 will failover at 1 5 10 20 25 30 test_51: failover in 1 sec Failing mds1 on oleg312-server Stopping /mnt/lustre-mds1 (opts:) on oleg312-server reboot facets: mds1 Failover mds1 to oleg312-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 test_51: failover in 5 sec Failing mds1 on oleg312-server Stopping /mnt/lustre-mds1 (opts:) on oleg312-server reboot facets: mds1 Failover mds1 to oleg312-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 test_51: failover in 10 sec Failing mds1 on oleg312-server Stopping /mnt/lustre-mds1 (opts:) on oleg312-server reboot facets: mds1 Failover mds1 to oleg312-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 test_51: failover in 20 sec Failing mds1 on oleg312-server Stopping /mnt/lustre-mds1 (opts:) on oleg312-server reboot facets: mds1 Failover mds1 to oleg312-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 test_51: failover in 25 sec Failing mds1 on oleg312-server Stopping /mnt/lustre-mds1 (opts:) on oleg312-server reboot facets: mds1 Failover mds1 to oleg312-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 test_51: failover in 30 sec Failing mds1 on oleg312-server Stopping /mnt/lustre-mds1 (opts:) on oleg312-server reboot facets: mds1 Failover mds1 to oleg312-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 writemany returned 0 PASS 51 (392s) == recovery-small test 52: failover OST under load ======= 12:53:26 (1781283206) writemany pid 46604 Failing ost1 on oleg312-server Stopping /mnt/lustre-ost1 (opts:) on oleg312-server reboot facets: ost1 Failover ost1 to oleg312-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0000 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec writemany succeeded writemany pid 47488 Failing ost1 on oleg312-server Stopping /mnt/lustre-ost1 (opts:) on oleg312-server reboot facets: ost1 Failover ost1 to oleg312-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0000 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec writemany succeeded writemany pid 48393 Failing ost1 on oleg312-server Stopping /mnt/lustre-ost1 (opts:) on oleg312-server reboot facets: ost1 Failover ost1 to oleg312-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0000 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec writemany succeeded Connected clients: oleg312-client.virtnet oleg312-client.virtnet PASS 52 (974s) == recovery-small test 53a: touch: drop rep ============== 13:09:40 (1781284180) fail_loc=0x157 Succeed in opening file "/mnt/lustre/f53a.recovery-small"(flags=O_RDWR, mode=755) fail_loc=0 PASS 53a (56s) == recovery-small test 53b: touch: drop rep ============== 13:10:37 (1781284237) fail_loc=0x157 Succeed in opening file "/mnt/lustre/f53b.recovery-small"(flags=O_RDWR, mode=755) fail_loc=0 PASS 53b (57s) == recovery-small test 53c: touch: drop rep ============== 13:11:35 (1781284295) fail_loc=0x157 Succeed in opening file "/mnt/lustre/f53c.recovery-small"(flags=O_RDWR, mode=755) fail_loc=0 PASS 53c (54s) == recovery-small test 54: back in time ================== 13:12:29 (1781284349) Starting client: oleg312-client.virtnet: -o user_xattr,flock oleg312-server@tcp:/lustre /mnt/lustre2 cat: /mnt/lustre2/f54.recovery-small.missing: No such file or directory Failing mds1 on oleg312-server Stopping /mnt/lustre-mds1 (opts:) on oleg312-server reboot facets: mds1 Failover mds1 to oleg312-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 54 (72s) == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 13:13:41 (1781284421) step1: testing ...... 4+0 records in 4+0 records out 134217728 bytes (134 MB, 128 MiB) copied, 12.692 s, 10.6 MB/s (dd_pid=53381, time=13)successful fail_loc=0x0000021d step2: testing ...... (dd_pid=53406, time=257)successful fail_loc=0 step3: testing ...... 4+0 records in 4+0 records out 134217728 bytes (134 MB, 128 MiB) copied, 279.212 s, 481 kB/s (dd_pid=53406, time=17)successful osc.lustre-OST0000-osc-ffff9f724370f800.max_dirty_mb=462 osc.lustre-OST0001-osc-ffff9f724370f800.max_dirty_mb=462 PASS 55 (309s) == recovery-small test 56: do not fail on getattr resend ========================================================== 13:18:50 (1781284730) fail_loc=0x80000136 File: /mnt/lustre/f56.recovery-small Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115373044689692 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-12 13:18:52.000000000 -0400 Modify: 2026-06-12 13:18:52.000000000 -0400 Change: 2026-06-12 13:18:52.000000000 -0400 Birth: 2026-06-12 13:18:52.000000000 -0400 fail_loc=0 PASS 56 (50s) == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 13:19:41 (1781284781) fail_loc=0x80000B00 Stopping client oleg312-client.virtnet /mnt/lustre (opts:) fail_loc=0x80000B00 Stopping /mnt/lustre-mds1 (opts:) on oleg312-server Failover mds1 to oleg312-server Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 pdsh@oleg312-client: oleg312-client: ssh exited with exit code 95 pdsh@oleg312-client: oleg312-client: ssh exited with exit code 95 /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 1414: kill: (54884) - No such process fail_loc=0 Starting client: oleg312-client.virtnet: -o user_xattr,flock oleg312-server@tcp:/lustre /mnt/lustre Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.112@tcp:/lustre 7542784 6144 7532544 1% /mnt/lustre PASS 57 (35s) == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 13:20:17 (1781284817) -rw-r--r-- 1 root root 0 Jun 12 13:20 /mnt/lustre/f58.recovery-small fail_loc=0x80000801 fail_loc=0 ldlm.namespaces.MGC192.168.203.112@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9f7245257800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9f7245257800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9f7245257800.early_lock_cancel=0 fail_loc=0x80000305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.203.112@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9f7245257800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9f7245257800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9f7245257800.early_lock_cancel=1 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.112@tcp:/lustre 7542784 6144 7532544 1% /mnt/lustre PASS 58 (21s) == recovery-small test 59: Read cancel race on client eviction ========================================================== 13:20:38 (1781284838) Starting client: oleg312-client.virtnet: -o user_xattr,flock oleg312-server@tcp:/lustre /mnt/lustre2 fail_loc=0x311 fail_loc=0 Stopping client oleg312-client.virtnet /mnt/lustre2 (opts:-f) PASS 59 (24s) == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 13:21:02 (1781284862) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl1' - open/close 619 (time 1781284879.51 total 10.02 last 61.77) - open/close 1295 (time 1781284889.51 total 20.02 last 67.60) - open/close 1994 (time 1781284899.52 total 30.02 last 69.89) - open/close 2595 (time 1781284909.52 total 40.02 last 60.10) - open/close 3245 (time 1781284919.53 total 50.04 last 64.92) - open/close 3923 (time 1781284929.54 total 60.05 last 67.69) - open/close 4573 (time 1781284939.55 total 70.06 last 64.93) total: 5000 open/close in 77.15 seconds: 64.81 ops/second Failing mds1 on oleg312-server Stopping /mnt/lustre-mds1 (opts:) on oleg312-server reboot facets: mds1 Failover mds1 to oleg312-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 - unlinked 0 (time 1781284955 ; total 0 ; last 0) total: 5000 unlinks in 154 seconds: 32.467533 unlinks/second 5000 unlinks in changelog lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 lustre-MDT0000: changelog user 'cl1' not found PASS 60 (269s) == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 13:25:31 (1781285131) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4864 2203648 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 10 open/close in 0.29 seconds: 34.97 ops/second Stopping /mnt/lustre-mds1 (opts:) on oleg312-server Failover mds1 to oleg312-server Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 61 (43s) == recovery-small test 65: lock enqueue for destroyed export ========================================================== 13:26:14 (1781285174) Starting client: oleg312-client.virtnet: -o user_xattr,flock oleg312-server@tcp:/lustre /mnt/lustre2 fail_loc=0x31e write: Input/output error fail_loc=0 192.168.203.112@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg312-client.virtnet /mnt/lustre2 (opts:) PASS 65 (24s) == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 13:26:38 (1781285198) fail_loc=0x80000157 fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff9f7245257800.import=connection=192.168.203.112@tcp stat: cannot statx '/mnt/lustre/f66.recovery-small': Input/output error /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: oleg312-client.virtnet fail_loc=0 PASS 66 (18s) == recovery-small test 67: connect vs import invalidate race ========================================================== 13:26:56 (1781285216) fail_loc=0x80000531 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Input/output error mdc.lustre-MDT0000-mdc-ffff9f7245257800.import=connection=192.168.203.112@tcp error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: No such device cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: No such device error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: No such device cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: No such device error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: No such device cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: No such device error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: No such device cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: No such device error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: No such device cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: No such device error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: No such device cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: No such device error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: No such device cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: No such device error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: No such device cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: No such device error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: No such device cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown Connected clients: oleg312-client.virtnet oleg312-client.virtnet PASS 67 (22s) == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 13:27:18 (1781285238) mgs.MGS.ir_timeout Failing ost1 on oleg312-server Stopping /mnt/lustre-ost1 (opts:) on oleg312-server reboot facets: ost1 Failover ost1 to oleg312-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0000 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 100 (51s) == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 13:28:09 (1781285289) mgs.MGS.ir_timeout Failing ost1 on oleg312-server Stopping /mnt/lustre-ost1 (opts:) on oleg312-server reboot facets: ost1 Failover ost1 to oleg312-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0000 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 101a (49s) == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 13:28:58 (1781285338) mgs.MGS.ir_timeout fail_loc=0x247 Failing ost1 on oleg312-server Stopping /mnt/lustre-ost1 (opts:) on oleg312-server reboot facets: ost1 Failover ost1 to oleg312-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0000 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 101b (75s) == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 13:30:13 (1781285413) mgs.MGS.ir_timeout Failing ost1 on oleg312-server Stopping /mnt/lustre-ost1 (opts:) on oleg312-server reboot facets: ost1 Failover ost1 to oleg312-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0000 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:) Stopping client oleg312-client.virtnet /mnt/lustre opts: Stopping /mnt/lustre-mds1 (opts:) on oleg312-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 Failing ost1 on oleg312-server Stopping /mnt/lustre-ost1 (opts:) on oleg312-server reboot facets: ost1 Failover ost1 to oleg312-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0000 pdsh@oleg312-client: oleg312-client: ssh exited with exit code 95 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid pdsh@oleg312-client: oleg312-client: ssh exited with exit code 95 Starting client oleg312-client.virtnet: -o user_xattr,flock oleg312-server@tcp:/lustre /mnt/lustre Started clients oleg312-client.virtnet: 192.168.203.112@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) PASS 102 (110s) == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 13:32:03 (1781285523) mgs.MGS.ir_timeout SKIP: recovery-small test_103 needs separate mgs and mds SKIP 103 (5s) == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 13:32:08 (1781285528) mgs.MGS.ir_timeout Stopping /mnt/lustre-ost1 (opts:) on oleg312-server Starting ost1: -o localrecov -onoir lustre-ost1/ost1 /mnt/lustre-ost1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0000 PASS 104 (28s) == recovery-small test 105: IR: NON IR clients support === 13:32:36 (1781285556) SKIP: recovery-small test_105 Needs multiple clients SKIP 105 (4s) == recovery-small test 106: lightweight connection support ========================================================== 13:32:40 (1781285560) fail_loc=0x805 Starting client: oleg312-client.virtnet: -o user_xattr,flock oleg312-server@tcp:/lustre /mnt/lustre2 fail_loc=0 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3584 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre debug=ha Failing mds1 on oleg312-server Stopping /mnt/lustre-mds1 (opts:) on oleg312-server reboot facets: mds1 Failover mds1 to oleg312-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 Debug log: 219 lines, 219 kept, 0 dropped, 0 bad. /mnt/lustre/f106.recovery-small has type file OK 192.168.203.112@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg312-client.virtnet /mnt/lustre2 (opts:) PASS 106 (53s) == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 13:33:33 (1781285613) fail_loc=0x119 fail_loc=0 Failing mds1 on oleg312-server Stopping /mnt/lustre-mds1 (opts:) on oleg312-server reboot facets: mds1 Failover mds1 to oleg312-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 107 (59s) == recovery-small test 108: client eviction don't crash == 13:34:33 (1781285673) dd: error writing '/mnt/lustre/d108.recovery-small/f108.recovery-small': Input/output error 13+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 1.39111 s, 9.0 MB/s PASS 108 (10s) == recovery-small test 110a: create remote directory: drop client req ========================================================== 13:34:43 (1781285683) SKIP: recovery-small test_110a needs >= 2 MDTs SKIP 110a (3s) == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 13:34:46 (1781285686) SKIP: recovery-small test_110b needs >= 2 MDTs SKIP 110b (4s) == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 13:34:50 (1781285690) SKIP: recovery-small test_110c needs >= 2 MDTs SKIP 110c (4s) == recovery-small test 110d: remove remote directory: drop client req ========================================================== 13:34:54 (1781285694) SKIP: recovery-small test_110d needs >= 2 MDTs SKIP 110d (3s) == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 13:34:57 (1781285697) SKIP: recovery-small test_110e needs >= 2 MDTs SKIP 110e (4s) == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 13:35:01 (1781285701) SKIP: recovery-small test_110f needs >= 2 MDTs SKIP 110f (4s) == recovery-small test 110g: drop reply during migration ========================================================== 13:35:05 (1781285705) SKIP: recovery-small test_110g needs >= 2 MDTs SKIP 110g (3s) == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 13:35:08 (1781285708) SKIP: recovery-small test_110h needs >= 2 MDTs SKIP 110h (4s) == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 13:35:12 (1781285712) SKIP: recovery-small test_110i needs >= 2 MDTs SKIP 110i (4s) == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 13:35:16 (1781285716) SKIP: recovery-small test_110j needs >= 2 MDTs SKIP 110j (4s) == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 13:35:20 (1781285720) SKIP: recovery-small test_110k needs >= 2 MDTS SKIP 110k (3s) == recovery-small test 110m: update resent vs original RPC race ========================================================== 13:35:24 (1781285724) SKIP: recovery-small test_110m needs at least 2 MDTs SKIP 110m (4s) == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 13:35:28 (1781285728) fail_loc=0x151 Stopping /mnt/lustre-mds1 (opts:) on oleg312-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: mount.lustre: mount lustre-mdt1/mdt1 at /mnt/lustre-mds1 failed: Input/output error oleg312-server: Is the MGS running? pdsh@oleg312-client: oleg312-server: ssh exited with exit code 5 Start of lustre-mdt1/mdt1 on mds1 failed 5 fail_loc=0 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 111 (31s) == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 13:35:59 (1781285759) timeout is 20/20 fail_val=20 fail_loc=0x80000214 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.112@tcp:/lustre 7542784 6144 7532544 1% /mnt/lustre PASS 112a (30s) == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 13:36:29 (1781285789) Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.112@tcp:/lustre 7542784 6144 7532544 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.030535 s, 134 kB/s fail_loc=0x8000051b fail_val=3 fail_val=0 fail_loc=0x8000051a 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 4.05542 s, 1.0 kB/s PASS 115a (11s) == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 13:36:40 (1781285800) Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.112@tcp:/lustre 7542784 6144 7530496 1% /mnt/lustre fail_loc=0x8000051b fail_val=4 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.112@tcp:/lustre 7542784 6144 7530496 1% /mnt/lustre fail_val=2 fail_loc=0x80000215 dd: error writing '/mnt/lustre/f115b.recovery-small': No space left on device 1+0 records in 0+0 records out 0 bytes copied, 4.08169 s, 0.0 kB/s PASS 115b (11s) == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 13:36:51 (1781285811) Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.112@tcp:/lustre 7542784 6144 7532544 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0295136 s, 139 kB/s fail_loc=0x8000050f fail_val=3 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0391208 s, 105 kB/s fail_val=0 fail_loc=0x8000051a PASS 115c (9s) == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 13:37:00 (1781285820) Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.112@tcp:/lustre 7542784 6144 7530496 1% /mnt/lustre fail_loc=0x8000050f fail_val=4 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.162085 s, 25.3 kB/s Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.112@tcp:/lustre 7542784 6144 7531520 1% /mnt/lustre fail_val=0 fail_loc=0x80000215 PASS 115d (9s) == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 13:37:10 (1781285830) Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.112@tcp:/lustre 7542784 6144 7532544 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0150234 s, 273 kB/s fail_loc=0x80000510 fail_val=3 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0430397 s, 95.2 kB/s fail_val=0 fail_loc=0x80000211 PASS 115e (9s) == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 13:37:19 (1781285839) Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.112@tcp:/lustre 7542784 6144 7532544 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0302458 s, 135 kB/s fail_loc=0x8000051b fail_val=3 fail_val=0 fail_loc=0x80000211 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 48.3101 s, 0.1 kB/s PASS 115f (56s) == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 13:38:16 (1781285896) Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.112@tcp:/lustre 7542784 6144 7532544 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0653286 s, 62.7 kB/s fail_loc=0x8000051c fail_val=3 fail_val=0 fail_loc=0 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 47.6827 s, 0.1 kB/s PASS 115g (56s) == recovery-small test 120: flock race: completion vs. evict ========================================================== 13:39:12 (1781285952) ** FLOCK REPLY vs. EVICTION race, lock set, CLEANUP cp first fail_loc=0x80000320 FLOCKS_TEST 5: SET write flock ** Evicting and re-connecting client ** /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Input/output error Connected clients: oleg312-client.virtnet fcntl cmd 7 failed: Input/output error exiting with rc = 5 fail_loc=0 ** FLOCK REPLY vs. EVICTION race, lock get, CLEANUP cp first ** Taking conflict ** FLOCKS_TEST 5: SET read flock fail_loc=0x80000320 FLOCKS_TEST 5: GET write flock ** Evicting and re-connecting client ** /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Input/output error Connected clients: oleg312-client.virtnet fail_loc=0 ** FLOCK REPLY vs. EVICTION race, lock unlock, CLEANUP cp first fail_loc=0x80000320 FLOCKS_TEST 5: UNLOCK write flock ** Evicting and re-connecting client ** /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Input/output error Connected clients: oleg312-client.virtnet fail_loc=0 ** FLOCK REPLY vs. EVICTION race, lock set, REPLY cp first fail_loc=0x80000321 FLOCKS_TEST 5: SET write flock ** Evicting and re-connecting client ** /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Input/output error error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown fcntl cmd 7 failed: Input/output error exiting with rc = 5 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown Connected clients: oleg312-client.virtnet oleg312-client.virtnet fail_loc=0 ** FLOCK REPLY vs. EVICTION race, lock get, REPLY cp first ** Taking conflict ** FLOCKS_TEST 5: SET read flock fail_loc=0x80000321 FLOCKS_TEST 5: GET write flock ** Evicting and re-connecting client ** /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Input/output error Connected clients: oleg312-client.virtnet fail_loc=0 ** FLOCK REPLY vs. EVICTION race, lock unlock, REPLY cp first fail_loc=0x80000321 FLOCKS_TEST 5: UNLOCK write flock ** Evicting and re-connecting client ** /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Input/output error Connected clients: oleg312-client.virtnet fail_loc=0 ** FLOCK REPLY vs. EVICTION race, lock set DEADLOCK, CLEANUP cp first fail_loc=0x80000322 FLOCKS_TEST 5: SET write flock ** Evicting and re-connecting client ** /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Input/output error error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown Connected clients: oleg312-client.virtnet fcntl cmd 7 failed: Input/output error exiting with rc = 5 fail_loc=0 ** FLOCK REPLY vs. EVICTION race, lock set DEADLOCK, REPLY cp first fail_loc=0x80000323 FLOCKS_TEST 5: SET write flock ** Evicting and re-connecting client ** /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Input/output error error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown fcntl cmd 7 failed: Resource deadlock avoided exiting with rc = 35 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown Connected clients: oleg312-client.virtnet oleg312-client.virtnet fail_loc=0 FLOCKS_TEST 5: SET write flock fail_loc=0x320 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Input/output error error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown Connected clients: oleg312-client.virtnet oleg312-client.virtnet fail_loc=0 PASS 120 (71s) == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 13:40:24 (1781286024) fail_loc=0x80000157 File: /mnt/lustre/f113.recovery-small Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144116010713088002 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-12 13:40:25.000000000 -0400 Modify: 2026-06-12 13:40:25.000000000 -0400 Change: 2026-06-12 13:40:25.000000000 -0400 Birth: 2026-06-12 13:40:25.000000000 -0400 fail_loc=0 fail_loc=0x8000031f Connected clients: oleg312-client.virtnet oleg312-client.virtnet PASS 113 (19s) == recovery-small test 130a: enqueue resend on not existing file ========================================================== 13:40:43 (1781286043) fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000157 stat: cannot statx '/mnt/lustre/d130a.recovery-small': No such file or directory PASS 130a (54s) == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 13:41:37 (1781286097) fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000157 fail_val=0 fail_loc=0x80000217 stat: cannot statx '/mnt/lustre/d130b.recovery-small': No such file or directory PASS 130b (53s) == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 13:42:30 (1781286150) fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000157 fail_val=0 fail_loc=0x80000217 PASS 130c (33s) == recovery-small test 132: long punch =================== 13:43:03 (1781286183) Starting client: oleg312-client.virtnet: -o user_xattr,flock oleg312-server@tcp:/lustre /mnt/lustre2 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.308176 s, 13.3 kB/s fail_val=120 fail_loc=0x80000236 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 119.024 s, 0.0 kB/s 192.168.203.112@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg312-client.virtnet /mnt/lustre2 (opts:) PASS 132 (131s) == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 13:45:15 (1781286315) 1+0 records in 1+0 records out 512 bytes copied, 0.0150164 s, 34.1 kB/s fail_loc=0x80000414 fail_val=4 fail_loc=0x80000414 fail_val=0 fail_loc=0x8000031e uname: write error: Input/output error uname: write error: Input/output error cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown dd: writing to '/mnt/lustre/f131.recovery-small': Cannot send after transport endpoint shutdown 1+0 records in 0+0 records out 0 bytes copied, 4.07798 s, 0.0 kB/s PASS 131 (10s) == recovery-small test 133: don't fail on flock resend === 13:45:26 (1781286326) bl lock multiop /mnt/lustre/f133.recovery-small vO_jc TMPPIPE=/tmp/multiop_open_wait_pipe.7155 fail_loc=0x80000157 waiting for multiop 87752 bl flock unlocked PASS 133 (54s) == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 13:46:19 (1781286379) SKIP: recovery-small test_134 Need 2+ clients, have 1 SKIP 134 (4s) == recovery-small test 135: DOM: open/create resend to return size ========================================================== 13:46:24 (1781286384) fail_loc=0x80000157 Succeed in opening file "/mnt/lustre/d135.recovery-small/f135.recovery-small"(flags=O_RDWR, mode=755) PASS 135 (54s) SKIP: recovery-small test_136 skipping excluded test 136 == recovery-small test 137: late resend must be skipped if already applied ========================================================== 13:47:19 (1781286439) Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.112@tcp:/lustre 7542784 6144 7532544 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.124656 s, 32.9 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.028607 s, 143 kB/s fail_loc=0x80000525 PASS 137 (50s) == recovery-small test 138: Umount MDT during recovery === 13:48:09 (1781286489) SKIP: recovery-small test_138 needs >= 2 MDTs SKIP 138 (4s) == recovery-small test 139: corrupted catid won't cause crash ========================================================== 13:48:13 (1781286493) SKIP: recovery-small test_139 needs >= 2 MDTs SKIP 139 (3s) == recovery-small test 140a: local mount is flagged properly ========================================================== 13:48:16 (1781286496) mdt.lustre-MDT0000.local_recovery=0 oleg312-server Starting client: oleg312-server: -o user_xattr,flock oleg312-server@tcp:/lustre /mnt/lustre2 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 1 clients with recovery disabled Stopping client oleg312-server /mnt/lustre2 (opts:) pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 mdt.lustre-MDT0000.local_recovery=1 oleg312-server Starting client: oleg312-server: -o user_xattr,flock oleg312-server@tcp:/lustre /mnt/lustre2 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 0 clients with recovery disabled Stopping client oleg312-server /mnt/lustre2 (opts:) pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 mdt.lustre-MDT0000.local_recovery=1 PASS 140a (33s) == recovery-small test 140b: local mount is excluded from recovery ========================================================== 13:48:49 (1781286529) mdt.lustre-MDT0000.local_recovery=0 oleg312-server Starting client: oleg312-server: -o user_xattr,flock oleg312-server@tcp:/lustre /mnt/lustre2 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3584 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Stopping client oleg312-server /mnt/lustre2 (opts:) pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Failing mds1 on oleg312-server Stopping /mnt/lustre-mds1 (opts:) on oleg312-server reboot facets: mds1 Failover mds1 to oleg312-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdt.lustre-MDT0000.local_recovery=1 PASS 140b (69s) == recovery-small test 141: do not lose locks on MGS restart ========================================================== 13:49:58 (1781286598) SKIP: recovery-small test_141 cannot run in local mode or from build tree SKIP 141 (4s) == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 13:50:02 (1781286602) fail_loc=0x165 Stopping /mnt/lustre-mds1 (opts:) on oleg312-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 142 (26s) == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 13:50:28 (1781286628) Stopping /mnt/lustre-mds1 (opts:) on oleg312-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 affected facets: mds1 oleg312-server: oleg312-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 oleg312-server: *.lustre-MDT0000.recovery_status status: COMPLETE PASS 143 (50s) == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 13:51:18 (1781286678) timeout=300 Failing ost1 on oleg312-server Stopping /mnt/lustre-ost1 (opts:) on oleg312-server reboot facets: ost1 Failover ost1 to oleg312-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0000 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Failing mds1 on oleg312-server Stopping /mnt/lustre-mds1 (opts:) on oleg312-server reboot facets: mds1 Failover mds1 to oleg312-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95274 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95275 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95276 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95277 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95278 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95279 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95280 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95281 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95282 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95283 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95284 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95285 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95286 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95287 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95288 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95289 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3637: 95292 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3615: 95290 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3615: 95294 Killed touch $DIR/$tdir/$tfile_$i Failing mds1 on oleg312-server Stopping /mnt/lustre-mds1 (opts:) on oleg312-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4330: 95295 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95291 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95297 Killed touch $DIR/$tdir/$tfile_$i reboot facets: mds1 Failover mds1 to oleg312-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3117: 95293 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 2185: 95296 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 2185: 95298 Killed touch $DIR/$tdir/$tfile_$i Started lustre-MDT0000 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3637: 95300 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3637: 95302 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3615: 95301 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3615: 95303 Killed touch $DIR/$tdir/$tfile_$i oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95299 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95304 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95305 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95306 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95307 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95308 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95309 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95310 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95311 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95312 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95313 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95314 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95315 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 95317 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3637: 95316 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3615: 95318 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3615: 95319 Killed touch $DIR/$tdir/$tfile_$i timeout=20 PASS 144a (270s) == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 13:55:48 (1781286948) SKIP: recovery-small test_145 needs >= 3 MDTs SKIP 145 (7s) == recovery-small test 147: Check client reconnect ======= 13:55:55 (1781286955) timeout=200 fail_loc=0x00000225 timeout=200 osc.lustre-OST0000-osc-ffff9f7245e31800.state= current_state: CONNECTING state_history: - [ 1781286864, FULL ] - [ 1781286895, CONNECTING ] - [ 1781286895, IDLE ] - [ 1781286962, CONNECTING ] - [ 1781286973, DISCONN ] - [ 1781286973, CONNECTING ] - [ 1781286989, DISCONN ] - [ 1781286989, CONNECTING ] - [ 1781287010, DISCONN ] - [ 1781287010, CONNECTING ] - [ 1781287036, DISCONN ] - [ 1781287036, CONNECTING ] - [ 1781287067, DISCONN ] - [ 1781287067, CONNECTING ] - [ 1781287103, DISCONN ] - [ 1781287103, CONNECTING ] 8 fail_loc=0 timeout=20 timeout=20 Connected clients: oleg312-client.virtnet oleg312-client.virtnet PASS 147 (189s) == recovery-small test 148: data corruption through resend ========================================================== 13:59:04 (1781287144) at_max=0 at_max=0 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0997515 s, 41.1 kB/s fail_loc=0x80000227 fail_val=22 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 22.2421 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.041455 s, 98.8 kB/s at_max=600 at_max=600 PASS 148 (41s) == recovery-small test 149: skip orphan removal at umount ========================================================== 13:59:45 (1781287185) SKIP: recovery-small test_149 needs >= 2 MDTs SKIP 149 (3s) == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 13:59:48 (1781287188) fail_loc=0x80000173 fail_val=20 lov.lustre-MDT0000-mdtlov.qos_threshold_rr=0 QoS allocation took 11 seconds lov.lustre-MDT0000-mdtlov.qos_threshold_rr=17% PASS 152 (35s) == recovery-small test complete, duration 6494 sec ======= 14:00:23 (1781287223)