-----============= acceptance-small: recovery-small ============----- Sat Feb 28 13:53:29 EST 2026 oleg409-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/recovery-small.*ex': No such file or directory excepting tests: 136 oleg409-client.virtnet: executing check_config_client /mnt/lustre oleg409-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg409-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8c9658889800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c9658889800.idle_timeout=debug disable quota as required oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 osd-ldiskfs.track_declares_assert=1 == recovery-small test 1: create, chmod, stat: drop req, drop rep ========================================================== 13:53:44 (1772304824) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x122 fail_loc=0 PASS 1 (53s) == recovery-small test 4: open: drop req, drop rep ======= 13:54:37 (1772304877) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x122 fail_loc=0 PASS 4 (18s) == recovery-small test 5: rename: drop req, drop rep ===== 13:54:55 (1772304895) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 PASS 5 (17s) == recovery-small test 6: link, unlink: drop req, drop rep ========================================================== 13:55:12 (1772304912) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 PASS 6 (33s) == recovery-small test 8: touch: drop rep (bug 1423) ===== 13:55:45 (1772304945) fail_loc=0x119 fail_loc=0 PASS 8 (9s) == recovery-small test 9: pause bulk on OST (bug 1420) === 13:55:54 (1772304954) timeout is 0/ fail_val=0 fail_loc=0x80000214 fail_loc=0 PASS 9 (9s) == recovery-small test 10a: finish request on server after client eviction (bug 1521) ========================================================== 13:56:03 (1772304963) ldlm.namespaces.MGC192.168.204.109@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8c9658889800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff8c9658889800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8c9658889800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8c9658889800.early_lock_cancel=0 fail_loc=0x305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.204.109@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8c9658889800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff8c9658889800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8c9658889800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c9658889800.early_lock_cancel=1 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Input/output error Connected clients: oleg409-client.virtnet /mnt/lustre has perms 0777 OK PASS 10a (104s) == recovery-small test 10b: re-send BL AST =============== 13:57:47 (1772305067) ldlm.namespaces.MGC192.168.204.109@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8c9658889800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff8c9658889800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8c9658889800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8c9658889800.early_lock_cancel=0 fail_loc=0x80000305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.204.109@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8c9658889800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff8c9658889800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8c9658889800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c9658889800.early_lock_cancel=1 Connected clients: oleg409-client.virtnet oleg409-client.virtnet /mnt/lustre has perms 0777 OK PASS 10b (13s) == recovery-small test 10c: re-send BL AST vs reconnect race (LU-5569) ========================================================== 13:58:01 (1772305081) ldlm.namespaces.MGC192.168.204.109@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8c9658889800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff8c9658889800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8c9658889800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8c9658889800.early_lock_cancel=0 fail_loc=0x80000305 mdc.lustre-MDT0000-mdc-ffff8c9658889800.import=connection=192.168.204.109@tcp fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.204.109@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8c9658889800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff8c9658889800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8c9658889800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c9658889800.early_lock_cancel=1 Connected clients: oleg409-client.virtnet oleg409-client.virtnet /mnt/lustre/d10c.recovery-small has perms 0777 OK PASS 10c (4s) == recovery-small test 10d: test failed blocking ast ===== 13:58:04 (1772305084) 7+0 records in 7+0 records out 7 bytes copied, 0.000145393 s, 48.1 kB/s Stopping client oleg409-client.virtnet /mnt/lustre (opts:) Starting client: oleg409-client.virtnet: -o user_xattr,flock oleg409-server@tcp:/lustre /mnt/lustre Starting client: oleg409-client.virtnet: -o user_xattr,flock oleg409-server@tcp:/lustre /mnt/lustre2 1+0 records in 1+0 records out 5 bytes copied, 0.00235688 s, 2.1 kB/s fail_err=71 ldlm.namespaces.MGC192.168.204.109@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8c96607d6800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8c9660c68800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff8c96607d6800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff8c9660c68800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8c96607d6800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8c9660c68800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8c96607d6800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8c9660c68800.early_lock_cancel=0 fail_loc=0x305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.204.109@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8c96607d6800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8c9660c68800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff8c96607d6800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff8c9660c68800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8c96607d6800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8c9660c68800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c96607d6800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c9660c68800.early_lock_cancel=1 uname: write error: Input/output error Connected clients: oleg409-client.virtnet osc.lustre-OST0000-osc-ffff8c96607d6800.state= current_state: IDLE state_history: - [ 1772305086, CONNECTING ] - [ 1772305086, FULL ] - [ 1772305107, CONNECTING ] - [ 1772305107, IDLE ] osc.lustre-OST0000-osc-ffff8c9660c68800.state= current_state: FULL state_history: - [ 1772305086, CONNECTING ] - [ 1772305086, FULL ] - [ 1772305107, CONNECTING ] - [ 1772305107, IDLE ] - [ 1772305107, CONNECTING ] - [ 1772305107, FULL ] recovery-small test_10d: @@@@@@ FAIL: no eviction: before:1772305084 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6459:error() = /home/green/git/lustre-release/lustre/tests/recovery-small.sh:270:test_10d() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6817:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6864:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6705:run_test() = /home/green/git/lustre-release/lustre/tests/recovery-small.sh:276:main() Dumping lctl log to /tmp/testlogs//recovery-small.test_10d.*.1772305108.log fail_err=0 192.168.204.109@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 rsync: chown "/tmp/testlogs/.recovery-small.test_10d.debug_log.oleg409-server.1772305108.log.AHTarC" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.recovery-small.test_10d.dmesg.oleg409-server.1772305108.log.ACo6no" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] Stopping client oleg409-client.virtnet /mnt/lustre2 (opts:) FAIL 10d (27s) == recovery-small test 10e: re-send BL AST vs reconnect race 2 ========================================================== 13:58:31 (1772305111) SKIP: recovery-small test_10e need two clients SKIP 10e (1s) == recovery-small test 11: wake up a thread waiting for completion after eviction (b=2460) ========================================================== 13:58:32 (1772305112) ldlm.namespaces.MGC192.168.204.109@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8c9660c68800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff8c9660c68800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8c9660c68800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8c9660c68800.early_lock_cancel=0 fail_loc=0x80000305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.204.109@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8c9660c68800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff8c9660c68800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8c9660c68800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c9660c68800.early_lock_cancel=1 PASS 11 (11s) == recovery-small test 12: recover from timed out resend in ptlrpcd (b=2494) ========================================================== 13:58:43 (1772305123) fail_loc=0x115 multiop /mnt/lustre/f12.recovery-small vOS_c TMPPIPE=/tmp/multiop_open_wait_pipe.7732 waiting for multiop 17804 clearing fail_loc on mds1 fail_loc=0 PASS 12 (41s) == recovery-small test 13: mdc_readpage restart test (bug 1138) ========================================================== 13:59:24 (1772305164) fail_loc=0x80000104 newentry fail_loc=0 PASS 13 (10s) == recovery-small test 14: mdc_readpage resend test (bug 1138) ========================================================== 13:59:34 (1772305174) fail_loc=0x80000106 newentry fail_loc=0 PASS 14 (2s) == recovery-small test 15: failed open (-ENOMEM) ========= 13:59:36 (1772305176) fail_loc=0x80000128 touch: cannot touch '/mnt/lustre/f15.recovery-small': Cannot allocate memory PASS 15 (3s) == recovery-small test 16: timeout bulk put, don't evict client (2732) ========================================================== 13:59:39 (1772305179) fail_loc=0x80000504 fail_loc=0 PASS 16 (67s) == recovery-small test 17a: timeout bulk get, don't evict client (2732) ========================================================== 14:00:46 (1772305246) at_max=20 fail_loc=0x80000503 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.109@tcp:/lustre 3833116 1524 3605496 1% /mnt/lustre at_max=600 PASS 17a (44s) == recovery-small test 17b: timeout bulk get, dont evict client (3582) ========================================================== 14:01:30 (1772305290) SKIP: recovery-small test_17b Needs multiple clients SKIP 17b (1s) == recovery-small test 18a: manual ost invalidate clears page cache immediately ========================================================== 14:01:31 (1772305291) PASS 18a (2s) == recovery-small test 18b: eviction and reconnect clears page cache (2766) ========================================================== 14:01:34 (1772305294) /mnt/lustre/d18b.recovery-small/f18b.recovery-small lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 20 0x14 0 recovery-small test_18b: @@@@@@ FAIL: /mnt/lustre/d18b.recovery-small/f18b.recovery-small: stripe_index 1 != 0 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6459:error() = /home/green/git/lustre-release/lustre/tests/recovery-small.sh:594:test_18b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6817:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6864:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6705:run_test() = /home/green/git/lustre-release/lustre/tests/recovery-small.sh:609:main() Dumping lctl log to /tmp/testlogs//recovery-small.test_18b.*.1772305315.log rsync: chown "/tmp/testlogs/.recovery-small.test_18b.debug_log.oleg409-server.1772305315.log.SVzyzE" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.recovery-small.test_18b.dmesg.oleg409-server.1772305315.log.jOXiFr" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 18b (23s) == recovery-small test 18c: Dropped connect reply after eviction handing (14755) ========================================================== 14:01:57 (1772305317) fail_loc=0x80000225 PASS 18c (12s) == recovery-small test 19a: test expired_lock_main on mds (2867) ========================================================== 14:02:09 (1772305329) Starting client: oleg409-client.virtnet: -o user_xattr,flock oleg409-server@tcp:/lustre /mnt/lustre2 fail_loc=0x304 fail_loc=0 192.168.204.109@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg409-client.virtnet /mnt/lustre2 (opts:) Connected clients: oleg409-client.virtnet oleg409-client.virtnet PASS 19a (104s) == recovery-small test 19b: test expired_lock_main on ost (2867) ========================================================== 14:03:53 (1772305433) Starting client: oleg409-client.virtnet: -o user_xattr,flock oleg409-server@tcp:/lustre /mnt/lustre2 fail_loc=0x304 fail_loc=0 192.168.204.109@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg409-client.virtnet /mnt/lustre2 (opts:) uname: write error: Input/output error Connected clients: oleg409-client.virtnet PASS 19b (105s) == recovery-small test 19c: check reconnect and lock resend do not trigger expired_lock_main ========================================================== 14:05:38 (1772305538) Starting client: oleg409-client.virtnet: -o user_xattr,flock oleg409-server@tcp:/lustre /mnt/lustre2 ldlm.namespaces.MGC192.168.204.109@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8c9647f65800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8c9660c68800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff8c9647f65800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff8c9660c68800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8c9647f65800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8c9660c68800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8c9647f65800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8c9660c68800.early_lock_cancel=0 File: /mnt/lustre/f19c.recovery-small Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 2c54f966h/743766374d Inode: 144115205339611140 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-02-28 14:05:38.000000000 -0500 Modify: 2026-02-28 14:05:38.000000000 -0500 Change: 2026-02-28 14:05:38.000000000 -0500 Birth: 2026-02-28 14:05:39.000000000 -0500 fail_loc=0x80000516 ldlm.namespaces.MGC192.168.204.109@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8c9647f65800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8c9660c68800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff8c9647f65800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff8c9660c68800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8c9647f65800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8c9660c68800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c9647f65800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c9660c68800.early_lock_cancel=1 192.168.204.109@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg409-client.virtnet /mnt/lustre2 (opts:) PASS 19c (9s) == recovery-small test 20a: ldlm_handle_enqueue error (should return error) ========================================================== 14:05:48 (1772305548) multiop /mnt/lustre/d20a.recovery-small/f20a.recovery-small vO_wc TMPPIPE=/tmp/multiop_open_wait_pipe.7732 fail_loc=0x80000308 write: Cannot allocate memory PASS 20a (2s) == recovery-small test 20b: ldlm_handle_enqueue error (should return error) ========================================================== 14:05:50 (1772305550) fail_loc=0x80000308 dd: failed to open '/mnt/lustre/d20b.recovery-small/f20b.recovery-small': Cannot allocate memory PASS 20b (22s) == recovery-small test 21a: drop close request while close and open are both in flight ========================================================== 14:06:12 (1772305572) multiop /mnt/lustre/d21a.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7732 fail_loc=0x80000129 fail_loc=0 fail_loc=0x80000115 fail_loc=0 /mnt/lustre/d21a.recovery-small-1/f has type file OK /mnt/lustre/d21a.recovery-small-2/f has type file OK PASS 21a (12s) == recovery-small test 21b: drop open request while close and open are both in flight ========================================================== 14:06:24 (1772305584) multiop /mnt/lustre/d21b.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7732 fail_loc=0x80000107 fail_loc=0 /mnt/lustre/d21b.recovery-small-1/f has type file OK /mnt/lustre/d21b.recovery-small-2/f has type file OK PASS 21b (173s) == recovery-small test 21c: drop both request while close and open are both in flight ========================================================== 14:09:17 (1772305757) multiop /mnt/lustre/d21c.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7732 fail_loc=0x80000107 fail_loc=0 fail_loc=0x80000115 fail_loc=0 /mnt/lustre/d21c.recovery-small-1/f has type file OK /mnt/lustre/d21c.recovery-small-2/f has type file OK PASS 21c (173s) == recovery-small test 21d: drop close reply while close and open are both in flight ========================================================== 14:12:10 (1772305930) multiop /mnt/lustre/d21d.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7732 fail_loc=0x80000129 fail_loc=0 fail_loc=0x80000122 fail_loc=0 /mnt/lustre/d21d.recovery-small-1/f has type file OK /mnt/lustre/d21d.recovery-small-2/f has type file OK PASS 21d (11s) == recovery-small test 21e: drop open reply while close and open are both in flight ========================================================== 14:12:21 (1772305941) multiop /mnt/lustre/d21e.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7732 fail_loc=0x80000119 fail_loc=0 /mnt/lustre/d21e.recovery-small-1/f has type file OK /mnt/lustre/d21e.recovery-small-2/f has type file OK PASS 21e (171s) == recovery-small test 21f: drop both reply while close and open are both in flight ========================================================== 14:15:12 (1772306112) multiop /mnt/lustre/d21f.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7732 fail_loc=0x80000119 fail_loc=0 fail_loc=0x80000122 fail_loc=0 /mnt/lustre/d21f.recovery-small-1/f has type file OK /mnt/lustre/d21f.recovery-small-2/f has type file OK PASS 21f (32s) == recovery-small test 21g: drop open reply and close request while close and open are both in flight ========================================================== 14:15:44 (1772306144) multiop /mnt/lustre/d21g.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7732 fail_loc=0x80000119 fail_loc=0 fail_loc=0x80000115 fail_loc=0 /mnt/lustre/d21g.recovery-small-1/f has type file OK /mnt/lustre/d21g.recovery-small-2/f has type file OK PASS 21g (12s) == recovery-small test 21h: drop open request and close reply while close and open are both in flight ========================================================== 14:15:56 (1772306156) multiop /mnt/lustre/d21h.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7732 fail_loc=0x80000107 fail_loc=0 fail_loc=0x80000122 fail_loc=0 /mnt/lustre/d21h.recovery-small-1/f has type file OK /mnt/lustre/d21h.recovery-small-2/f has type file OK PASS 21h (12s) == recovery-small test 22: drop close request and do mknod ========================================================== 14:16:08 (1772306168) fail_loc=0x80000115 fail_loc=0 PASS 22 (10s) == recovery-small test 23: client hang when close a file after mds crash ========================================================== 14:16:18 (1772306178) multiop /mnt/lustre/f23.recovery-small vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7732 fail_val=0 fail_loc=0x123 fail_loc=0 Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server reboot facets: mds1 Failover mds1 to oleg409-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23 (33s) == recovery-small test 24a: fsync error (should return error) ========================================================== 14:16:51 (1772306211) multiop /mnt/lustre/d24a.recovery-small/f24a.recovery-small vOwy_wyc TMPPIPE=/tmp/multiop_open_wait_pipe.7732 fsync: Input/output error fail_loc=0x0 Connected clients: oleg409-client.virtnet oleg409-client.virtnet PASS 24a (3s) == recovery-small test 24b: test dirty page discard due to client eviction ========================================================== 14:16:54 (1772306214) multiop /mnt/lustre/d24b.recovery-small/f24b.recovery-small-1 vOw8192_yc TMPPIPE=/tmp/multiop_open_wait_pipe.7732 multiop /mnt/lustre/d24b.recovery-small/f24b.recovery-small-2 vOw8192_c TMPPIPE=/tmp/multiop_open_wait_pipe.7732 fsync: Input/output error close: Input/output error fail_loc=0x0 Connected clients: oleg409-client.virtnet oleg409-client.virtnet [ 1529.916523] Lustre: 2267:0:(llite_lib.c:3595:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.109@tcp:/lustre/fid: [0x200000405:0x20:0x0]// may get corrupted (rc -108) PASS 24b (3s) == recovery-small test 26a: evict dead exports =========== 14:16:57 (1772306217) SKIP: recovery-small test_26a msg and ost1 are at the same node SKIP 26a (2s) == recovery-small test 26b: evict dead exports =========== 14:16:59 (1772306219) SKIP: recovery-small test_26b msg and ost1 are at the same node SKIP 26b (1s) == recovery-small test 27: fail LOV while using OSC's ==== 14:17:00 (1772306220) Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server reboot facets: mds1 Failover mds1 to oleg409-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0x80000407 waiting for fail_loc Waiting 90s for '-2147482617' Waiting 80s for '-2147482617' Waiting 70s for '-2147482617' Waiting 60s for '-2147482617' Waiting 50s for '-2147482617' Waiting 30s for '-2147482617' Waiting 20s for '-2147482617' Waiting 10s for '-2147482617' Waiting 0s for '-2147482617' Update not seen after 90s: want '-2147482617' got '2147484679' Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server reboot facets: mds1 Failover mds1 to oleg409-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 27 (137s) == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 14:19:17 (1772306357) ldlm.namespaces.MGC192.168.204.109@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8c9660c68800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff8c9660c68800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8c9660c68800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8c9660c68800.early_lock_cancel=0 fail_loc=0x80000305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.204.109@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8c9660c68800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff8c9660c68800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8c9660c68800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c9660c68800.early_lock_cancel=1 fail_loc=0x8000012f Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server reboot facets: mds1 Failover mds1 to oleg409-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 28 (48s) == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 14:20:05 (1772306405) fail_loc=0x80000711 Stopping /mnt/lustre-mds1 (opts:) on oleg409-server Failover mds1 to oleg409-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg409-server: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg409-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec PASS 29a (26s) == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 14:20:31 (1772306431) fail_loc=0x80000711 Stopping /mnt/lustre-ost1 (opts:) on oleg409-server Failover ost1 to oleg409-server Starting ost1: -o localrecov -o abort_recovery /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 pdsh@oleg409-client: oleg409-client: ssh exited with exit code 5 first stat failed: 5 PASS 29b (20s) == recovery-small test 50: failover MDS under load ======= 14:20:51 (1772306451) writemany pid 41824 Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server reboot facets: mds1 Failover mds1 to oleg409-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server reboot facets: mds1 Failover mds1 to oleg409-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server reboot facets: mds1 Failover mds1 to oleg409-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec writemany returned 0 PASS 50 (249s) == recovery-small test 51: failover MDS during recovery == 14:25:00 (1772306700) fail_loc=0x00001310 Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server reboot facets: mds1 Failover mds1 to oleg409-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 will failover at 1 5 10 20 25 30 test_51: failover in 1 sec Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server reboot facets: mds1 Failover mds1 to oleg409-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 test_51: failover in 5 sec Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server reboot facets: mds1 Failover mds1 to oleg409-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 test_51: failover in 10 sec Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server reboot facets: mds1 Failover mds1 to oleg409-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 test_51: failover in 20 sec Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server reboot facets: mds1 Failover mds1 to oleg409-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 test_51: failover in 25 sec Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server reboot facets: mds1 Failover mds1 to oleg409-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 test_51: failover in 30 sec Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server reboot facets: mds1 Failover mds1 to oleg409-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 writemany returned 0 PASS 51 (242s) == recovery-small test 52: failover OST under load ======= 14:29:02 (1772306942) writemany pid 48348 Failing ost1 on oleg409-server Stopping /mnt/lustre-ost1 (opts:) on oleg409-server reboot facets: ost1 Failover ost1 to oleg409-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec writemany succeeded writemany pid 49242 Failing ost1 on oleg409-server Stopping /mnt/lustre-ost1 (opts:) on oleg409-server reboot facets: ost1 Failover ost1 to oleg409-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec writemany succeeded writemany pid 50141 Failing ost1 on oleg409-server Stopping /mnt/lustre-ost1 (opts:) on oleg409-server reboot facets: ost1 Failover ost1 to oleg409-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec writemany succeeded Connected clients: oleg409-client.virtnet oleg409-client.virtnet PASS 52 (967s) == recovery-small test 53a: touch: drop rep ============== 14:45:09 (1772307909) fail_loc=0x157 Succeed in opening file "/mnt/lustre/f53a.recovery-small"(flags=O_RDWR, mode=755) fail_loc=0 PASS 53a (49s) == recovery-small test 53b: touch: drop rep ============== 14:45:58 (1772307958) fail_loc=0x157 Succeed in opening file "/mnt/lustre/f53b.recovery-small"(flags=O_RDWR, mode=755) fail_loc=0 PASS 53b (49s) == recovery-small test 53c: touch: drop rep ============== 14:46:47 (1772308007) fail_loc=0x157 Succeed in opening file "/mnt/lustre/f53c.recovery-small"(flags=O_RDWR, mode=755) fail_loc=0 PASS 53c (49s) == recovery-small test 54: back in time ================== 14:47:36 (1772308056) Starting client: oleg409-client.virtnet: -o user_xattr,flock oleg409-server@tcp:/lustre /mnt/lustre2 cat: /mnt/lustre2/f54.recovery-small.missing: No such file or directory Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server reboot facets: mds1 Failover mds1 to oleg409-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 54 (39s) == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 14:48:15 (1772308095) step1: testing ...... 4+0 records in 4+0 records out 134217728 bytes (134 MB, 128 MiB) copied, 7.42022 s, 18.1 MB/s (dd_pid=55171, time=8)successful fail_loc=0x0000021d step2: testing ...... (dd_pid=55190, time=65)successful fail_loc=0 step3: testing ...... 4+0 records in 4+0 records out 134217728 bytes (134 MB, 128 MiB) copied, 74.1548 s, 1.8 MB/s (dd_pid=55190, time=9)successful osc.lustre-OST0000-osc-ffff8c9660c68800.max_dirty_mb=462 osc.lustre-OST0001-osc-ffff8c9660c68800.max_dirty_mb=462 PASS 55 (89s) == recovery-small test 56: do not fail on getattr resend ========================================================== 14:49:44 (1772308184) fail_loc=0x80000136 File: /mnt/lustre/f56.recovery-small Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205339681557 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-02-28 14:49:45.000000000 -0500 Modify: 2026-02-28 14:49:45.000000000 -0500 Change: 2026-02-28 14:49:45.000000000 -0500 Birth: 2026-02-28 14:49:45.000000000 -0500 fail_loc=0 PASS 56 (44s) == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 14:50:28 (1772308228) fail_loc=0x80000B00 Stopping client oleg409-client.virtnet /mnt/lustre (opts:) fail_loc=0x80000B00 Stopping /mnt/lustre-mds1 (opts:) on oleg409-server Failover mds1 to oleg409-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 pdsh@oleg409-client: oleg409-client: ssh exited with exit code 95 pdsh@oleg409-client: oleg409-client: ssh exited with exit code 95 /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 1414: kill: (56454) - No such process fail_loc=0 Starting client: oleg409-client.virtnet: -o user_xattr,flock oleg409-server@tcp:/lustre /mnt/lustre Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.109@tcp:/lustre 7666232 3744 7210296 1% /mnt/lustre PASS 57 (22s) == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 14:50:50 (1772308250) -rw-r--r-- 1 root root 0 Feb 28 14:50 /mnt/lustre/f58.recovery-small fail_loc=0x80000801 fail_loc=0 ldlm.namespaces.MGC192.168.204.109@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8c9645163800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff8c9645163800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8c9645163800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8c9645163800.early_lock_cancel=0 fail_loc=0x80000305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.204.109@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8c9645163800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff8c9645163800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8c9645163800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c9645163800.early_lock_cancel=1 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.109@tcp:/lustre 7666232 3744 7210296 1% /mnt/lustre PASS 58 (16s) == recovery-small test 59: Read cancel race on client eviction ========================================================== 14:51:06 (1772308266) Starting client: oleg409-client.virtnet: -o user_xattr,flock oleg409-server@tcp:/lustre /mnt/lustre2 fail_loc=0x311 fail_loc=0 Stopping client oleg409-client.virtnet /mnt/lustre2 (opts:-f) PASS 59 (14s) == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 14:51:20 (1772308280) striped dir -i0 -c2 -H crush /mnt/lustre/d60.recovery-small mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' - open/close 2644 (time 1772308294.63 total 10.00 last 264.38) total: 5000 open/close in 19.58 seconds: 255.30 ops/second Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server reboot facets: mds1 Failover mds1 to oleg409-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 - unlinked 0 (time 1772308311 ; total 0 ; last 0) total: 5000 unlinks in 42 seconds: 119.047623 unlinks/second 5000 unlinks in changelog lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0001: changelog user 'cl1' not found lustre-MDT0000: changelog user 'cl1' not found PASS 60 (82s) == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 14:52:42 (1772308362) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2912 1284776 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4536 1283152 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1700 3605320 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2048 3604972 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3748 7210292 1% /mnt/lustre total: 10 open/close in 0.11 seconds: 95.11 ops/second Stopping /mnt/lustre-mds1 (opts:) on oleg409-server Failover mds1 to oleg409-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 1524: [: too many arguments PASS 61 (26s) == recovery-small test 65: lock enqueue for destroyed export ========================================================== 14:53:08 (1772308388) Starting client: oleg409-client.virtnet: -o user_xattr,flock oleg409-server@tcp:/lustre /mnt/lustre2 fail_loc=0x31e write: Input/output error fail_loc=0 192.168.204.109@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg409-client.virtnet /mnt/lustre2 (opts:) PASS 65 (16s) == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 14:53:24 (1772308404) fail_loc=0x80000157 fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8c9645163800.import=connection=192.168.204.109@tcp stat: cannot statx '/mnt/lustre/f66.recovery-small': Input/output error /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: oleg409-client.virtnet fail_loc=0 PASS 66 (11s) == recovery-small test 67: connect vs import invalidate race ========================================================== 14:53:35 (1772308415) fail_loc=0x80000531 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Input/output error mdc.lustre-MDT0000-mdc-ffff8c9645163800.import=connection=192.168.204.109@tcp error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: No such device cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: No such device error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: No such device cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: No such device error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: No such device cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: No such device error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: No such device cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: No such device error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: No such device cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: No such device error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: No such device cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: No such device error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: No such device cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: No such device error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: No such device cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: No such device error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: No such device cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: No such device error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: No such device cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown Connected clients: oleg409-client.virtnet oleg409-client.virtnet PASS 67 (17s) == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 14:53:52 (1772308432) mgs.MGS.ir_timeout Failing ost1 on oleg409-server Stopping /mnt/lustre-ost1 (opts:) on oleg409-server reboot facets: ost1 Failover ost1 to oleg409-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 100 (28s) == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 14:54:20 (1772308460) mgs.MGS.ir_timeout Failing ost1 on oleg409-server Stopping /mnt/lustre-ost1 (opts:) on oleg409-server reboot facets: ost1 Failover ost1 to oleg409-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 101a (28s) == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 14:54:48 (1772308488) mgs.MGS.ir_timeout fail_loc=0x247 Failing ost1 on oleg409-server Stopping /mnt/lustre-ost1 (opts:) on oleg409-server reboot facets: ost1 Failover ost1 to oleg409-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 101b (52s) == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 14:55:40 (1772308540) mgs.MGS.ir_timeout Failing ost1 on oleg409-server Stopping /mnt/lustre-ost1 (opts:) on oleg409-server reboot facets: ost1 Failover ost1 to oleg409-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Stopping clients: oleg409-client.virtnet /mnt/lustre (opts:) Stopping client oleg409-client.virtnet /mnt/lustre opts: Stopping /mnt/lustre-mds1 (opts:) on oleg409-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 Failing ost1 on oleg409-server Stopping /mnt/lustre-ost1 (opts:) on oleg409-server reboot facets: ost1 Failover ost1 to oleg409-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 pdsh@oleg409-client: oleg409-client: ssh exited with exit code 95 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid pdsh@oleg409-client: oleg409-client: ssh exited with exit code 95 Starting client oleg409-client.virtnet: -o user_xattr,flock oleg409-server@tcp:/lustre /mnt/lustre Started clients oleg409-client.virtnet: 192.168.204.109@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) PASS 102 (58s) == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 14:56:38 (1772308598) mgs.MGS.ir_timeout SKIP: recovery-small test_103 needs separate mgs and mds SKIP 103 (2s) == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 14:56:40 (1772308600) mgs.MGS.ir_timeout Stopping /mnt/lustre-ost1 (opts:) on oleg409-server Starting ost1: -o localrecov -onoir /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 PASS 104 (12s) == recovery-small test 105: IR: NON IR clients support === 14:56:52 (1772308612) SKIP: recovery-small test_105 Needs multiple clients SKIP 105 (2s) == recovery-small test 106: lightweight connection support ========================================================== 14:56:54 (1772308614) fail_loc=0x805 Starting client: oleg409-client.virtnet: -o user_xattr,flock oleg409-server@tcp:/lustre /mnt/lustre2 fail_loc=0 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2292 1285396 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4536 1283152 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1704 3605316 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2048 3604972 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3752 7210288 1% /mnt/lustre debug=ha Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server reboot facets: mds1 Failover mds1 to oleg409-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 Debug log: 211 lines, 211 kept, 0 dropped, 0 bad. /mnt/lustre/f106.recovery-small has type file OK 192.168.204.109@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg409-client.virtnet /mnt/lustre2 (opts:) PASS 106 (40s) == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 14:57:34 (1772308654) fail_loc=0x119 fail_loc=0 Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server reboot facets: mds1 Failover mds1 to oleg409-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 107 (33s) == recovery-small test 108: client eviction don't crash == 14:58:08 (1772308688) dd: error writing '/mnt/lustre/d108.recovery-small/f108.recovery-small': Input/output error 16+0 records in 15+0 records out 16703488 bytes (17 MB, 16 MiB) copied, 2.60329 s, 6.4 MB/s PASS 108 (7s) == recovery-small test 110a: create remote directory: drop client req ========================================================== 14:58:15 (1772308695) fail_val=0 fail_loc=0x123 fail_loc=0 PASS 110a (62s) == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 14:59:17 (1772308757) fail_loc=0x119 fail_loc=0 PASS 110b (60s) == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 15:00:17 (1772308817) fail_loc=0x1701 fail_loc=0 PASS 110c (9s) == recovery-small test 110d: remove remote directory: drop client req ========================================================== 15:00:26 (1772308826) fail_val=0 fail_loc=0x123 fail_loc=0 PASS 110d (61s) == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 15:01:27 (1772308887) fail_loc=0x119 fail_loc=0 PASS 110e (61s) == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 15:02:28 (1772308948) fail_loc=0x1701 fail_loc=0 PASS 110f (10s) == recovery-small test 110g: drop reply during migration ========================================================== 15:02:38 (1772308958) fail_loc=0x119 fail_loc=0 PASS 110g (60s) == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 15:03:38 (1772309018) 0+1 records in 0+1 records out 158 bytes copied, 0.0106161 s, 14.9 kB/s fail_loc=0x1701 rename returned 0: Success fail_loc=0 Can't lstat /mnt/lustre/d110h.recovery-small/source_dir/src_file: No such file or directory PASS 110h (10s) == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 15:03:48 (1772309028) fail_loc=0x1701 rename returned 0: Success fail_loc=0 Can't lstat /mnt/lustre/d110i.recovery-small/source_dir/src_dir: No such file or directory /mnt/lustre/d110i.recovery-small/target_dir/tgt_dir has type dir OK /mnt/lustre/d110i.recovery-small/target_dir/tgt_dir/a has type file OK PASS 110i (10s) == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 15:03:58 (1772309038) fail_loc=0x1701 fail_loc=0 /mnt/lustre/d110j.recovery-small/remote_dir/remote_file has type file OK PASS 110j (9s) == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 15:04:07 (1772309047) Stopping /mnt/lustre-mds2 (opts:) on oleg409-server fail_loc=0x80001103 Starting mds2: -o localrecov -o abort_recovery /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0001 fail_loc=0 Stopping /mnt/lustre-mds2 (opts:) on oleg409-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0001 Starting client: oleg409-client.virtnet: -o user_xattr,flock oleg409-server@tcp:/lustre /mnt/lustre PASS 110k (25s) == recovery-small test 110m: update resent vs original RPC race ========================================================== 15:04:32 (1772309072) fail_loc=0x80000525 conn_uuid=192.168.204.109@tcp osp.lustre-MDT0000-osp-MDT0001.import=connection=192.168.204.109@tcp PASS 110m (7s) == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 15:04:39 (1772309079) fail_loc=0x151 Stopping /mnt/lustre-mds1 (opts:) on oleg409-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: Input/output error oleg409-server: Is the MGS running? pdsh@oleg409-client: oleg409-server: ssh exited with exit code 5 Start of /dev/mapper/mds1_flakey on mds1 failed 5 fail_loc=0 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 111 (13s) == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 15:04:52 (1772309092) timeout is 20/20 fail_val=20 fail_loc=0x80000214 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.109@tcp:/lustre 7666232 3756 7210160 1% /mnt/lustre PASS 112a (24s) == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 15:05:16 (1772309116) Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.109@tcp:/lustre 7666232 3756 7210160 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00930092 s, 440 kB/s fail_loc=0x8000051b fail_val=3 fail_val=0 fail_loc=0x8000051a 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 4.01156 s, 1.0 kB/s PASS 115a (6s) == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 15:05:22 (1772309122) Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.109@tcp:/lustre 7666232 3860 7210028 1% /mnt/lustre fail_loc=0x8000051b fail_val=4 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.109@tcp:/lustre 7666232 3860 7210028 1% /mnt/lustre fail_val=2 fail_loc=0x80000215 dd: error writing '/mnt/lustre/f115b.recovery-small': No space left on device 1+0 records in 0+0 records out 0 bytes copied, 4.05208 s, 0.0 kB/s PASS 115b (6s) == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 15:05:28 (1772309128) Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.109@tcp:/lustre 7666232 3760 7210280 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00553061 s, 741 kB/s fail_loc=0x8000050f fail_val=3 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00600576 s, 682 kB/s fail_val=0 fail_loc=0x8000051a PASS 115c (3s) == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 15:05:31 (1772309131) Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.109@tcp:/lustre 7666232 3760 7210252 1% /mnt/lustre fail_loc=0x8000050f fail_val=4 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0115638 s, 354 kB/s Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.109@tcp:/lustre 7666232 3760 7210252 1% /mnt/lustre fail_val=0 fail_loc=0x80000215 PASS 115d (4s) == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 15:05:35 (1772309135) Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.109@tcp:/lustre 7666232 3768 7210244 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00684919 s, 598 kB/s fail_loc=0x80000510 fail_val=3 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00600859 s, 682 kB/s fail_val=0 fail_loc=0x80000211 PASS 115e (4s) == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 15:05:39 (1772309139) Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.109@tcp:/lustre 7666232 3768 7210244 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00912472 s, 449 kB/s fail_loc=0x8000051b fail_val=3 fail_val=0 fail_loc=0x80000211 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 4.06788 s, 1.0 kB/s PASS 115f (6s) == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 15:05:45 (1772309145) Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.109@tcp:/lustre 7666232 3776 7210236 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00640078 s, 640 kB/s fail_loc=0x8000051c fail_val=3 fail_val=0 fail_loc=0 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 55.4852 s, 0.1 kB/s PASS 115g (58s) == recovery-small test 120: flock race: completion vs. evict ========================================================== 15:06:43 (1772309203) ** FLOCK REPLY vs. EVICTION race, lock set, CLEANUP cp first fail_loc=0x80000320 FLOCKS_TEST 5: SET write flock ** Evicting and re-connecting client ** /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Input/output error error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown Connected clients: oleg409-client.virtnet fcntl cmd 7 failed: Input/output error exiting with rc = 5 fail_loc=0 ** FLOCK REPLY vs. EVICTION race, lock get, CLEANUP cp first ** Taking conflict ** FLOCKS_TEST 5: SET read flock fail_loc=0x80000320 FLOCKS_TEST 5: GET write flock ** Evicting and re-connecting client ** /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Input/output error Connected clients: oleg409-client.virtnet fail_loc=0 ** FLOCK REPLY vs. EVICTION race, lock unlock, CLEANUP cp first fail_loc=0x80000320 FLOCKS_TEST 5: UNLOCK write flock ** Evicting and re-connecting client ** /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Input/output error Connected clients: oleg409-client.virtnet fail_loc=0 ** FLOCK REPLY vs. EVICTION race, lock set, REPLY cp first fail_loc=0x80000321 FLOCKS_TEST 5: SET write flock ** Evicting and re-connecting client ** /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Input/output error error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown fcntl cmd 7 failed: Input/output error exiting with rc = 5 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown Connected clients: oleg409-client.virtnet oleg409-client.virtnet fail_loc=0 ** FLOCK REPLY vs. EVICTION race, lock get, REPLY cp first ** Taking conflict ** FLOCKS_TEST 5: SET read flock fail_loc=0x80000321 FLOCKS_TEST 5: GET write flock ** Evicting and re-connecting client ** /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Input/output error Connected clients: oleg409-client.virtnet fail_loc=0 ** FLOCK REPLY vs. EVICTION race, lock unlock, REPLY cp first fail_loc=0x80000321 FLOCKS_TEST 5: UNLOCK write flock ** Evicting and re-connecting client ** /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Input/output error Connected clients: oleg409-client.virtnet fail_loc=0 ** FLOCK REPLY vs. EVICTION race, lock set DEADLOCK, CLEANUP cp first fail_loc=0x80000322 FLOCKS_TEST 5: SET write flock ** Evicting and re-connecting client ** /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Input/output error Connected clients: oleg409-client.virtnet fcntl cmd 7 failed: Input/output error exiting with rc = 5 fail_loc=0 ** FLOCK REPLY vs. EVICTION race, lock set DEADLOCK, REPLY cp first fail_loc=0x80000323 FLOCKS_TEST 5: SET write flock ** Evicting and re-connecting client ** /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Input/output error error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown fcntl cmd 7 failed: Resource deadlock avoided exiting with rc = 35 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown Connected clients: oleg409-client.virtnet oleg409-client.virtnet fail_loc=0 FLOCKS_TEST 5: SET write flock fail_loc=0x320 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Input/output error error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3678: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access '/mnt/lustre/recon': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown Connected clients: oleg409-client.virtnet oleg409-client.virtnet fail_loc=0 PASS 120 (62s) == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 15:07:45 (1772309265) fail_loc=0x80000157 File: /mnt/lustre/f113.recovery-small Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144116010713088002 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-02-28 15:07:46.000000000 -0500 Modify: 2026-02-28 15:07:46.000000000 -0500 Change: 2026-02-28 15:07:46.000000000 -0500 Birth: 2026-02-28 15:07:46.000000000 -0500 fail_loc=0 fail_loc=0x8000031f Connected clients: oleg409-client.virtnet oleg409-client.virtnet PASS 113 (64s) == recovery-small test 130a: enqueue resend on not existing file ========================================================== 15:08:49 (1772309329) striped dir -i0 -c2 -H all_char /mnt/lustre/d130a.recovery-small fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000157 stat: cannot statx '/mnt/lustre/d130a.recovery-small': No such file or directory PASS 130a (61s) == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 15:09:50 (1772309390) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d130b.recovery-small fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000157 fail_val=0 fail_loc=0x80000217 stat: cannot statx '/mnt/lustre/d130b.recovery-small': No such file or directory PASS 130b (62s) == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 15:10:52 (1772309452) fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000157 fail_val=0 fail_loc=0x80000217 PASS 130c (26s) == recovery-small test 132: long punch =================== 15:11:18 (1772309478) Starting client: oleg409-client.virtnet: -o user_xattr,flock oleg409-server@tcp:/lustre /mnt/lustre2 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0112618 s, 364 kB/s fail_val=120 fail_loc=0x80000236 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 119.02 s, 0.0 kB/s 192.168.204.109@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg409-client.virtnet /mnt/lustre2 (opts:) PASS 132 (123s) == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 15:13:21 (1772309601) 1+0 records in 1+0 records out 512 bytes copied, 0.00284375 s, 180 kB/s fail_loc=0x80000414 fail_val=4 fail_loc=0x80000414 fail_val=0 fail_loc=0x8000031e Connected clients: oleg409-client.virtnet oleg409-client.virtnet dd: writing to '/mnt/lustre/f131.recovery-small': Cannot send after transport endpoint shutdown 1+0 records in 0+0 records out 0 bytes copied, 4.07106 s, 0.0 kB/s PASS 131 (6s) == recovery-small test 133: don't fail on flock resend === 15:13:27 (1772309607) bl lock multiop /mnt/lustre/f133.recovery-small vO_jc TMPPIPE=/tmp/multiop_open_wait_pipe.7732 fail_loc=0x80000157 waiting for multiop 94480 bl flock unlocked PASS 133 (42s) == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 15:14:09 (1772309649) SKIP: recovery-small test_134 Need 2+ clients, have 1 SKIP 134 (1s) == recovery-small test 135: DOM: open/create resend to return size ========================================================== 15:14:10 (1772309650) fail_loc=0x80000157 Succeed in opening file "/mnt/lustre/d135.recovery-small/f135.recovery-small"(flags=O_RDWR, mode=755) PASS 135 (20s) SKIP: recovery-small test_136 skipping excluded test 136 == recovery-small test 137: late resend must be skipped if already applied ========================================================== 15:14:31 (1772309671) Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.109@tcp:/lustre 7666232 3784 7210256 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00630977 s, 649 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00927377 s, 442 kB/s fail_loc=0x80000525 PASS 137 (19s) == recovery-small test 138: Umount MDT during recovery === 15:14:50 (1772309690) Stopping clients: oleg409-client.virtnet /mnt/lustre (opts:) Stopping client oleg409-client.virtnet /mnt/lustre opts: fail_loc=0x724 fail_val=5 Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server reboot facets: mds1 Failover mds1 to oleg409-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg409-server fail_loc=0 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client oleg409-client.virtnet: -o user_xattr,flock oleg409-server@tcp:/lustre /mnt/lustre Started clients oleg409-client.virtnet: 192.168.204.109@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) PASS 138 (108s) == recovery-small test 139: corrupted catid won't cause crash ========================================================== 15:16:38 (1772309798) Stopping /mnt/lustre-mds1 (opts:) on oleg409-server fail_val=0x68 fail_loc=0x80002106 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 139 (9s) == recovery-small test 140a: local mount is flagged properly ========================================================== 15:16:47 (1772309807) mdt.lustre-MDT0000.local_recovery=0 mdt.lustre-MDT0001.local_recovery=0 oleg409-server Starting client: oleg409-server: -o user_xattr,flock oleg409-server@tcp:/lustre /mnt/lustre2 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 1 clients with recovery disabled Stopping client oleg409-server /mnt/lustre2 (opts:) pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 mdt.lustre-MDT0000.local_recovery=1 mdt.lustre-MDT0001.local_recovery=1 oleg409-server Starting client: oleg409-server: -o user_xattr,flock oleg409-server@tcp:/lustre /mnt/lustre2 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 0 clients with recovery disabled Stopping client oleg409-server /mnt/lustre2 (opts:) pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 mdt.lustre-MDT0000.local_recovery=1 mdt.lustre-MDT0001.local_recovery=1 PASS 140a (10s) == recovery-small test 140b: local mount is excluded from recovery ========================================================== 15:16:57 (1772309817) mdt.lustre-MDT0000.local_recovery=0 mdt.lustre-MDT0001.local_recovery=0 oleg409-server Starting client: oleg409-server: -o user_xattr,flock oleg409-server@tcp:/lustre /mnt/lustre2 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2428 1285260 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2184 1285504 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1732 3605288 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2060 3604960 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3792 7210248 1% /mnt/lustre Stopping client oleg409-server /mnt/lustre2 (opts:) pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server reboot facets: mds1 Failover mds1 to oleg409-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdt.lustre-MDT0000.local_recovery=1 mdt.lustre-MDT0001.local_recovery=1 PASS 140b (37s) == recovery-small test 141: do not lose locks on MGS restart ========================================================== 15:17:34 (1772309854) SKIP: recovery-small test_141 cannot run in local mode or from build tree SKIP 141 (1s) == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 15:17:35 (1772309855) fail_loc=0x165 Stopping /mnt/lustre-mds1 (opts:) on oleg409-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 142 (9s) == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 15:17:44 (1772309864) Stopping /mnt/lustre-mds1 (opts:) on oleg409-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 affected facets: mds1 oleg409-server: oleg409-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 oleg409-server: *.lustre-MDT0000.recovery_status status: RECOVERING oleg409-server: Waiting 1470 secs for *.lustre-MDT0000.recovery_status recovery done. status: RECOVERING oleg409-server: *.lustre-MDT0000.recovery_status status: COMPLETE PASS 143 (16s) == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 15:18:00 (1772309880) striped dir -i0 -c1 -H all_char /mnt/lustre/d144a.recovery-small timeout=300 Failing ost1 on oleg409-server Stopping /mnt/lustre-ost1 (opts:) on oleg409-server reboot facets: ost1 Failover ost1 to oleg409-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server reboot facets: mds1 Failover mds1 to oleg409-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103421 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103422 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103423 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103424 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103425 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103426 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103427 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103428 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103429 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103430 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103431 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103432 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103435 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3637: 103433 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3637: 103434 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3637: 103436 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3637: 103437 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3637: 103438 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3615: 103439 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3615: 103440 Killed touch $DIR/$tdir/$tfile_$i Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4330: 103441 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4330: 103442 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103443 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103444 Killed touch $DIR/$tdir/$tfile_$i reboot facets: mds1 Failover mds1 to oleg409-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103445 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103446 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103447 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103448 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103449 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103450 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103451 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103452 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103453 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103454 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103455 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4169: 103456 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3637: 103457 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3637: 103458 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3637: 103459 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3637: 103460 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3637: 103462 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3615: 103461 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3615: 103463 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4298: 103465 Killed touch $DIR/$tdir/$tfile_$i (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4274: 103464 Killed touch $DIR/$tdir/$tfile_$i (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4274: 103466 Killed touch $DIR/$tdir/$tfile_$i (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7591: 103467 Killed touch $DIR/$tdir/$tfile_$i (wd: ~) timeout=20 PASS 144a (193s) == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 15:21:13 (1772310073) SKIP: recovery-small test_145 needs >= 3 MDTs SKIP 145 (3s) == recovery-small test 147: Check client reconnect ======= 15:21:16 (1772310076) timeout=200 fail_loc=0x00000225 timeout=200 osc.lustre-OST0000-osc-ffff8c96464a2000.state= current_state: CONNECTING state_history: - [ 1772310022, FULL ] - [ 1772310047, CONNECTING ] - [ 1772310047, IDLE ] - [ 1772310079, CONNECTING ] - [ 1772310090, DISCONN ] - [ 1772310090, CONNECTING ] - [ 1772310106, DISCONN ] - [ 1772310106, CONNECTING ] - [ 1772310127, DISCONN ] - [ 1772310127, CONNECTING ] - [ 1772310153, DISCONN ] - [ 1772310153, CONNECTING ] - [ 1772310184, DISCONN ] - [ 1772310184, CONNECTING ] - [ 1772310220, DISCONN ] - [ 1772310220, CONNECTING ] 8 fail_loc=0 timeout=20 timeout=20 Connected clients: oleg409-client.virtnet oleg409-client.virtnet PASS 147 (185s) == recovery-small test 148: data corruption through resend ========================================================== 15:24:21 (1772310261) at_max=0 at_max=0 oleg409-server: error: read_param: '/sys/fs/lustre/obdfilter/lustre-OST0000/writethrough_cache_enable': Operation not supported oleg409-server: error: set_param: setting /sys/fs/lustre/obdfilter/lustre-OST0000/writethrough_cache_enable=0: Operation not supported pdsh@oleg409-client: oleg409-server: ssh exited with exit code 95 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00959912 s, 427 kB/s fail_loc=0x80000227 fail_val=22 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 22.0733 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00696818 s, 588 kB/s at_max=600 at_max=600 oleg409-server: error: set_param: setting : no value pdsh@oleg409-client: oleg409-server: ssh exited with exit code 22 PASS 148 (29s) == recovery-small test 149: skip orphan removal at umount ========================================================== 15:24:50 (1772310290) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d149.recovery-small Stopping /mnt/lustre-mds2 (opts:-f) on oleg409-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg409-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0001 PASS 149 (33s) == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 15:25:23 (1772310323) striped dir -i0 -c1 -H crush /mnt/lustre/d152.recovery-small striped dir -i0 -c1 -H all_char /mnt/lustre/d152.recovery-small/rr striped dir -i0 -c1 -H all_char /mnt/lustre/d152.recovery-small/qos fail_loc=0x80000173 fail_val=20 lov.lustre-MDT0000-mdtlov.qos_threshold_rr=0 lov.lustre-MDT0001-mdtlov.qos_threshold_rr=0 QoS allocation took 10 seconds lov.lustre-MDT0000-mdtlov.qos_threshold_rr=17% lov.lustre-MDT0001-mdtlov.qos_threshold_rr=17% PASS 152 (25s) == recovery-small test complete, duration 5538 sec ======= 15:25:48 (1772310348) recovery-small: FAIL: test_10d no eviction: before:1772305084 recovery-small: FAIL: test_18b /mnt/lustre/d18b.recovery-small/f18b.recovery-small: stripe_index 1 != 0