-----============= acceptance-small: recovery-small ============----- Thu Jan 22 04:14:09 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg406-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/recovery-small.*ex': No such file or directory excepting tests: 136 === recovery-small: start setup 04:14:26 (1769073266) === oleg406-client.virtnet: executing check_config_client /mnt/lustre oleg406-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg406-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9413d92db800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9413d92db800.idle_timeout=debug disable quota as required oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all === recovery-small: finish setup 04:14:56 (1769073296) === == recovery-small test 1: create, chmod, stat: drop req, drop rep ========================================================== 04:14:58 (1769073298) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x122 fail_loc=0 PASS 1 (130s) == recovery-small test 4: open: drop req, drop rep ======= 04:17:08 (1769073428) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x122 fail_loc=0 PASS 4 (48s) == recovery-small test 5: rename: drop req, drop rep ===== 04:17:56 (1769073476) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 PASS 5 (47s) == recovery-small test 6: link, unlink: drop req, drop rep ========================================================== 04:18:43 (1769073523) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 PASS 6 (86s) == recovery-small test 8: touch: drop rep (bug 1423) ===== 04:20:09 (1769073609) fail_loc=0x119 fail_loc=0 PASS 8 (29s) == recovery-small test 9: pause bulk on OST (bug 1420) === 04:20:38 (1769073638) timeout is 0 fail_val=0 fail_loc=0x80000214 fail_loc=0 PASS 9 (18s) == recovery-small test 10a: finish request on server after client eviction (bug 1521) ========================================================== 04:20:56 (1769073656) ldlm.namespaces.lustre-MDT0000-mdc-ffff9413d92db800.lru_max_age=3900s console_ratelimit=0 ldlm.namespaces.MGC192.168.204.106@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9413d92db800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9413d92db800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9413d92db800.early_lock_cancel=0 fail_loc=0x305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.204.106@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9413d92db800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9413d92db800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9413d92db800.early_lock_cancel=1 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4167: /mnt/lustre/recon: Input/output error Connected clients: oleg406-client.virtnet /mnt/lustre has perms 0777 OK [ 689.123356] Lustre: mdt00_003: service thread pid 6096 was inactive for 43.661 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 757.409995] Lustre: mdt00_003: service thread pid 6096 completed after 111.948s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). ldlm.namespaces.lustre-MDT0000-mdc-ffff9413d92db800.lru_max_age=600000 console_ratelimit=1 PASS 10a (126s) == recovery-small test 10b: re-send BL AST =============== 04:23:03 (1769073783) ldlm.namespaces.MGC192.168.204.106@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9413d92db800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9413d92db800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9413d92db800.early_lock_cancel=0 fail_loc=0x80000305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.204.106@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9413d92db800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9413d92db800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9413d92db800.early_lock_cancel=1 Connected clients: oleg406-client.virtnet oleg406-client.virtnet /mnt/lustre has perms 0777 OK PASS 10b (29s) == recovery-small test 10c: re-send BL AST vs reconnect race (LU-5569) ========================================================== 04:23:31 (1769073811) ldlm.namespaces.MGC192.168.204.106@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9413d92db800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9413d92db800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9413d92db800.early_lock_cancel=0 fail_loc=0x80000305 mdc.lustre-MDT0000-mdc-ffff9413d92db800.import=connection=192.168.204.106@tcp fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.204.106@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9413d92db800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9413d92db800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9413d92db800.early_lock_cancel=1 Connected clients: oleg406-client.virtnet oleg406-client.virtnet /mnt/lustre/d10c.recovery-small has perms 0777 OK PASS 10c (11s) == recovery-small test 10d: test failed blocking ast ===== 04:23:42 (1769073822) 7+0 records in 7+0 records out 7 bytes copied, 0.00113606 s, 6.2 kB/s Stopping client oleg406-client.virtnet /mnt/lustre (opts:) Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre2 1+0 records in 1+0 records out 5 bytes copied, 0.0123486 s, 0.4 kB/s fail_err=71 ldlm.namespaces.MGC192.168.204.106@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9413d12f5000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9413d92d8000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9413d12f5000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9413d92d8000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9413d12f5000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9413d92d8000.early_lock_cancel=0 fail_loc=0x305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.204.106@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9413d12f5000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9413d92d8000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9413d12f5000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9413d92d8000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9413d12f5000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9413d92d8000.early_lock_cancel=1 uname: write error: Input/output error Connected clients: oleg406-client.virtnet fail_err=0 192.168.204.106@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg406-client.virtnet /mnt/lustre2 (opts:) PASS 10d (21s) == recovery-small test 10e: re-send BL AST vs reconnect race 2 ========================================================== 04:24:03 (1769073843) SKIP: recovery-small test_10e need two clients SKIP 10e (6s) == recovery-small test 11: wake up a thread waiting for completion after eviction (b=2460) ========================================================== 04:24:09 (1769073849) ldlm.namespaces.MGC192.168.204.106@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9413d12f5000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9413d12f5000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9413d12f5000.early_lock_cancel=0 fail_loc=0x80000305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.204.106@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9413d12f5000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9413d12f5000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9413d12f5000.early_lock_cancel=1 PASS 11 (29s) == recovery-small test 12: recover from timed out resend in ptlrpcd (b=2494) ========================================================== 04:24:38 (1769073878) fail_loc=0x115 multiop /mnt/lustre/f12.recovery-small vOS_c TMPPIPE=/tmp/multiop_open_wait_pipe.7584 waiting for multiop 17611 clearing fail_loc on mds1 fail_loc=0 PASS 12 (46s) == recovery-small test 13: mdc_readpage restart test (bug 1138) ========================================================== 04:25:24 (1769073924) fail_loc=0x80000104 newentry fail_loc=0 PASS 13 (28s) == recovery-small test 14: mdc_readpage resend test (bug 1138) ========================================================== 04:25:52 (1769073952) fail_loc=0x80000106 newentry fail_loc=0 PASS 14 (11s) == recovery-small test 15: failed open (-ENOMEM) ========= 04:26:03 (1769073963) fail_loc=0x80000128 touch: cannot touch '/mnt/lustre/f15.recovery-small': Cannot allocate memory PASS 15 (9s) == recovery-small test 16: timeout bulk put, don't evict client (2732) ========================================================== 04:26:13 (1769073973) fail_loc=0x80000504 fail_loc=0 PASS 16 (50s) == recovery-small test 17a: timeout bulk get, don't evict client (2732) ========================================================== 04:27:03 (1769074023) at_max=20 fail_loc=0x80000503 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.106@tcp:/lustre 7542784 6144 7532544 1% /mnt/lustre at_max=600 PASS 17a (58s) == recovery-small test 17b: timeout bulk get, dont evict client (3582) ========================================================== 04:28:01 (1769074081) SKIP: recovery-small test_17b Needs multiple clients SKIP 17b (4s) == recovery-small test 18a: manual ost invalidate clears page cache immediately ========================================================== 04:28:05 (1769074085) PASS 18a (10s) == recovery-small test 18b: eviction and reconnect clears page cache (2766) ========================================================== 04:28:15 (1769074095) PASS 18b (33s) == recovery-small test 18c: Dropped connect reply after eviction handing (14755) ========================================================== 04:28:49 (1769074129) fail_loc=0x80000225 PASS 18c (27s) == recovery-small test 19a: test expired_lock_main on mds (2867) ========================================================== 04:29:16 (1769074156) Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre2 fail_loc=0x304 fail_loc=0 192.168.204.106@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg406-client.virtnet /mnt/lustre2 (opts:) Connected clients: oleg406-client.virtnet oleg406-client.virtnet PASS 19a (120s) == recovery-small test 19b: test expired_lock_main on ost (2867) ========================================================== 04:31:16 (1769074276) Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre2 fail_loc=0x304 fail_loc=0 192.168.204.106@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg406-client.virtnet /mnt/lustre2 (opts:) Connected clients: oleg406-client.virtnet oleg406-client.virtnet PASS 19b (118s) == recovery-small test 19c: check reconnect and lock resend do not trigger expired_lock_main ========================================================== 04:33:14 (1769074394) Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre2 ldlm.namespaces.MGC192.168.204.106@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9413c34d7000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9413d12f5000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9413c34d7000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9413d12f5000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9413c34d7000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9413d12f5000.early_lock_cancel=0 File: /mnt/lustre/f19c.recovery-small Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205322833924 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-22 04:33:17.000000000 -0500 Modify: 2026-01-22 04:33:17.000000000 -0500 Change: 2026-01-22 04:33:17.000000000 -0500 Birth: 2026-01-22 04:33:17.000000000 -0500 fail_loc=0x80000516 ldlm.namespaces.MGC192.168.204.106@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9413c34d7000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9413d12f5000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9413c34d7000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9413d12f5000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9413c34d7000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9413d12f5000.early_lock_cancel=1 192.168.204.106@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg406-client.virtnet /mnt/lustre2 (opts:) PASS 19c (21s) == recovery-small test 20a: ldlm_handle_enqueue error (should return error) ========================================================== 04:33:35 (1769074415) multiop /mnt/lustre/d20a.recovery-small/f20a.recovery-small vO_wc TMPPIPE=/tmp/multiop_open_wait_pipe.7584 fail_loc=0x80000308 write: Cannot allocate memory PASS 20a (11s) == recovery-small test 20b: ldlm_handle_enqueue error (should return error) ========================================================== 04:33:46 (1769074426) fail_loc=0x80000308 dd: failed to open '/mnt/lustre/d20b.recovery-small/f20b.recovery-small': Cannot allocate memory PASS 20b (10s) == recovery-small test 21a: drop close request while close and open are both in flight ========================================================== 04:33:57 (1769074437) multiop /mnt/lustre/d21a.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7584 fail_loc=0x80000129 fail_loc=0 fail_loc=0x80000115 fail_loc=0 /mnt/lustre/d21a.recovery-small-1/f has type file OK /mnt/lustre/d21a.recovery-small-2/f has type file OK PASS 21a (31s) == recovery-small test 21b: drop open request while close and open are both in flight ========================================================== 04:34:28 (1769074468) multiop /mnt/lustre/d21b.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7584 fail_loc=0x80000107 fail_loc=0 /mnt/lustre/d21b.recovery-small-1/f has type file OK /mnt/lustre/d21b.recovery-small-2/f has type file OK PASS 21b (148s) == recovery-small test 21c: drop both request while close and open are both in flight ========================================================== 04:36:56 (1769074616) multiop /mnt/lustre/d21c.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7584 fail_loc=0x80000107 fail_loc=0 fail_loc=0x80000115 fail_loc=0 /mnt/lustre/d21c.recovery-small-1/f has type file OK /mnt/lustre/d21c.recovery-small-2/f has type file OK PASS 21c (34s) == recovery-small test 21d: drop close reply while close and open are both in flight ========================================================== 04:37:30 (1769074650) multiop /mnt/lustre/d21d.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7584 fail_loc=0x80000129 fail_loc=0 fail_loc=0x80000122 fail_loc=0 /mnt/lustre/d21d.recovery-small-1/f has type file OK /mnt/lustre/d21d.recovery-small-2/f has type file OK PASS 21d (34s) == recovery-small test 21e: drop open reply while close and open are both in flight ========================================================== 04:38:04 (1769074684) multiop /mnt/lustre/d21e.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7584 fail_loc=0x80000119 fail_loc=0 /mnt/lustre/d21e.recovery-small-1/f has type file OK /mnt/lustre/d21e.recovery-small-2/f has type file OK PASS 21e (141s) == recovery-small test 21f: drop both reply while close and open are both in flight ========================================================== 04:40:25 (1769074825) multiop /mnt/lustre/d21f.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7584 fail_loc=0x80000119 fail_loc=0 fail_loc=0x80000122 fail_loc=0 /mnt/lustre/d21f.recovery-small-1/f has type file OK /mnt/lustre/d21f.recovery-small-2/f has type file OK PASS 21f (32s) == recovery-small test 21g: drop open reply and close request while close and open are both in flight ========================================================== 04:40:57 (1769074857) multiop /mnt/lustre/d21g.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7584 fail_loc=0x80000119 fail_loc=0 fail_loc=0x80000115 fail_loc=0 /mnt/lustre/d21g.recovery-small-1/f has type file OK /mnt/lustre/d21g.recovery-small-2/f has type file OK PASS 21g (28s) == recovery-small test 21h: drop open request and close reply while close and open are both in flight ========================================================== 04:41:26 (1769074886) multiop /mnt/lustre/d21h.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7584 fail_loc=0x80000107 fail_loc=0 fail_loc=0x80000122 fail_loc=0 /mnt/lustre/d21h.recovery-small-1/f has type file OK /mnt/lustre/d21h.recovery-small-2/f has type file OK PASS 21h (28s) == recovery-small test 22: drop close request and do mknod ========================================================== 04:41:53 (1769074913) fail_loc=0x80000115 fail_loc=0 PASS 22 (27s) == recovery-small test 23: client hang when close a file after mds crash ========================================================== 04:42:20 (1769074940) multiop /mnt/lustre/f23.recovery-small vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7584 fail_val=0 fail_loc=0x123 fail_loc=0 Failing mds1 on oleg406-server Stopping /mnt/lustre-mds1 (opts:) on oleg406-server 04:42:34 (1769074954) shut down facet: mds1 facet_host: oleg406-server facet_failover_host: oleg406-server Failover mds1 to oleg406-server mount facets: mds1