-----============= acceptance-small: recovery-small ============----- Wed Apr 23 21:02:37 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg244-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/recovery-small.*ex: No such file or directory excepting tests: 136 === recovery-small: start setup 21:02:42 (1745456562) === oleg244-client.virtnet: executing check_config_client /mnt/lustre oleg244-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg244-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-a845463a-a675-47f7-9c5e-a2bd1bce373e.idle_timeout=debug osc.lustre-OST0001-osc-a845463a-a675-47f7-9c5e-a2bd1bce373e.idle_timeout=debug disable quota as required oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all === recovery-small: finish setup 21:02:50 (1745456570) === == recovery-small test 1: create, chmod, stat: drop req, drop rep ========================================================== 21:02:50 (1745456570) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x122 fail_loc=0 PASS 1 (101s) == recovery-small test 4: open: drop req, drop rep ======= 21:04:31 (1745456671) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x122 fail_loc=0 PASS 4 (35s) == recovery-small test 5: rename: drop req, drop rep ===== 21:05:06 (1745456706) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 PASS 5 (35s) == recovery-small test 6: link, unlink: drop req, drop rep ========================================================== 21:05:41 (1745456741) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 PASS 6 (69s) == recovery-small test 8: touch: drop rep (bug 1423) ===== 21:06:50 (1745456810) fail_loc=0x119 fail_loc=0 PASS 8 (19s) == recovery-small test 9: pause bulk on OST (bug 1420) === 21:07:09 (1745456829) timeout is 0 fail_val=0 fail_loc=0x80000214 fail_loc=0 PASS 9 (9s) == recovery-small test 10a: finish request on server after client eviction (bug 1521) ========================================================== 21:07:18 (1745456838) ldlm.namespaces.lustre-MDT0000-mdc-a845463a-a675-47f7-9c5e-a2bd1bce373e.lru_max_age=3900s console_ratelimit=0 ldlm.namespaces.MGC192.168.202.144@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-a845463a-a675-47f7-9c5e-a2bd1bce373e.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-a845463a-a675-47f7-9c5e-a2bd1bce373e.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-a845463a-a675-47f7-9c5e-a2bd1bce373e.early_lock_cancel=0 fail_loc=0x305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.202.144@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-a845463a-a675-47f7-9c5e-a2bd1bce373e.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-a845463a-a675-47f7-9c5e-a2bd1bce373e.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-a845463a-a675-47f7-9c5e-a2bd1bce373e.early_lock_cancel=1 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4125: /mnt/lustre/recon: Input/output error Connected clients: oleg244-client.virtnet /mnt/lustre has perms 0777 OK [ 408.082318] Lustre: ll_ost00_001: service thread pid 8432 was inactive for 40.041 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 404.497595] Lustre: mdt00_004: service thread pid 15314 was inactive for 40.054 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 476.553554] Lustre: mdt00_004: service thread pid 15314 completed after 112.110s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). ldlm.namespaces.lustre-MDT0000-mdc-a845463a-a675-47f7-9c5e-a2bd1bce373e.lru_max_age=600000 console_ratelimit=1 PASS 10a (115s) == recovery-small test 10b: re-send BL AST =============== 21:09:13 (1745456953) ldlm.namespaces.MGC192.168.202.144@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-a845463a-a675-47f7-9c5e-a2bd1bce373e.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-a845463a-a675-47f7-9c5e-a2bd1bce373e.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-a845463a-a675-47f7-9c5e-a2bd1bce373e.early_lock_cancel=0 fail_loc=0x80000305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.202.144@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-a845463a-a675-47f7-9c5e-a2bd1bce373e.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-a845463a-a675-47f7-9c5e-a2bd1bce373e.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-a845463a-a675-47f7-9c5e-a2bd1bce373e.early_lock_cancel=1 Connected clients: oleg244-client.virtnet oleg244-client.virtnet /mnt/lustre has perms 0777 OK PASS 10b (17s) == recovery-small test 10c: re-send BL AST vs reconnect race (LU-5569) ========================================================== 21:09:30 (1745456970) ldlm.namespaces.MGC192.168.202.144@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-a845463a-a675-47f7-9c5e-a2bd1bce373e.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-a845463a-a675-47f7-9c5e-a2bd1bce373e.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-a845463a-a675-47f7-9c5e-a2bd1bce373e.early_lock_cancel=0 fail_loc=0x80000305 mdc.lustre-MDT0000-mdc-a845463a-a675-47f7-9c5e-a2bd1bce373e.import=connection=192.168.202.144@tcp fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.202.144@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-a845463a-a675-47f7-9c5e-a2bd1bce373e.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-a845463a-a675-47f7-9c5e-a2bd1bce373e.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-a845463a-a675-47f7-9c5e-a2bd1bce373e.early_lock_cancel=1 Connected clients: oleg244-client.virtnet oleg244-client.virtnet /mnt/lustre/d10c.recovery-small has perms 0777 OK PASS 10c (3s) == recovery-small test 10d: test failed blocking ast ===== 21:09:33 (1745456973) 7+0 records in 7+0 records out 7 bytes (7 B) copied, 0.000134111 s, 52.2 kB/s Stopping client oleg244-client.virtnet /mnt/lustre (opts:) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre2 1+0 records in 1+0 records out 5 bytes (5 B) copied, 0.00333718 s, 1.5 kB/s fail_err=71 ldlm.namespaces.MGC192.168.202.144@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-cac4f518-5264-4e13-95ac-eda93970e04f.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-cac4f518-5264-4e13-95ac-eda93970e04f.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-cac4f518-5264-4e13-95ac-eda93970e04f.early_lock_cancel=0 fail_loc=0x305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.202.144@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-cac4f518-5264-4e13-95ac-eda93970e04f.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-cac4f518-5264-4e13-95ac-eda93970e04f.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-cac4f518-5264-4e13-95ac-eda93970e04f.early_lock_cancel=1 uname: write error: Input/output error Connected clients: oleg244-client.virtnet fail_err=0 192.168.202.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg244-client.virtnet /mnt/lustre2 (opts:) PASS 10d (4s) == recovery-small test 10e: re-send BL AST vs reconnect race 2 ========================================================== 21:09:37 (1745456977) SKIP: recovery-small test_10e need two clients SKIP 10e (0s) == recovery-small test 11: wake up a thread waiting for completion after eviction (b=2460) ========================================================== 21:09:38 (1745456978) ldlm.namespaces.MGC192.168.202.144@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=0 fail_loc=0x80000305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.202.144@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=1 PASS 11 (19s) == recovery-small test 12: recover from timed out resend in ptlrpcd (b=2494) ========================================================== 21:09:57 (1745456997) fail_loc=0x115 multiop /mnt/lustre/f12.recovery-small vOS_c TMPPIPE=/tmp/multiop_open_wait_pipe.7026 waiting for multiop 18047 clearing fail_loc on mds1 fail_loc=0 PASS 12 (40s) == recovery-small test 13: mdc_readpage restart test (bug 1138) ========================================================== 21:10:38 (1745457038) fail_loc=0x80000104 newentry fail_loc=0 PASS 13 (18s) == recovery-small test 14: mdc_readpage resend test (bug 1138) ========================================================== 21:10:56 (1745457056) fail_loc=0x80000106 newentry fail_loc=0 PASS 14 (2s) == recovery-small test 15: failed open (-ENOMEM) ========= 21:10:58 (1745457058) fail_loc=0x80000128 touch: cannot touch '/mnt/lustre/f15.recovery-small': Cannot allocate memory PASS 15 (1s) == recovery-small test 16: timeout bulk put, don't evict client (2732) ========================================================== 21:10:59 (1745457059) fail_loc=0x80000504 fail_loc=0 PASS 16 (39s) == recovery-small test 17a: timeout bulk get, don't evict client (2732) ========================================================== 21:11:38 (1745457098) at_max=20 fail_loc=0x80000503 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.144@tcp:/lustre 7542784 6144 7532544 1% /mnt/lustre at_max=600 PASS 17a (43s) == recovery-small test 17b: timeout bulk get, dont evict client (3582) ========================================================== 21:12:21 (1745457141) SKIP: recovery-small test_17b Needs multiple clients SKIP 17b (1s) == recovery-small test 18a: manual ost invalidate clears page cache immediately ========================================================== 21:12:22 (1745457142) PASS 18a (1s) == recovery-small test 18b: eviction and reconnect clears page cache (2766) ========================================================== 21:12:23 (1745457143) PASS 18b (24s) == recovery-small test 18c: Dropped connect reply after eviction handing (14755) ========================================================== 21:12:47 (1745457167) fail_loc=0x80000225 PASS 18c (15s) == recovery-small test 19a: test expired_lock_main on mds (2867) ========================================================== 21:13:02 (1745457182) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre2 fail_loc=0x304 fail_loc=0 192.168.202.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg244-client.virtnet /mnt/lustre2 (opts:) Connected clients: oleg244-client.virtnet oleg244-client.virtnet PASS 19a (103s) == recovery-small test 19b: test expired_lock_main on ost (2867) ========================================================== 21:14:45 (1745457285) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre2 fail_loc=0x304 fail_loc=0 192.168.202.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg244-client.virtnet /mnt/lustre2 (opts:) Connected clients: oleg244-client.virtnet oleg244-client.virtnet PASS 19b (103s) == recovery-small test 19c: check reconnect and lock resend do not trigger expired_lock_main ========================================================== 21:16:28 (1745457388) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre2 ldlm.namespaces.MGC192.168.202.144@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-feb075de-2f85-45e3-bc2c-be3d85efa682.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-feb075de-2f85-45e3-bc2c-be3d85efa682.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-feb075de-2f85-45e3-bc2c-be3d85efa682.early_lock_cancel=0 File: '/mnt/lustre/f19c.recovery-small' Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205322833924 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-23 21:16:28.000000000 -0400 Modify: 2025-04-23 21:16:28.000000000 -0400 Change: 2025-04-23 21:16:28.000000000 -0400 Birth: - fail_loc=0x80000516 ldlm.namespaces.MGC192.168.202.144@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-feb075de-2f85-45e3-bc2c-be3d85efa682.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-feb075de-2f85-45e3-bc2c-be3d85efa682.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-feb075de-2f85-45e3-bc2c-be3d85efa682.early_lock_cancel=1 192.168.202.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg244-client.virtnet /mnt/lustre2 (opts:) PASS 19c (8s) == recovery-small test 20a: ldlm_handle_enqueue error (should return error) ========================================================== 21:16:36 (1745457396) multiop /mnt/lustre/d20a.recovery-small/f20a.recovery-small vO_wc TMPPIPE=/tmp/multiop_open_wait_pipe.7026 fail_loc=0x80000308 write: Cannot allocate memory PASS 20a (2s) == recovery-small test 20b: ldlm_handle_enqueue error (should return error) ========================================================== 21:16:38 (1745457398) fail_loc=0x80000308 dd: failed to open '/mnt/lustre/d20b.recovery-small/f20b.recovery-small': Cannot allocate memory PASS 20b (2s) == recovery-small test 21a: drop close request while close and open are both in flight ========================================================== 21:16:40 (1745457400) multiop /mnt/lustre/d21a.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7026 fail_loc=0x80000129 fail_loc=0 fail_loc=0x80000115 fail_loc=0 /mnt/lustre/d21a.recovery-small-1/f has type file OK /mnt/lustre/d21a.recovery-small-2/f has type file OK PASS 21a (19s) == recovery-small test 21b: drop open request while close and open are both in flight ========================================================== 21:16:59 (1745457419) multiop /mnt/lustre/d21b.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7026 fail_loc=0x80000107 fail_loc=0 /mnt/lustre/d21b.recovery-small-1/f has type file OK /mnt/lustre/d21b.recovery-small-2/f has type file OK PASS 21b (182s) == recovery-small test 21c: drop both request while close and open are both in flight ========================================================== 21:20:01 (1745457601) multiop /mnt/lustre/d21c.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7026 fail_loc=0x80000107 fail_loc=0 fail_loc=0x80000115 fail_loc=0 /mnt/lustre/d21c.recovery-small-1/f has type file OK /mnt/lustre/d21c.recovery-small-2/f has type file OK PASS 21c (21s) == recovery-small test 21d: drop close reply while close and open are both in flight ========================================================== 21:20:22 (1745457622) multiop /mnt/lustre/d21d.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7026 fail_loc=0x80000129 fail_loc=0 fail_loc=0x80000122 fail_loc=0 /mnt/lustre/d21d.recovery-small-1/f has type file OK /mnt/lustre/d21d.recovery-small-2/f has type file OK PASS 21d (20s) == recovery-small test 21e: drop open reply while close and open are both in flight ========================================================== 21:20:42 (1745457642) multiop /mnt/lustre/d21e.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7026 fail_loc=0x80000119 fail_loc=0 /mnt/lustre/d21e.recovery-small-1/f has type file OK /mnt/lustre/d21e.recovery-small-2/f has type file OK PASS 21e (172s) == recovery-small test 21f: drop both reply while close and open are both in flight ========================================================== 21:23:34 (1745457814) multiop /mnt/lustre/d21f.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7026 fail_loc=0x80000119 fail_loc=0 fail_loc=0x80000122 fail_loc=0 /mnt/lustre/d21f.recovery-small-1/f has type file OK /mnt/lustre/d21f.recovery-small-2/f has type file OK PASS 21f (20s) == recovery-small test 21g: drop open reply and close request while close and open are both in flight ========================================================== 21:23:54 (1745457834) multiop /mnt/lustre/d21g.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7026 fail_loc=0x80000119 fail_loc=0 fail_loc=0x80000115 fail_loc=0 /mnt/lustre/d21g.recovery-small-1/f has type file OK /mnt/lustre/d21g.recovery-small-2/f has type file OK PASS 21g (19s) == recovery-small test 21h: drop open request and close reply while close and open are both in flight ========================================================== 21:24:13 (1745457853) multiop /mnt/lustre/d21h.recovery-small-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7026 fail_loc=0x80000107 fail_loc=0 fail_loc=0x80000122 fail_loc=0 /mnt/lustre/d21h.recovery-small-1/f has type file OK /mnt/lustre/d21h.recovery-small-2/f has type file OK PASS 21h (20s) == recovery-small test 22: drop close request and do mknod ========================================================== 21:24:33 (1745457873) fail_loc=0x80000115 fail_loc=0 PASS 22 (18s) == recovery-small test 23: client hang when close a file after mds crash ========================================================== 21:24:51 (1745457891) multiop /mnt/lustre/f23.recovery-small vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7026 fail_val=0 fail_loc=0x123 fail_loc=0 Failing mds1 on oleg244-server Stopping /mnt/lustre-mds1 (opts:) on oleg244-server 21:24:58 (1745457898) shut down facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover mds1 to oleg244-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-MDT0000 21:25:10 (1745457910) targets are mounted 21:25:10 (1745457910) facet_failover done oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23 (24s) == recovery-small test 24a: fsync error (should return error) ========================================================== 21:25:15 (1745457915) multiop /mnt/lustre/d24a.recovery-small/f24a.recovery-small vOwy_wyc TMPPIPE=/tmp/multiop_open_wait_pipe.7026 fsync: Input/output error fail_loc=0x0 Connected clients: oleg244-client.virtnet oleg244-client.virtnet PASS 24a (2s) == recovery-small test 24b: test dirty page discard due to client eviction ========================================================== 21:25:17 (1745457917) multiop /mnt/lustre/d24b.recovery-small/f24b.recovery-small-1 vOw8192_yc TMPPIPE=/tmp/multiop_open_wait_pipe.7026 multiop /mnt/lustre/d24b.recovery-small/f24b.recovery-small-2 vOw8192_c TMPPIPE=/tmp/multiop_open_wait_pipe.7026 fsync: Input/output error close: Input/output error fail_loc=0x0 Connected clients: oleg244-client.virtnet oleg244-client.virtnet [ 1447.155859] Lustre: 1960:0:(llite_lib.c:4114:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.144@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108) PASS 24b (2s) == recovery-small test 26a: evict dead exports =========== 21:25:19 (1745457919) SKIP: recovery-small test_26a msg and ost1 are at the same node SKIP 26a (0s) == recovery-small test 26b: evict dead exports =========== 21:25:19 (1745457919) SKIP: recovery-small test_26b msg and ost1 are at the same node SKIP 26b (1s) == recovery-small test 27: fail LOV while using OSC's ==== 21:25:20 (1745457920) Failing mds1 on oleg244-server Stopping /mnt/lustre-mds1 (opts:) on oleg244-server 21:25:23 (1745457923) shut down facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover mds1 to oleg244-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-MDT0000 21:25:35 (1745457935) targets are mounted 21:25:35 (1745457935) facet_failover done fail_loc=0x80000407 waiting for fail_loc Waiting 90s for '-2147482617' Waiting 80s for '-2147482617' Waiting 70s for '-2147482617' Waiting 60s for '-2147482617' Waiting 50s for '-2147482617' Waiting 40s for '-2147482617' Waiting 30s for '-2147482617' Waiting 20s for '-2147482617' Waiting 10s for '-2147482617' Waiting 0s for '-2147482617' Update not seen after 90s: want '-2147482617' got '2147484679' Failing mds1 on oleg244-server Stopping /mnt/lustre-mds1 (opts:) on oleg244-server 21:27:07 (1745458027) shut down facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover mds1 to oleg244-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-MDT0000 21:27:20 (1745458040) targets are mounted 21:27:20 (1745458040) facet_failover done PASS 27 (124s) == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 21:27:24 (1745458044) ldlm.namespaces.MGC192.168.202.144@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=0 fail_loc=0x80000305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.202.144@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-563dc96c-f6a4-4727-b192-f8266cf689b8.early_lock_cancel=1 fail_loc=0x8000012f Failing mds1 on oleg244-server Stopping /mnt/lustre-mds1 (opts:) on oleg244-server 21:27:43 (1745458063) shut down facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover mds1 to oleg244-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-MDT0000 21:27:56 (1745458076) targets are mounted 21:27:56 (1745458076) facet_failover done oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 28 (36s) == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 21:28:00 (1745458080) fail_loc=0x80000711 Stopping /mnt/lustre-mds1 (opts:) on oleg244-server Failover mds1 to oleg244-server oleg244-server.virtnet Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-MDT0000 pdsh@oleg244-client: oleg244-client: ssh exited with exit code 5 first stat failed: 5 oleg244-server: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg244-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec PASS 29a (10s) == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 21:28:10 (1745458090) fail_loc=0x80000711 Stopping /mnt/lustre-ost1 (opts:) on oleg244-server Failover ost1 to oleg244-server oleg244-server.virtnet Starting ost1: -o localrecov -o abort_recovery lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0000 pdsh@oleg244-client: oleg244-client: ssh exited with exit code 5 first stat failed: 5 PASS 29b (11s) == recovery-small test 50: failover MDS under load ======= 21:28:21 (1745458101) writemany pid 11748 Failing mds1 on oleg244-server Stopping /mnt/lustre-mds1 (opts:) on oleg244-server 21:28:33 (1745458113) shut down facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover mds1 to oleg244-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-MDT0000 21:28:45 (1745458125) targets are mounted 21:28:45 (1745458125) facet_failover done oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds1 on oleg244-server Stopping /mnt/lustre-mds1 (opts:) on oleg244-server 21:29:52 (1745458192) shut down facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover mds1 to oleg244-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-MDT0000 21:30:04 (1745458204) targets are mounted 21:30:04 (1745458204) facet_failover done oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds1 on oleg244-server Stopping /mnt/lustre-mds1 (opts:) on oleg244-server 21:31:12 (1745458272) shut down facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover mds1 to oleg244-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-MDT0000 21:31:24 (1745458284) targets are mounted 21:31:24 (1745458284) facet_failover done oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec writemany returned 0 PASS 50 (211s) == recovery-small test 51: failover MDS during recovery == 21:31:52 (1745458312) fail_loc=0x00001310 Failing mds1 on oleg244-server Stopping /mnt/lustre-mds1 (opts:) on oleg244-server 21:31:55 (1745458315) shut down facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover mds1 to oleg244-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-MDT0000 21:32:07 (1745458327) targets are mounted 21:32:07 (1745458327) facet_failover done will failover at 1 5 10 20 25 30 test_51: failover in 1 sec Failing mds1 on oleg244-server Stopping /mnt/lustre-mds1 (opts:) on oleg244-server 21:32:10 (1745458330) shut down facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover mds1 to oleg244-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-MDT0000 21:32:22 (1745458342) targets are mounted 21:32:22 (1745458342) facet_failover done test_51: failover in 5 sec Failing mds1 on oleg244-server Stopping /mnt/lustre-mds1 (opts:) on oleg244-server 21:32:29 (1745458349) shut down facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover mds1 to oleg244-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-MDT0000 21:32:41 (1745458361) targets are mounted 21:32:41 (1745458361) facet_failover done test_51: failover in 10 sec Failing mds1 on oleg244-server Stopping /mnt/lustre-mds1 (opts:) on oleg244-server 21:32:53 (1745458373) shut down facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover mds1 to oleg244-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-MDT0000 21:33:05 (1745458385) targets are mounted 21:33:05 (1745458385) facet_failover done test_51: failover in 20 sec Failing mds1 on oleg244-server Stopping /mnt/lustre-mds1 (opts:) on oleg244-server 21:33:27 (1745458407) shut down facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover mds1 to oleg244-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-MDT0000 21:33:40 (1745458420) targets are mounted 21:33:40 (1745458420) facet_failover done test_51: failover in 25 sec Failing mds1 on oleg244-server Stopping /mnt/lustre-mds1 (opts:) on oleg244-server 21:34:06 (1745458446) shut down facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover mds1 to oleg244-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-MDT0000 21:34:19 (1745458459) targets are mounted 21:34:19 (1745458459) facet_failover done test_51: failover in 30 sec Failing mds1 on oleg244-server Stopping /mnt/lustre-mds1 (opts:) on oleg244-server 21:34:50 (1745458490) shut down facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover mds1 to oleg244-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-MDT0000 21:35:04 (1745458504) targets are mounted 21:35:04 (1745458504) facet_failover done writemany returned 0 PASS 51 (213s) == recovery-small test 52: failover OST under load ======= 21:35:25 (1745458525) writemany pid 18405 Failing ost1 on oleg244-server Stopping /mnt/lustre-ost1 (opts:) on oleg244-server 21:35:38 (1745458538) shut down facet: ost1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover ost1 to oleg244-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0000 21:35:51 (1745458551) targets are mounted 21:35:51 (1745458551) facet_failover done oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec writemany succeeded writemany pid 19478 Failing ost1 on oleg244-server Stopping /mnt/lustre-ost1 (opts:) on oleg244-server 21:41:07 (1745458867) shut down facet: ost1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover ost1 to oleg244-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0000 21:41:20 (1745458880) targets are mounted 21:41:20 (1745458880) facet_failover done oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec writemany succeeded writemany pid 20556 Failing ost1 on oleg244-server Stopping /mnt/lustre-ost1 (opts:) on oleg244-server 21:46:38 (1745459198) shut down facet: ost1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover ost1 to oleg244-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0000 21:46:50 (1745459210) targets are mounted 21:46:50 (1745459210) facet_failover done oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec writemany succeeded Connected clients: oleg244-client.virtnet oleg244-client.virtnet PASS 52 (964s) == recovery-small test 53a: touch: drop rep ============== 21:51:29 (1745459489) fail_loc=0x157 Succeed in opening file "/mnt/lustre/f53a.recovery-small"(flags=O_RDWR, mode=755) fail_loc=0 PASS 53a (19s) == recovery-small test 53b: touch: drop rep ============== 21:51:48 (1745459508) fail_loc=0x157 Succeed in opening file "/mnt/lustre/f53b.recovery-small"(flags=O_RDWR, mode=755) fail_loc=0 PASS 53b (18s) == recovery-small test 53c: touch: drop rep ============== 21:52:06 (1745459526) fail_loc=0x157 Succeed in opening file "/mnt/lustre/f53c.recovery-small"(flags=O_RDWR, mode=755) fail_loc=0 PASS 53c (18s) == recovery-small test 54: back in time ================== 21:52:24 (1745459544) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre2 cat: /mnt/lustre2/f54.recovery-small.missing: No such file or directory Failing mds1 on oleg244-server Stopping /mnt/lustre-mds1 (opts:) on oleg244-server 21:52:36 (1745459556) shut down facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover mds1 to oleg244-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-MDT0000 21:52:49 (1745459569) targets are mounted 21:52:49 (1745459569) facet_failover done oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 54 (30s) == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 21:52:54 (1745459574) step1: testing ...... 4+0 records in 4+0 records out 134217728 bytes (134 MB) copied, 2.85365 s, 47.0 MB/s (dd_pid=25979, time=3)successful fail_loc=0x0000021d step2: testing ...... (dd_pid=25995, time=257)successful fail_loc=0 step3: testing ...... 4+0 records in 4+0 records out 134217728 bytes (134 MB) copied, 275.141 s, 488 kB/s (dd_pid=25995, time=17)successful osc.lustre-OST0000-osc-563dc96c-f6a4-4727-b192-f8266cf689b8.max_dirty_mb=467 osc.lustre-OST0001-osc-563dc96c-f6a4-4727-b192-f8266cf689b8.max_dirty_mb=467 PASS 55 (282s) == recovery-small test 56: do not fail on getattr resend ========================================================== 21:57:36 (1745459856) fail_loc=0x80000136 File: '/mnt/lustre/f56.recovery-small' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115708589020857 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-23 21:57:37.000000000 -0400 Modify: 2025-04-23 21:57:37.000000000 -0400 Change: 2025-04-23 21:57:37.000000000 -0400 Birth: - fail_loc=0 PASS 56 (42s) == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 21:58:19 (1745459899) fail_loc=0x80000B00 Stopping client oleg244-client.virtnet /mnt/lustre (opts:) fail_loc=0x80000B00 Stopping /mnt/lustre-mds1 (opts:) on oleg244-server Failover mds1 to oleg244-server oleg244-server.virtnet Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-MDT0000 pdsh@oleg244-client: oleg244-client: ssh exited with exit code 95 pdsh@oleg244-client: oleg244-client: ssh exited with exit code 95 /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 1445: kill: (27795) - No such process fail_loc=0 Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.144@tcp:/lustre 7542784 8192 7530496 1% /mnt/lustre PASS 57 (11s) == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 21:58:30 (1745459910) -rw-r--r-- 1 root root 0 Apr 23 21:58 /mnt/lustre/f58.recovery-small fail_loc=0x80000801 fail_loc=0 ldlm.namespaces.MGC192.168.202.144@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ebe0154c-5485-4796-8239-30ad5f6dbb3a.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ebe0154c-5485-4796-8239-30ad5f6dbb3a.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ebe0154c-5485-4796-8239-30ad5f6dbb3a.early_lock_cancel=0 fail_loc=0x80000305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.202.144@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ebe0154c-5485-4796-8239-30ad5f6dbb3a.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ebe0154c-5485-4796-8239-30ad5f6dbb3a.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ebe0154c-5485-4796-8239-30ad5f6dbb3a.early_lock_cancel=1 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.144@tcp:/lustre 7542784 8192 7530496 1% /mnt/lustre PASS 58 (19s) == recovery-small test 59: Read cancel race on client eviction ========================================================== 21:58:49 (1745459929) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre2 fail_loc=0x311 fail_loc=0 Stopping client oleg244-client.virtnet /mnt/lustre2 (opts:-f) PASS 59 (13s) == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 21:59:02 (1745459942) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl1' - open/close 2443 (time 1745459954.55 total 10.00 last 244.26) total: 5000 open/close in 19.52 seconds: 256.09 ops/second Failing mds1 on oleg244-server Stopping /mnt/lustre-mds1 (opts:) on oleg244-server 21:59:32 (1745459972) shut down facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover mds1 to oleg244-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-MDT0000 21:59:45 (1745459985) targets are mounted 21:59:45 (1745459985) facet_failover done - unlinked 0 (time 1745459970 ; total 0 ; last 0) total: 5000 unlinks in 55 seconds: 90.909088 unlinks/second 5000 unlinks in changelog lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 lustre-MDT0000: changelog user 'cl1' not found PASS 60 (88s) == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 22:00:30 (1745460030) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 5504 2203008 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 4096 3765248 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 4096 3765248 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 8192 7530496 1% /mnt/lustre total: 10 open/close in 0.11 seconds: 92.83 ops/second Stopping /mnt/lustre-mds1 (opts:) on oleg244-server Failover mds1 to oleg244-server oleg244-server.virtnet Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 61 (13s) == recovery-small test 65: lock enqueue for destroyed export ========================================================== 22:00:43 (1745460043) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre2 fail_loc=0x31e write: Input/output error fail_loc=0 192.168.202.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg244-client.virtnet /mnt/lustre2 (opts:) PASS 65 (18s) == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 22:01:01 (1745460061) fail_loc=0x80000157 fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ebe0154c-5485-4796-8239-30ad5f6dbb3a.import=connection=192.168.202.144@tcp /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4125: /mnt/lustre/recon: Input/output error stat: cannot stat '/mnt/lustre/f66.recovery-small': Input/output error Connected clients: oleg244-client.virtnet fail_loc=0 PASS 66 (7s) == recovery-small test 67: connect vs import invalidate race ========================================================== 22:01:08 (1745460068) fail_loc=0x80000531 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4125: /mnt/lustre/recon: Input/output error mdc.lustre-MDT0000-mdc-ebe0154c-5485-4796-8239-30ad5f6dbb3a.import=connection=192.168.202.144@tcp error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access /mnt/lustre/recon: Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4125: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access /mnt/lustre/recon: Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4125: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access /mnt/lustre/recon: Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4125: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access /mnt/lustre/recon: Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4125: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access /mnt/lustre/recon: Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4125: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access /mnt/lustre/recon: Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4125: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access /mnt/lustre/recon: Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4125: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access /mnt/lustre/recon: Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4125: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access /mnt/lustre/recon: Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4125: /mnt/lustre/recon: Cannot send after transport endpoint shutdown error: invalid path '/mnt/lustre': Cannot send after transport endpoint shutdown bash: /mnt/lustre/recon: Cannot send after transport endpoint shutdown cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access /mnt/lustre/recon: Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/recon': Cannot send after transport endpoint shutdown Connected clients: oleg244-client.virtnet oleg244-client.virtnet PASS 67 (16s) == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 22:01:24 (1745460084) mgs.MGS.ir_timeout Failing ost1 on oleg244-server Stopping /mnt/lustre-ost1 (opts:) on oleg244-server 22:01:27 (1745460087) shut down facet: ost1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover ost1 to oleg244-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0000 22:01:42 (1745460102) targets are mounted 22:01:42 (1745460102) facet_failover done oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 100 (25s) == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 22:01:49 (1745460109) mgs.MGS.ir_timeout Failing ost1 on oleg244-server Stopping /mnt/lustre-ost1 (opts:) on oleg244-server 22:01:52 (1745460112) shut down facet: ost1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover ost1 to oleg244-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0000 22:02:05 (1745460125) targets are mounted 22:02:05 (1745460125) facet_failover done