== sanity-lfsck test 30: LFSCK can recover the orphans from backend /lost+found ========================================================== 02:54:11 (1785394451) ##### The namespace LFSCK will move the orphans from backend /lost+found directory to normal client visible namespace or to global visible ./lustre/lost+found/MDTxxxx/ directory ##### Inject failure stub on MDT0 to simulate the case that directory d0 has no linkEA entry, then the LFSCK will move it into .lustre/lost+found/MDTxxxx/ later. fail_loc=0x161d fail_loc=0 Inject failure stub on MDT0 to simulate the case that the object's name entry will be removed, but not destroy the object. Then backend e2fsck will handle it as orphan and add them into the backend /lost+found directory. fail_loc=0x1624 fail_loc=0 192.168.201.129@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-mds1 (opts:) on oleg129-server run e2fsck on mds1 e2fsck -d -v -t -t -f -y /dev/mapper/mds1_flakey -m8 oleg129-server: e2fsck 1.47.3-wc2 (11-Nov-2025) oleg129-server: Use max possible thread num: 1 instead Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 1], used inodes 287/40000 [Thread 0] jumping to group 0 [Thread 0] e2fsck_pass1_run:2316: increase inode 7 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 81 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 82 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 82 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 83 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 114 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 118 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 118 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 119 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 120 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 121 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 122 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 123 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 123 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 124 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 124 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 125 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 125 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 126 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 126 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 127 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 127 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 128 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 128 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 129 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 129 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 130 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 130 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 131 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 131 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 132 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 133 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 133 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 134 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 134 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 135 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 135 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 136 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 136 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 137 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 137 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 138 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 138 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 139 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 139 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 140 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 140 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 141 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 141 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 142 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 142 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 143 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 143 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 144 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 144 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 145 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 145 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 146 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 147 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 147 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 148 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 150 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2316: increase inode 151 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 151 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 152 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 152 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 153 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 153 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 154 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 154 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 158 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 158 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 159 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 159 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 160 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 161 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 161 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 162 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 162 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 163 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 163 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 164 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 164 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 165 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 165 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 166 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 166 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 167 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 167 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 168 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 168 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 169 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 169 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 170 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 170 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 171 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 171 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 172 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 172 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 173 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 173 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 174 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 174 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 175 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 175 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 176 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 176 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 177 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 177 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 178 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 179 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 179 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 180 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 180 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 181 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 181 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 182 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 182 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 183 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 183 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 184 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 184 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 185 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 185 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 186 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 186 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 187 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 187 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 188 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 188 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 189 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 189 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 190 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 190 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 191 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 191 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 192 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 192 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 193 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 193 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 194 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 194 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 198 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 199 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 200 badness 0 to 2 for 10084 [Thread 0] group 0 finished [Thread 0] e2fsck_pass1_run:2316: increase inode 20036 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 20036 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 20073 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2316: increase inode 20074 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2316: increase inode 20075 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2316: increase inode 20076 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2316: increase inode 20078 badness 0 to 1 for 1005c [Thread 0] group 1 finished [Thread 0] Pass 1: Memory used: 264k/0k (146k/119k), time: 0.03/ 0.00/ 0.00 [Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 33.38MB/s [Thread 0] Scanned group range [0, 1], used inodes 287/388 Pass 2: Checking directory structure Paoleg129-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (2289664, 277) != expected (2289664, 278) oleg129-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (2289664, 277) != expected (2289664, 278) oleg129-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (2289664, 277) != expected (2289664, 278) pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 ss 2: Memory used: 264k/0k (97k/168k), time: 0.05/ 0.00/ 0.00 Pass 2: I/O read: 1MB, write: 0MB, rate: 19.93MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 264k/0k (97k/168k), time: 0.10/ 0.00/ 0.01 Unconnected directory inode 20074 (was in /ROOT/d30.sanity-lfsck/foo) Connect to /lost+found? yes Unconnected directory inode 20078 (was in /lost+found/#20074) Connect to /lost+found? yes Pass 3A: Memory used: 264k/0k (98k/167k), time: 0.00/ 0.00/ 0.00 Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 3: Memory used: 264k/0k (96k/169k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 1MB, write: 0MB, rate: 2087.68MB/s Pass 4: Checking reference counts Unattached inode 206 Connect to /lost+found? yes Inode 206 ref count is 2, should be 1. Fix? yes Unattached inode 207 Connect to /lost+found? yes Inode 207 ref count is 2, should be 1. Fix? yes Unattached inode 211 Connect to /lost+found? yes Inode 211 ref count is 2, should be 1. Fix? yes Unattached inode 219 Connect to /lost+found? yes Inode 219 ref count is 2, should be 1. Fix? yes Inode 20073 ref count is 1, should be 2. Fix? yes Inode 20078 ref count is 3, should be 2. Fix? yes Pass 4: Memory used: 264k/0k (83k/182k), time: 0.01/ 0.00/ 0.00 Pass 4: I/O read: 1MB, write: 1MB, rate: 87.26MB/s Pass 5: Checking group summary information Pass 5: Memory used: 372k/0k (82k/291k), time: 0.01/ 0.00/ 0.00 Pass 5: I/O read: 1MB, write: 1MB, rate: 102.86MB/s Update quota info for quota type 0? yes Update quota info for quota type 1? yes Update quota info for quota type 2? yes lustre-MDT0000: ***** FILE SYSTEM WAS MODIFIED ***** 287 inodes used (0.72%, out of 40000) 6 non-contiguous files (2.1%) 0 non-contiguous directories (0.0%) # of inodes with ind/dind/tind blocks: 0/0/0 Extent depth histogram: 209/1 12642 blocks used (50.57%, out of 25000) 0 bad blocks 1 large file 154 regular files 123 directories 0 character device files 0 block device files 0 fifos 4294967294 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 271 files Memory used: 372k/0k (85k/288k), time: 0.16/ 0.01/ 0.01 I/O read: 1MB, write: 1MB, rate: 6.43MB/s oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Trigger namespace LFSCK to recover backend orphans Started LFSCK on the device lustre-MDT0000: scrub namespace Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre File: /mnt/lustre/d30.sanity-lfsck/foo/f0 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272414920866 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 02:54:13.000000000 -0400 Modify: 2026-07-30 02:54:13.000000000 -0400 Change: 2026-07-30 02:55:08.000000000 -0400 Birth: 2026-07-30 02:54:13.000000000 -0400 total 12 144115188109410307 dr-x------ 4 root root 4096 Dec 31 1969 . 144115205306056705 drwx------ 3 root root 4096 Jul 30 02:51 MDT0000 162129603815538689 drwx------ 2 root root 4096 Jul 30 02:52 MDT0001 d0 should become orphan under .lustre/lost+found/MDT0000/ total 16 144115205306056705 drwx------ 3 root root 4096 Jul 30 02:51 . 144115188109410307 dr-x------ 4 root root 4096 Dec 31 1969 .. 144115272414920830 -rw-r--r-- 1 root root 6 Jul 30 02:49 [0x2000013a3:0x7e:0x0]-O-0 144115272414920847 -rw-r--r-- 1 root root 0 Jul 30 02:51 [0x2000013a3:0x8f:0x0]-O-0 144115272414920867 drwxr-xr-x 3 root root 4096 Jul 30 02:55 [0x2000013a3:0xa3:0x0]-[0x2000013a3:0xa1:0x0]-D-0 File: /mnt/lustre/.lustre/lost+found/MDT0000/[0x2000013a3:0xa3:0x0]-[0x2000013a3:0xa1:0x0]-D-0/d1 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115272414920869 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 02:54:16.000000000 -0400 Modify: 2026-07-30 02:54:16.000000000 -0400 Change: 2026-07-30 02:55:08.000000000 -0400 Birth: 2026-07-30 02:54:16.000000000 -0400 File: /mnt/lustre/.lustre/lost+found/MDT0000/[0x2000013a3:0xa3:0x0]-[0x2000013a3:0xa1:0x0]-D-0/f1 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272414920868 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 02:54:16.000000000 -0400 Modify: 2026-07-30 02:54:16.000000000 -0400 Change: 2026-07-30 02:55:08.000000000 -0400 Birth: 2026-07-30 02:54:15.000000000 -0400