== sanity-lfsck test 30: LFSCK can recover the orphans from backend /lost+found ========================================================== 05:29:18 (1743499758) ##### The namespace LFSCK will move the orphans from backend /lost+found directory to normal client visible namespace or to global visible ./lustre/lost+found/MDTxxxx/ directory ##### Inject failure stub on MDT0 to simulate the case that directory d0 has no linkEA entry, then the LFSCK will move it into .lustre/lost+found/MDTxxxx/ later. fail_loc=0x161d fail_loc=0 Inject failure stub on MDT0 to simulate the case that the object's name entry will be removed, but not destroy the object. Then backend e2fsck will handle it as orphan and add them into the backend /lost+found directory. fail_loc=0x1624 fail_loc=0 192.168.206.159@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg659-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-mds1 (opts:) on oleg659-server run e2fsck on mds1 e2fsck -d -v -t -t -f -y /dev/mapper/mds1_flakey -m8 oleg659-server: e2fsck 1.47.1-wc2 (08-Nov-2024) oleg659-server: Use max possible thread num: 1 instead Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 1], used inodes 287/40000 [Thread 0] jumping to group 0 [Thread 0] e2fsck_pass1_run:2675: increase inode 81 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 82 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 83 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 112 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 133 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 137 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 138 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 139 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 140 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 141 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 142 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 143 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 144 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 145 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 146 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 147 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 148 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 149 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 150 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 151 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 152 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 153 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 154 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 155 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 156 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 157 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 158 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 159 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 160 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 161 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 162 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 163 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 168 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 169 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 170 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 171 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 172 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 173 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 174 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 175 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 176 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 177 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 178 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 179 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 180 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 181 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 182 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 183 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 184 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 185 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 186 badness 0 to 2 oleg659-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (2285568, 277) != expected (2285568, 278) for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 187 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 188 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 189 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 190 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 191 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 192 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 193 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 194 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 195 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 196 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 197 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 198 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 199 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 200 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 201 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 202 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 203 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 207 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 208 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 209 badness 0 to 2 for 10084 [Thread 0] group 0 finished [Thread 0] e2fsck_pass1_run:2675: increase inode 20054 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 20055 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 20056 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 20057 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 20059 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 20060 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 20061 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 20062 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 20063 badness 0 to 2 for 10084 [Thread 0] group 1 finished [Thread 0] Pass 1: Memory used: 264k/0k (144k/121k), time: 0.01/ 0.00/ 0.00 [Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 134.68MB/s [Thread 0] Scanned group range [0, 1], used inodes 287/388 Pass 2: Checking directory structure Pass 2: Memory used: 264k/0k (97k/168k), time: 0.02/ 0.00/ 0.00 Pass 2: I/O read: 1MB, write: 0MB, rate: 65.93MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 264k/0k (97k/168k), time: 0.03/ 0.00/ 0.01 Unconnected directory inode 20067 (was in /ROOT/d30.sanity-lfsck/foo) Connect to /lost+found? yes Unconnected directory inode 20069 (was in /lost+found/#20067) Connect to /lost+found? yes Pass 3A: Memory used: 264k/0k (97k/168k), time: 0.00/ 0.00/ 0.00 Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 3: Memory used: 264k/0k (95k/170k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 1MB, write: 0MB, rate: 2036.66MB/s Pass 4: Checking reference counts Unattached inode 215 Connect to /lost+found? yes Inode 215 ref count is 2, should be 1. Fix? yes Unattached inode 217 Connect to /lost+found? yes Inode 217 ref count is 2, should be 1. Fix? yes Unattached inode 220 Connect to /lost+found? yes Inode 220 ref count is 2, should be 1. Fix? yes Unattached inode 228 Connect to /lost+found? yes Inode 228 ref count is 2, should be 1. Fix? yes Inode 20065 ref count is 1, should be 2. Fix? yes Inode 20069 ref count is 3, should be 2. Fix? yes Pass 4: Memory used: 264k/0k (84k/181k), time: 0.00/ 0.00/ 0.00 Pass 4: I/O read: 1MB, write: 1MB, rate: 599.16MB/s Pass 5: Checking group summary information Pass 5: Memory used: 372k/0k (83k/290k), time: 0.00/ 0.00/ 0.00 Pass 5: I/O read: 1MB, write: 1MB, rate: 359.32MB/s Update quota info for quota type 0? yes Update oleg659-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (2285568, 277) != expected (2285568, 278) oleg659-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (2285568, 277) != expected (2285568, 278) pdsh@oleg659-client: oleg659-server: ssh exited with exit code 1 quota info for quota type 1? yes Update quota info for quota type 2? yes lustre-MDT0000: ***** FILE SYSTEM WAS MODIFIED ***** 287 inodes used (0.72%, out of 40000) 5 non-contiguous files (1.7%) 0 non-contiguous directories (0.0%) # of inodes with ind/dind/tind blocks: 1/0/0 12644 blocks used (50.58%, out of 25000) 0 bad blocks 1 large file 154 regular files 123 directories 0 character device files 0 block device files 0 fifos 4294967294 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 271 files Memory used: 372k/0k (86k/287k), time: 0.05/ 0.01/ 0.01 I/O read: 1MB, write: 1MB, rate: 22.10MB/s oleg659-server: oleg659-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg659-client: oleg659-server: ssh exited with exit code 1 Trigger namespace LFSCK to recover backend orphans Started LFSCK on the device lustre-MDT0000: scrub namespace Starting client: oleg659-client.virtnet: -o user_xattr,flock 192.168.206.159@tcp:/lustre /mnt/lustre File: /mnt/lustre/d30.sanity-lfsck/foo/f0 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272414920880 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-01 05:29:21.000000000 -0400 Modify: 2025-04-01 05:29:21.000000000 -0400 Change: 2025-04-01 05:29:58.000000000 -0400 Birth: 2025-04-01 05:29:20.000000000 -0400 total 12 144115188109410307 dr-x------ 4 root root 4096 Dec 31 1969 . 144115205306056705 drwx------ 3 root root 4096 Apr 1 05:27 MDT0000 162129603815538689 drwx------ 2 root root 4096 Apr 1 05:28 MDT0001 d0 should become orphan under .lustre/lost+found/MDT0000/ total 16 144115205306056705 drwx------ 3 root root 4096 Apr 1 05:27 . 144115188109410307 dr-x------ 4 root root 4096 Dec 31 1969 .. 144115272414920844 -rw-r--r-- 1 root root 6 Apr 1 05:25 [0x2000013a3:0x8c:0x0]-O-0 144115272414920861 -rw-r--r-- 1 root root 0 Apr 1 05:27 [0x2000013a3:0x9d:0x0]-O-0 144115272414920881 drwxr-xr-x 3 root root 4096 Apr 1 05:29 [0x2000013a3:0xb1:0x0]-[0x2000013a3:0xaf:0x0]-D-0 File: /mnt/lustre/.lustre/lost+found/MDT0000/[0x2000013a3:0xb1:0x0]-[0x2000013a3:0xaf:0x0]-D-0/d1 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115272414920883 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-01 05:29:23.000000000 -0400 Modify: 2025-04-01 05:29:23.000000000 -0400 Change: 2025-04-01 05:29:58.000000000 -0400 Birth: 2025-04-01 05:29:22.000000000 -0400 File: /mnt/lustre/.lustre/lost+found/MDT0000/[0x2000013a3:0xb1:0x0]-[0x2000013a3:0xaf:0x0]-D-0/f1 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272414920882 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-01 05:29:23.000000000 -0400 Modify: 2025-04-01 05:29:23.000000000 -0400 Change: 2025-04-01 05:29:58.000000000 -0400 Birth: 2025-04-01 05:29:22.000000000 -0400