== sanity-lfsck test 30: LFSCK can recover the orphans from backend /lost+found ========================================================== 18:29:16 (1746138556) ##### The namespace LFSCK will move the orphans from backend /lost+found directory to normal client visible namespace or to global visible ./lustre/lost+found/MDTxxxx/ directory ##### Inject failure stub on MDT0 to simulate the case that directory d0 has no linkEA entry, then the LFSCK will move it into .lustre/lost+found/MDTxxxx/ later. fail_loc=0x161d fail_loc=0 Inject failure stub on MDT0 to simulate the case that the object's name entry will be removed, but not destroy the object. Then backend e2fsck will handle it as orphan and add them into the backend /lost+found directory. fail_loc=0x1624 fail_loc=0 192.168.204.103@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg403-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-mds1 (opts:) on oleg403-server run e2fsck on mds1 e2fsck -d -v -t -t -f -y /dev/mapper/mds1_flakey -m8 oleg403-server: e2fsck 1.46.2.wc5 (26-Mar-2022) oleg403-server: Use max possible thread num: 1 instead Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 2) [Thread 0] jumping to group 0 [Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 83 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 114 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 121 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 122 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 123 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 124 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 125 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 126 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 127 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 130 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 144 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 145 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 146 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 147 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 152 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 154 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 155 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 156 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 157 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 158 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 159 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 160 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 161 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 162 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 163 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 164 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 165 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 166 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 167 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 168 badness 0 to 2 for 10084 [Thread 0] e2oleg403-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (2306048, 279) != expected (2306048, 280) oleg403-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (2306048, 279) != expected (2306048, 280) oleg403-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (2306048, 279) != expected (2306048, 280) fsck_pass1_run:2564: increase inode 169 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 170 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 171 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 172 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 173 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 174 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 175 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 176 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 177 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 178 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 179 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 180 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 181 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 182 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 183 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 184 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 185 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 186 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 187 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 191 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 192 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 193 badness 0 to 2 for 10084 [Thread 0] group 1 finished [Thread 0] e2fsck_pass1_run:2564: increase inode 20036 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 20073 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 20075 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 20076 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 20077 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 20078 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 20079 badness 0 to 2 for 10084 [Thread 0] group 2 finished [Thread 0] Pass 1: Memory used: 264k/0k (131k/134k), time: 0.01/ 0.00/ 0.00 [Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 108.39MB/s [Thread 0] Scanned group range [0, 2), inodes 388 Pass 2: Checking directory structure Pass 2: Memory used: 264k/0k (88k/177k), time: 0.01/ 0.00/ 0.01 Pass 2: I/O read: 1MB, write: 0MB, rate: 110.12MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 264k/0k (88k/177k), time: 0.03/ 0.01/ 0.01 Unconnected directory inode 20083 (was in /ROOT/d30.sanity-lfsck/foo) Connect to /lost+found? yes Unconnected directory inode 20085 (was in /lost+found/#20083) Connect to /lost+found? yes Pass 3A: Memory used: 264k/0k (88k/177k), time: 0.00/ 0.00/ 0.00 Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 3: Memory used: 264k/0k (86k/179k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 1MB, write: 0MB, rate: 539.08MB/s Pass 4: Checking reference counts Unattached inode 195 Connect to /lost+found? yes Inode 195 ref count is 2, should be 1. Fix? yes Unattached inode 200 Connect to /lost+found? yes Inode 200 ref count is 2, should be 1. Fix? yes Unattached inode 204 Connect to /lost+found? yes Inode 204 ref count is 2, should be 1. Fix? yes Unattached inode 212 Connect to /lost+found? yes Inode 212 ref count is 2, should be 1. Fix? yes Inode 20081 ref count is 1, should be 2. Fix? yes Inode 20085 ref count is 3, should be 2. Fix? yes Pass 4: Memory used: 264k/0k (69k/196k), time: 0.00/ 0.00/ 0.00 Pass 4: I/O read: 1MB, write: 1MB, rate: 396.98MB/s Pass 5: Checking group summary information Pass 5: Memory used: 264k/0k (68k/197k), time: 0.00/ 0.00/ 0.00 Pass 5: I/O read: 1MB, write: 1MB, rate: 302.21MB/s Update quota info for quota type 0? yes Update quota info for quota type 1? yes Uppdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 date quota info for quota type 2? yes lustre-MDT0000: ***** FILE SYSTEM WAS MODIFIED ***** 289 inodes used (0.72%, out of 40000) 5 non-contiguous files (1.7%) 0 non-contiguous directories (0.0%) # of inodes with ind/dind/tind blocks: 1/0/0 12649 blocks used (50.60%, out of 25000) 0 bad blocks 1 large file 156 regular files 123 directories 0 character device files 0 block device files 0 fifos 4294967294 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 273 files Memory used: 264k/0k (67k/198k), time: 0.05/ 0.01/ 0.02 I/O read: 1MB, write: 1MB, rate: 20.63MB/s oleg403-server: oleg403-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Trigger namespace LFSCK to recover backend orphans Started LFSCK on the device lustre-MDT0000: scrub namespace Starting client: oleg403-client.virtnet: -o user_xattr,flock 192.168.204.103@tcp:/lustre /mnt/lustre File: '/mnt/lustre/d30.sanity-lfsck/foo/f0' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272414920881 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-01 18:29:17.000000000 -0400 Modify: 2025-05-01 18:29:17.000000000 -0400 Change: 2025-05-01 18:29:28.000000000 -0400 Birth: - total 12 144115188109410307 dr-x------ 4 root root 4096 Dec 31 1969 . 144115205306056705 drwx------ 3 root root 4096 May 1 18:28 MDT0000 162129603815538689 drwx------ 2 root root 4096 May 1 18:28 MDT0001 d0 should become orphan under .lustre/lost+found/MDT0000/ total 16 144115205306056705 drwx------ 3 root root 4096 May 1 18:28 . 144115188109410307 dr-x------ 4 root root 4096 Dec 31 1969 .. 144115272414920845 -rw-r--r-- 1 root root 6 May 1 18:27 [0x2000013a3:0x8d:0x0]-O-0 144115272414920862 -rw-r--r-- 1 root root 0 May 1 18:28 [0x2000013a3:0x9e:0x0]-O-0 144115272414920882 drwxr-xr-x 3 root root 4096 May 1 18:29 [0x2000013a3:0xb2:0x0]-[0x2000013a3:0xb0:0x0]-D-0 File: '/mnt/lustre/.lustre/lost+found/MDT0000/[0x2000013a3:0xb2:0x0]-[0x2000013a3:0xb0:0x0]-D-0/d1' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115272414920884 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-01 18:29:18.000000000 -0400 Modify: 2025-05-01 18:29:18.000000000 -0400 Change: 2025-05-01 18:29:28.000000000 -0400 Birth: - File: '/mnt/lustre/.lustre/lost+found/MDT0000/[0x2000013a3:0xb2:0x0]-[0x2000013a3:0xb0:0x0]-D-0/f1' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272414920883 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-01 18:29:18.000000000 -0400 Modify: 2025-05-01 18:29:18.000000000 -0400 Change: 2025-05-01 18:29:28.000000000 -0400 Birth: -