== sanity-lfsck test 30: LFSCK can recover the orphans from backend /lost+found ========================================================== 13:37:26 (1747849046) ##### The namespace LFSCK will move the orphans from backend /lost+found directory to normal client visible namespace or to global visible ./lustre/lost+found/MDTxxxx/ directory ##### Inject failure stub on MDT0 to simulate the case that directory d0 has no linkEA entry, then the LFSCK will move it into .lustre/lost+found/MDTxxxx/ later. fail_loc=0x161d fail_loc=0 Inject failure stub on MDT0 to simulate the case that the object's name entry will be removed, but not destroy the object. Then backend e2fsck will handle it as orphan and add them into the backend /lost+found directory. fail_loc=0x1624 fail_loc=0 192.168.202.104@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg204-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-mds1 (opts:) on oleg204-server run e2fsck on mds1 e2fsck -d -v -t -t -f -y /dev/mapper/mds1_flakey -m8 oleg204-server: e2fsck 1.46.2.wc5 (26-Mar-2022) oleg204-server: Use max possible thread num: 1 instead Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 2) [Thread 0] jumping to group 0 [Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 83 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 117 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 121 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 122 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 123 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 124 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 125 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 126 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 127 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 130 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 144 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 145 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 146 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 147 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 148 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 154 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 155 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 156 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 157 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 158 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 159 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 160 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 161 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 162 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 163 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 164 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 165 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 166 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 167 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 168 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 169 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 170 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 171 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 172 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 173 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 174 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 175 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 176 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 177 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 178 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 179 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 180 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 181 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 182 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 183 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 184 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 185 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 186 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 187 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 188 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 190 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 191 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 192 badness 0 to 2 for 10084 [Thread 0] group 1 finished [Thread 0] e2fsck_pass1_run:2564: increase inode 20036 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 20070 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 20071 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 20072 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 20074 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 20075 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 20076 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 20077 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 20078 badness 0 to 2 for 10084 [Thread 0] group 2 finished [Thread 0] Pass 1: Memory used: 264k/0k (130k/135k), time: 0.01/ 0.00/ 0.01 [Thread 0] Pass 1: I/O read: 8MB, write: 0MB, rate: 677.62MB/s [Thread 0] Scanned group range [0, 2), inodes 7353 Pass 2: Checking directory structure Pass 2: Memory used: 264k/0k (87k/178k), time: 0.00/ 0.00/ 0.00 Pass 2: I/O read: 1MB, write: 0MB, rate: 253.36MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 264k/0k (87k/178k), time: 0.02/ 0.01/ 0.01 Unconnected directory inode 20081 (was in /ROOT/d30.sanity-lfsck/foo) Connect to /lost+found? yes Unconnected directory inode 20082 (was in /lost+found/#20081) Connect to /lost+found? yes Pass 3A: Memory used: 264k/0k (87k/178k), time: 0.00/ 0.00/ 0.00 Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 3: Memory used: 264k/0k (85k/180k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 1MB, write: 0MB, rate: 2433.09MB/s Pass 4: Checking reference counts Unattached inode 198 Connect to /lost+found? yes Inode 198 ref count is 2, should be 1. Fix? yes Unattached inode 199 Connect to /lost+found? yes Inode 199 ref count is 2, should be 1. Fix? yes Unattached inode 200 Connect to /lost+found? yes Inode 200 ref count is 2, should be 1. Fix? yes Unattached inode 208 Connect to /lost+found? yes Inode 208 ref count is 2, should be 1. Fix? yes Inode 20080 ref count is 1, should be 2. Fix? yes Inode 20082 ref count is 3, should be 2. Fix? yes Pass 4: Memory used: 264k/0k (69k/196k), time: 0.00/ 0.00/ 0.00 Pass 4: I/O read: 1MB, write: 1MB, rate: 542.59MB/s Pass 5: Checking group summary information Pass 5: Memory used: 264k/0k (68k/197k), time: 0.00/ 0.00/ 0.00 Pass 5: I/O read: 1MB, write: 1MB, rate: 519.75MB/s lustre-MDT0000: ***** FILE SYSTEM WAS MODIFIED ***** 282 inodes usepdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 d (0.70%, out of 40000) 7 non-contiguous files (2.5%) 0 non-contiguous directories (0.0%) # of inodes with ind/dind/tind blocks: 1/0/0 12579 blocks used (50.32%, out of 25000) 0 bad blocks 1 large file 150 regular files 122 directories 0 character device files 0 block device files 0 fifos 4294967294 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 266 files Memory used: 264k/0k (67k/198k), time: 0.03/ 0.01/ 0.01 I/O read: 1MB, write: 1MB, rate: 36.40MB/s oleg204-server: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Trigger namespace LFSCK to recover backend orphans Started LFSCK on the device lustre-MDT0000: scrub namespace Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre File: '/mnt/lustre/d30.sanity-lfsck/foo/f0' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115507279167648 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-21 13:37:26.000000000 -0400 Modify: 2025-05-21 13:37:26.000000000 -0400 Change: 2025-05-21 13:37:47.000000000 -0400 Birth: - total 8 144115188109410307 dr-x------ 3 root root 4096 Dec 31 1969 . 144115205289279489 drwx------ 3 root root 4096 May 21 13:37 MDT0000 d0 should become orphan under .lustre/lost+found/MDT0000/ total 16 144115205289279489 drwx------ 3 root root 4096 May 21 13:37 . 144115188109410307 dr-x------ 3 root root 4096 Dec 31 1969 .. 144115507279167623 -rw-r--r-- 1 root root 6 May 21 13:36 [0x200004a52:0x87:0x0]-O-0 144115507279167633 -rw-r--r-- 1 root root 0 May 21 13:37 [0x200004a52:0x91:0x0]-O-0 144115507279167649 drwxr-xr-x 3 root root 4096 May 21 13:37 [0x200004a52:0xa1:0x0]-[0x200004a52:0x9f:0x0]-D-0 File: '/mnt/lustre/.lustre/lost+found/MDT0000/[0x200004a52:0xa1:0x0]-[0x200004a52:0x9f:0x0]-D-0/d1' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115507279167651 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-21 13:37:27.000000000 -0400 Modify: 2025-05-21 13:37:27.000000000 -0400 Change: 2025-05-21 13:37:47.000000000 -0400 Birth: - File: '/mnt/lustre/.lustre/lost+found/MDT0000/[0x200004a52:0xa1:0x0]-[0x200004a52:0x9f:0x0]-D-0/f1' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115507279167650 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-21 13:37:27.000000000 -0400 Modify: 2025-05-21 13:37:27.000000000 -0400 Change: 2025-05-21 13:37:47.000000000 -0400 Birth: -