-----============= acceptance-small: sanity ============----- Wed Jun 3 18:03:25 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg433-server: /home/green/git/lustre-release/lustre/tests/except/sanity.56aa.ex oleg433-server: /home/green/git/lustre-release/lustre/tests/except/sanity.ex - see CLIENT_VERSION > v2_15_63-134-gdacc4b6d38 (34682169 > 34553734) for LU-15963, go 312 - see CLIENT_VERSION > v2_15_63-239-gab4ede45b4 (34682169 > 34553839) for LU-17856, go 56wa 56xe - see CLIENT_VERSION > v2_16_54-54-ga2e3a2f5a3 (34682169 > 34616886) for LU-14590, go 0d - see CLIENT_VERSION >= v2_17_50-196-g63ee4af3bb (34682169 >= 34681540) for LU-19620, go 60b - see CLIENT_VERSION >= v2_17_50-71-g2cbe29c282 (34682169 >= 34681415) for LU-19644, go 150ib - see CLIENT_VERSION >= v2_17_51-74 (34682169 >= 34681674) for LU-20010, go 56aa - need MDS1_VERSION <= 2.16.61-1-g89cf292a8c2 (34682169 <= 34618625) for LU-18938, skip 360 - see MDS1_VERSION <= CLIENT_VERSION (34682169 <= 34682169) for LU-18562, go 270a - need MDS1_VERSION < v2_14_55-100-g8a84c7f9c7 (34682169 < 34486116) for LU-14927, skip 0f excepting tests: 42a 42c 42b 118c 118d 407 119i 817 411a 130b 130c 130d 130e 130f 130g skipping tests SLOW=no: 27m 60i 64b 68 71 135 136 230d 300o 842 51b 51c 51e 834 === sanity: start setup 18:03:33 (1780524213) === oleg433-client.virtnet: executing check_config_client /mnt/lustre oleg433-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg433-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff972052442000.idle_timeout=debug osc.lustre-OST0001-osc-ffff972052442000.idle_timeout=debug disable quota as required oleg433-server: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity: finish setup 18:03:45 (1780524225) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8377] preparing for tests involving mounts mke2fs 1.47.3-wc2 (11-Nov-2025) debug=all == sanity test 33h: temp file is located on the same MDT as target (crush) ========================================================== 18:03:47 (1780524227) striped dir -i1 -c2 -H crush /mnt/lustre/d33h.sanity pattern .f33h.sanity.XXXXXX /mnt/lustre/d33h.sanity/.f33h.sanity.EGZHNN MDT index mismatch 0 != 1 pattern f33h.sanity.XXXXXXXX 1/80 MDT index mismatches, expect ~2 pattern .f33h.sanity.XXXXXX pattern f33h.sanity.XXXXXXXX 80/80 matches, expect ~80 for crush pattern=.f33h.sanity....XXX pattern=f33h.sanity....XXXXX 80/80 matches, expect ~80 for crush PASS 33h (19s) == sanity test 33hh: temp file is located on the same MDT as target (crush2) ========================================================== 18:04:05 (1780524245) striped dir -i1 -c2 -H crush2 /mnt/lustre/d33hh.sanity pattern .f33hh.sanity.XXXXXX pattern f33hh.sanity.XXXXXXXX 0/80 MDT index mismatches, expect ~2 pattern .f33hh.sanity.XXXXXX pattern f33hh.sanity.XXXXXXXX 39/80 matches, expect ~40 for crush2 pattern=.f33hh.sanity....XXX pattern=f33hh.sanity....XXXXX 39/80 matches, expect ~40 for crush2 PASS 33hh (18s) == sanity test 51b: exceed 64k subdirectory nlink limit on create, verify unlink ========================================================== 18:04:23 (1780524263) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 553205 1141 552064 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 549273 484 548789 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 118763 971 117792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 118762 970 117792 1% /mnt/lustre[OST:1] filesystem_summary: 237209 1625 235584 1% /mnt/lustre - mkdir 1826 (time 1780524275.35 total 10.00 last 182.53) - mkdir 3401 (time 1780524285.35 total 20.01 last 157.47) - mkdir 5012 (time 1780524295.35 total 30.01 last 161.08) - mkdir 6763 (time 1780524305.35 total 40.01 last 175.08) - mkdir 8231 (time 1780524315.35 total 50.01 last 146.75) - mkdir 9803 (time 1780524325.35 total 60.01 last 157.20) - mkdir 11395 (time 1780524335.36 total 70.02 last 159.14) - mkdir 12978 (time 1780524345.37 total 80.02 last 158.16) - mkdir 14454 (time 1780524355.37 total 90.03 last 147.53) - mkdir 16161 (time 1780524365.37 total 100.03 last 170.66) - mkdir 17543 (time 1780524375.38 total 110.04 last 138.11) - mkdir 18737 (time 1780524385.38 total 120.04 last 119.38) - mkdir 19769 (time 1780524395.39 total 130.05 last 103.10) - mkdir 20000 (time 1780524397.55 total 132.21 last 106.83) - mkdir 21175 (time 1780524407.56 total 142.22 last 117.44) - mkdir 22421 (time 1780524417.56 total 152.22 last 124.57) - mkdir 23761 (time 1780524427.57 total 162.23 last 133.89) - mkdir 25652 (time 1780524437.62 total 172.27 last 188.23) - mkdir 27501 (time 1780524447.62 total 182.28 last 184.84) - mkdir 29198 (time 1780524457.63 total 192.28 last 169.57) - mkdir 30000 (time 1780524462.38 total 197.04 last 168.74) - mkdir 32269 (time 1780524472.42 total 207.08 last 225.91) total: 32818 mkdir in 209.34 seconds: 156.77 ops/second - mkdir 1794 (time 1780524275.34 total 10.00 last 179.36) - mkdir 3358 (time 1780524285.34 total 20.00 last 156.40) - mkdir 4980 (time 1780524295.34 total 30.00 last 162.18) - mkdir 6764 (time 1780524305.34 total 40.01 last 178.36) - mkdir 8256 (time 1780524315.35 total 50.01 last 149.15) - mkdir 9772 (time 1780524325.35 total 60.01 last 151.59) - mkdir 11392 (time 1780524335.35 total 70.02 last 161.92) - mkdir 13017 (time 1780524345.35 total 80.02 last 162.48) - mkdir 14493 (time 1780524355.36 total 90.02 last 147.53) - mkdir 16141 (time 1780524365.36 total 100.02 last 164.75) - mkdir 17522 (time 1780524375.37 total 110.03 last 138.05) - mkdir 18719 (time 1780524385.38 total 120.04 last 119.53) - mkdir 19779 (time 1780524395.39 total 130.05 last 105.89) - mkdir 20000 (time 1780524397.45 total 132.12 last 107.00) - mkdir 21180 (time 1780524407.46 total 142.12 last 117.96) - mkdir 22393 (time 1780524417.46 total 152.12 last 121.29) - mkdir 23716 (time 1780524427.48 total 162.14 last 132.08) - mkdir 25590 (time 1780524437.48 total 172.14 last 187.32) - mkdir 27428 (time 1780524447.48 total 182.14 last 183.78) - mkdir 29118 (time 1780524457.49 total 192.15 last 168.91) - mkdir 30000 (time 1780524462.59 total 197.25 last 172.98) - mkdir 32272 (time 1780524472.59 total 207.25 last 227.16) total: 32818 mkdir in 209.45 seconds: 156.69 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210176 489728 1718400 23% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 553212 1148 552064 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 304803 66121 238682 22% /mnt/lustre[MDT:1] lustre-OST0000_UUID 118763 971 117792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 118762 970 117792 1% /mnt/lustre[OST:1] filesystem_summary: 302853 67269 235584 23% /mnt/lustre nlink before: 65638, created before: 65637 - unlinked 0 (time 1780524719 ; total 0 ; last 0) - unlinked 10000 (time 1780524792 ; total 73 ; last 73) - unlinked 20000 (time 1780524848 ; total 129 ; last 56) - unlinked 30000 (time 1780524904 ; total 185 ; last 56) total: 32768 unlinks in 201 seconds: 163.024872 unlinks/second - unlinked 0 (time 1780524719 ; total 0 ; last 0) - unlinked 10000 (time 1780524793 ; total 74 ; last 74) - unlinked 20000 (time 1780524848 ; total 129 ; last 55) - unlinked 30000 (time 1780524905 ; total 186 ; last 57) total: 32768 unlinks in 201 seconds: 163.024872 unlinks/second nlink between: 102 - unlinked 0 (time 1780524921 ; total 0 ; last 0) total: 50 unlinks in 0 seconds: inf unlinks/second - unlinked 0 (time 1780524921 ; total 0 ; last 0) total: 50 unlinks in 0 seconds: inf unlinks/second nlink after: 2 rmdir(/mnt/lustre/d51b.sanity/d.1.0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second rmdir(/mnt/lustre/d51b.sanity/d.0.0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2209152 4224 2202880 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 553212 1148 552064 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 490618 485 490133 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 118763 971 117792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 118762 970 117792 1% /mnt/lustre[OST:1] filesystem_summary: 237217 1633 235584 1% /mnt/lustre PASS 51b (663s) == sanity test 51f: check many open files limit ========== 18:15:26 (1780524926) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d51f.sanity ulimit_old=1024 numfree= 490133, max=100000 changed ulimit from 1024 to 100100 - open/keep 3805 (time 1780524938.01 total 10.00 last 380.43) - open/keep 7511 (time 1780524948.02 total 20.00 last 370.51) - open/keep 10000 (time 1780524954.79 total 26.77 last 367.71) - open/keep 13702 (time 1780524964.79 total 36.78 last 370.12) - open/keep 17271 (time 1780524974.79 total 46.78 last 356.86) - open/keep 20000 (time 1780524982.42 total 54.41 last 357.47) - open/keep 23507 (time 1780524992.43 total 64.41 last 350.61) - open/keep 28110 (time 1780525002.43 total 74.41 last 460.29) - open/keep 30000 (time 1780525007.37 total 79.36 last 382.04) - open/keep 33651 (time 1780525017.38 total 89.36 last 364.97) - open/keep 37677 (time 1780525027.38 total 99.37 last 402.52) - open/keep 40000 (time 1780525033.21 total 105.20 last 398.42) - open/keep 43701 (time 1780525043.21 total 115.20 last 370.04) total: 45379 open/keep in 120.02 seconds: 378.10 ops/second - closed 7236 (time 1780525058.03 total 10.00 last -3646.17) - closed 10000 (time 1780525061.19 total 13.16 last 875.22) - closed 17384 (time 1780525071.19 total 23.16 last 738.37) - closed 20000 (time 1780525074.88 total 26.85 last 709.64) - closed 26781 (time 1780525084.88 total 36.85 last 677.90) - closed 30000 (time 1780525089.22 total 41.18 last 742.46) - closed 37898 (time 1780525099.22 total 51.19 last 789.39) - closed 40000 (time 1780525101.56 total 53.53 last 898.89) total: 45379 close in 59.72 seconds: 759.88 close/second - unlinked 0 (time 1780525111 ; total 0 ; last 0) - unlinked 10000 (time 1780525131 ; total 20 ; last 20) - unlinked 20000 (time 1780525149 ; total 38 ; last 18) - unlinked 30000 (time 1780525168 ; total 57 ; last 19) - unlinked 40000 (time 1780525187 ; total 76 ; last 19) unlink(/mnt/lustre/d51f.sanity/f45379) error: No such file or directory total: 45379 unlinks in 86 seconds: 527.662781 unlinks/second PASS 51f (272s) == sanity test 57b: default LOV EAs are stored inside large inodes ============================================================= 18:19:58 (1780525198) SKIP: sanity test_57b ldiskfs only test SKIP 57b (1s) resend_count is set to 4 4 resend_count is set to 4 4 resend_count is set to 4 4 resend_count is set to 4 4 resend_count is set to 4 4 == sanity test 160i: changelog user register/unregister race ========================================================== 18:20:00 (1780525200) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' fail_loc=0x10001315 fail_val=1 lustre-MDT0000: clear the changelog for cl1 of all records mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm lustre-MDT0000: Deregistered changelog user #1 lustre-MDT0001: clear the changelog for cl1 of all records Registered 2 changelog users: 'cl1 cl2 cl1 cl2' lustre-MDT0001: Deregistered changelog user #1 cl2 3 (1) cl2 2 (1) verify changelogs added: 3->7 verify changelogs added: 2->6 lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001: Deregistered changelog user #2 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 mdd.lustre-MDT0000.changelog_mask=MARK mdd.lustre-MDT0001.changelog_mask=MARK lustre-MDT0001: changelog user 'cl1' not found lustre-MDT0000: changelog user 'cl1' not found mdd.lustre-MDT0000.changelog_mask=MARK mdd.lustre-MDT0001.changelog_mask=MARK PASS 160i (13s) == sanity test 230o: dir split =========================== 18:20:14 (1780525214) lod.lustre-MDT0000-mdtlov.mdt_hash=crush lod.lustre-MDT0001-mdtlov.mdt_hash=crush mdt.lustre-MDT0000.enable_dir_restripe=1 mdt.lustre-MDT0001.enable_dir_restripe=1 total: 50 create in 0.11 seconds: 447.41 ops/second total: 50 mkdir in 0.23 seconds: 218.83 ops/second Waiting 300s for 'crush' Updated after 2s: want 'crush' got 'crush' 50 split sc=1->2 expect<=110 short=0 mdt.lustre-MDT0000.enable_dir_restripe=0 mdt.lustre-MDT0001.enable_dir_restripe=0 PASS 230o (6s) == sanity test 230p: dir merge =========================== 18:20:21 (1780525221) lod.lustre-MDT0000-mdtlov.mdt_hash=crush lod.lustre-MDT0001-mdtlov.mdt_hash=crush mdt.lustre-MDT0000.enable_dir_restripe=1 mdt.lustre-MDT0001.enable_dir_restripe=1 striped dir -i0 -c2 -H crush /mnt/lustre/d230p.sanity total: 50 create in 0.12 seconds: 402.33 ops/second total: 50 mkdir in 0.09 seconds: 531.99 ops/second restripe /mnt/lustre/d230p.sanity to 1 stripes Waiting 210s for 'crush,fixed' Updated after 2s: want 'crush,fixed' got 'crush,fixed' 50 merge sc=2->1 expect<=110 short=0 Splitting the dir back to 2 stripes. Waiting 210s for 'crush' Updated after 2s: want 'crush' got 'crush' mdt.lustre-MDT0000.enable_dir_restripe=0 mdt.lustre-MDT0001.enable_dir_restripe=0 PASS 230p (10s) == sanity test 230q: dir auto split ====================== 18:20:30 (1780525230) mdt.lustre-MDT0000.enable_dir_auto_split=1 mdt.lustre-MDT0001.enable_dir_auto_split=1 mdt.lustre-MDT0000.dir_split_count=50 mdt.lustre-MDT0001.dir_split_count=50 mdt.lustre-MDT0000.dir_split_delta=2 mdt.lustre-MDT0001.dir_split_delta=2 mdt.lustre-MDT0000.dir_restripe_nsonly=0 mdt.lustre-MDT0001.dir_restripe_nsonly=0 lod.lustre-MDT0000-mdtlov.mdt_hash=crush lod.lustre-MDT0001-mdtlov.mdt_hash=crush total: 75 create in 0.14 seconds: 523.86 ops/second Waiting 200s for 'crush' Updated after 7s: want 'crush' got 'crush' 36/75 on lustre-MDT0001, sc=2, exp=44 fixed layout directory won't auto split mdt.lustre-MDT0000.dir_restripe_nsonly=1 mdt.lustre-MDT0001.dir_restripe_nsonly=1 mdt.lustre-MDT0000.dir_split_delta=4 mdt.lustre-MDT0001.dir_split_delta=4 mdt.lustre-MDT0000.dir_split_count=50000 mdt.lustre-MDT0001.dir_split_count=50000 PASS 230q (14s) == sanity test 230x: dir migration check space =========== 18:20:44 (1780525244) total: 100 mkdir in 0.36 seconds: 281.45 ops/second lfs migrate: /mnt/lustre/d230x.sanity migrate failed: No space left on device (28) PASS 230x (23s) == sanity test 270a: DoM: basic functionality tests ====== 18:21:07 (1780525267) 192+0 records in 192+0 records out 196608 bytes (197 kB, 192 KiB) copied, 0.08213 s, 2.4 MB/s 3+0 records in 3+0 records out 196608 bytes (197 kB, 192 KiB) copied, 0.055158 s, 3.6 MB/s 1984+0 records in 1984+0 records out 2031616 bytes (2.0 MB, 1.9 MiB) copied, 0.629833 s, 3.2 MB/s 31+0 records in 31+0 records out 2031616 bytes (2.0 MB, 1.9 MiB) copied, 0.483953 s, 4.2 MB/s PASS 270a (5s) == sanity test 272b: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 18:21:12 (1780525272) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0903525 s, 23.2 MB/s PASS 272b (4s) == sanity test 272c: DoM migration: DOM file to the OST-striped file (composite) ========================================================== 18:21:16 (1780525276) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.092496 s, 22.7 MB/s PASS 272c (4s) == sanity test 272d: DoM mirroring: OST-striped mirror to DOM file ========================================================== 18:21:20 (1780525280) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0803576 s, 26.1 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x5cd0:0x0]: Invalid argument (22) PASS 272d (3s) == sanity test 300ug: migrate overstriped dirs =========== 18:21:24 (1780525284) lmv_stripe_count: 8 lmv_stripe_offset: 0 lmv_hash_type: crush,fixed mdtidx FID[seq:oid:ver] 0 [0x200000400:0x68:0x0] 1 [0x240000401:0x68:0x0] 0 [0x200000400:0x69:0x0] 1 [0x240000401:0x69:0x0] 0 [0x200000400:0x6a:0x0] 1 [0x240000401:0x6a:0x0] 0 [0x200000400:0x6b:0x0] 1 [0x240000401:0x6b:0x0] PASS 300ug (3s) == sanity test 808: Check trusted.som xattr not logged in Changelogs ========================================================== 18:21:28 (1780525288) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl3 cl3' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313198 s, 33.5 MB/s lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0001: Deregistered changelog user #3 lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0000: Deregistered changelog user #3 mdd.lustre-MDT0000.changelog_mask=MARK mdd.lustre-MDT0001.changelog_mask=MARK PASS 808 (6s) == sanity test complete, duration 1089 sec =============== 18:21:34 (1780525294) === sanity: start cleanup 18:21:35 (1780525295) === === sanity: finish cleanup 18:21:45 (1780525305) === debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck