-----============= acceptance-small: sanity ============----- Wed Jun 3 18:04:02 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg309-server: /home/green/git/lustre-release/lustre/tests/except/sanity.56aa.ex oleg309-server: /home/green/git/lustre-release/lustre/tests/except/sanity.ex - see CLIENT_VERSION > v2_15_63-134-gdacc4b6d38 (34682169 > 34553734) for LU-15963, go 312 - see CLIENT_VERSION > v2_15_63-239-gab4ede45b4 (34682169 > 34553839) for LU-17856, go 56wa 56xe - see CLIENT_VERSION > v2_16_54-54-ga2e3a2f5a3 (34682169 > 34616886) for LU-14590, go 0d - see CLIENT_VERSION >= v2_17_50-196-g63ee4af3bb (34682169 >= 34681540) for LU-19620, go 60b - see CLIENT_VERSION >= v2_17_50-71-g2cbe29c282 (34682169 >= 34681415) for LU-19644, go 150ib - see CLIENT_VERSION >= v2_17_51-74 (34682169 >= 34681674) for LU-20010, go 56aa - need MDS1_VERSION <= 2.16.61-1-g89cf292a8c2 (34682169 <= 34618625) for LU-18938, skip 360 - see MDS1_VERSION <= CLIENT_VERSION (34682169 <= 34682169) for LU-18562, go 270a - need MDS1_VERSION < v2_14_55-100-g8a84c7f9c7 (34682169 < 34486116) for LU-14927, skip 0f excepting tests: 42a 42c 42b 118c 118d 407 119i 817 411a 130b 130c 130d 130e 130f 130g skipping tests SLOW=no: 27m 60i 64b 68 71 135 136 230d 300o 842 51b 51c 51e 834 === sanity: start setup 18:04:14 (1780524254) === oleg309-client.virtnet: executing check_config_client /mnt/lustre oleg309-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg309-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff946a1098f000.idle_timeout=debug osc.lustre-OST0001-osc-ffff946a1098f000.idle_timeout=debug disable quota as required oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity: finish setup 18:04:29 (1780524269) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7579] preparing for tests involving mounts mke2fs 1.47.3-wc2 (11-Nov-2025) debug=all == sanity test 33h: temp file is located on the same MDT as target (crush) ========================================================== 18:04:31 (1780524271) SKIP: sanity test_33h needs >= 2 MDTs SKIP 33h (1s) == sanity test 33hh: temp file is located on the same MDT as target (crush2) ========================================================== 18:04:32 (1780524272) SKIP: sanity test_33hh needs >= 2 MDTs SKIP 33hh (1s) == sanity test 51b: exceed 64k subdirectory nlink limit on create, verify unlink ========================================================== 18:04:34 (1780524274) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 534698 333 534365 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118185 393 117792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 118185 393 117792 1% /mnt/lustre[OST:1] filesystem_summary: 235917 333 235584 1% /mnt/lustre - mkdir 1408 (time 1780524286.06 total 10.00 last 140.79) - mkdir 2761 (time 1780524296.06 total 20.00 last 135.26) - mkdir 3978 (time 1780524306.06 total 30.00 last 121.69) - mkdir 5306 (time 1780524316.06 total 40.00 last 132.80) - mkdir 6618 (time 1780524326.06 total 50.01 last 131.18) - mkdir 7792 (time 1780524336.07 total 60.01 last 117.32) - mkdir 8991 (time 1780524346.08 total 70.02 last 119.80) - mkdir 10000 (time 1780524354.94 total 78.88 last 113.91) - mkdir 11070 (time 1780524364.94 total 88.89 last 106.92) - mkdir 12184 (time 1780524374.95 total 98.89 last 111.31) - mkdir 13373 (time 1780524384.95 total 108.89 last 118.90) - mkdir 14486 (time 1780524394.95 total 118.89 last 111.30) - mkdir 15575 (time 1780524404.96 total 128.90 last 108.84) - mkdir 16489 (time 1780524414.98 total 138.92 last 91.25) - mkdir 17438 (time 1780524424.98 total 148.92 last 94.88) - mkdir 18449 (time 1780524434.98 total 158.92 last 101.07) - mkdir 19461 (time 1780524445.00 total 168.94 last 101.00) - mkdir 20000 (time 1780524450.47 total 174.41 last 98.54) - mkdir 21047 (time 1780524460.48 total 184.42 last 104.63) - mkdir 22133 (time 1780524470.48 total 194.42 last 108.54) - mkdir 23174 (time 1780524480.49 total 204.43 last 104.05) - mkdir 24200 (time 1780524490.50 total 214.44 last 102.45) - mkdir 25251 (time 1780524500.51 total 224.45 last 105.02) - mkdir 26293 (time 1780524510.52 total 234.46 last 104.08) - mkdir 27334 (time 1780524520.58 total 244.52 last 103.48) - mkdir 28411 (time 1780524530.58 total 254.52 last 107.70) - mkdir 29414 (time 1780524540.58 total 264.52 last 100.29) - mkdir 30000 (time 1780524546.82 total 270.76 last 93.92) - mkdir 31009 (time 1780524556.83 total 280.77 last 100.83) - mkdir 32188 (time 1780524566.83 total 290.77 last 117.88) total: 32818 mkdir in 296.18 seconds: 110.81 ops/second - mkdir 1419 (time 1780524286.06 total 10.01 last 141.83) - mkdir 2757 (time 1780524296.06 total 20.01 last 133.76) - mkdir 3942 (time 1780524306.07 total 30.01 last 118.42) - mkdir 5217 (time 1780524316.08 total 40.02 last 127.42) - mkdir 6512 (time 1780524326.09 total 50.03 last 129.38) - mkdir 7689 (time 1780524336.09 total 60.04 last 117.64) - mkdir 8861 (time 1780524346.09 total 70.04 last 117.20) - mkdir 9998 (time 1780524356.10 total 80.04 last 113.60) - mkdir 11134 (time 1780524366.12 total 90.06 last 113.44) - mkdir 12194 (time 1780524376.12 total 100.06 last 105.99) - mkdir 13377 (time 1780524386.12 total 110.06 last 118.30) - mkdir 14532 (time 1780524396.12 total 120.07 last 115.42) - mkdir 15647 (time 1780524406.13 total 130.07 last 111.46) - mkdir 16595 (time 1780524416.13 total 140.07 last 94.79) - mkdir 17567 (time 1780524426.13 total 150.07 last 97.18) - mkdir 18578 (time 1780524436.14 total 160.09 last 100.96) - mkdir 19589 (time 1780524446.14 total 170.09 last 101.10) - mkdir 20000 (time 1780524450.38 total 174.32 last 96.99) - mkdir 21033 (time 1780524460.38 total 184.33 last 103.29) - mkdir 22076 (time 1780524470.39 total 194.33 last 104.23) - mkdir 23120 (time 1780524480.40 total 204.34 last 104.33) - mkdir 24163 (time 1780524490.41 total 214.36 last 104.12) - mkdir 25180 (time 1780524500.42 total 224.36 last 101.64) - mkdir 26224 (time 1780524510.43 total 234.37 last 104.28) - mkdir 27256 (time 1780524520.45 total 244.39 last 103.05) - mkdir 28287 (time 1780524530.45 total 254.39 last 103.07) - mkdir 29295 (time 1780524540.45 total 264.39 last 100.77) - mkdir 30000 (time 1780524548.02 total 271.96 last 93.12) - mkdir 31045 (time 1780524558.02 total 281.96 last 104.49) - mkdir 32203 (time 1780524568.03 total 291.97 last 115.70) total: 32818 mkdir in 296.84 seconds: 110.56 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 489856 1718272 23% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 304109 65969 238140 22% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118185 393 117792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 118184 392 117792 1% /mnt/lustre[OST:1] filesystem_summary: 301553 65969 235584 22% /mnt/lustre nlink before: 65638, created before: 65637 - unlinked 0 (time 1780524846 ; total 0 ; last 0) - unlinked 10000 (time 1780524924 ; total 78 ; last 78) - unlinked 20000 (time 1780524998 ; total 152 ; last 74) - unlinked 30000 (time 1780525066 ; total 220 ; last 68) total: 32768 unlinks in 240 seconds: 136.533340 unlinks/second - unlinked 0 (time 1780524846 ; total 0 ; last 0) - unlinked 10000 (time 1780524924 ; total 78 ; last 78) - unlinked 20000 (time 1780524999 ; total 153 ; last 75) - unlinked 30000 (time 1780525069 ; total 223 ; last 70) total: 32768 unlinks in 242 seconds: 135.404953 unlinks/second nlink between: 102 - unlinked 0 (time 1780525089 ; total 0 ; last 0) total: 50 unlinks in 0 seconds: inf unlinks/second - unlinked 0 (time 1780525089 ; total 0 ; last 0) total: 50 unlinks in 0 seconds: inf unlinks/second nlink after: 2 rmdir(/mnt/lustre/d51b.sanity/d.1.0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second rmdir(/mnt/lustre/d51b.sanity/d.0.0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2208768 2176 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 526795 333 526462 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118185 393 117792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 118184 392 117792 1% /mnt/lustre[OST:1] filesystem_summary: 235917 333 235584 1% /mnt/lustre PASS 51b (820s) == sanity test 51f: check many open files limit ========== 18:18:15 (1780525095) ulimit_old=1024 numfree= 526462, max=100000 changed ulimit from 1024 to 100100 - open/keep 2901 (time 1780525106.70 total 10.00 last 290.01) - open/keep 6215 (time 1780525116.71 total 20.01 last 331.31) - open/keep 9614 (time 1780525126.71 total 30.01 last 339.85) - open/keep 12567 (time 1780525136.71 total 40.01 last 295.16) - open/keep 15154 (time 1780525146.71 total 50.01 last 258.65) - open/keep 17806 (time 1780525156.72 total 60.02 last 265.15) - open/keep 20000 (time 1780525163.96 total 67.26 last 302.92) - open/keep 22970 (time 1780525173.96 total 77.26 last 296.90) - open/keep 25796 (time 1780525183.97 total 87.27 last 282.51) - open/keep 28409 (time 1780525193.97 total 97.27 last 261.24) - open/keep 30000 (time 1780525199.80 total 103.10 last 272.64) - open/keep 32753 (time 1780525209.81 total 113.11 last 275.20) total: 34840 open/keep in 120.00 seconds: 290.33 ops/second - closed 4950 (time 1780525226.70 total 10.00 last -2780.04) - closed 10000 (time 1780525234.83 total 18.13 last 620.96) - closed 15595 (time 1780525244.83 total 28.13 last 559.49) - closed 20000 (time 1780525253.86 total 37.16 last 488.22) - closed 24744 (time 1780525263.86 total 47.16 last 474.38) - closed 29840 (time 1780525273.86 total 57.16 last 509.51) - closed 30000 (time 1780525274.16 total 57.46 last 537.00) total: 34840 close in 67.13 seconds: 518.95 close/second - unlinked 0 (time 1780525287 ; total 0 ; last 0) - unlinked 10000 (time 1780525318 ; total 31 ; last 31) - unlinked 20000 (time 1780525346 ; total 59 ; last 28) - unlinked 30000 (time 1780525371 ; total 84 ; last 25) unlink(/mnt/lustre/d51f.sanity/f34840) error: No such file or directory total: 34840 unlinks in 98 seconds: 355.510193 unlinks/second PASS 51f (291s) == sanity test 57b: default LOV EAs are stored inside large inodes ============================================================= 18:23:06 (1780525386) SKIP: sanity test_57b ldiskfs only test SKIP 57b (1s) resend_count is set to 4 4 resend_count is set to 4 4 resend_count is set to 4 4 resend_count is set to 4 4 resend_count is set to 4 4 == sanity test 160i: changelog user register/unregister race ========================================================== 18:23:09 (1780525389) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl1' fail_loc=0x10001315 fail_val=1 lustre-MDT0000: clear the changelog for cl1 of all records mdd.lustre-MDT0000.changelog_mask=+hsm lustre-MDT0000: Deregistered changelog user #1 Registered 1 changelog users: 'cl1 cl2' cl2 3 (0) verify changelogs added: 3->7 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 mdd.lustre-MDT0000.changelog_mask=MARK lustre-MDT0000: changelog user 'cl1' not found mdd.lustre-MDT0000.changelog_mask=MARK PASS 160i (11s) == sanity test 230o: dir split =========================== 18:23:21 (1780525401) SKIP: sanity test_230o needs >= 2 MDTs SKIP 230o (1s) == sanity test 230p: dir merge =========================== 18:23:22 (1780525402) SKIP: sanity test_230p needs >= 2 MDTs SKIP 230p (1s) == sanity test 230q: dir auto split ====================== 18:23:23 (1780525403) SKIP: sanity test_230q needs >= 2 MDTs SKIP 230q (2s) == sanity test 230x: dir migration check space =========== 18:23:25 (1780525405) SKIP: sanity test_230x needs >= 2 MDTs SKIP 230x (1s) == sanity test 270a: DoM: basic functionality tests ====== 18:23:26 (1780525406) 192+0 records in 192+0 records out 196608 bytes (197 kB, 192 KiB) copied, 0.0710889 s, 2.8 MB/s 3+0 records in 3+0 records out 196608 bytes (197 kB, 192 KiB) copied, 0.0571055 s, 3.4 MB/s 1984+0 records in 1984+0 records out 2031616 bytes (2.0 MB, 1.9 MiB) copied, 0.590735 s, 3.4 MB/s 31+0 records in 31+0 records out 2031616 bytes (2.0 MB, 1.9 MiB) copied, 0.46843 s, 4.3 MB/s PASS 270a (6s) == sanity test 272b: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 18:23:32 (1780525412) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0836015 s, 25.1 MB/s PASS 272b (4s) == sanity test 272c: DoM migration: DOM file to the OST-striped file (composite) ========================================================== 18:23:36 (1780525416) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106637 s, 19.7 MB/s PASS 272c (3s) == sanity test 272d: DoM mirroring: OST-striped mirror to DOM file ========================================================== 18:23:39 (1780525419) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118568 s, 17.7 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000401:0x18894:0x0]: Invalid argument (22) PASS 272d (4s) == sanity test 300ug: migrate overstriped dirs =========== 18:23:44 (1780525424) SKIP: sanity test_300ug needs >= 2 MDTs SKIP 300ug (2s) == sanity test 808: Check trusted.som xattr not logged in Changelogs ========================================================== 18:23:47 (1780525427) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl3' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338924 s, 30.9 MB/s lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0000: Deregistered changelog user #3 mdd.lustre-MDT0000.changelog_mask=MARK PASS 808 (5s) == sanity test complete, duration 1189 sec =============== 18:23:52 (1780525432) === sanity: start cleanup 18:23:52 (1780525432) === === sanity: finish cleanup 18:23:54 (1780525434) === debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck