== sanity test 398c: run fio to test AIO ================= 05:14:05 (1780996445) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.422326 s, 99.3 MB/s osc.lustre-OST0000-osc-ffff947f0e687800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=174669: Tue Jun 9 05:17:04 2026 write: IOPS=15, BW=60.5KiB/s (61.9kB/s)(10.0MiB/169264msec); 0 zone resets slat (usec): min=41, max=19131, avg=395.92, stdev=1178.16 clat (msec): min=167, max=1906, avg=1056.41, stdev=251.24 lat (msec): min=168, max=1906, avg=1056.81, stdev=251.22 clat percentiles (msec): | 1.00th=[ 535], 5.00th=[ 676], 10.00th=[ 768], 20.00th=[ 852], | 30.00th=[ 919], 40.00th=[ 986], 50.00th=[ 1036], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1385], 95.00th=[ 1502], | 99.00th=[ 1703], 99.50th=[ 1787], 99.90th=[ 1888], 99.95th=[ 1888], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 127, per=24.91%, avg=60.04, stdev=26.39, samples=335 iops : min= 1, max= 31, avg=14.52, stdev= 6.62, samples=335 lat (msec) : 250=0.16%, 500=0.70%, 750=7.30%, 1000=34.80%, 2000=57.03% cpu : usr=0.05%, sys=0.32%, ctx=2153, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=174670: Tue Jun 9 05:17:04 2026 write: IOPS=15, BW=60.5KiB/s (61.0kB/s)(10.0MiB/169199msec); 0 zone resets slat (usec): min=36, max=14871, avg=410.05, stdev=1087.89 clat (msec): min=115, max=1904, avg=1056.55, stdev=254.87 lat (msec): min=116, max=1905, avg=1056.97, stdev=254.82 clat percentiles (msec): | 1.00th=[ 523], 5.00th=[ 676], 10.00th=[ 760], 20.00th=[ 852], | 30.00th=[ 919], 40.00th=[ 978], 50.00th=[ 1028], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1267], 90.00th=[ 1401], 95.00th=[ 1519], | 99.00th=[ 1670], 99.50th=[ 1770], 99.90th=[ 1888], 99.95th=[ 1888], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 142, per=24.99%, avg=60.24, stdev=26.45, samples=334 iops : min= 1, max= 35, avg=14.57, stdev= 6.63, samples=334 lat (msec) : 250=0.47%, 500=0.39%, 750=7.97%, 1000=35.98%, 2000=55.20% cpu : usr=0.05%, sys=0.30%, ctx=2259, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=174671: Tue Jun 9 05:17:04 2026 write: IOPS=15, BW=60.5KiB/s (61.0kB/s)(10.0MiB/169258msec); 0 zone resets slat (usec): min=35, max=22819, avg=386.86, stdev=1305.10 clat (msec): min=283, max=1945, avg=1056.39, stdev=250.48 lat (msec): min=284, max=1945, avg=1056.78, stdev=250.46 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 659], 10.00th=[ 768], 20.00th=[ 860], | 30.00th=[ 919], 40.00th=[ 986], 50.00th=[ 1028], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1401], 95.00th=[ 1502], | 99.00th=[ 1670], 99.50th=[ 1770], 99.90th=[ 1905], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 7, max= 128, per=25.30%, avg=60.97, stdev=31.43, samples=330 iops : min= 1, max= 32, avg=14.75, stdev= 7.87, samples=330 lat (msec) : 500=0.90%, 750=7.58%, 1000=35.27%, 2000=56.25% cpu : usr=0.05%, sys=0.29%, ctx=1856, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=174672: Tue Jun 9 05:17:04 2026 write: IOPS=15, BW=60.5KiB/s (61.9kB/s)(10.0MiB/169267msec); 0 zone resets slat (usec): min=51, max=16244, avg=334.36, stdev=960.92 clat (msec): min=315, max=1934, avg=1056.72, stdev=249.23 lat (msec): min=318, max=1934, avg=1057.05, stdev=249.24 clat percentiles (msec): | 1.00th=[ 527], 5.00th=[ 676], 10.00th=[ 768], 20.00th=[ 860], | 30.00th=[ 919], 40.00th=[ 978], 50.00th=[ 1028], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1401], 95.00th=[ 1502], | 99.00th=[ 1703], 99.50th=[ 1787], 99.90th=[ 1871], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 7, max= 128, per=25.60%, avg=61.70, stdev=29.74, samples=326 iops : min= 1, max= 32, avg=14.93, stdev= 7.45, samples=326 lat (msec) : 500=0.94%, 750=7.73%, 1000=35.90%, 2000=55.43% cpu : usr=0.06%, sys=0.28%, ctx=1791, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=242KiB/s (248kB/s), 60.5KiB/s-60.5KiB/s (61.9kB/s-61.0kB/s), io=40.0MiB (41.9MB), run=169199-169267msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=174706: Tue Jun 9 05:19:25 2026 read: IOPS=9, BW=38.4KiB/s (39.3kB/s)(5048KiB/131387msec) slat (usec): min=3010, max=68736, avg=18002.15, stdev=9153.87 clat (msec): min=24, max=999, avg=503.75, stdev=124.24 lat (msec): min=35, max=1019, avg=521.76, stdev=125.17 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 338], 10.00th=[ 372], 20.00th=[ 405], | 30.00th=[ 435], 40.00th=[ 464], 50.00th=[ 485], 60.00th=[ 518], | 70.00th=[ 550], 80.00th=[ 600], 90.00th=[ 676], 95.00th=[ 743], | 99.00th=[ 827], 99.50th=[ 844], 99.90th=[ 885], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 7, max= 95, per=25.20%, avg=38.31, stdev=17.90, samples=257 iops : min= 1, max= 23, avg= 9.01, stdev= 4.50, samples=257 write: IOPS=9, BW=39.5KiB/s (40.5kB/s)(5192KiB/131387msec); 0 zone resets slat (usec): min=53, max=31488, avg=606.49, stdev=1465.78 clat (msec): min=230, max=2056, avg=1108.92, stdev=268.88 lat (msec): min=230, max=2057, avg=1109.53, stdev=268.87 clat percentiles (msec): | 1.00th=[ 550], 5.00th=[ 709], 10.00th=[ 793], 20.00th=[ 877], | 30.00th=[ 944], 40.00th=[ 1020], 50.00th=[ 1083], 60.00th=[ 1167], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1469], 95.00th=[ 1603], | 99.00th=[ 1787], 99.50th=[ 1854], 99.90th=[ 2039], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 6, max= 102, per=24.67%, avg=38.98, stdev=17.69, samples=259 iops : min= 1, max= 25, avg= 9.18, stdev= 4.46, samples=259 lat (msec) : 50=0.12%, 100=0.23%, 250=0.27%, 500=26.48%, 750=23.79% lat (msec) : 1000=17.15%, 2000=31.88%, >=2000=0.08% cpu : usr=0.12%, sys=0.83%, ctx=4887, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=174707: Tue Jun 9 05:19:25 2026 read: IOPS=9, BW=37.8KiB/s (38.7kB/s)(4976KiB/131781msec) slat (usec): min=3614, max=62898, avg=18296.04, stdev=9378.57 clat (msec): min=34, max=1038, avg=506.08, stdev=125.32 lat (msec): min=47, max=1047, avg=524.40, stdev=126.62 clat percentiles (msec): | 1.00th=[ 249], 5.00th=[ 334], 10.00th=[ 363], 20.00th=[ 405], | 30.00th=[ 443], 40.00th=[ 464], 50.00th=[ 493], 60.00th=[ 523], | 70.00th=[ 558], 80.00th=[ 609], 90.00th=[ 667], 95.00th=[ 726], | 99.00th=[ 827], 99.50th=[ 877], 99.90th=[ 1003], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 6, max= 112, per=24.74%, avg=37.60, stdev=18.98, samples=256 iops : min= 1, max= 28, avg= 8.84, stdev= 4.78, samples=256 write: IOPS=9, BW=39.9KiB/s (40.9kB/s)(5264KiB/131781msec); 0 zone resets slat (usec): min=53, max=15579, avg=827.85, stdev=1813.00 clat (msec): min=271, max=2034, avg=1103.06, stdev=276.32 lat (msec): min=280, max=2034, avg=1103.89, stdev=276.24 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 676], 10.00th=[ 768], 20.00th=[ 860], | 30.00th=[ 936], 40.00th=[ 1020], 50.00th=[ 1083], 60.00th=[ 1167], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1469], 95.00th=[ 1569], | 99.00th=[ 1854], 99.50th=[ 1972], 99.90th=[ 2022], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 6, max= 103, per=25.14%, avg=39.72, stdev=19.70, samples=258 iops : min= 1, max= 25, avg= 9.36, stdev= 4.94, samples=258 lat (msec) : 50=0.12%, 100=0.20%, 250=0.20%, 500=25.23%, 750=25.86% lat (msec) : 1000=15.86%, 2000=32.38%, >=2000=0.16% cpu : usr=0.13%, sys=0.88%, ctx=5077, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=174708: Tue Jun 9 05:19:25 2026 read: IOPS=9, BW=37.8KiB/s (38.7kB/s)(4984KiB/131830msec) slat (usec): min=3873, max=72623, avg=17993.07, stdev=9477.79 clat (msec): min=17, max=963, avg=503.23, stdev=119.34 lat (msec): min=29, max=977, avg=521.24, stdev=120.83 clat percentiles (msec): | 1.00th=[ 264], 5.00th=[ 334], 10.00th=[ 376], 20.00th=[ 409], | 30.00th=[ 439], 40.00th=[ 464], 50.00th=[ 489], 60.00th=[ 518], | 70.00th=[ 550], 80.00th=[ 600], 90.00th=[ 659], 95.00th=[ 726], | 99.00th=[ 818], 99.50th=[ 860], 99.90th=[ 944], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 7, max= 119, per=25.15%, avg=38.22, stdev=19.68, samples=254 iops : min= 1, max= 29, avg= 8.98, stdev= 4.95, samples=254 write: IOPS=9, BW=39.9KiB/s (40.8kB/s)(5256KiB/131830msec); 0 zone resets slat (usec): min=57, max=18914, avg=719.88, stdev=1617.24 clat (msec): min=263, max=2084, avg=1106.88, stdev=271.04 lat (msec): min=263, max=2085, avg=1107.61, stdev=271.05 clat percentiles (msec): | 1.00th=[ 498], 5.00th=[ 701], 10.00th=[ 785], 20.00th=[ 869], | 30.00th=[ 936], 40.00th=[ 1020], 50.00th=[ 1099], 60.00th=[ 1167], | 70.00th=[ 1250], 80.00th=[ 1334], 90.00th=[ 1469], 95.00th=[ 1552], | 99.00th=[ 1737], 99.50th=[ 1854], 99.90th=[ 2039], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 103, per=24.92%, avg=39.38, stdev=17.86, samples=259 iops : min= 1, max= 25, avg= 9.28, stdev= 4.49, samples=259 lat (msec) : 20=0.04%, 50=0.04%, 100=0.04%, 250=0.20%, 500=26.84% lat (msec) : 750=23.44%, 1000=16.99%, 2000=32.30%, >=2000=0.12% cpu : usr=0.13%, sys=0.86%, ctx=5045, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=174709: Tue Jun 9 05:19:25 2026 read: IOPS=9, BW=38.5KiB/s (39.5kB/s)(5060KiB/131285msec) slat (usec): min=59, max=81229, avg=18034.77, stdev=9580.56 clat (msec): min=25, max=989, avg=506.47, stdev=128.10 lat (msec): min=35, max=1014, avg=524.52, stdev=129.37 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 330], 10.00th=[ 372], 20.00th=[ 414], | 30.00th=[ 443], 40.00th=[ 464], 50.00th=[ 498], 60.00th=[ 523], | 70.00th=[ 558], 80.00th=[ 600], 90.00th=[ 667], 95.00th=[ 743], | 99.00th=[ 860], 99.50th=[ 894], 99.90th=[ 978], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 7, max= 95, per=25.14%, avg=38.21, stdev=18.37, samples=258 iops : min= 1, max= 23, avg= 8.99, stdev= 4.61, samples=258 write: IOPS=9, BW=39.5KiB/s (40.4kB/s)(5180KiB/131285msec); 0 zone resets slat (usec): min=50, max=20733, avg=615.66, stdev=1452.12 clat (msec): min=124, max=2039, avg=1107.90, stdev=271.41 lat (msec): min=124, max=2040, avg=1108.52, stdev=271.44 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 718], 10.00th=[ 793], 20.00th=[ 869], | 30.00th=[ 944], 40.00th=[ 1036], 50.00th=[ 1116], 60.00th=[ 1167], | 70.00th=[ 1250], 80.00th=[ 1334], 90.00th=[ 1452], 95.00th=[ 1569], | 99.00th=[ 1754], 99.50th=[ 1921], 99.90th=[ 1955], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 88, per=24.52%, avg=38.74, stdev=16.58, samples=260 iops : min= 1, max= 22, avg= 9.12, stdev= 4.20, samples=260 lat (msec) : 50=0.12%, 100=0.16%, 250=0.66%, 500=25.16%, 750=24.65% lat (msec) : 1000=16.91%, 2000=32.30%, >=2000=0.04% cpu : usr=0.12%, sys=0.84%, ctx=4776, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=152KiB/s (156kB/s), 37.8KiB/s-38.5KiB/s (38.7kB/s-39.5kB/s), io=19.6MiB (20.5MB), run=131285-131830msec WRITE: bw=158KiB/s (162kB/s), 39.5KiB/s-39.9KiB/s (40.4kB/s-40.9kB/s), io=20.4MiB (21.4MB), run=131285-131830msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=174732: Tue Jun 9 05:19:36 2026 read: IOPS=2, BW=89.1MiB/s (93.4MB/s)(40.0MiB/449msec) slat (nsec): min=32755k, max=32755k, avg=32754581.00, stdev= 0.00 clat (nsec): min=398788k, max=398788k, avg=398787923.00, stdev= 0.00 lat (nsec): min=431547k, max=431547k, avg=431546545.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 401], 10.00th=[ 401], 20.00th=[ 401], | 30.00th=[ 401], 40.00th=[ 401], 50.00th=[ 401], 60.00th=[ 401], | 70.00th=[ 401], 80.00th=[ 401], 90.00th=[ 401], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.46%, ctx=37, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=89.1MiB/s (93.4MB/s), 89.1MiB/s-89.1MiB/s (93.4MB/s-93.4MB/s), io=40.0MiB (41.9MB), run=449-449msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout