== sanity test 398c: run fio to test AIO ================= 20:46:33 (1763343993) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.469631 s, 89.3 MB/s osc.lustre-OST0000-osc-ffff90f667f6c000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147168: Sun Nov 16 20:49:31 2025 write: IOPS=15, BW=60.3KiB/s (61.7kB/s)(10.0MiB/169895msec); 0 zone resets slat (usec): min=27, max=11017, avg=221.83, stdev=581.83 clat (msec): min=101, max=2225, avg=1061.08, stdev=237.54 lat (msec): min=101, max=2225, avg=1061.31, stdev=237.54 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 701], 10.00th=[ 802], 20.00th=[ 894], | 30.00th=[ 961], 40.00th=[ 1020], 50.00th=[ 1070], 60.00th=[ 1116], | 70.00th=[ 1167], 80.00th=[ 1234], 90.00th=[ 1334], 95.00th=[ 1435], | 99.00th=[ 1603], 99.50th=[ 1720], 99.90th=[ 2140], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 8, max= 183, per=24.96%, avg=60.15, stdev=24.35, samples=333 iops : min= 2, max= 45, avg=14.51, stdev= 6.09, samples=333 lat (msec) : 250=0.35%, 500=2.15%, 750=4.57%, 1000=29.61%, 2000=63.16% lat (msec) : >=2000=0.16% cpu : usr=0.04%, sys=0.29%, ctx=1827, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147169: Sun Nov 16 20:49:31 2025 write: IOPS=15, BW=60.3KiB/s (61.7kB/s)(10.0MiB/169859msec); 0 zone resets slat (usec): min=27, max=9935, avg=218.43, stdev=553.83 clat (msec): min=153, max=2037, avg=1060.46, stdev=232.30 lat (msec): min=155, max=2040, avg=1060.68, stdev=232.31 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 701], 10.00th=[ 802], 20.00th=[ 885], | 30.00th=[ 953], 40.00th=[ 1020], 50.00th=[ 1070], 60.00th=[ 1116], | 70.00th=[ 1183], 80.00th=[ 1250], 90.00th=[ 1334], 95.00th=[ 1418], | 99.00th=[ 1586], 99.50th=[ 1636], 99.90th=[ 1720], 99.95th=[ 1720], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 178, per=24.88%, avg=59.96, stdev=24.38, samples=334 iops : min= 1, max= 44, avg=14.47, stdev= 6.10, samples=334 lat (msec) : 250=0.94%, 500=1.13%, 750=4.65%, 1000=30.74%, 2000=62.50% lat (msec) : >=2000=0.04% cpu : usr=0.05%, sys=0.30%, ctx=1773, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147170: Sun Nov 16 20:49:31 2025 write: IOPS=15, BW=60.3KiB/s (61.7kB/s)(10.0MiB/169857msec); 0 zone resets slat (usec): min=27, max=9283, avg=208.70, stdev=613.25 clat (msec): min=224, max=2309, avg=1060.35, stdev=238.17 lat (msec): min=224, max=2309, avg=1060.56, stdev=238.18 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 684], 10.00th=[ 793], 20.00th=[ 894], | 30.00th=[ 961], 40.00th=[ 1011], 50.00th=[ 1062], 60.00th=[ 1116], | 70.00th=[ 1183], 80.00th=[ 1250], 90.00th=[ 1334], 95.00th=[ 1418], | 99.00th=[ 1603], 99.50th=[ 1703], 99.90th=[ 2198], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 173, per=24.89%, avg=59.99, stdev=30.59, samples=334 iops : min= 1, max= 43, avg=14.47, stdev= 7.67, samples=334 lat (msec) : 250=0.62%, 500=1.91%, 750=5.08%, 1000=30.35%, 2000=61.84% lat (msec) : >=2000=0.20% cpu : usr=0.05%, sys=0.28%, ctx=1695, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147171: Sun Nov 16 20:49:31 2025 write: IOPS=15, BW=60.3KiB/s (61.7kB/s)(10.0MiB/169883msec); 0 zone resets slat (usec): min=26, max=13532, avg=250.76, stdev=725.68 clat (msec): min=233, max=1889, avg=1060.68, stdev=228.09 lat (msec): min=234, max=1889, avg=1060.93, stdev=228.08 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 709], 10.00th=[ 802], 20.00th=[ 894], | 30.00th=[ 961], 40.00th=[ 1020], 50.00th=[ 1070], 60.00th=[ 1116], | 70.00th=[ 1183], 80.00th=[ 1234], 90.00th=[ 1334], 95.00th=[ 1401], | 99.00th=[ 1569], 99.50th=[ 1636], 99.90th=[ 1888], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 7, max= 176, per=24.74%, avg=59.62, stdev=26.41, samples=336 iops : min= 1, max= 44, avg=14.39, stdev= 6.60, samples=336 lat (msec) : 250=0.62%, 500=1.52%, 750=5.08%, 1000=29.73%, 2000=63.05% cpu : usr=0.06%, sys=0.30%, ctx=2003, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=241KiB/s (247kB/s), 60.3KiB/s-60.3KiB/s (61.7kB/s-61.7kB/s), io=40.0MiB (41.9MB), run=169857-169895msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147206: Sun Nov 16 20:51:33 2025 read: IOPS=11, BW=44.4KiB/s (45.5kB/s)(5048KiB/113631msec) slat (usec): min=3016, max=79540, avg=13980.85, stdev=6224.75 clat (msec): min=21, max=803, avg=447.55, stdev=95.35 lat (msec): min=33, max=820, avg=461.54, stdev=95.85 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 313], 10.00th=[ 338], 20.00th=[ 376], | 30.00th=[ 401], 40.00th=[ 426], 50.00th=[ 447], 60.00th=[ 468], | 70.00th=[ 493], 80.00th=[ 518], 90.00th=[ 558], 95.00th=[ 609], | 99.00th=[ 684], 99.50th=[ 735], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 7, max= 104, per=24.94%, avg=43.89, stdev=17.95, samples=224 iops : min= 1, max= 26, avg=10.45, stdev= 4.50, samples=224 write: IOPS=11, BW=45.7KiB/s (46.8kB/s)(5192KiB/113631msec); 0 zone resets slat (usec): min=30, max=15936, avg=324.37, stdev=895.00 clat (msec): min=204, max=1535, avg=948.89, stdev=220.13 lat (msec): min=205, max=1535, avg=949.22, stdev=220.09 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 584], 10.00th=[ 667], 20.00th=[ 751], | 30.00th=[ 835], 40.00th=[ 894], 50.00th=[ 953], 60.00th=[ 1003], | 70.00th=[ 1053], 80.00th=[ 1133], 90.00th=[ 1234], 95.00th=[ 1334], | 99.00th=[ 1452], 99.50th=[ 1469], 99.90th=[ 1469], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 6, max= 111, per=24.61%, avg=45.04, stdev=19.70, samples=225 iops : min= 1, max= 27, avg=10.74, stdev= 4.94, samples=225 lat (msec) : 50=0.12%, 100=0.20%, 250=0.59%, 500=36.13%, 750=22.03% lat (msec) : 1000=20.35%, 2000=20.59% cpu : usr=0.12%, sys=0.73%, ctx=4828, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147207: Sun Nov 16 20:51:33 2025 read: IOPS=10, BW=43.8KiB/s (44.9kB/s)(4976KiB/113480msec) slat (usec): min=3043, max=43946, avg=14039.68, stdev=5941.77 clat (msec): min=25, max=1114, avg=446.47, stdev=104.38 lat (msec): min=34, max=1126, avg=460.52, stdev=104.76 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 300], 10.00th=[ 330], 20.00th=[ 372], | 30.00th=[ 397], 40.00th=[ 422], 50.00th=[ 443], 60.00th=[ 460], | 70.00th=[ 485], 80.00th=[ 523], 90.00th=[ 567], 95.00th=[ 609], | 99.00th=[ 718], 99.50th=[ 852], 99.90th=[ 1083], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 119, per=24.63%, avg=43.35, stdev=19.47, samples=224 iops : min= 1, max= 29, avg=10.31, stdev= 4.88, samples=224 write: IOPS=11, BW=46.4KiB/s (47.5kB/s)(5264KiB/113480msec); 0 zone resets slat (usec): min=32, max=16671, avg=312.60, stdev=811.96 clat (msec): min=148, max=1589, avg=941.71, stdev=220.80 lat (msec): min=148, max=1589, avg=942.03, stdev=220.77 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 567], 10.00th=[ 651], 20.00th=[ 751], | 30.00th=[ 827], 40.00th=[ 894], 50.00th=[ 944], 60.00th=[ 1003], | 70.00th=[ 1053], 80.00th=[ 1116], 90.00th=[ 1234], 95.00th=[ 1334], | 99.00th=[ 1418], 99.50th=[ 1452], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 7, max= 111, per=25.14%, avg=46.00, stdev=19.12, samples=224 iops : min= 1, max= 27, avg=10.97, stdev= 4.82, samples=224 lat (msec) : 50=0.04%, 100=0.31%, 250=0.51%, 500=36.13%, 750=21.29% lat (msec) : 1000=20.90%, 2000=20.82% cpu : usr=0.10%, sys=0.78%, ctx=5043, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147208: Sun Nov 16 20:51:33 2025 read: IOPS=10, BW=43.9KiB/s (44.0kB/s)(4984KiB/113481msec) slat (usec): min=30, max=80793, avg=13886.27, stdev=6187.87 clat (msec): min=10, max=884, avg=445.30, stdev=99.73 lat (msec): min=24, max=902, avg=459.20, stdev=100.34 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 296], 10.00th=[ 330], 20.00th=[ 372], | 30.00th=[ 401], 40.00th=[ 422], 50.00th=[ 443], 60.00th=[ 464], | 70.00th=[ 489], 80.00th=[ 523], 90.00th=[ 567], 95.00th=[ 600], | 99.00th=[ 693], 99.50th=[ 709], 99.90th=[ 869], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 7, max= 104, per=24.89%, avg=43.81, stdev=20.37, samples=223 iops : min= 1, max= 26, avg=10.43, stdev= 5.13, samples=223 write: IOPS=11, BW=46.3KiB/s (47.4kB/s)(5256KiB/113481msec); 0 zone resets slat (usec): min=33, max=13100, avg=300.18, stdev=812.59 clat (msec): min=179, max=1582, avg=945.24, stdev=219.30 lat (msec): min=179, max=1583, avg=945.54, stdev=219.27 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 575], 10.00th=[ 676], 20.00th=[ 776], | 30.00th=[ 835], 40.00th=[ 894], 50.00th=[ 936], 60.00th=[ 995], | 70.00th=[ 1045], 80.00th=[ 1116], 90.00th=[ 1250], 95.00th=[ 1334], | 99.00th=[ 1435], 99.50th=[ 1452], 99.90th=[ 1519], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 7, max= 119, per=25.09%, avg=45.91, stdev=18.71, samples=223 iops : min= 1, max= 29, avg=10.95, stdev= 4.70, samples=223 lat (msec) : 20=0.04%, 100=0.35%, 250=1.17%, 500=35.12%, 750=20.43% lat (msec) : 1000=23.32%, 2000=19.57% cpu : usr=0.08%, sys=0.75%, ctx=4913, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147209: Sun Nov 16 20:51:33 2025 read: IOPS=11, BW=44.9KiB/s (45.9kB/s)(5060KiB/112815msec) slat (usec): min=3684, max=79573, avg=14001.79, stdev=6240.06 clat (msec): min=20, max=865, avg=446.05, stdev=100.92 lat (msec): min=37, max=891, avg=460.07, stdev=101.45 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 305], 10.00th=[ 334], 20.00th=[ 372], | 30.00th=[ 401], 40.00th=[ 422], 50.00th=[ 443], 60.00th=[ 464], | 70.00th=[ 489], 80.00th=[ 518], 90.00th=[ 567], 95.00th=[ 617], | 99.00th=[ 693], 99.50th=[ 709], 99.90th=[ 852], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 7, max= 112, per=25.42%, avg=44.74, stdev=19.45, samples=221 iops : min= 1, max= 28, avg=10.67, stdev= 4.88, samples=221 write: IOPS=11, BW=45.9KiB/s (47.0kB/s)(5180KiB/112815msec); 0 zone resets slat (usec): min=29, max=12349, avg=296.99, stdev=757.47 clat (msec): min=191, max=1637, avg=941.75, stdev=217.95 lat (msec): min=191, max=1638, avg=942.04, stdev=217.93 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 592], 10.00th=[ 667], 20.00th=[ 760], | 30.00th=[ 827], 40.00th=[ 885], 50.00th=[ 936], 60.00th=[ 995], | 70.00th=[ 1053], 80.00th=[ 1116], 90.00th=[ 1234], 95.00th=[ 1334], | 99.00th=[ 1418], 99.50th=[ 1469], 99.90th=[ 1536], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 7, max= 103, per=24.83%, avg=45.43, stdev=17.89, samples=223 iops : min= 1, max= 25, avg=10.84, stdev= 4.49, samples=223 lat (msec) : 50=0.08%, 100=0.51%, 250=0.31%, 500=36.09%, 750=21.95% lat (msec) : 1000=21.45%, 2000=19.61% cpu : usr=0.09%, sys=0.76%, ctx=4746, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=177KiB/s (181kB/s), 43.8KiB/s-44.9KiB/s (44.9kB/s-45.9kB/s), io=19.6MiB (20.5MB), run=112815-113631msec WRITE: bw=184KiB/s (188kB/s), 45.7KiB/s-46.4KiB/s (46.8kB/s-47.5kB/s), io=20.4MiB (21.4MB), run=112815-113631msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147228: Sun Nov 16 20:51:40 2025 read: IOPS=1, BW=66.8MiB/s (70.0MB/s)(40.0MiB/599msec) slat (nsec): min=23620k, max=23620k, avg=23620170.00, stdev= 0.00 clat (nsec): min=555564k, max=555564k, avg=555564149.00, stdev= 0.00 lat (nsec): min=579188k, max=579188k, avg=579187941.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 558], 5.00th=[ 558], 10.00th=[ 558], 20.00th=[ 558], | 30.00th=[ 558], 40.00th=[ 558], 50.00th=[ 558], 60.00th=[ 558], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.51%, ctx=43, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=66.8MiB/s (70.0MB/s), 66.8MiB/s-66.8MiB/s (70.0MB/s-70.0MB/s), io=40.0MiB (41.9MB), run=599-599msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout