== sanity test 398c: run fio to test AIO ================= 22:25:19 (1789611919) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.396233 s, 106 MB/s osc.lustre-OST0000-osc-ffffa098a06cd000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145929: Wed Sep 16 22:27:46 2026 write: IOPS=18, BW=74.0KiB/s (75.8kB/s)(10.0MiB/138290msec); 0 zone resets slat (usec): min=23, max=14799, avg=185.95, stdev=706.96 clat (msec): min=70, max=1911, avg=863.51, stdev=236.80 lat (msec): min=70, max=1912, avg=863.70, stdev=236.82 clat percentiles (msec): | 1.00th=[ 262], 5.00th=[ 502], 10.00th=[ 584], 20.00th=[ 676], | 30.00th=[ 751], 40.00th=[ 818], 50.00th=[ 860], 60.00th=[ 911], | 70.00th=[ 961], 80.00th=[ 1045], 90.00th=[ 1150], 95.00th=[ 1234], | 99.00th=[ 1586], 99.50th=[ 1670], 99.90th=[ 1804], 99.95th=[ 1804], | 99.99th=[ 1905] bw ( KiB/s): min= 16, max= 262, per=25.13%, avg=74.13, stdev=30.34, samples=272 iops : min= 4, max= 65, avg=18.09, stdev= 7.62, samples=272 lat (msec) : 100=0.12%, 250=0.12%, 500=4.45%, 750=25.12%, 1000=44.41% lat (msec) : 2000=25.78% cpu : usr=0.05%, sys=0.25%, ctx=1730, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145930: Wed Sep 16 22:27:46 2026 write: IOPS=18, BW=73.9KiB/s (75.7kB/s)(10.0MiB/138542msec); 0 zone resets slat (usec): min=21, max=14621, avg=152.82, stdev=566.77 clat (msec): min=212, max=1757, avg=864.43, stdev=228.01 lat (msec): min=212, max=1758, avg=864.58, stdev=228.04 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 514], 10.00th=[ 600], 20.00th=[ 684], | 30.00th=[ 751], 40.00th=[ 810], 50.00th=[ 860], 60.00th=[ 919], | 70.00th=[ 969], 80.00th=[ 1045], 90.00th=[ 1150], 95.00th=[ 1234], | 99.00th=[ 1401], 99.50th=[ 1552], 99.90th=[ 1687], 99.95th=[ 1720], | 99.99th=[ 1754] bw ( KiB/s): min= 7, max= 256, per=25.14%, avg=74.15, stdev=32.92, samples=272 iops : min= 1, max= 64, avg=18.09, stdev= 8.28, samples=272 lat (msec) : 250=0.47%, 500=3.67%, 750=25.04%, 1000=44.73%, 2000=26.09% cpu : usr=0.05%, sys=0.21%, ctx=1574, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145931: Wed Sep 16 22:27:46 2026 write: IOPS=18, BW=73.0KiB/s (75.8kB/s)(10.0MiB/138396msec); 0 zone resets slat (usec): min=23, max=11106, avg=175.25, stdev=629.75 clat (msec): min=249, max=1719, avg=863.34, stdev=227.91 lat (msec): min=249, max=1719, avg=863.52, stdev=227.95 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 531], 10.00th=[ 600], 20.00th=[ 667], | 30.00th=[ 751], 40.00th=[ 810], 50.00th=[ 869], 60.00th=[ 911], | 70.00th=[ 961], 80.00th=[ 1036], 90.00th=[ 1133], 95.00th=[ 1234], | 99.00th=[ 1569], 99.50th=[ 1653], 99.90th=[ 1670], 99.95th=[ 1687], | 99.99th=[ 1720] bw ( KiB/s): min= 7, max= 254, per=25.41%, avg=74.97, stdev=33.71, samples=269 iops : min= 1, max= 63, avg=18.30, stdev= 8.45, samples=269 lat (msec) : 250=0.16%, 500=3.79%, 750=25.55%, 1000=45.39%, 2000=25.12% cpu : usr=0.05%, sys=0.22%, ctx=1769, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145932: Wed Sep 16 22:27:46 2026 write: IOPS=18, BW=74.1KiB/s (75.9kB/s)(10.0MiB/138218msec); 0 zone resets slat (usec): min=22, max=17902, avg=188.50, stdev=690.89 clat (msec): min=264, max=1648, avg=862.32, stdev=233.14 lat (msec): min=265, max=1648, avg=862.51, stdev=233.14 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 489], 10.00th=[ 592], 20.00th=[ 676], | 30.00th=[ 760], 40.00th=[ 810], 50.00th=[ 860], 60.00th=[ 911], | 70.00th=[ 969], 80.00th=[ 1036], 90.00th=[ 1167], 95.00th=[ 1250], | 99.00th=[ 1469], 99.50th=[ 1569], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1653] bw ( KiB/s): min= 7, max= 256, per=25.14%, avg=74.15, stdev=35.70, samples=272 iops : min= 1, max= 64, avg=18.10, stdev= 8.95, samples=272 lat (msec) : 500=5.35%, 750=23.71%, 1000=45.82%, 2000=25.12% cpu : usr=0.04%, sys=0.25%, ctx=1724, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=296KiB/s (303kB/s), 73.9KiB/s-74.1KiB/s (75.7kB/s-75.9kB/s), io=40.0MiB (41.9MB), run=138218-138542msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145962: Wed Sep 16 22:29:25 2026 read: IOPS=14, BW=56.6KiB/s (57.0kB/s)(5048KiB/89185msec) slat (usec): min=46, max=48452, avg=12757.29, stdev=7007.07 clat (msec): min=6, max=1048, avg=349.93, stdev=105.33 lat (msec): min=25, max=1085, avg=362.71, stdev=106.42 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 226], 10.00th=[ 249], 20.00th=[ 275], | 30.00th=[ 300], 40.00th=[ 321], 50.00th=[ 342], 60.00th=[ 359], | 70.00th=[ 384], 80.00th=[ 409], 90.00th=[ 456], 95.00th=[ 523], | 99.00th=[ 726], 99.50th=[ 877], 99.90th=[ 995], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 7, max= 143, per=25.10%, avg=55.98, stdev=24.20, samples=177 iops : min= 1, max= 35, avg=13.53, stdev= 6.08, samples=177 write: IOPS=14, BW=58.2KiB/s (59.6kB/s)(5192KiB/89185msec); 0 zone resets slat (usec): min=25, max=13456, avg=213.90, stdev=611.12 clat (msec): min=172, max=1415, avg=745.22, stdev=211.30 lat (msec): min=173, max=1415, avg=745.43, stdev=211.30 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 435], 10.00th=[ 493], 20.00th=[ 575], | 30.00th=[ 625], 40.00th=[ 659], 50.00th=[ 718], 60.00th=[ 785], | 70.00th=[ 844], 80.00th=[ 936], 90.00th=[ 1028], 95.00th=[ 1133], | 99.00th=[ 1267], 99.50th=[ 1301], 99.90th=[ 1401], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 15, max= 135, per=25.14%, avg=58.31, stdev=23.10, samples=175 iops : min= 3, max= 33, avg=14.11, stdev= 5.79, samples=175 lat (msec) : 10=0.04%, 20=0.04%, 50=0.35%, 100=0.31%, 250=4.92% lat (msec) : 500=46.56%, 750=24.96%, 1000=16.68%, 2000=6.13% cpu : usr=0.11%, sys=0.65%, ctx=4518, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145963: Wed Sep 16 22:29:25 2026 read: IOPS=13, BW=55.4KiB/s (56.7kB/s)(4976KiB/89861msec) slat (usec): min=2481, max=51704, avg=12836.07, stdev=6988.96 clat (msec): min=33, max=838, avg=344.58, stdev=86.77 lat (msec): min=37, max=852, avg=357.43, stdev=87.66 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 226], 10.00th=[ 249], 20.00th=[ 275], | 30.00th=[ 300], 40.00th=[ 317], 50.00th=[ 338], 60.00th=[ 359], | 70.00th=[ 380], 80.00th=[ 409], 90.00th=[ 451], 95.00th=[ 485], | 99.00th=[ 600], 99.50th=[ 651], 99.90th=[ 827], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 7, max= 127, per=24.38%, avg=54.38, stdev=24.41, samples=178 iops : min= 1, max= 31, avg=13.13, stdev= 6.10, samples=178 write: IOPS=14, BW=58.6KiB/s (59.0kB/s)(5264KiB/89861msec); 0 zone resets slat (usec): min=23, max=24270, avg=272.25, stdev=997.92 clat (msec): min=93, max=1398, avg=751.96, stdev=214.79 lat (msec): min=94, max=1398, avg=752.23, stdev=214.79 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 430], 10.00th=[ 498], 20.00th=[ 584], | 30.00th=[ 634], 40.00th=[ 684], 50.00th=[ 726], 60.00th=[ 776], | 70.00th=[ 844], 80.00th=[ 927], 90.00th=[ 1045], 95.00th=[ 1167], | 99.00th=[ 1284], 99.50th=[ 1318], 99.90th=[ 1385], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 8, max= 127, per=24.86%, avg=57.68, stdev=22.29, samples=178 iops : min= 2, max= 31, avg=13.96, stdev= 5.60, samples=178 lat (msec) : 50=0.12%, 100=0.27%, 250=5.04%, 500=46.60%, 750=24.69% lat (msec) : 1000=16.29%, 2000=6.99% cpu : usr=0.13%, sys=0.73%, ctx=4767, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145964: Wed Sep 16 22:29:25 2026 read: IOPS=13, BW=55.5KiB/s (56.9kB/s)(4984KiB/89764msec) slat (usec): min=2070, max=43850, avg=12773.35, stdev=6884.77 clat (msec): min=12, max=877, avg=346.19, stdev=88.01 lat (msec): min=20, max=890, avg=358.97, stdev=88.82 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 226], 10.00th=[ 249], 20.00th=[ 279], | 30.00th=[ 300], 40.00th=[ 321], 50.00th=[ 338], 60.00th=[ 359], | 70.00th=[ 380], 80.00th=[ 414], 90.00th=[ 451], 95.00th=[ 489], | 99.00th=[ 600], 99.50th=[ 634], 99.90th=[ 785], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 7, max= 128, per=24.94%, avg=55.62, stdev=26.18, samples=176 iops : min= 1, max= 32, avg=13.44, stdev= 6.57, samples=176 write: IOPS=14, BW=58.6KiB/s (59.0kB/s)(5256KiB/89764msec); 0 zone resets slat (usec): min=26, max=9489, avg=200.94, stdev=507.49 clat (msec): min=226, max=1390, avg=749.31, stdev=218.54 lat (msec): min=226, max=1390, avg=749.52, stdev=218.53 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 409], 10.00th=[ 489], 20.00th=[ 575], | 30.00th=[ 634], 40.00th=[ 667], 50.00th=[ 718], 60.00th=[ 776], | 70.00th=[ 852], 80.00th=[ 944], 90.00th=[ 1053], 95.00th=[ 1150], | 99.00th=[ 1284], 99.50th=[ 1318], 99.90th=[ 1385], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 7, max= 128, per=24.76%, avg=57.44, stdev=23.19, samples=178 iops : min= 1, max= 32, avg=13.90, stdev= 5.79, samples=178 lat (msec) : 20=0.04%, 50=0.23%, 100=0.08%, 250=5.31%, 500=46.76% lat (msec) : 750=24.96%, 1000=15.04%, 2000=7.58% cpu : usr=0.11%, sys=0.70%, ctx=4360, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145965: Wed Sep 16 22:29:25 2026 read: IOPS=14, BW=56.7KiB/s (58.0kB/s)(5060KiB/89274msec) slat (usec): min=2366, max=47976, avg=12602.39, stdev=6940.45 clat (msec): min=28, max=1201, avg=351.80, stdev=97.13 lat (msec): min=33, max=1206, avg=364.41, stdev=97.97 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 234], 10.00th=[ 255], 20.00th=[ 284], | 30.00th=[ 305], 40.00th=[ 321], 50.00th=[ 342], 60.00th=[ 359], | 70.00th=[ 384], 80.00th=[ 414], 90.00th=[ 456], 95.00th=[ 489], | 99.00th=[ 634], 99.50th=[ 802], 99.90th=[ 1167], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 7, max= 112, per=25.47%, avg=56.80, stdev=21.60, samples=175 iops : min= 1, max= 28, avg=13.74, stdev= 5.38, samples=175 write: IOPS=14, BW=58.0KiB/s (59.4kB/s)(5180KiB/89274msec); 0 zone resets slat (usec): min=31, max=14510, avg=236.38, stdev=715.45 clat (msec): min=192, max=1416, avg=743.94, stdev=215.07 lat (msec): min=192, max=1416, avg=744.18, stdev=215.06 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 426], 10.00th=[ 493], 20.00th=[ 567], | 30.00th=[ 617], 40.00th=[ 659], 50.00th=[ 701], 60.00th=[ 768], | 70.00th=[ 852], 80.00th=[ 919], 90.00th=[ 1045], 95.00th=[ 1150], | 99.00th=[ 1267], 99.50th=[ 1318], 99.90th=[ 1401], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 7, max= 159, per=24.79%, avg=57.52, stdev=22.80, samples=177 iops : min= 1, max= 39, avg=13.92, stdev= 5.71, samples=177 lat (msec) : 50=0.12%, 100=0.20%, 250=4.10%, 500=48.44%, 750=25.23% lat (msec) : 1000=15.16%, 2000=6.76% cpu : usr=0.11%, sys=0.66%, ctx=4595, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=223KiB/s (229kB/s), 55.4KiB/s-56.7KiB/s (56.7kB/s-58.0kB/s), io=19.6MiB (20.5MB), run=89185-89861msec WRITE: bw=232KiB/s (238kB/s), 58.0KiB/s-58.6KiB/s (59.4kB/s-59.0kB/s), io=20.4MiB (21.4MB), run=89185-89861msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145982: Wed Sep 16 22:29:36 2026 read: IOPS=2, BW=91.1MiB/s (95.5MB/s)(40.0MiB/439msec) slat (nsec): min=37572k, max=37572k, avg=37572036.00, stdev= 0.00 clat (nsec): min=368315k, max=368315k, avg=368315038.00, stdev= 0.00 lat (nsec): min=405894k, max=405894k, avg=405893904.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 368], 10.00th=[ 368], 20.00th=[ 368], | 30.00th=[ 368], 40.00th=[ 368], 50.00th=[ 368], 60.00th=[ 368], | 70.00th=[ 368], 80.00th=[ 368], 90.00th=[ 368], 95.00th=[ 368], | 99.00th=[ 368], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.88%, ctx=6, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=91.1MiB/s (95.5MB/s), 91.1MiB/s-91.1MiB/s (95.5MB/s-95.5MB/s), io=40.0MiB (41.9MB), run=439-439msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout