== sanity test 398c: run fio to test AIO ================= 02:13:53 (1788934433) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.510262 s, 82.2 MB/s osc.lustre-OST0000-osc-ffff9b2b92747800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145084: Wed Sep 9 02:16:27 2026 write: IOPS=17, BW=71.0KiB/s (73.7kB/s)(10.0MiB/142315msec); 0 zone resets slat (usec): min=22, max=25497, avg=194.05, stdev=869.11 clat (msec): min=202, max=1892, avg=888.27, stdev=268.40 lat (msec): min=202, max=1892, avg=888.46, stdev=268.46 clat percentiles (msec): | 1.00th=[ 268], 5.00th=[ 518], 10.00th=[ 600], 20.00th=[ 684], | 30.00th=[ 743], 40.00th=[ 793], 50.00th=[ 852], 60.00th=[ 927], | 70.00th=[ 1003], 80.00th=[ 1083], 90.00th=[ 1234], 95.00th=[ 1385], | 99.00th=[ 1653], 99.50th=[ 1770], 99.90th=[ 1821], 99.95th=[ 1821], | 99.99th=[ 1888] bw ( KiB/s): min= 7, max= 256, per=25.57%, avg=73.39, stdev=33.12, samples=275 iops : min= 1, max= 64, avg=17.91, stdev= 8.32, samples=275 lat (msec) : 250=0.62%, 500=3.83%, 750=26.91%, 1000=38.16%, 2000=30.47% cpu : usr=0.04%, sys=0.25%, ctx=1813, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145085: Wed Sep 9 02:16:27 2026 write: IOPS=18, BW=72.2KiB/s (73.0kB/s)(10.0MiB/141780msec); 0 zone resets slat (usec): min=22, max=21667, avg=202.17, stdev=761.48 clat (msec): min=83, max=1868, avg=885.41, stdev=270.38 lat (msec): min=84, max=1868, avg=885.61, stdev=270.40 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 518], 10.00th=[ 600], 20.00th=[ 684], | 30.00th=[ 743], 40.00th=[ 793], 50.00th=[ 860], 60.00th=[ 919], | 70.00th=[ 1003], 80.00th=[ 1083], 90.00th=[ 1250], 95.00th=[ 1368], | 99.00th=[ 1670], 99.50th=[ 1737], 99.90th=[ 1804], 99.95th=[ 1821], | 99.99th=[ 1871] bw ( KiB/s): min= 7, max= 238, per=25.10%, avg=72.02, stdev=30.24, samples=280 iops : min= 1, max= 59, avg=17.57, stdev= 7.62, samples=280 lat (msec) : 100=0.04%, 250=1.02%, 500=3.44%, 750=27.15%, 1000=38.36% lat (msec) : 2000=30.00% cpu : usr=0.06%, sys=0.24%, ctx=2028, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145086: Wed Sep 9 02:16:27 2026 write: IOPS=17, BW=71.9KiB/s (73.6kB/s)(10.0MiB/142384msec); 0 zone resets slat (usec): min=21, max=33985, avg=186.81, stdev=968.96 clat (msec): min=217, max=1854, avg=888.78, stdev=264.59 lat (msec): min=217, max=1854, avg=888.97, stdev=264.61 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 550], 10.00th=[ 600], 20.00th=[ 676], | 30.00th=[ 743], 40.00th=[ 802], 50.00th=[ 852], 60.00th=[ 927], | 70.00th=[ 1003], 80.00th=[ 1083], 90.00th=[ 1234], 95.00th=[ 1385], | 99.00th=[ 1636], 99.50th=[ 1737], 99.90th=[ 1821], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 7, max= 256, per=25.11%, avg=72.07, stdev=31.61, samples=280 iops : min= 1, max= 64, avg=17.58, stdev= 7.98, samples=280 lat (msec) : 250=0.62%, 500=3.24%, 750=27.93%, 1000=37.62%, 2000=30.59% cpu : usr=0.05%, sys=0.22%, ctx=1771, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145087: Wed Sep 9 02:16:27 2026 write: IOPS=17, BW=71.9KiB/s (73.6kB/s)(10.0MiB/142419msec); 0 zone resets slat (usec): min=22, max=26270, avg=193.75, stdev=868.61 clat (msec): min=203, max=1869, avg=889.03, stdev=267.21 lat (msec): min=204, max=1869, avg=889.23, stdev=267.24 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 550], 10.00th=[ 609], 20.00th=[ 684], | 30.00th=[ 743], 40.00th=[ 802], 50.00th=[ 860], 60.00th=[ 919], | 70.00th=[ 995], 80.00th=[ 1083], 90.00th=[ 1234], 95.00th=[ 1385], | 99.00th=[ 1703], 99.50th=[ 1787], 99.90th=[ 1871], 99.95th=[ 1871], | 99.99th=[ 1871] bw ( KiB/s): min= 7, max= 184, per=24.83%, avg=71.27, stdev=31.95, samples=283 iops : min= 1, max= 46, avg=17.38, stdev= 8.04, samples=283 lat (msec) : 250=0.98%, 500=2.97%, 750=26.84%, 1000=40.08%, 2000=29.14% cpu : usr=0.05%, sys=0.25%, ctx=1924, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=288KiB/s (295kB/s), 71.9KiB/s-72.2KiB/s (73.6kB/s-73.0kB/s), io=40.0MiB (41.9MB), run=141780-142419msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145117: Wed Sep 9 02:18:10 2026 read: IOPS=13, BW=54.6KiB/s (55.9kB/s)(5048KiB/92402msec) slat (usec): min=36, max=57674, avg=13008.59, stdev=7413.84 clat (msec): min=33, max=1326, avg=363.30, stdev=119.09 lat (msec): min=43, max=1346, avg=376.32, stdev=120.37 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 220], 10.00th=[ 245], 20.00th=[ 284], | 30.00th=[ 309], 40.00th=[ 330], 50.00th=[ 351], 60.00th=[ 376], | 70.00th=[ 397], 80.00th=[ 435], 90.00th=[ 498], 95.00th=[ 550], | 99.00th=[ 735], 99.50th=[ 995], 99.90th=[ 1284], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 7, max= 118, per=25.30%, avg=54.40, stdev=23.10, samples=182 iops : min= 1, max= 29, avg=13.11, stdev= 5.81, samples=182 write: IOPS=14, BW=56.2KiB/s (57.5kB/s)(5192KiB/92402msec); 0 zone resets slat (usec): min=30, max=23230, avg=260.53, stdev=1039.55 clat (msec): min=185, max=1561, avg=771.58, stdev=206.30 lat (msec): min=186, max=1562, avg=771.84, stdev=206.28 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 468], 10.00th=[ 514], 20.00th=[ 592], | 30.00th=[ 659], 40.00th=[ 709], 50.00th=[ 760], 60.00th=[ 818], | 70.00th=[ 869], 80.00th=[ 927], 90.00th=[ 1028], 95.00th=[ 1133], | 99.00th=[ 1418], 99.50th=[ 1469], 99.90th=[ 1552], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 7, max= 119, per=24.97%, avg=55.94, stdev=21.28, samples=182 iops : min= 1, max= 29, avg=13.50, stdev= 5.34, samples=182 lat (msec) : 50=0.23%, 100=0.47%, 250=4.92%, 500=43.32%, 750=24.34% lat (msec) : 1000=20.70%, 2000=6.02% cpu : usr=0.08%, sys=0.73%, ctx=4945, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145118: Wed Sep 9 02:18:10 2026 read: IOPS=13, BW=53.4KiB/s (54.7kB/s)(4976KiB/93204msec) slat (usec): min=2301, max=67739, avg=13109.88, stdev=7349.76 clat (msec): min=17, max=879, avg=366.57, stdev=98.87 lat (msec): min=27, max=897, avg=379.69, stdev=100.00 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 230], 10.00th=[ 257], 20.00th=[ 288], | 30.00th=[ 313], 40.00th=[ 338], 50.00th=[ 359], 60.00th=[ 376], | 70.00th=[ 401], 80.00th=[ 435], 90.00th=[ 489], 95.00th=[ 550], | 99.00th=[ 659], 99.50th=[ 760], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 7, max= 127, per=24.63%, avg=52.96, stdev=24.20, samples=184 iops : min= 1, max= 31, avg=12.76, stdev= 6.05, samples=184 write: IOPS=14, BW=56.5KiB/s (57.8kB/s)(5264KiB/93204msec); 0 zone resets slat (usec): min=24, max=10997, avg=236.52, stdev=754.12 clat (msec): min=295, max=1564, avg=772.08, stdev=199.89 lat (msec): min=295, max=1565, avg=772.32, stdev=199.88 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 477], 10.00th=[ 518], 20.00th=[ 592], | 30.00th=[ 659], 40.00th=[ 718], 50.00th=[ 768], 60.00th=[ 818], | 70.00th=[ 869], 80.00th=[ 927], 90.00th=[ 1020], 95.00th=[ 1099], | 99.00th=[ 1385], 99.50th=[ 1469], 99.90th=[ 1552], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 15, max= 134, per=25.12%, avg=56.27, stdev=23.66, samples=184 iops : min= 3, max= 33, avg=13.59, stdev= 5.96, samples=184 lat (msec) : 20=0.04%, 50=0.04%, 100=0.12%, 250=3.98%, 500=44.10% lat (msec) : 750=24.02%, 1000=21.76%, 2000=5.94% cpu : usr=0.08%, sys=0.70%, ctx=4446, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145119: Wed Sep 9 02:18:10 2026 read: IOPS=13, BW=53.5KiB/s (54.8kB/s)(4984KiB/93208msec) slat (usec): min=2160, max=66997, avg=13187.82, stdev=7602.47 clat (msec): min=24, max=807, avg=363.17, stdev=98.65 lat (msec): min=28, max=822, avg=376.36, stdev=99.74 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 230], 10.00th=[ 257], 20.00th=[ 288], | 30.00th=[ 313], 40.00th=[ 334], 50.00th=[ 355], 60.00th=[ 376], | 70.00th=[ 401], 80.00th=[ 430], 90.00th=[ 485], 95.00th=[ 542], | 99.00th=[ 634], 99.50th=[ 776], 99.90th=[ 810], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 7, max= 128, per=24.77%, avg=53.26, stdev=25.39, samples=184 iops : min= 1, max= 32, avg=12.83, stdev= 6.42, samples=184 write: IOPS=14, BW=56.4KiB/s (57.7kB/s)(5256KiB/93208msec); 0 zone resets slat (usec): min=30, max=15458, avg=236.23, stdev=746.29 clat (msec): min=242, max=1554, avg=775.31, stdev=203.23 lat (msec): min=242, max=1554, avg=775.55, stdev=203.24 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 468], 10.00th=[ 527], 20.00th=[ 609], | 30.00th=[ 667], 40.00th=[ 718], 50.00th=[ 768], 60.00th=[ 810], | 70.00th=[ 869], 80.00th=[ 927], 90.00th=[ 1028], 95.00th=[ 1133], | 99.00th=[ 1368], 99.50th=[ 1452], 99.90th=[ 1536], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 8, max= 128, per=25.00%, avg=55.99, stdev=22.61, samples=184 iops : min= 2, max= 32, avg=13.52, stdev= 5.65, samples=184 lat (msec) : 50=0.20%, 100=0.16%, 250=3.83%, 500=44.61%, 750=23.63% lat (msec) : 1000=21.60%, 2000=5.98% cpu : usr=0.12%, sys=0.67%, ctx=4209, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145120: Wed Sep 9 02:18:10 2026 read: IOPS=13, BW=54.5KiB/s (55.8kB/s)(5060KiB/92903msec) slat (usec): min=2103, max=61525, avg=13111.09, stdev=7520.27 clat (msec): min=9, max=883, avg=364.56, stdev=98.28 lat (msec): min=16, max=887, avg=377.67, stdev=99.37 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 234], 10.00th=[ 262], 20.00th=[ 288], | 30.00th=[ 313], 40.00th=[ 334], 50.00th=[ 355], 60.00th=[ 376], | 70.00th=[ 401], 80.00th=[ 430], 90.00th=[ 481], 95.00th=[ 542], | 99.00th=[ 659], 99.50th=[ 793], 99.90th=[ 877], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 7, max= 124, per=25.33%, avg=54.45, stdev=23.00, samples=183 iops : min= 1, max= 31, avg=13.13, stdev= 5.81, samples=183 write: IOPS=13, BW=55.8KiB/s (57.1kB/s)(5180KiB/92903msec); 0 zone resets slat (usec): min=27, max=28202, avg=252.35, stdev=1023.90 clat (msec): min=298, max=1663, avg=775.91, stdev=211.85 lat (msec): min=298, max=1663, avg=776.17, stdev=211.86 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 464], 10.00th=[ 523], 20.00th=[ 600], | 30.00th=[ 651], 40.00th=[ 718], 50.00th=[ 760], 60.00th=[ 818], | 70.00th=[ 869], 80.00th=[ 927], 90.00th=[ 1036], 95.00th=[ 1133], | 99.00th=[ 1485], 99.50th=[ 1552], 99.90th=[ 1653], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 15, max= 111, per=24.57%, avg=55.04, stdev=21.06, samples=184 iops : min= 3, max= 27, avg=13.28, stdev= 5.28, samples=184 lat (msec) : 10=0.04%, 50=0.08%, 100=0.20%, 250=3.67%, 500=45.90% lat (msec) : 750=23.20%, 1000=20.59%, 2000=6.33% cpu : usr=0.11%, sys=0.68%, ctx=4380, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=215KiB/s (220kB/s), 53.4KiB/s-54.6KiB/s (54.7kB/s-55.9kB/s), io=19.6MiB (20.5MB), run=92402-93208msec WRITE: bw=224KiB/s (230kB/s), 55.8KiB/s-56.5KiB/s (57.1kB/s-57.8kB/s), io=20.4MiB (21.4MB), run=92402-93208msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145138: Wed Sep 9 02:18:20 2026 read: IOPS=1, BW=64.5MiB/s (67.7MB/s)(40.0MiB/620msec) slat (nsec): min=19798k, max=19798k, avg=19797995.00, stdev= 0.00 clat (nsec): min=586860k, max=586860k, avg=586860117.00, stdev= 0.00 lat (nsec): min=606664k, max=606664k, avg=606663787.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 584], 5.00th=[ 584], 10.00th=[ 584], 20.00th=[ 584], | 30.00th=[ 584], 40.00th=[ 584], 50.00th=[ 584], 60.00th=[ 584], | 70.00th=[ 584], 80.00th=[ 584], 90.00th=[ 584], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.94%, ctx=42, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=64.5MiB/s (67.7MB/s), 64.5MiB/s-64.5MiB/s (67.7MB/s-67.7MB/s), io=40.0MiB (41.9MB), run=620-620msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout