== sanity test 398c: run fio to test AIO ================= 20:34:28 (1781224468) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.631627 s, 66.4 MB/s osc.lustre-OST0000-osc-ffff9ca144ce8000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145691: Thu Jun 11 20:36:52 2026 write: IOPS=19, BW=76.9KiB/s (78.8kB/s)(10.0MiB/133090msec); 0 zone resets slat (usec): min=37, max=20505, avg=212.97, stdev=717.51 clat (msec): min=294, max=2749, avg=830.92, stdev=289.73 lat (msec): min=294, max=2749, avg=831.14, stdev=289.76 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 489], 10.00th=[ 567], 20.00th=[ 651], | 30.00th=[ 701], 40.00th=[ 743], 50.00th=[ 785], 60.00th=[ 835], | 70.00th=[ 877], 80.00th=[ 953], 90.00th=[ 1099], 95.00th=[ 1284], | 99.00th=[ 2106], 99.50th=[ 2265], 99.90th=[ 2635], 99.95th=[ 2735], | 99.99th=[ 2735] bw ( KiB/s): min= 7, max= 199, per=24.82%, avg=76.21, stdev=30.49, samples=265 iops : min= 1, max= 49, avg=18.63, stdev= 7.68, samples=265 lat (msec) : 500=5.35%, 750=36.52%, 1000=42.54%, 2000=14.22%, >=2000=1.37% cpu : usr=0.04%, sys=0.29%, ctx=2164, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145692: Thu Jun 11 20:36:52 2026 write: IOPS=19, BW=76.0KiB/s (78.8kB/s)(10.0MiB/133034msec); 0 zone resets slat (usec): min=36, max=16113, avg=208.03, stdev=690.36 clat (msec): min=132, max=2750, avg=830.51, stdev=291.34 lat (msec): min=132, max=2750, avg=830.72, stdev=291.34 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 493], 10.00th=[ 575], 20.00th=[ 642], | 30.00th=[ 693], 40.00th=[ 743], 50.00th=[ 785], 60.00th=[ 835], | 70.00th=[ 885], 80.00th=[ 944], 90.00th=[ 1116], 95.00th=[ 1267], | 99.00th=[ 2123], 99.50th=[ 2333], 99.90th=[ 2735], 99.95th=[ 2735], | 99.99th=[ 2735] bw ( KiB/s): min= 7, max= 152, per=25.11%, avg=77.09, stdev=29.26, samples=262 iops : min= 1, max= 38, avg=18.85, stdev= 7.35, samples=262 lat (msec) : 250=0.04%, 500=5.04%, 750=37.19%, 1000=42.50%, 2000=14.02% lat (msec) : >=2000=1.21% cpu : usr=0.05%, sys=0.26%, ctx=2082, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145693: Thu Jun 11 20:36:52 2026 write: IOPS=19, BW=77.0KiB/s (78.9kB/s)(10.0MiB/132945msec); 0 zone resets slat (usec): min=38, max=11200, avg=203.88, stdev=501.44 clat (msec): min=101, max=2749, avg=830.35, stdev=291.97 lat (msec): min=101, max=2750, avg=830.56, stdev=291.97 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 510], 10.00th=[ 575], 20.00th=[ 642], | 30.00th=[ 701], 40.00th=[ 743], 50.00th=[ 785], 60.00th=[ 835], | 70.00th=[ 877], 80.00th=[ 936], 90.00th=[ 1099], 95.00th=[ 1301], | 99.00th=[ 2165], 99.50th=[ 2299], 99.90th=[ 2668], 99.95th=[ 2668], | 99.99th=[ 2735] bw ( KiB/s): min= 8, max= 247, per=25.20%, avg=77.38, stdev=33.61, samples=261 iops : min= 2, max= 61, avg=18.92, stdev= 8.43, samples=261 lat (msec) : 250=0.55%, 500=3.71%, 750=37.77%, 1000=42.58%, 2000=13.91% lat (msec) : >=2000=1.48% cpu : usr=0.04%, sys=0.27%, ctx=2583, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145694: Thu Jun 11 20:36:52 2026 write: IOPS=19, BW=76.9KiB/s (78.8kB/s)(10.0MiB/133089msec); 0 zone resets slat (usec): min=37, max=20258, avg=222.00, stdev=799.91 clat (msec): min=219, max=2654, avg=831.34, stdev=288.34 lat (msec): min=219, max=2654, avg=831.56, stdev=288.33 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 502], 10.00th=[ 575], 20.00th=[ 659], | 30.00th=[ 701], 40.00th=[ 743], 50.00th=[ 785], 60.00th=[ 827], | 70.00th=[ 877], 80.00th=[ 944], 90.00th=[ 1099], 95.00th=[ 1250], | 99.00th=[ 2072], 99.50th=[ 2366], 99.90th=[ 2601], 99.95th=[ 2601], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 175, per=25.01%, avg=76.79, stdev=30.71, samples=263 iops : min= 1, max= 43, avg=18.78, stdev= 7.74, samples=263 lat (msec) : 250=0.04%, 500=4.30%, 750=36.95%, 1000=43.32%, 2000=13.91% lat (msec) : >=2000=1.48% cpu : usr=0.05%, sys=0.29%, ctx=2279, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=308KiB/s (315kB/s), 76.9KiB/s-77.0KiB/s (78.8kB/s-78.9kB/s), io=40.0MiB (41.9MB), run=132945-133090msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145725: Thu Jun 11 20:38:51 2026 read: IOPS=11, BW=46.3KiB/s (47.4kB/s)(5048KiB/108939msec) slat (usec): min=58, max=67442, avg=14927.64, stdev=9028.77 clat (msec): min=35, max=1080, avg=426.35, stdev=137.57 lat (msec): min=46, max=1099, avg=441.29, stdev=140.20 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 218], 10.00th=[ 279], 20.00th=[ 326], | 30.00th=[ 355], 40.00th=[ 380], 50.00th=[ 405], 60.00th=[ 443], | 70.00th=[ 485], 80.00th=[ 535], 90.00th=[ 600], 95.00th=[ 667], | 99.00th=[ 852], 99.50th=[ 885], 99.90th=[ 1020], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 135, per=25.55%, avg=46.49, stdev=23.34, samples=213 iops : min= 1, max= 33, avg=11.16, stdev= 5.88, samples=213 write: IOPS=11, BW=47.7KiB/s (48.8kB/s)(5192KiB/108939msec); 0 zone resets slat (usec): min=52, max=10325, avg=245.17, stdev=546.79 clat (msec): min=227, max=1877, avg=912.75, stdev=276.67 lat (msec): min=228, max=1877, avg=913.00, stdev=276.66 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 477], 10.00th=[ 575], 20.00th=[ 684], | 30.00th=[ 768], 40.00th=[ 844], 50.00th=[ 902], 60.00th=[ 961], | 70.00th=[ 1028], 80.00th=[ 1099], 90.00th=[ 1267], 95.00th=[ 1435], | 99.00th=[ 1720], 99.50th=[ 1770], 99.90th=[ 1838], 99.95th=[ 1871], | 99.99th=[ 1871] bw ( KiB/s): min= 7, max= 104, per=24.88%, avg=47.27, stdev=20.42, samples=215 iops : min= 1, max= 26, avg=11.35, stdev= 5.13, samples=215 lat (msec) : 50=0.04%, 100=0.31%, 250=3.20%, 500=35.90%, 750=22.73% lat (msec) : 1000=20.16%, 2000=17.66% cpu : usr=0.09%, sys=0.69%, ctx=4828, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145726: Thu Jun 11 20:38:51 2026 read: IOPS=11, BW=45.3KiB/s (46.4kB/s)(4976KiB/109831msec) slat (usec): min=2103, max=62450, avg=14881.24, stdev=8544.11 clat (msec): min=18, max=885, avg=430.79, stdev=129.29 lat (msec): min=50, max=924, avg=445.68, stdev=131.34 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 247], 10.00th=[ 284], 20.00th=[ 330], | 30.00th=[ 363], 40.00th=[ 388], 50.00th=[ 414], 60.00th=[ 451], | 70.00th=[ 481], 80.00th=[ 531], 90.00th=[ 600], 95.00th=[ 667], | 99.00th=[ 785], 99.50th=[ 810], 99.90th=[ 877], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 7, max= 111, per=24.50%, avg=44.60, stdev=22.91, samples=218 iops : min= 1, max= 27, avg=10.68, stdev= 5.76, samples=218 write: IOPS=11, BW=47.9KiB/s (49.1kB/s)(5264KiB/109831msec); 0 zone resets slat (usec): min=41, max=15446, avg=251.34, stdev=661.35 clat (msec): min=243, max=1877, avg=911.46, stdev=271.60 lat (msec): min=244, max=1877, avg=911.71, stdev=271.59 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 468], 10.00th=[ 592], 20.00th=[ 693], | 30.00th=[ 768], 40.00th=[ 835], 50.00th=[ 902], 60.00th=[ 961], | 70.00th=[ 1028], 80.00th=[ 1099], 90.00th=[ 1250], 95.00th=[ 1401], | 99.00th=[ 1687], 99.50th=[ 1737], 99.90th=[ 1871], 99.95th=[ 1871], | 99.99th=[ 1871] bw ( KiB/s): min= 7, max= 104, per=24.91%, avg=47.33, stdev=20.55, samples=218 iops : min= 1, max= 26, avg=11.37, stdev= 5.15, samples=218 lat (msec) : 20=0.04%, 50=0.08%, 100=0.31%, 250=2.38%, 500=36.17% lat (msec) : 750=22.93%, 1000=20.31%, 2000=17.77% cpu : usr=0.10%, sys=0.65%, ctx=4636, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145727: Thu Jun 11 20:38:51 2026 read: IOPS=11, BW=45.5KiB/s (46.6kB/s)(4984KiB/109629msec) slat (usec): min=2863, max=57867, avg=14498.38, stdev=8111.13 clat (msec): min=20, max=908, avg=426.03, stdev=128.87 lat (msec): min=26, max=915, avg=440.53, stdev=130.89 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 234], 10.00th=[ 284], 20.00th=[ 330], | 30.00th=[ 359], 40.00th=[ 384], 50.00th=[ 409], 60.00th=[ 439], | 70.00th=[ 477], 80.00th=[ 535], 90.00th=[ 600], 95.00th=[ 651], | 99.00th=[ 760], 99.50th=[ 827], 99.90th=[ 885], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 7, max= 159, per=24.90%, avg=45.32, stdev=22.99, samples=216 iops : min= 1, max= 39, avg=10.87, stdev= 5.80, samples=216 write: IOPS=11, BW=47.9KiB/s (49.1kB/s)(5256KiB/109629msec); 0 zone resets slat (usec): min=37, max=12291, avg=306.56, stdev=938.32 clat (msec): min=145, max=1817, avg=914.67, stdev=276.95 lat (msec): min=146, max=1817, avg=914.98, stdev=276.99 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 460], 10.00th=[ 575], 20.00th=[ 693], | 30.00th=[ 760], 40.00th=[ 835], 50.00th=[ 902], 60.00th=[ 969], | 70.00th=[ 1036], 80.00th=[ 1099], 90.00th=[ 1267], 95.00th=[ 1418], | 99.00th=[ 1670], 99.50th=[ 1737], 99.90th=[ 1787], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 7, max= 127, per=24.82%, avg=47.16, stdev=21.98, samples=218 iops : min= 1, max= 31, avg=11.33, stdev= 5.52, samples=218 lat (msec) : 50=0.23%, 100=0.23%, 250=3.01%, 500=35.47%, 750=23.24% lat (msec) : 1000=20.27%, 2000=17.54% cpu : usr=0.10%, sys=0.68%, ctx=4838, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145728: Thu Jun 11 20:38:51 2026 read: IOPS=11, BW=46.1KiB/s (47.2kB/s)(5060KiB/109769msec) slat (usec): min=2007, max=62879, avg=14440.83, stdev=8536.37 clat (msec): min=35, max=1015, avg=433.74, stdev=122.31 lat (msec): min=42, max=1033, avg=448.19, stdev=124.36 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 255], 10.00th=[ 296], 20.00th=[ 338], | 30.00th=[ 368], 40.00th=[ 388], 50.00th=[ 418], 60.00th=[ 451], | 70.00th=[ 485], 80.00th=[ 535], 90.00th=[ 592], 95.00th=[ 651], | 99.00th=[ 785], 99.50th=[ 835], 99.90th=[ 902], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 7, max= 136, per=25.36%, avg=46.16, stdev=20.92, samples=215 iops : min= 1, max= 34, avg=11.07, stdev= 5.27, samples=215 write: IOPS=11, BW=47.2KiB/s (48.3kB/s)(5180KiB/109769msec); 0 zone resets slat (usec): min=43, max=10664, avg=250.76, stdev=523.15 clat (msec): min=301, max=1797, avg=915.06, stdev=275.64 lat (msec): min=301, max=1798, avg=915.31, stdev=275.66 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 472], 10.00th=[ 575], 20.00th=[ 701], | 30.00th=[ 768], 40.00th=[ 835], 50.00th=[ 902], 60.00th=[ 961], | 70.00th=[ 1028], 80.00th=[ 1116], 90.00th=[ 1267], 95.00th=[ 1435], | 99.00th=[ 1703], 99.50th=[ 1737], 99.90th=[ 1804], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 7, max= 111, per=24.51%, avg=46.57, stdev=21.49, samples=218 iops : min= 1, max= 27, avg=11.18, stdev= 5.41, samples=218 lat (msec) : 50=0.04%, 100=0.04%, 250=2.07%, 500=37.11%, 750=23.24% lat (msec) : 1000=20.39%, 2000=17.11% cpu : usr=0.09%, sys=0.61%, ctx=4759, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=183KiB/s (187kB/s), 45.3KiB/s-46.3KiB/s (46.4kB/s-47.4kB/s), io=19.6MiB (20.5MB), run=108939-109831msec WRITE: bw=190KiB/s (195kB/s), 47.2KiB/s-47.9KiB/s (48.3kB/s-49.1kB/s), io=20.4MiB (21.4MB), run=108939-109831msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145749: Thu Jun 11 20:39:02 2026 read: IOPS=2, BW=99.8MiB/s (105MB/s)(40.0MiB/401msec) slat (nsec): min=21711k, max=21711k, avg=21710599.00, stdev= 0.00 clat (nsec): min=357661k, max=357661k, avg=357660697.00, stdev= 0.00 lat (nsec): min=379377k, max=379377k, avg=379376615.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[ 359], | 30.00th=[ 359], 40.00th=[ 359], 50.00th=[ 359], 60.00th=[ 359], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.75%, ctx=31, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=99.8MiB/s (105MB/s), 99.8MiB/s-99.8MiB/s (105MB/s-105MB/s), io=40.0MiB (41.9MB), run=401-401msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout