== sanity test 398c: run fio to test AIO ================= 01:54:13 (1783144453) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.220713 s, 190 MB/s osc.lustre-OST0000-osc-ffff9e3fe6943000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147911: Sat Jul 4 01:55:19 2026 write: IOPS=41, BW=165KiB/s (168kB/s)(10.0MiB/62235msec); 0 zone resets slat (usec): min=36, max=6565, avg=226.83, stdev=295.25 clat (msec): min=116, max=655, avg=388.21, stdev=81.13 lat (msec): min=116, max=655, avg=388.44, stdev=81.15 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 255], 10.00th=[ 288], 20.00th=[ 321], | 30.00th=[ 351], 40.00th=[ 372], 50.00th=[ 388], 60.00th=[ 409], | 70.00th=[ 430], 80.00th=[ 460], 90.00th=[ 489], 95.00th=[ 510], | 99.00th=[ 567], 99.50th=[ 592], 99.90th=[ 625], 99.95th=[ 642], | 99.99th=[ 659] bw ( KiB/s): min= 48, max= 313, per=24.91%, avg=163.88, stdev=37.00, samples=123 iops : min= 12, max= 78, avg=40.76, stdev= 9.27, samples=123 lat (msec) : 250=4.84%, 500=87.50%, 750=7.66% cpu : usr=0.09%, sys=0.56%, ctx=3086, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147912: Sat Jul 4 01:55:19 2026 write: IOPS=41, BW=165KiB/s (169kB/s)(10.0MiB/62112msec); 0 zone resets slat (usec): min=38, max=4405, avg=230.25, stdev=260.46 clat (msec): min=94, max=655, avg=387.49, stdev=80.95 lat (msec): min=95, max=655, avg=387.72, stdev=80.96 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 255], 10.00th=[ 288], 20.00th=[ 321], | 30.00th=[ 351], 40.00th=[ 372], 50.00th=[ 388], 60.00th=[ 409], | 70.00th=[ 435], 80.00th=[ 456], 90.00th=[ 485], 95.00th=[ 510], | 99.00th=[ 567], 99.50th=[ 584], 99.90th=[ 642], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 103, max= 313, per=24.98%, avg=164.34, stdev=33.78, samples=123 iops : min= 25, max= 78, avg=40.87, stdev= 8.46, samples=123 lat (msec) : 100=0.04%, 250=4.80%, 500=88.16%, 750=6.99% cpu : usr=0.09%, sys=0.61%, ctx=3174, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147913: Sat Jul 4 01:55:19 2026 write: IOPS=41, BW=165KiB/s (169kB/s)(10.0MiB/62056msec); 0 zone resets slat (usec): min=35, max=6557, avg=221.06, stdev=262.21 clat (msec): min=75, max=716, avg=387.41, stdev=83.72 lat (msec): min=75, max=716, avg=387.63, stdev=83.72 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 243], 10.00th=[ 284], 20.00th=[ 330], | 30.00th=[ 351], 40.00th=[ 372], 50.00th=[ 388], 60.00th=[ 409], | 70.00th=[ 435], 80.00th=[ 456], 90.00th=[ 489], 95.00th=[ 514], | 99.00th=[ 567], 99.50th=[ 592], 99.90th=[ 659], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 88, max= 376, per=25.01%, avg=164.58, stdev=38.19, samples=123 iops : min= 22, max= 94, avg=40.93, stdev= 9.53, samples=123 lat (msec) : 100=0.20%, 250=5.23%, 500=86.64%, 750=7.93% cpu : usr=0.07%, sys=0.59%, ctx=3173, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147914: Sat Jul 4 01:55:19 2026 write: IOPS=41, BW=165KiB/s (168kB/s)(10.0MiB/62235msec); 0 zone resets slat (usec): min=44, max=6681, avg=227.26, stdev=279.17 clat (msec): min=99, max=656, avg=388.30, stdev=82.13 lat (msec): min=99, max=656, avg=388.53, stdev=82.13 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 257], 10.00th=[ 288], 20.00th=[ 326], | 30.00th=[ 351], 40.00th=[ 372], 50.00th=[ 393], 60.00th=[ 409], | 70.00th=[ 435], 80.00th=[ 460], 90.00th=[ 489], 95.00th=[ 510], | 99.00th=[ 567], 99.50th=[ 600], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 659] bw ( KiB/s): min= 87, max= 312, per=24.91%, avg=163.93, stdev=35.30, samples=123 iops : min= 21, max= 78, avg=40.77, stdev= 8.84, samples=123 lat (msec) : 100=0.08%, 250=4.73%, 500=87.77%, 750=7.42% cpu : usr=0.08%, sys=0.60%, ctx=3190, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=658KiB/s (674kB/s), 165KiB/s-165KiB/s (168kB/s-169kB/s), io=40.0MiB (41.9MB), run=62056-62235msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147932: Sat Jul 4 01:55:58 2026 read: IOPS=35, BW=144KiB/s (147kB/s)(5048KiB/35063msec) slat (usec): min=1181, max=13595, avg=4060.68, stdev=1698.09 clat (msec): min=8, max=283, avg=135.32, stdev=33.88 lat (msec): min=13, max=289, avg=139.38, stdev=34.34 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 91], 10.00th=[ 99], 20.00th=[ 107], | 30.00th=[ 116], 40.00th=[ 123], 50.00th=[ 130], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 163], 90.00th=[ 182], 95.00th=[ 197], | 99.00th=[ 234], 99.50th=[ 241], 99.90th=[ 255], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 48, max= 224, per=25.11%, avg=143.39, stdev=43.35, samples=69 iops : min= 12, max= 56, avg=35.61, stdev=10.91, samples=69 write: IOPS=37, BW=148KiB/s (152kB/s)(5192KiB/35063msec); 0 zone resets slat (usec): min=46, max=4257, avg=238.72, stdev=267.53 clat (msec): min=95, max=668, avg=295.48, stdev=91.23 lat (msec): min=95, max=669, avg=295.72, stdev=91.23 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 178], 10.00th=[ 197], 20.00th=[ 222], | 30.00th=[ 243], 40.00th=[ 259], 50.00th=[ 279], 60.00th=[ 300], | 70.00th=[ 330], 80.00th=[ 363], 90.00th=[ 405], 95.00th=[ 456], | 99.00th=[ 600], 99.50th=[ 617], 99.90th=[ 651], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 79, max= 208, per=24.76%, avg=147.35, stdev=34.34, samples=69 iops : min= 19, max= 52, avg=36.61, stdev= 8.64, samples=69 lat (msec) : 10=0.04%, 50=0.20%, 100=5.55%, 250=60.23%, 500=32.19% lat (msec) : 750=1.80% cpu : usr=0.12%, sys=1.43%, ctx=4949, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147933: Sat Jul 4 01:55:58 2026 read: IOPS=35, BW=142KiB/s (145kB/s)(4976KiB/35101msec) slat (usec): min=1267, max=13677, avg=4099.83, stdev=1712.50 clat (msec): min=3, max=266, avg=134.34, stdev=35.38 lat (msec): min=5, max=273, avg=138.44, stdev=35.91 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 87], 10.00th=[ 96], 20.00th=[ 107], | 30.00th=[ 115], 40.00th=[ 124], 50.00th=[ 131], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 161], 90.00th=[ 180], 95.00th=[ 201], | 99.00th=[ 230], 99.50th=[ 236], 99.90th=[ 257], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 47, max= 264, per=24.61%, avg=140.51, stdev=46.93, samples=69 iops : min= 11, max= 66, avg=34.90, stdev=11.76, samples=69 write: IOPS=37, BW=150KiB/s (154kB/s)(5264KiB/35101msec); 0 zone resets slat (usec): min=59, max=2590, avg=230.83, stdev=208.08 clat (msec): min=27, max=657, avg=294.94, stdev=93.20 lat (msec): min=27, max=657, avg=295.18, stdev=93.21 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 169], 10.00th=[ 192], 20.00th=[ 220], | 30.00th=[ 241], 40.00th=[ 259], 50.00th=[ 279], 60.00th=[ 305], | 70.00th=[ 338], 80.00th=[ 368], 90.00th=[ 414], 95.00th=[ 464], | 99.00th=[ 575], 99.50th=[ 600], 99.90th=[ 651], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 71, max= 224, per=24.91%, avg=148.20, stdev=34.28, samples=69 iops : min= 17, max= 56, avg=36.83, stdev= 8.66, samples=69 lat (msec) : 4=0.04%, 10=0.12%, 20=0.08%, 50=0.31%, 100=6.29% lat (msec) : 250=59.73%, 500=31.72%, 750=1.72% cpu : usr=0.15%, sys=1.42%, ctx=4974, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147934: Sat Jul 4 01:55:58 2026 read: IOPS=35, BW=142KiB/s (146kB/s)(4984KiB/34981msec) slat (usec): min=54, max=14764, avg=4129.03, stdev=1779.05 clat (msec): min=7, max=398, avg=134.95, stdev=36.42 lat (msec): min=11, max=401, avg=139.08, stdev=36.89 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 88], 10.00th=[ 95], 20.00th=[ 106], | 30.00th=[ 113], 40.00th=[ 123], 50.00th=[ 132], 60.00th=[ 140], | 70.00th=[ 150], 80.00th=[ 165], 90.00th=[ 182], 95.00th=[ 197], | 99.00th=[ 232], 99.50th=[ 247], 99.90th=[ 326], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 48, max= 296, per=25.10%, avg=143.33, stdev=44.83, samples=69 iops : min= 12, max= 74, avg=35.61, stdev=11.22, samples=69 write: IOPS=37, BW=150KiB/s (154kB/s)(5256KiB/34981msec); 0 zone resets slat (usec): min=57, max=3800, avg=242.21, stdev=272.23 clat (msec): min=62, max=662, avg=293.42, stdev=91.78 lat (msec): min=62, max=663, avg=293.67, stdev=91.79 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 171], 10.00th=[ 194], 20.00th=[ 222], | 30.00th=[ 239], 40.00th=[ 257], 50.00th=[ 275], 60.00th=[ 305], | 70.00th=[ 334], 80.00th=[ 363], 90.00th=[ 414], 95.00th=[ 447], | 99.00th=[ 584], 99.50th=[ 609], 99.90th=[ 659], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 72, max= 224, per=25.17%, avg=149.74, stdev=39.42, samples=69 iops : min= 18, max= 56, avg=37.20, stdev= 9.94, samples=69 lat (msec) : 10=0.04%, 20=0.08%, 50=0.27%, 100=6.80%, 250=60.04% lat (msec) : 500=31.17%, 750=1.60% cpu : usr=0.14%, sys=1.48%, ctx=4940, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147935: Sat Jul 4 01:55:58 2026 read: IOPS=36, BW=145KiB/s (148kB/s)(5060KiB/34955msec) slat (usec): min=1199, max=12772, avg=4100.67, stdev=1733.93 clat (msec): min=5, max=349, avg=134.82, stdev=35.97 lat (msec): min=8, max=353, avg=138.92, stdev=36.52 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 85], 10.00th=[ 94], 20.00th=[ 108], | 30.00th=[ 116], 40.00th=[ 125], 50.00th=[ 132], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 161], 90.00th=[ 178], 95.00th=[ 194], | 99.00th=[ 239], 99.50th=[ 249], 99.90th=[ 342], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 64, max= 304, per=25.44%, avg=145.29, stdev=46.28, samples=69 iops : min= 16, max= 76, avg=36.10, stdev=11.56, samples=69 write: IOPS=37, BW=148KiB/s (152kB/s)(5180KiB/34955msec); 0 zone resets slat (usec): min=59, max=3726, avg=231.17, stdev=225.43 clat (msec): min=75, max=640, avg=295.08, stdev=90.71 lat (msec): min=76, max=640, avg=295.31, stdev=90.72 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 176], 10.00th=[ 194], 20.00th=[ 222], | 30.00th=[ 241], 40.00th=[ 257], 50.00th=[ 279], 60.00th=[ 305], | 70.00th=[ 334], 80.00th=[ 368], 90.00th=[ 418], 95.00th=[ 456], | 99.00th=[ 584], 99.50th=[ 600], 99.90th=[ 634], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 55, max= 232, per=24.88%, avg=148.03, stdev=37.93, samples=69 iops : min= 13, max= 58, avg=36.77, stdev= 9.59, samples=69 lat (msec) : 10=0.04%, 20=0.04%, 50=0.47%, 100=6.17%, 250=60.12% lat (msec) : 500=31.48%, 750=1.68% cpu : usr=0.13%, sys=1.45%, ctx=4962, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=572KiB/s (585kB/s), 142KiB/s-145KiB/s (145kB/s-148kB/s), io=19.6MiB (20.5MB), run=34955-35101msec WRITE: bw=595KiB/s (609kB/s), 148KiB/s-150KiB/s (152kB/s-154kB/s), io=20.4MiB (21.4MB), run=34955-35101msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147943: Sat Jul 4 01:56:02 2026 read: IOPS=6, BW=276MiB/s (289MB/s)(40.0MiB/145msec) slat (nsec): min=13505k, max=13505k, avg=13505295.00, stdev= 0.00 clat (nsec): min=124940k, max=124940k, avg=124940427.00, stdev= 0.00 lat (nsec): min=138475k, max=138475k, avg=138475103.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 126], 10.00th=[ 126], 20.00th=[ 126], | 30.00th=[ 126], 40.00th=[ 126], 50.00th=[ 126], 60.00th=[ 126], | 70.00th=[ 126], 80.00th=[ 126], 90.00th=[ 126], 95.00th=[ 126], | 99.00th=[ 126], 99.50th=[ 126], 99.90th=[ 126], 99.95th=[ 126], | 99.99th=[ 126] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=8.33%, ctx=8, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=276MiB/s (289MB/s), 276MiB/s-276MiB/s (289MB/s-289MB/s), io=40.0MiB (41.9MB), run=145-145msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout