== sanity test 398c: run fio to test AIO ================= 14:21:35 (1785954095) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 1.08865 s, 38.5 MB/s osc.lustre-OST0000-osc-ffff95d3ca926000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179237: Wed Aug 5 14:22:27 2026 write: IOPS=62, BW=250KiB/s (256kB/s)(10.0MiB/40987msec); 0 zone resets slat (usec): min=47, max=24762, avg=363.96, stdev=1491.26 clat (msec): min=32, max=507, avg=255.68, stdev=63.46 lat (msec): min=34, max=508, avg=256.05, stdev=63.44 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 153], 10.00th=[ 176], 20.00th=[ 207], | 30.00th=[ 224], 40.00th=[ 236], 50.00th=[ 247], 60.00th=[ 268], | 70.00th=[ 288], 80.00th=[ 309], 90.00th=[ 338], 95.00th=[ 355], | 99.00th=[ 430], 99.50th=[ 493], 99.90th=[ 506], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 126, max= 422, per=24.87%, avg=248.21, stdev=65.05, samples=80 iops : min= 31, max= 105, avg=61.63, stdev=16.27, samples=80 lat (msec) : 50=0.08%, 100=0.27%, 250=50.86%, 500=48.55%, 750=0.23% cpu : usr=0.07%, sys=0.85%, ctx=1608, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179238: Wed Aug 5 14:22:27 2026 write: IOPS=62, BW=250KiB/s (256kB/s)(10.0MiB/41002msec); 0 zone resets slat (usec): min=49, max=27487, avg=397.99, stdev=1579.25 clat (msec): min=36, max=488, avg=255.68, stdev=64.98 lat (msec): min=36, max=488, avg=256.09, stdev=64.88 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 146], 10.00th=[ 174], 20.00th=[ 205], | 30.00th=[ 224], 40.00th=[ 236], 50.00th=[ 253], 60.00th=[ 268], | 70.00th=[ 284], 80.00th=[ 309], 90.00th=[ 334], 95.00th=[ 368], | 99.00th=[ 430], 99.50th=[ 481], 99.90th=[ 489], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 135, max= 475, per=24.89%, avg=248.39, stdev=56.53, samples=80 iops : min= 33, max= 118, avg=61.64, stdev=14.14, samples=80 lat (msec) : 50=0.04%, 100=0.23%, 250=48.83%, 500=50.90% cpu : usr=0.16%, sys=1.10%, ctx=2441, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179239: Wed Aug 5 14:22:27 2026 write: IOPS=62, BW=250KiB/s (256kB/s)(10.0MiB/40966msec); 0 zone resets slat (usec): min=48, max=27661, avg=321.58, stdev=1322.46 clat (msec): min=75, max=500, avg=255.44, stdev=63.77 lat (msec): min=76, max=500, avg=255.77, stdev=63.70 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 148], 10.00th=[ 182], 20.00th=[ 207], | 30.00th=[ 222], 40.00th=[ 234], 50.00th=[ 251], 60.00th=[ 268], | 70.00th=[ 284], 80.00th=[ 305], 90.00th=[ 338], 95.00th=[ 359], | 99.00th=[ 435], 99.50th=[ 481], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 127, max= 403, per=24.84%, avg=247.89, stdev=63.56, samples=80 iops : min= 31, max= 100, avg=61.54, stdev=15.91, samples=80 lat (msec) : 100=0.08%, 250=49.57%, 500=50.23%, 750=0.12% cpu : usr=0.14%, sys=0.86%, ctx=1514, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179240: Wed Aug 5 14:22:27 2026 write: IOPS=62, BW=250KiB/s (256kB/s)(10.0MiB/40988msec); 0 zone resets slat (usec): min=49, max=30001, avg=338.89, stdev=1360.48 clat (msec): min=28, max=498, avg=255.75, stdev=63.07 lat (msec): min=28, max=499, avg=256.09, stdev=63.05 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 155], 10.00th=[ 174], 20.00th=[ 207], | 30.00th=[ 224], 40.00th=[ 239], 50.00th=[ 251], 60.00th=[ 268], | 70.00th=[ 288], 80.00th=[ 309], 90.00th=[ 338], 95.00th=[ 363], | 99.00th=[ 422], 99.50th=[ 439], 99.90th=[ 493], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 140, max= 459, per=24.85%, avg=248.05, stdev=55.46, samples=80 iops : min= 35, max= 114, avg=61.61, stdev=13.84, samples=80 lat (msec) : 50=0.04%, 100=0.16%, 250=49.22%, 500=50.59% cpu : usr=0.12%, sys=0.86%, ctx=1877, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=999KiB/s (1023kB/s), 250KiB/s-250KiB/s (256kB/s-256kB/s), io=40.0MiB (41.9MB), run=40966-41002msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179258: Wed Aug 5 14:23:38 2026 read: IOPS=23, BW=94.8KiB/s (97.1kB/s)(5048KiB/53252msec) slat (msec): min=3, max=101, avg=24.60, stdev=15.81 clat (msec): min=34, max=685, avg=314.23, stdev=94.25 lat (msec): min=48, max=751, avg=338.83, stdev=98.50 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 180], 10.00th=[ 205], 20.00th=[ 234], | 30.00th=[ 257], 40.00th=[ 284], 50.00th=[ 309], 60.00th=[ 330], | 70.00th=[ 355], 80.00th=[ 384], 90.00th=[ 435], 95.00th=[ 489], | 99.00th=[ 584], 99.50th=[ 625], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 30, max= 181, per=25.01%, avg=94.03, stdev=29.25, samples=105 iops : min= 7, max= 45, avg=22.92, stdev= 7.40, samples=105 write: IOPS=24, BW=97.5KiB/s (99.8kB/s)(5192KiB/53252msec); 0 zone resets slat (usec): min=51, max=30900, avg=370.95, stdev=1518.94 clat (msec): min=59, max=767, avg=322.59, stdev=92.97 lat (msec): min=59, max=767, avg=322.97, stdev=92.98 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 186], 10.00th=[ 215], 20.00th=[ 249], | 30.00th=[ 271], 40.00th=[ 292], 50.00th=[ 313], 60.00th=[ 334], | 70.00th=[ 359], 80.00th=[ 393], 90.00th=[ 439], 95.00th=[ 493], | 99.00th=[ 600], 99.50th=[ 651], 99.90th=[ 760], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 31, max= 183, per=24.73%, avg=96.93, stdev=32.55, samples=105 iops : min= 7, max= 45, avg=23.65, stdev= 8.15, samples=105 lat (msec) : 50=0.08%, 100=0.04%, 250=23.20%, 500=72.34%, 750=4.26% lat (msec) : 1000=0.08% cpu : usr=0.17%, sys=1.35%, ctx=3692, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179259: Wed Aug 5 14:23:38 2026 read: IOPS=23, BW=93.5KiB/s (95.7kB/s)(4976KiB/53230msec) slat (msec): min=3, max=120, avg=24.70, stdev=16.10 clat (msec): min=62, max=813, avg=313.30, stdev=94.81 lat (msec): min=83, max=821, avg=338.01, stdev=99.04 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 178], 10.00th=[ 207], 20.00th=[ 239], | 30.00th=[ 259], 40.00th=[ 279], 50.00th=[ 305], 60.00th=[ 326], | 70.00th=[ 351], 80.00th=[ 380], 90.00th=[ 435], 95.00th=[ 481], | 99.00th=[ 600], 99.50th=[ 625], 99.90th=[ 735], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 14, max= 215, per=24.64%, avg=92.66, stdev=33.28, samples=105 iops : min= 3, max= 53, avg=22.58, stdev= 8.35, samples=105 write: IOPS=24, BW=98.9KiB/s (101kB/s)(5264KiB/53230msec); 0 zone resets slat (usec): min=51, max=78424, avg=481.68, stdev=2492.09 clat (msec): min=70, max=760, avg=323.82, stdev=97.17 lat (msec): min=70, max=760, avg=324.30, stdev=97.18 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 192], 10.00th=[ 220], 20.00th=[ 245], | 30.00th=[ 271], 40.00th=[ 292], 50.00th=[ 317], 60.00th=[ 338], | 70.00th=[ 359], 80.00th=[ 388], 90.00th=[ 443], 95.00th=[ 498], | 99.00th=[ 634], 99.50th=[ 684], 99.90th=[ 760], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 23, max= 207, per=25.10%, avg=98.41, stdev=34.58, samples=105 iops : min= 5, max= 51, avg=24.03, stdev= 8.71, samples=105 lat (msec) : 100=0.12%, 250=24.57%, 500=70.78%, 750=4.34%, 1000=0.20% cpu : usr=0.13%, sys=1.32%, ctx=3778, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179260: Wed Aug 5 14:23:38 2026 read: IOPS=23, BW=93.6KiB/s (95.8kB/s)(4984KiB/53274msec) slat (msec): min=3, max=104, avg=24.61, stdev=15.65 clat (msec): min=43, max=727, avg=315.46, stdev=95.38 lat (msec): min=85, max=799, avg=340.07, stdev=99.30 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 180], 10.00th=[ 205], 20.00th=[ 236], | 30.00th=[ 259], 40.00th=[ 284], 50.00th=[ 305], 60.00th=[ 326], | 70.00th=[ 355], 80.00th=[ 384], 90.00th=[ 443], 95.00th=[ 485], | 99.00th=[ 592], 99.50th=[ 676], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 22, max= 183, per=24.69%, avg=92.82, stdev=33.32, samples=105 iops : min= 5, max= 45, avg=22.62, stdev= 8.34, samples=105 write: IOPS=24, BW=98.7KiB/s (101kB/s)(5256KiB/53274msec); 0 zone resets slat (usec): min=50, max=17521, avg=377.90, stdev=1202.01 clat (msec): min=101, max=753, avg=321.89, stdev=93.07 lat (msec): min=101, max=753, avg=322.27, stdev=93.08 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 188], 10.00th=[ 213], 20.00th=[ 245], | 30.00th=[ 271], 40.00th=[ 292], 50.00th=[ 317], 60.00th=[ 338], | 70.00th=[ 359], 80.00th=[ 393], 90.00th=[ 439], 95.00th=[ 481], | 99.00th=[ 617], 99.50th=[ 667], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 29, max= 206, per=24.87%, avg=97.50, stdev=35.05, samples=105 iops : min= 7, max= 51, avg=23.80, stdev= 8.83, samples=105 lat (msec) : 50=0.04%, 100=0.04%, 250=23.75%, 500=72.30%, 750=3.79% lat (msec) : 1000=0.08% cpu : usr=0.14%, sys=1.30%, ctx=3635, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179261: Wed Aug 5 14:23:38 2026 read: IOPS=23, BW=95.3KiB/s (97.6kB/s)(5060KiB/53085msec) slat (usec): min=92, max=130479, avg=24894.92, stdev=16419.58 clat (msec): min=30, max=858, avg=314.28, stdev=98.69 lat (msec): min=31, max=889, avg=339.18, stdev=103.18 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 176], 10.00th=[ 205], 20.00th=[ 232], | 30.00th=[ 257], 40.00th=[ 279], 50.00th=[ 305], 60.00th=[ 330], | 70.00th=[ 355], 80.00th=[ 384], 90.00th=[ 443], 95.00th=[ 485], | 99.00th=[ 600], 99.50th=[ 659], 99.90th=[ 844], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 31, max= 191, per=25.13%, avg=94.50, stdev=31.35, samples=105 iops : min= 7, max= 47, avg=23.05, stdev= 7.89, samples=105 write: IOPS=24, BW=97.6KiB/s (99.9kB/s)(5180KiB/53085msec); 0 zone resets slat (usec): min=53, max=44039, avg=355.33, stdev=1616.31 clat (msec): min=111, max=791, avg=323.54, stdev=93.80 lat (msec): min=111, max=791, avg=323.90, stdev=93.85 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 194], 10.00th=[ 211], 20.00th=[ 249], | 30.00th=[ 271], 40.00th=[ 292], 50.00th=[ 313], 60.00th=[ 338], | 70.00th=[ 368], 80.00th=[ 397], 90.00th=[ 443], 95.00th=[ 485], | 99.00th=[ 584], 99.50th=[ 701], 99.90th=[ 760], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 31, max= 206, per=24.72%, avg=96.90, stdev=31.63, samples=105 iops : min= 7, max= 51, avg=23.64, stdev= 7.97, samples=105 lat (msec) : 50=0.20%, 100=0.12%, 250=23.75%, 500=71.95%, 750=3.75% lat (msec) : 1000=0.23% cpu : usr=0.18%, sys=1.49%, ctx=3861, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=377KiB/s (386kB/s), 93.5KiB/s-95.3KiB/s (95.7kB/s-97.6kB/s), io=19.6MiB (20.5MB), run=53085-53274msec WRITE: bw=392KiB/s (402kB/s), 97.5KiB/s-98.9KiB/s (99.8kB/s-101kB/s), io=20.4MiB (21.4MB), run=53085-53274msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179274: Wed Aug 5 14:23:56 2026 read: IOPS=1, BW=72.3MiB/s (75.8MB/s)(40.0MiB/553msec) slat (nsec): min=96112k, max=96112k, avg=96111682.00, stdev= 0.00 clat (nsec): min=413955k, max=413955k, avg=413955150.00, stdev= 0.00 lat (nsec): min=510072k, max=510072k, avg=510071689.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 414], 10.00th=[ 414], 20.00th=[ 414], | 30.00th=[ 414], 40.00th=[ 414], 50.00th=[ 414], 60.00th=[ 414], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=1.81%, ctx=8, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=72.3MiB/s (75.8MB/s), 72.3MiB/s-72.3MiB/s (75.8MB/s-75.8MB/s), io=40.0MiB (41.9MB), run=553-553msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout