== sanity test 398c: run fio to test AIO ================= 07:17:13 (1788607033) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.4602 s, 91.1 MB/s osc.lustre-OST0000-osc-ffff9a1490965800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=182045: Sat Sep 5 07:17:51 2026 write: IOPS=92, BW=369KiB/s (378kB/s)(10.0MiB/27770msec); 0 zone resets slat (usec): min=50, max=31545, avg=358.97, stdev=1129.52 clat (msec): min=17, max=318, avg=172.64, stdev=34.43 lat (msec): min=17, max=319, avg=173.00, stdev=34.45 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 126], 10.00th=[ 133], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 169], 60.00th=[ 178], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 222], 95.00th=[ 241], | 99.00th=[ 268], 99.50th=[ 284], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 321] bw ( KiB/s): min= 255, max= 469, per=24.94%, avg=367.60, stdev=56.13, samples=55 iops : min= 63, max= 117, avg=91.55, stdev=14.14, samples=55 lat (msec) : 20=0.04%, 50=0.20%, 100=0.20%, 250=97.70%, 500=1.88% cpu : usr=0.19%, sys=1.28%, ctx=2752, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182046: Sat Sep 5 07:17:51 2026 write: IOPS=92, BW=369KiB/s (377kB/s)(10.0MiB/27785msec); 0 zone resets slat (usec): min=51, max=25339, avg=306.18, stdev=1027.81 clat (msec): min=28, max=310, avg=172.83, stdev=33.92 lat (msec): min=28, max=310, avg=173.14, stdev=33.90 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 125], 10.00th=[ 134], 20.00th=[ 146], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 178], | 70.00th=[ 186], 80.00th=[ 194], 90.00th=[ 220], 95.00th=[ 239], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 309], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 250, max= 501, per=24.88%, avg=366.78, stdev=60.77, samples=55 iops : min= 62, max= 125, avg=91.33, stdev=15.31, samples=55 lat (msec) : 50=0.08%, 100=0.23%, 250=97.30%, 500=2.38% cpu : usr=0.15%, sys=1.14%, ctx=2230, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182047: Sat Sep 5 07:17:51 2026 write: IOPS=92, BW=370KiB/s (379kB/s)(10.0MiB/27676msec); 0 zone resets slat (usec): min=49, max=29900, avg=361.24, stdev=1190.37 clat (msec): min=29, max=362, avg=172.39, stdev=36.34 lat (msec): min=29, max=362, avg=172.76, stdev=36.32 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 126], 10.00th=[ 133], 20.00th=[ 144], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 180], | 70.00th=[ 186], 80.00th=[ 197], 90.00th=[ 222], 95.00th=[ 241], | 99.00th=[ 266], 99.50th=[ 300], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 254, max= 486, per=25.15%, avg=370.67, stdev=54.50, samples=54 iops : min= 63, max= 121, avg=92.31, stdev=13.74, samples=54 lat (msec) : 50=0.27%, 100=1.09%, 250=95.86%, 500=2.77% cpu : usr=0.21%, sys=1.43%, ctx=2953, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182048: Sat Sep 5 07:17:51 2026 write: IOPS=92, BW=369KiB/s (377kB/s)(10.0MiB/27788msec); 0 zone resets slat (usec): min=50, max=41927, avg=330.88, stdev=1270.99 clat (msec): min=13, max=300, avg=172.61, stdev=33.54 lat (msec): min=14, max=300, avg=172.95, stdev=33.55 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 129], 10.00th=[ 136], 20.00th=[ 146], | 30.00th=[ 153], 40.00th=[ 161], 50.00th=[ 169], 60.00th=[ 178], | 70.00th=[ 186], 80.00th=[ 194], 90.00th=[ 224], 95.00th=[ 236], | 99.00th=[ 259], 99.50th=[ 268], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 255, max= 508, per=24.88%, avg=366.76, stdev=60.58, samples=55 iops : min= 63, max= 127, avg=91.33, stdev=15.21, samples=55 lat (msec) : 20=0.04%, 50=0.08%, 100=0.62%, 250=97.19%, 500=2.07% cpu : usr=0.16%, sys=1.22%, ctx=2632, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1474KiB/s (1509kB/s), 369KiB/s-370KiB/s (377kB/s-379kB/s), io=40.0MiB (41.9MB), run=27676-27788msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=182062: Sat Sep 5 07:18:34 2026 read: IOPS=36, BW=146KiB/s (150kB/s)(5048KiB/34519msec) slat (usec): min=2706, max=70497, avg=15408.04, stdev=8552.46 clat (msec): min=73, max=413, avg=202.76, stdev=52.60 lat (msec): min=79, max=448, avg=218.18, stdev=55.47 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 126], 10.00th=[ 140], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 197], 60.00th=[ 211], | 70.00th=[ 224], 80.00th=[ 245], 90.00th=[ 275], 95.00th=[ 296], | 99.00th=[ 347], 99.50th=[ 355], 99.90th=[ 401], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 70, max= 246, per=25.03%, avg=145.40, stdev=37.90, samples=68 iops : min= 17, max= 61, avg=35.85, stdev= 9.51, samples=68 write: IOPS=37, BW=150KiB/s (154kB/s)(5192KiB/34519msec); 0 zone resets slat (usec): min=54, max=17419, avg=366.26, stdev=1041.05 clat (msec): min=76, max=417, avg=209.89, stdev=51.30 lat (msec): min=77, max=421, avg=210.26, stdev=51.28 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 140], 10.00th=[ 150], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 192], 50.00th=[ 205], 60.00th=[ 218], | 70.00th=[ 232], 80.00th=[ 255], 90.00th=[ 279], 95.00th=[ 305], | 99.00th=[ 342], 99.50th=[ 368], 99.90th=[ 401], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 87, max= 247, per=24.71%, avg=149.51, stdev=40.35, samples=68 iops : min= 21, max= 61, avg=36.87, stdev=10.07, samples=68 lat (msec) : 100=0.66%, 250=79.53%, 500=19.80% cpu : usr=0.24%, sys=2.00%, ctx=4484, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182063: Sat Sep 5 07:18:34 2026 read: IOPS=36, BW=145KiB/s (148kB/s)(4976KiB/34408msec) slat (usec): min=3109, max=67626, avg=15075.98, stdev=8433.54 clat (msec): min=79, max=426, avg=202.58, stdev=53.34 lat (msec): min=85, max=436, avg=217.67, stdev=55.36 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 126], 10.00th=[ 140], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 197], 60.00th=[ 211], | 70.00th=[ 226], 80.00th=[ 243], 90.00th=[ 275], 95.00th=[ 305], | 99.00th=[ 359], 99.50th=[ 376], 99.90th=[ 384], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 62, max= 248, per=24.81%, avg=144.13, stdev=39.57, samples=67 iops : min= 15, max= 62, avg=35.54, stdev= 9.94, samples=67 write: IOPS=38, BW=153KiB/s (157kB/s)(5264KiB/34408msec); 0 zone resets slat (usec): min=53, max=16150, avg=373.03, stdev=979.78 clat (msec): min=93, max=436, avg=210.16, stdev=52.02 lat (msec): min=94, max=436, avg=210.53, stdev=52.01 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 140], 10.00th=[ 153], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 215], | 70.00th=[ 232], 80.00th=[ 251], 90.00th=[ 275], 95.00th=[ 305], | 99.00th=[ 363], 99.50th=[ 380], 99.90th=[ 435], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 69, max= 238, per=25.30%, avg=153.09, stdev=37.64, samples=67 iops : min= 17, max= 59, avg=37.79, stdev= 9.40, samples=67 lat (msec) : 100=0.43%, 250=80.78%, 500=18.79% cpu : usr=0.25%, sys=1.74%, ctx=4594, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182064: Sat Sep 5 07:18:34 2026 read: IOPS=36, BW=144KiB/s (148kB/s)(4984KiB/34494msec) slat (usec): min=2931, max=75535, avg=15194.43, stdev=8651.06 clat (msec): min=77, max=386, avg=203.03, stdev=51.88 lat (msec): min=82, max=413, avg=218.24, stdev=54.12 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 127], 10.00th=[ 140], 20.00th=[ 161], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 199], 60.00th=[ 211], | 70.00th=[ 226], 80.00th=[ 245], 90.00th=[ 271], 95.00th=[ 300], | 99.00th=[ 355], 99.50th=[ 359], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 87, max= 222, per=24.94%, avg=144.90, stdev=34.83, samples=67 iops : min= 21, max= 55, avg=35.75, stdev= 8.73, samples=67 write: IOPS=38, BW=152KiB/s (156kB/s)(5256KiB/34494msec); 0 zone resets slat (usec): min=56, max=36972, avg=418.07, stdev=1495.11 clat (msec): min=73, max=401, avg=209.26, stdev=49.88 lat (msec): min=74, max=401, avg=209.69, stdev=49.92 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 138], 10.00th=[ 150], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 192], 50.00th=[ 203], 60.00th=[ 218], | 70.00th=[ 232], 80.00th=[ 251], 90.00th=[ 279], 95.00th=[ 300], | 99.00th=[ 338], 99.50th=[ 359], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 62, max= 247, per=25.12%, avg=151.99, stdev=38.68, samples=67 iops : min= 15, max= 61, avg=37.51, stdev= 9.66, samples=67 lat (msec) : 100=0.39%, 250=81.17%, 500=18.44% cpu : usr=0.23%, sys=1.97%, ctx=4634, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182065: Sat Sep 5 07:18:34 2026 read: IOPS=36, BW=147KiB/s (150kB/s)(5060KiB/34439msec) slat (usec): min=89, max=99955, avg=15435.62, stdev=8851.36 clat (msec): min=24, max=404, avg=205.33, stdev=56.61 lat (msec): min=24, max=422, avg=220.77, stdev=59.02 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 124], 10.00th=[ 138], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 186], 50.00th=[ 199], 60.00th=[ 213], | 70.00th=[ 228], 80.00th=[ 249], 90.00th=[ 284], 95.00th=[ 313], | 99.00th=[ 359], 99.50th=[ 372], 99.90th=[ 401], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 47, max= 239, per=25.27%, avg=146.84, stdev=32.38, samples=67 iops : min= 11, max= 59, avg=36.21, stdev= 8.08, samples=67 write: IOPS=37, BW=150KiB/s (154kB/s)(5180KiB/34439msec); 0 zone resets slat (usec): min=57, max=13360, avg=346.89, stdev=947.98 clat (msec): min=58, max=467, avg=209.08, stdev=50.58 lat (msec): min=58, max=467, avg=209.43, stdev=50.56 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 140], 10.00th=[ 150], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 192], 50.00th=[ 205], 60.00th=[ 215], | 70.00th=[ 228], 80.00th=[ 249], 90.00th=[ 279], 95.00th=[ 296], | 99.00th=[ 347], 99.50th=[ 363], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 86, max= 288, per=24.75%, avg=149.75, stdev=40.82, samples=67 iops : min= 21, max= 72, avg=36.96, stdev=10.27, samples=67 lat (msec) : 50=0.16%, 100=0.55%, 250=79.49%, 500=19.80% cpu : usr=0.25%, sys=1.97%, ctx=4532, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=581KiB/s (595kB/s), 144KiB/s-147KiB/s (148kB/s-150kB/s), io=19.6MiB (20.5MB), run=34408-34519msec WRITE: bw=605KiB/s (620kB/s), 150KiB/s-153KiB/s (154kB/s-157kB/s), io=20.4MiB (21.4MB), run=34408-34519msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=182075: Sat Sep 5 07:18:43 2026 read: IOPS=1, BW=68.5MiB/s (71.8MB/s)(40.0MiB/584msec) slat (nsec): min=69957k, max=69957k, avg=69956876.00, stdev= 0.00 clat (nsec): min=486858k, max=486858k, avg=486857510.00, stdev= 0.00 lat (nsec): min=556821k, max=556821k, avg=556821034.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 489], 5.00th=[ 489], 10.00th=[ 489], 20.00th=[ 489], | 30.00th=[ 489], 40.00th=[ 489], 50.00th=[ 489], 60.00th=[ 489], | 70.00th=[ 489], 80.00th=[ 489], 90.00th=[ 489], 95.00th=[ 489], | 99.00th=[ 489], 99.50th=[ 489], 99.90th=[ 489], 99.95th=[ 489], | 99.99th=[ 489] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.23%, ctx=5, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=68.5MiB/s (71.8MB/s), 68.5MiB/s-68.5MiB/s (71.8MB/s-71.8MB/s), io=40.0MiB (41.9MB), run=584-584msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout