== sanity test 398c: run fio to test AIO ================= 13:30:32 (1782235832) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.690621 s, 60.7 MB/s osc.lustre-OST0000-osc-ffff95bf82d30800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180143: Tue Jun 23 13:31:13 2026 write: IOPS=85, BW=344KiB/s (352kB/s)(10.0MiB/29809msec); 0 zone resets slat (usec): min=50, max=20180, avg=310.07, stdev=971.41 clat (msec): min=97, max=337, avg=185.16, stdev=44.12 lat (msec): min=97, max=341, avg=185.47, stdev=44.10 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 120], 10.00th=[ 130], 20.00th=[ 146], | 30.00th=[ 159], 40.00th=[ 169], 50.00th=[ 180], 60.00th=[ 194], | 70.00th=[ 209], 80.00th=[ 228], 90.00th=[ 249], 95.00th=[ 259], | 99.00th=[ 288], 99.50th=[ 296], 99.90th=[ 317], 99.95th=[ 326], | 99.99th=[ 338] bw ( KiB/s): min= 191, max= 512, per=24.91%, avg=342.24, stdev=71.74, samples=59 iops : min= 47, max= 128, avg=85.19, stdev=17.95, samples=59 lat (msec) : 100=1.21%, 250=89.34%, 500=9.45% cpu : usr=0.17%, sys=1.09%, ctx=2093, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180144: Tue Jun 23 13:31:13 2026 write: IOPS=85, BW=344KiB/s (352kB/s)(10.0MiB/29806msec); 0 zone resets slat (usec): min=46, max=17643, avg=320.87, stdev=855.13 clat (msec): min=19, max=311, avg=184.75, stdev=42.56 lat (msec): min=20, max=312, avg=185.07, stdev=42.54 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 122], 10.00th=[ 130], 20.00th=[ 146], | 30.00th=[ 163], 40.00th=[ 171], 50.00th=[ 184], 60.00th=[ 192], | 70.00th=[ 205], 80.00th=[ 224], 90.00th=[ 245], 95.00th=[ 257], | 99.00th=[ 279], 99.50th=[ 292], 99.90th=[ 296], 99.95th=[ 300], | 99.99th=[ 313] bw ( KiB/s): min= 108, max= 510, per=24.93%, avg=342.59, stdev=72.52, samples=59 iops : min= 27, max= 127, avg=85.27, stdev=18.11, samples=59 lat (msec) : 20=0.04%, 50=0.12%, 100=0.82%, 250=91.91%, 500=7.11% cpu : usr=0.13%, sys=1.16%, ctx=2785, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180145: Tue Jun 23 13:31:13 2026 write: IOPS=85, BW=344KiB/s (352kB/s)(10.0MiB/29785msec); 0 zone resets slat (usec): min=39, max=14512, avg=269.09, stdev=725.48 clat (msec): min=60, max=334, avg=184.96, stdev=42.83 lat (msec): min=60, max=334, avg=185.23, stdev=42.84 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 118], 10.00th=[ 130], 20.00th=[ 144], | 30.00th=[ 159], 40.00th=[ 171], 50.00th=[ 182], 60.00th=[ 194], | 70.00th=[ 209], 80.00th=[ 222], 90.00th=[ 243], 95.00th=[ 259], | 99.00th=[ 292], 99.50th=[ 296], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 334] bw ( KiB/s): min= 207, max= 512, per=24.90%, avg=342.19, stdev=80.73, samples=59 iops : min= 51, max= 128, avg=85.17, stdev=20.22, samples=59 lat (msec) : 100=0.08%, 250=92.23%, 500=7.70% cpu : usr=0.14%, sys=0.97%, ctx=2155, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180146: Tue Jun 23 13:31:13 2026 write: IOPS=86, BW=346KiB/s (354kB/s)(10.0MiB/29616msec); 0 zone resets slat (usec): min=44, max=17065, avg=320.77, stdev=875.16 clat (msec): min=20, max=383, avg=184.60, stdev=44.08 lat (msec): min=21, max=383, avg=184.92, stdev=44.10 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 120], 10.00th=[ 129], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 171], 50.00th=[ 182], 60.00th=[ 194], | 70.00th=[ 209], 80.00th=[ 224], 90.00th=[ 243], 95.00th=[ 259], | 99.00th=[ 292], 99.50th=[ 305], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 384] bw ( KiB/s): min= 231, max= 507, per=25.20%, avg=346.21, stdev=72.77, samples=58 iops : min= 57, max= 126, avg=86.14, stdev=18.19, samples=58 lat (msec) : 50=0.27%, 100=0.23%, 250=92.42%, 500=7.07% cpu : usr=0.19%, sys=1.28%, ctx=2434, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1374KiB/s (1407kB/s), 344KiB/s-346KiB/s (352kB/s-354kB/s), io=40.0MiB (41.9MB), run=29616-29809msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180161: Tue Jun 23 13:31:56 2026 read: IOPS=38, BW=153KiB/s (156kB/s)(5048KiB/33055msec) slat (usec): min=3177, max=53946, avg=14583.15, stdev=7610.29 clat (msec): min=14, max=365, avg=194.09, stdev=45.80 lat (msec): min=24, max=386, avg=208.68, stdev=47.31 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 129], 10.00th=[ 140], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 180], 50.00th=[ 190], 60.00th=[ 201], | 70.00th=[ 215], 80.00th=[ 232], 90.00th=[ 257], 95.00th=[ 275], | 99.00th=[ 317], 99.50th=[ 342], 99.90th=[ 363], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 44, max= 231, per=25.03%, avg=151.69, stdev=37.46, samples=65 iops : min= 11, max= 57, avg=37.46, stdev= 9.37, samples=65 write: IOPS=39, BW=157KiB/s (161kB/s)(5192KiB/33055msec); 0 zone resets slat (usec): min=41, max=12631, avg=350.42, stdev=779.17 clat (msec): min=46, max=354, avg=200.81, stdev=44.45 lat (msec): min=46, max=355, avg=201.17, stdev=44.45 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 133], 10.00th=[ 150], 20.00th=[ 163], | 30.00th=[ 176], 40.00th=[ 188], 50.00th=[ 197], 60.00th=[ 209], | 70.00th=[ 222], 80.00th=[ 239], 90.00th=[ 262], 95.00th=[ 279], | 99.00th=[ 321], 99.50th=[ 338], 99.90th=[ 342], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 51, max= 240, per=24.83%, avg=156.69, stdev=37.84, samples=65 iops : min= 12, max= 60, avg=38.69, stdev= 9.52, samples=65 lat (msec) : 20=0.08%, 50=0.04%, 100=0.27%, 250=86.60%, 500=13.01% cpu : usr=0.25%, sys=1.98%, ctx=4488, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180162: Tue Jun 23 13:31:56 2026 read: IOPS=37, BW=151KiB/s (154kB/s)(4976KiB/32997msec) slat (usec): min=2578, max=55439, avg=14991.52, stdev=7788.50 clat (msec): min=25, max=356, avg=194.13, stdev=45.72 lat (msec): min=41, max=370, avg=209.13, stdev=47.17 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 128], 10.00th=[ 140], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 203], | 70.00th=[ 215], 80.00th=[ 232], 90.00th=[ 255], 95.00th=[ 275], | 99.00th=[ 309], 99.50th=[ 317], 99.90th=[ 351], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 72, max= 223, per=24.74%, avg=149.91, stdev=35.47, samples=65 iops : min= 18, max= 55, avg=37.00, stdev= 8.89, samples=65 write: IOPS=39, BW=160KiB/s (163kB/s)(5264KiB/32997msec); 0 zone resets slat (usec): min=54, max=22943, avg=363.82, stdev=989.05 clat (msec): min=40, max=351, avg=202.13, stdev=44.49 lat (msec): min=40, max=351, avg=202.50, stdev=44.42 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 138], 10.00th=[ 150], 20.00th=[ 167], | 30.00th=[ 178], 40.00th=[ 188], 50.00th=[ 197], 60.00th=[ 209], | 70.00th=[ 222], 80.00th=[ 236], 90.00th=[ 262], 95.00th=[ 288], | 99.00th=[ 326], 99.50th=[ 338], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 94, max= 232, per=25.23%, avg=159.20, stdev=34.86, samples=65 iops : min= 23, max= 58, avg=39.34, stdev= 8.73, samples=65 lat (msec) : 50=0.16%, 100=0.43%, 250=86.64%, 500=12.77% cpu : usr=0.28%, sys=1.82%, ctx=4485, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180163: Tue Jun 23 13:31:56 2026 read: IOPS=37, BW=152KiB/s (155kB/s)(4984KiB/32844msec) slat (usec): min=2875, max=59972, avg=14591.82, stdev=7645.63 clat (msec): min=80, max=364, avg=193.38, stdev=44.78 lat (msec): min=104, max=377, avg=207.98, stdev=46.33 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 126], 10.00th=[ 138], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 190], 60.00th=[ 201], | 70.00th=[ 215], 80.00th=[ 230], 90.00th=[ 255], 95.00th=[ 275], | 99.00th=[ 313], 99.50th=[ 326], 99.90th=[ 347], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 36, max= 243, per=25.02%, avg=151.60, stdev=37.87, samples=65 iops : min= 9, max= 60, avg=37.43, stdev= 9.46, samples=65 write: IOPS=40, BW=160KiB/s (164kB/s)(5256KiB/32844msec); 0 zone resets slat (usec): min=53, max=9896, avg=370.41, stdev=803.66 clat (msec): min=66, max=350, avg=199.16, stdev=45.43 lat (msec): min=66, max=350, avg=199.53, stdev=45.45 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 132], 10.00th=[ 144], 20.00th=[ 161], | 30.00th=[ 174], 40.00th=[ 186], 50.00th=[ 197], 60.00th=[ 207], | 70.00th=[ 220], 80.00th=[ 234], 90.00th=[ 262], 95.00th=[ 284], | 99.00th=[ 321], 99.50th=[ 330], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 59, max= 256, per=25.23%, avg=159.18, stdev=42.47, samples=65 iops : min= 14, max= 64, avg=39.31, stdev=10.67, samples=65 lat (msec) : 100=0.27%, 250=87.58%, 500=12.15% cpu : usr=0.25%, sys=1.75%, ctx=4108, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180164: Tue Jun 23 13:31:56 2026 read: IOPS=38, BW=153KiB/s (157kB/s)(5060KiB/33068msec) slat (usec): min=3393, max=54437, avg=14713.25, stdev=7664.78 clat (msec): min=23, max=357, avg=194.52, stdev=44.68 lat (msec): min=35, max=375, avg=209.24, stdev=46.41 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 129], 10.00th=[ 140], 20.00th=[ 157], | 30.00th=[ 169], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 201], | 70.00th=[ 213], 80.00th=[ 230], 90.00th=[ 255], 95.00th=[ 275], | 99.00th=[ 313], 99.50th=[ 326], 99.90th=[ 351], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 44, max= 231, per=25.09%, avg=152.06, stdev=33.31, samples=65 iops : min= 11, max= 57, avg=37.54, stdev= 8.24, samples=65 write: IOPS=39, BW=157KiB/s (160kB/s)(5180KiB/33068msec); 0 zone resets slat (usec): min=53, max=14995, avg=362.68, stdev=943.24 clat (msec): min=89, max=392, avg=200.97, stdev=44.91 lat (msec): min=89, max=392, avg=201.33, stdev=44.94 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 136], 10.00th=[ 146], 20.00th=[ 163], | 30.00th=[ 174], 40.00th=[ 186], 50.00th=[ 199], 60.00th=[ 209], | 70.00th=[ 222], 80.00th=[ 236], 90.00th=[ 259], 95.00th=[ 284], | 99.00th=[ 326], 99.50th=[ 342], 99.90th=[ 372], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 51, max= 246, per=24.62%, avg=155.32, stdev=40.28, samples=65 iops : min= 12, max= 61, avg=38.34, stdev=10.15, samples=65 lat (msec) : 50=0.04%, 100=0.59%, 250=86.95%, 500=12.42% cpu : usr=0.24%, sys=1.95%, ctx=4482, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=607KiB/s (621kB/s), 151KiB/s-153KiB/s (154kB/s-157kB/s), io=19.6MiB (20.5MB), run=32844-33068msec WRITE: bw=632KiB/s (647kB/s), 157KiB/s-160KiB/s (160kB/s-164kB/s), io=20.4MiB (21.4MB), run=32844-33068msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180173: Tue Jun 23 13:32:07 2026 read: IOPS=2, BW=92.4MiB/s (96.9MB/s)(40.0MiB/433msec) slat (nsec): min=28568k, max=28568k, avg=28567627.00, stdev= 0.00 clat (nsec): min=384678k, max=384678k, avg=384678499.00, stdev= 0.00 lat (nsec): min=413251k, max=413251k, avg=413251069.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 384], 10.00th=[ 384], 20.00th=[ 384], | 30.00th=[ 384], 40.00th=[ 384], 50.00th=[ 384], 60.00th=[ 384], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.78%, ctx=13, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=92.4MiB/s (96.9MB/s), 92.4MiB/s-92.4MiB/s (96.9MB/s-96.9MB/s), io=40.0MiB (41.9MB), run=433-433msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout