== sanity test 398c: run fio to test AIO ================= 09:32:21 (1787751141) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.543437 s, 77.2 MB/s osc.lustre-OST0000-osc-ffff9bbfe13c2800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=178693: Wed Aug 26 09:33:04 2026 write: IOPS=80, BW=322KiB/s (329kB/s)(10.0MiB/31835msec); 0 zone resets slat (usec): min=27, max=21079, avg=161.25, stdev=747.81 clat (msec): min=27, max=356, avg=197.99, stdev=46.40 lat (msec): min=27, max=356, avg=198.15, stdev=46.38 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 128], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 171], 40.00th=[ 184], 50.00th=[ 197], 60.00th=[ 207], | 70.00th=[ 220], 80.00th=[ 239], 90.00th=[ 259], 95.00th=[ 279], | 99.00th=[ 317], 99.50th=[ 347], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 355] bw ( KiB/s): min= 183, max= 509, per=24.93%, avg=320.54, stdev=70.21, samples=63 iops : min= 45, max= 127, avg=79.78, stdev=17.57, samples=63 lat (msec) : 50=0.16%, 100=0.04%, 250=86.80%, 500=13.01% cpu : usr=0.13%, sys=0.74%, ctx=1617, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178694: Wed Aug 26 09:33:04 2026 write: IOPS=80, BW=324KiB/s (331kB/s)(10.0MiB/31643msec); 0 zone resets slat (usec): min=30, max=18752, avg=180.68, stdev=702.78 clat (msec): min=43, max=367, avg=197.35, stdev=48.44 lat (msec): min=43, max=367, avg=197.53, stdev=48.42 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 126], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 171], 40.00th=[ 184], 50.00th=[ 197], 60.00th=[ 205], | 70.00th=[ 220], 80.00th=[ 236], 90.00th=[ 259], 95.00th=[ 284], | 99.00th=[ 313], 99.50th=[ 338], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 368] bw ( KiB/s): min= 173, max= 512, per=25.12%, avg=323.06, stdev=71.53, samples=62 iops : min= 43, max= 128, avg=80.39, stdev=17.88, samples=62 lat (msec) : 50=0.04%, 100=1.37%, 250=83.98%, 500=14.61% cpu : usr=0.12%, sys=0.74%, ctx=1871, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178695: Wed Aug 26 09:33:04 2026 write: IOPS=80, BW=322KiB/s (329kB/s)(10.0MiB/31832msec); 0 zone resets slat (usec): min=28, max=11903, avg=174.75, stdev=620.01 clat (msec): min=23, max=345, avg=198.02, stdev=46.21 lat (msec): min=23, max=345, avg=198.19, stdev=46.18 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 127], 10.00th=[ 146], 20.00th=[ 161], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 194], 60.00th=[ 207], | 70.00th=[ 218], 80.00th=[ 239], 90.00th=[ 257], 95.00th=[ 279], | 99.00th=[ 321], 99.50th=[ 338], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 199, max= 507, per=24.93%, avg=320.57, stdev=69.04, samples=63 iops : min= 49, max= 126, avg=79.76, stdev=17.29, samples=63 lat (msec) : 50=0.20%, 100=0.98%, 250=85.39%, 500=13.44% cpu : usr=0.17%, sys=0.66%, ctx=1561, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178696: Wed Aug 26 09:33:04 2026 write: IOPS=80, BW=322KiB/s (329kB/s)(10.0MiB/31846msec); 0 zone resets slat (usec): min=28, max=18335, avg=217.67, stdev=797.64 clat (msec): min=19, max=361, avg=198.07, stdev=46.48 lat (msec): min=19, max=361, avg=198.29, stdev=46.49 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 127], 10.00th=[ 144], 20.00th=[ 159], | 30.00th=[ 169], 40.00th=[ 184], 50.00th=[ 197], 60.00th=[ 207], | 70.00th=[ 222], 80.00th=[ 239], 90.00th=[ 262], 95.00th=[ 275], | 99.00th=[ 313], 99.50th=[ 342], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 175, max= 470, per=24.93%, avg=320.62, stdev=65.23, samples=63 iops : min= 43, max= 117, avg=79.79, stdev=16.32, samples=63 lat (msec) : 20=0.04%, 50=0.12%, 100=0.39%, 250=85.78%, 500=13.67% cpu : usr=0.19%, sys=0.99%, ctx=2993, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1286KiB/s (1317kB/s), 322KiB/s-324KiB/s (329kB/s-331kB/s), io=40.0MiB (41.9MB), run=31643-31846msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=178711: Wed Aug 26 09:33:48 2026 read: IOPS=36, BW=145KiB/s (148kB/s)(5048KiB/34836msec) slat (usec): min=2647, max=48184, avg=14678.87, stdev=7987.55 clat (msec): min=79, max=437, avg=205.41, stdev=51.43 lat (msec): min=85, max=478, avg=220.09, stdev=53.55 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 131], 10.00th=[ 144], 20.00th=[ 161], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 201], 60.00th=[ 215], | 70.00th=[ 230], 80.00th=[ 251], 90.00th=[ 275], 95.00th=[ 292], | 99.00th=[ 334], 99.50th=[ 355], 99.90th=[ 430], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 71, max= 230, per=25.07%, avg=144.13, stdev=38.07, samples=69 iops : min= 17, max= 57, avg=35.49, stdev= 9.55, samples=69 write: IOPS=37, BW=149KiB/s (153kB/s)(5192KiB/34836msec); 0 zone resets slat (usec): min=31, max=11587, avg=218.46, stdev=712.49 clat (msec): min=99, max=371, avg=212.51, stdev=47.31 lat (msec): min=99, max=371, avg=212.73, stdev=47.32 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 142], 10.00th=[ 157], 20.00th=[ 174], | 30.00th=[ 184], 40.00th=[ 194], 50.00th=[ 207], 60.00th=[ 220], | 70.00th=[ 234], 80.00th=[ 253], 90.00th=[ 279], 95.00th=[ 305], | 99.00th=[ 334], 99.50th=[ 338], 99.90th=[ 368], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 59, max= 231, per=24.78%, avg=148.45, stdev=40.22, samples=69 iops : min= 14, max= 57, avg=36.57, stdev=10.07, samples=69 lat (msec) : 100=0.47%, 250=78.75%, 500=20.78% cpu : usr=0.21%, sys=1.52%, ctx=4085, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=178712: Wed Aug 26 09:33:48 2026 read: IOPS=35, BW=144KiB/s (147kB/s)(4976KiB/34633msec) slat (usec): min=33, max=70020, avg=14348.11, stdev=8001.03 clat (msec): min=28, max=416, avg=206.01, stdev=51.20 lat (msec): min=34, max=438, avg=220.36, stdev=53.00 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 133], 10.00th=[ 146], 20.00th=[ 161], | 30.00th=[ 176], 40.00th=[ 188], 50.00th=[ 201], 60.00th=[ 215], | 70.00th=[ 230], 80.00th=[ 249], 90.00th=[ 275], 95.00th=[ 300], | 99.00th=[ 334], 99.50th=[ 338], 99.90th=[ 405], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 63, max= 246, per=24.87%, avg=142.99, stdev=36.58, samples=68 iops : min= 15, max= 61, avg=35.19, stdev= 9.15, samples=68 write: IOPS=37, BW=152KiB/s (156kB/s)(5264KiB/34633msec); 0 zone resets slat (usec): min=33, max=15109, avg=231.09, stdev=743.38 clat (msec): min=31, max=380, avg=211.54, stdev=49.87 lat (msec): min=31, max=380, avg=211.78, stdev=49.91 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 138], 10.00th=[ 150], 20.00th=[ 169], | 30.00th=[ 182], 40.00th=[ 197], 50.00th=[ 207], 60.00th=[ 222], | 70.00th=[ 236], 80.00th=[ 253], 90.00th=[ 279], 95.00th=[ 300], | 99.00th=[ 334], 99.50th=[ 347], 99.90th=[ 363], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 63, max= 269, per=25.37%, avg=151.97, stdev=40.67, samples=68 iops : min= 15, max= 67, avg=37.46, stdev=10.23, samples=68 lat (msec) : 50=0.08%, 100=0.62%, 250=78.83%, 500=20.47% cpu : usr=0.19%, sys=1.48%, ctx=4189, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=178713: Wed Aug 26 09:33:48 2026 read: IOPS=35, BW=143KiB/s (146kB/s)(4984KiB/34841msec) slat (usec): min=3004, max=54675, avg=14543.37, stdev=7663.99 clat (msec): min=83, max=383, avg=205.57, stdev=50.58 lat (msec): min=94, max=401, avg=220.12, stdev=52.84 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 133], 10.00th=[ 144], 20.00th=[ 161], | 30.00th=[ 174], 40.00th=[ 186], 50.00th=[ 201], 60.00th=[ 215], | 70.00th=[ 234], 80.00th=[ 251], 90.00th=[ 275], 95.00th=[ 296], | 99.00th=[ 330], 99.50th=[ 334], 99.90th=[ 376], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 51, max= 231, per=24.84%, avg=142.84, stdev=35.93, samples=69 iops : min= 12, max= 57, avg=35.16, stdev= 9.03, samples=69 write: IOPS=37, BW=151KiB/s (154kB/s)(5256KiB/34841msec); 0 zone resets slat (usec): min=31, max=11637, avg=228.31, stdev=723.08 clat (msec): min=96, max=377, avg=211.98, stdev=47.63 lat (msec): min=97, max=377, avg=212.22, stdev=47.62 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 142], 10.00th=[ 155], 20.00th=[ 169], | 30.00th=[ 186], 40.00th=[ 197], 50.00th=[ 207], 60.00th=[ 222], | 70.00th=[ 236], 80.00th=[ 255], 90.00th=[ 279], 95.00th=[ 296], | 99.00th=[ 326], 99.50th=[ 338], 99.90th=[ 372], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 59, max= 214, per=24.99%, avg=149.70, stdev=38.73, samples=69 iops : min= 14, max= 53, avg=36.88, stdev= 9.71, samples=69 lat (msec) : 100=0.35%, 250=78.12%, 500=21.52% cpu : usr=0.23%, sys=1.44%, ctx=4424, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=178714: Wed Aug 26 09:33:48 2026 read: IOPS=36, BW=145KiB/s (149kB/s)(5060KiB/34813msec) slat (usec): min=2883, max=64051, avg=14371.62, stdev=7680.94 clat (msec): min=74, max=395, avg=205.52, stdev=50.66 lat (msec): min=79, max=410, avg=219.90, stdev=52.59 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 133], 10.00th=[ 146], 20.00th=[ 163], | 30.00th=[ 176], 40.00th=[ 188], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 230], 80.00th=[ 249], 90.00th=[ 275], 95.00th=[ 296], | 99.00th=[ 338], 99.50th=[ 342], 99.90th=[ 368], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 71, max= 231, per=25.13%, avg=144.51, stdev=35.39, samples=69 iops : min= 17, max= 57, avg=35.58, stdev= 8.83, samples=69 write: IOPS=37, BW=149KiB/s (152kB/s)(5180KiB/34813msec); 0 zone resets slat (usec): min=30, max=8267, avg=186.97, stdev=518.11 clat (msec): min=97, max=426, avg=212.28, stdev=49.38 lat (msec): min=97, max=426, avg=212.47, stdev=49.38 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 142], 10.00th=[ 155], 20.00th=[ 169], | 30.00th=[ 184], 40.00th=[ 197], 50.00th=[ 209], 60.00th=[ 222], | 70.00th=[ 236], 80.00th=[ 253], 90.00th=[ 279], 95.00th=[ 305], | 99.00th=[ 338], 99.50th=[ 351], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 63, max= 253, per=24.71%, avg=148.03, stdev=38.03, samples=69 iops : min= 15, max= 63, avg=36.48, stdev= 9.54, samples=69 lat (msec) : 100=0.51%, 250=78.91%, 500=20.59% cpu : usr=0.25%, sys=1.34%, ctx=3938, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=576KiB/s (590kB/s), 143KiB/s-145KiB/s (146kB/s-149kB/s), io=19.6MiB (20.5MB), run=34633-34841msec WRITE: bw=600KiB/s (614kB/s), 149KiB/s-152KiB/s (152kB/s-156kB/s), io=20.4MiB (21.4MB), run=34633-34841msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=178724: Wed Aug 26 09:33:59 2026 read: IOPS=1, BW=75.6MiB/s (79.3MB/s)(40.0MiB/529msec) slat (nsec): min=39744k, max=39744k, avg=39743706.00, stdev= 0.00 clat (nsec): min=452664k, max=452664k, avg=452664258.00, stdev= 0.00 lat (nsec): min=492414k, max=492414k, avg=492413564.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 451], 10.00th=[ 451], 20.00th=[ 451], | 30.00th=[ 451], 40.00th=[ 451], 50.00th=[ 451], 60.00th=[ 451], | 70.00th=[ 451], 80.00th=[ 451], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.65%, ctx=5, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=75.6MiB/s (79.3MB/s), 75.6MiB/s-75.6MiB/s (79.3MB/s-79.3MB/s), io=40.0MiB (41.9MB), run=529-529msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout