== sanity test 398c: run fio to test AIO ================= 18:48:57 (1787870937) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 1.04672 s, 40.1 MB/s osc.lustre-OST0000-osc-ffff8e2388572000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180670: Thu Aug 27 18:49:39 2026 write: IOPS=87, BW=350KiB/s (359kB/s)(10.0MiB/29225msec); 0 zone resets slat (usec): min=30, max=13145, avg=260.02, stdev=903.85 clat (msec): min=33, max=303, avg=182.17, stdev=36.67 lat (msec): min=33, max=303, avg=182.44, stdev=36.63 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 130], 10.00th=[ 144], 20.00th=[ 155], | 30.00th=[ 163], 40.00th=[ 169], 50.00th=[ 178], 60.00th=[ 190], | 70.00th=[ 197], 80.00th=[ 213], 90.00th=[ 232], 95.00th=[ 247], | 99.00th=[ 279], 99.50th=[ 288], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 305] bw ( KiB/s): min= 247, max= 510, per=24.98%, avg=348.53, stdev=54.51, samples=57 iops : min= 61, max= 127, avg=86.75, stdev=13.60, samples=57 lat (msec) : 50=0.08%, 100=1.48%, 250=93.95%, 500=4.49% cpu : usr=0.21%, sys=1.04%, ctx=2927, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180671: Thu Aug 27 18:49:39 2026 write: IOPS=87, BW=349KiB/s (357kB/s)(10.0MiB/29347msec); 0 zone resets slat (usec): min=28, max=24692, avg=203.25, stdev=862.46 clat (msec): min=23, max=301, avg=182.02, stdev=35.76 lat (msec): min=23, max=301, avg=182.23, stdev=35.79 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 127], 10.00th=[ 144], 20.00th=[ 155], | 30.00th=[ 163], 40.00th=[ 169], 50.00th=[ 180], 60.00th=[ 188], | 70.00th=[ 199], 80.00th=[ 209], 90.00th=[ 230], 95.00th=[ 245], | 99.00th=[ 275], 99.50th=[ 292], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 254, max= 507, per=24.86%, avg=346.84, stdev=51.83, samples=57 iops : min= 63, max= 126, avg=86.35, stdev=12.94, samples=57 lat (msec) : 50=0.12%, 100=0.98%, 250=95.00%, 500=3.91% cpu : usr=0.17%, sys=0.91%, ctx=2373, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180672: Thu Aug 27 18:49:39 2026 write: IOPS=87, BW=349KiB/s (358kB/s)(10.0MiB/29318msec); 0 zone resets slat (usec): min=30, max=12141, avg=173.07, stdev=598.38 clat (msec): min=84, max=301, avg=181.97, stdev=34.54 lat (msec): min=85, max=302, avg=182.14, stdev=34.54 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 132], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 165], 40.00th=[ 171], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 197], 80.00th=[ 205], 90.00th=[ 228], 95.00th=[ 245], | 99.00th=[ 275], 99.50th=[ 288], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 215, max= 500, per=24.88%, avg=347.12, stdev=54.53, samples=57 iops : min= 53, max= 125, avg=86.39, stdev=13.61, samples=57 lat (msec) : 100=1.05%, 250=94.73%, 500=4.22% cpu : usr=0.18%, sys=0.78%, ctx=1890, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180673: Thu Aug 27 18:49:39 2026 write: IOPS=87, BW=350KiB/s (359kB/s)(10.0MiB/29248msec); 0 zone resets slat (usec): min=31, max=13486, avg=203.21, stdev=781.23 clat (msec): min=16, max=303, avg=182.18, stdev=35.41 lat (msec): min=16, max=303, avg=182.38, stdev=35.40 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 130], 10.00th=[ 144], 20.00th=[ 157], | 30.00th=[ 165], 40.00th=[ 171], 50.00th=[ 180], 60.00th=[ 188], | 70.00th=[ 197], 80.00th=[ 211], 90.00th=[ 230], 95.00th=[ 245], | 99.00th=[ 275], 99.50th=[ 292], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 191, max= 471, per=24.96%, avg=348.25, stdev=49.46, samples=57 iops : min= 47, max= 117, avg=86.70, stdev=12.35, samples=57 lat (msec) : 20=0.04%, 50=0.12%, 100=0.78%, 250=94.80%, 500=4.26% cpu : usr=0.21%, sys=0.89%, ctx=2533, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1396KiB/s (1429kB/s), 349KiB/s-350KiB/s (357kB/s-359kB/s), io=40.0MiB (41.9MB), run=29225-29347msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180687: Thu Aug 27 18:50:21 2026 read: IOPS=38, BW=154KiB/s (157kB/s)(5048KiB/32871msec) slat (usec): min=3111, max=65226, avg=13807.30, stdev=7558.41 clat (msec): min=24, max=398, avg=192.88, stdev=47.00 lat (msec): min=41, max=413, avg=206.70, stdev=48.95 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 124], 10.00th=[ 138], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 188], 60.00th=[ 201], | 70.00th=[ 213], 80.00th=[ 234], 90.00th=[ 255], 95.00th=[ 279], | 99.00th=[ 326], 99.50th=[ 334], 99.90th=[ 351], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 86, max= 239, per=25.18%, avg=153.11, stdev=36.63, samples=65 iops : min= 21, max= 59, avg=37.83, stdev= 9.16, samples=65 write: IOPS=39, BW=158KiB/s (162kB/s)(5192KiB/32871msec); 0 zone resets slat (usec): min=37, max=16832, avg=247.20, stdev=856.95 clat (msec): min=90, max=413, avg=201.37, stdev=46.03 lat (msec): min=90, max=413, avg=201.62, stdev=46.01 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 136], 10.00th=[ 144], 20.00th=[ 161], | 30.00th=[ 174], 40.00th=[ 186], 50.00th=[ 199], 60.00th=[ 209], | 70.00th=[ 224], 80.00th=[ 241], 90.00th=[ 264], 95.00th=[ 279], | 99.00th=[ 326], 99.50th=[ 347], 99.90th=[ 397], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 79, max= 239, per=24.92%, avg=157.74, stdev=33.26, samples=65 iops : min= 19, max= 59, avg=38.98, stdev= 8.37, samples=65 lat (msec) : 50=0.08%, 100=0.47%, 250=86.25%, 500=13.20% cpu : usr=0.27%, sys=1.76%, ctx=4464, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180688: Thu Aug 27 18:50:21 2026 read: IOPS=37, BW=151KiB/s (155kB/s)(4976KiB/32978msec) slat (usec): min=166, max=172169, avg=14265.24, stdev=8947.20 clat (msec): min=26, max=389, avg=194.58, stdev=50.04 lat (msec): min=26, max=409, avg=208.85, stdev=51.47 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 122], 10.00th=[ 136], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 180], 50.00th=[ 190], 60.00th=[ 205], | 70.00th=[ 218], 80.00th=[ 232], 90.00th=[ 259], 95.00th=[ 288], | 99.00th=[ 321], 99.50th=[ 347], 99.90th=[ 368], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 71, max= 271, per=24.50%, avg=148.97, stdev=39.82, samples=65 iops : min= 17, max= 67, avg=36.78, stdev=10.04, samples=65 write: IOPS=39, BW=160KiB/s (163kB/s)(5264KiB/32978msec); 0 zone resets slat (usec): min=32, max=43837, avg=255.22, stdev=1369.59 clat (msec): min=26, max=350, avg=201.56, stdev=47.53 lat (msec): min=26, max=350, avg=201.82, stdev=47.60 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 131], 10.00th=[ 144], 20.00th=[ 161], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 199], 60.00th=[ 211], | 70.00th=[ 224], 80.00th=[ 243], 90.00th=[ 264], 95.00th=[ 288], | 99.00th=[ 326], 99.50th=[ 342], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 71, max= 207, per=25.08%, avg=158.74, stdev=32.11, samples=65 iops : min= 17, max= 51, avg=39.25, stdev= 8.04, samples=65 lat (msec) : 50=0.23%, 100=0.86%, 250=84.53%, 500=14.38% cpu : usr=0.29%, sys=1.80%, ctx=4419, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180689: Thu Aug 27 18:50:21 2026 read: IOPS=38, BW=152KiB/s (156kB/s)(4984KiB/32750msec) slat (usec): min=39, max=87004, avg=14020.40, stdev=8122.92 clat (msec): min=20, max=381, avg=193.94, stdev=49.13 lat (msec): min=25, max=416, avg=207.97, stdev=51.18 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 126], 10.00th=[ 138], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 186], 60.00th=[ 201], | 70.00th=[ 215], 80.00th=[ 232], 90.00th=[ 264], 95.00th=[ 288], | 99.00th=[ 326], 99.50th=[ 330], 99.90th=[ 359], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 79, max= 251, per=24.94%, avg=151.65, stdev=41.23, samples=65 iops : min= 19, max= 62, avg=37.46, stdev=10.35, samples=65 write: IOPS=40, BW=160KiB/s (164kB/s)(5256KiB/32750msec); 0 zone resets slat (usec): min=30, max=15186, avg=234.89, stdev=812.20 clat (msec): min=40, max=394, avg=200.84, stdev=48.40 lat (msec): min=41, max=394, avg=201.08, stdev=48.41 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 132], 10.00th=[ 144], 20.00th=[ 161], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 197], 60.00th=[ 209], | 70.00th=[ 222], 80.00th=[ 243], 90.00th=[ 266], 95.00th=[ 288], | 99.00th=[ 338], 99.50th=[ 359], 99.90th=[ 384], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 79, max= 240, per=25.09%, avg=158.80, stdev=37.14, samples=65 iops : min= 19, max= 60, avg=39.25, stdev= 9.31, samples=65 lat (msec) : 50=0.12%, 100=0.70%, 250=84.49%, 500=14.69% cpu : usr=0.30%, sys=1.88%, ctx=4346, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180690: Thu Aug 27 18:50:21 2026 read: IOPS=38, BW=154KiB/s (157kB/s)(5060KiB/32924msec) slat (usec): min=2911, max=59154, avg=14263.68, stdev=7780.94 clat (msec): min=25, max=379, avg=194.16, stdev=46.85 lat (msec): min=40, max=410, avg=208.43, stdev=48.61 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 129], 10.00th=[ 140], 20.00th=[ 155], | 30.00th=[ 165], 40.00th=[ 176], 50.00th=[ 186], 60.00th=[ 201], | 70.00th=[ 218], 80.00th=[ 234], 90.00th=[ 259], 95.00th=[ 284], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 351], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 79, max= 276, per=25.22%, avg=153.31, stdev=37.18, samples=65 iops : min= 19, max= 69, avg=37.88, stdev= 9.32, samples=65 write: IOPS=39, BW=157KiB/s (161kB/s)(5180KiB/32924msec); 0 zone resets slat (usec): min=32, max=19133, avg=231.31, stdev=922.49 clat (msec): min=97, max=365, avg=200.34, stdev=45.63 lat (msec): min=97, max=365, avg=200.57, stdev=45.64 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 140], 10.00th=[ 150], 20.00th=[ 161], | 30.00th=[ 171], 40.00th=[ 184], 50.00th=[ 194], 60.00th=[ 207], | 70.00th=[ 220], 80.00th=[ 234], 90.00th=[ 262], 95.00th=[ 288], | 99.00th=[ 338], 99.50th=[ 351], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 39, max= 222, per=24.68%, avg=156.22, stdev=36.05, samples=65 iops : min= 9, max= 55, avg=38.60, stdev= 9.10, samples=65 lat (msec) : 50=0.04%, 100=0.23%, 250=87.19%, 500=12.54% cpu : usr=0.27%, sys=1.73%, ctx=4608, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=609KiB/s (623kB/s), 151KiB/s-154KiB/s (155kB/s-157kB/s), io=19.6MiB (20.5MB), run=32750-32978msec WRITE: bw=634KiB/s (649kB/s), 157KiB/s-160KiB/s (161kB/s-164kB/s), io=20.4MiB (21.4MB), run=32750-32978msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180699: Thu Aug 27 18:50:32 2026 read: IOPS=2, BW=89.9MiB/s (94.3MB/s)(40.0MiB/445msec) slat (nsec): min=35900k, max=35900k, avg=35900339.00, stdev= 0.00 clat (nsec): min=391943k, max=391943k, avg=391943108.00, stdev= 0.00 lat (nsec): min=427874k, max=427874k, avg=427873803.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 393], 10.00th=[ 393], 20.00th=[ 393], | 30.00th=[ 393], 40.00th=[ 393], 50.00th=[ 393], 60.00th=[ 393], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.15%, ctx=3, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=89.9MiB/s (94.3MB/s), 89.9MiB/s-89.9MiB/s (94.3MB/s-94.3MB/s), io=40.0MiB (41.9MB), run=445-445msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout