== sanity test 398c: run fio to test AIO ================= 14:35:49 (1787855749) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.558332 s, 75.1 MB/s osc.lustre-OST0000-osc-ffff88e6007e3000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180078: Thu Aug 27 14:36:26 2026 write: IOPS=96, BW=387KiB/s (397kB/s)(10.0MiB/26431msec); 0 zone resets slat (usec): min=28, max=13211, avg=286.96, stdev=809.64 clat (msec): min=18, max=324, avg=164.02, stdev=32.33 lat (msec): min=19, max=324, avg=164.31, stdev=32.37 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 115], 10.00th=[ 126], 20.00th=[ 138], | 30.00th=[ 148], 40.00th=[ 155], 50.00th=[ 165], 60.00th=[ 169], | 70.00th=[ 180], 80.00th=[ 190], 90.00th=[ 205], 95.00th=[ 222], | 99.00th=[ 241], 99.50th=[ 279], 99.90th=[ 309], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 256, max= 544, per=24.91%, avg=385.92, stdev=52.99, samples=52 iops : min= 64, max= 136, avg=96.10, stdev=13.28, samples=52 lat (msec) : 20=0.04%, 50=0.04%, 100=0.74%, 250=98.40%, 500=0.78% cpu : usr=0.21%, sys=1.12%, ctx=3007, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180079: Thu Aug 27 14:36:26 2026 write: IOPS=97, BW=389KiB/s (398kB/s)(10.0MiB/26335msec); 0 zone resets slat (usec): min=30, max=19218, avg=235.88, stdev=805.14 clat (msec): min=17, max=310, avg=163.60, stdev=33.21 lat (msec): min=17, max=310, avg=163.85, stdev=33.19 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 116], 10.00th=[ 124], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 155], 50.00th=[ 163], 60.00th=[ 171], | 70.00th=[ 180], 80.00th=[ 190], 90.00th=[ 203], 95.00th=[ 220], | 99.00th=[ 253], 99.50th=[ 284], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 248, max= 520, per=25.11%, avg=388.98, stdev=56.33, samples=52 iops : min= 62, max= 130, avg=96.88, stdev=14.12, samples=52 lat (msec) : 20=0.04%, 50=0.04%, 100=0.78%, 250=98.12%, 500=1.02% cpu : usr=0.18%, sys=0.98%, ctx=2288, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180080: Thu Aug 27 14:36:26 2026 write: IOPS=97, BW=391KiB/s (400kB/s)(10.0MiB/26216msec); 0 zone resets slat (usec): min=22, max=15638, avg=245.04, stdev=736.91 clat (msec): min=22, max=324, avg=163.43, stdev=33.75 lat (msec): min=23, max=324, avg=163.67, stdev=33.74 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 113], 10.00th=[ 123], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 155], 50.00th=[ 163], 60.00th=[ 171], | 70.00th=[ 180], 80.00th=[ 190], 90.00th=[ 203], 95.00th=[ 222], | 99.00th=[ 249], 99.50th=[ 300], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 248, max= 512, per=25.11%, avg=388.90, stdev=55.07, samples=52 iops : min= 62, max= 128, avg=96.85, stdev=13.78, samples=52 lat (msec) : 50=0.04%, 100=1.64%, 250=97.54%, 500=0.78% cpu : usr=0.14%, sys=1.08%, ctx=2198, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180081: Thu Aug 27 14:36:26 2026 write: IOPS=97, BW=388KiB/s (398kB/s)(10.0MiB/26378msec); 0 zone resets slat (usec): min=27, max=14704, avg=248.59, stdev=769.83 clat (msec): min=26, max=306, avg=163.76, stdev=32.29 lat (msec): min=26, max=306, avg=164.01, stdev=32.28 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 117], 10.00th=[ 125], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 155], 50.00th=[ 163], 60.00th=[ 171], | 70.00th=[ 180], 80.00th=[ 190], 90.00th=[ 203], 95.00th=[ 222], | 99.00th=[ 247], 99.50th=[ 259], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 256, max= 512, per=24.99%, avg=387.06, stdev=56.87, samples=52 iops : min= 64, max= 128, avg=96.38, stdev=14.24, samples=52 lat (msec) : 50=0.20%, 100=0.59%, 250=98.52%, 500=0.70% cpu : usr=0.17%, sys=1.10%, ctx=2449, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1550KiB/s (1587kB/s), 387KiB/s-391KiB/s (397kB/s-400kB/s), io=40.0MiB (41.9MB), run=26216-26431msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180095: Thu Aug 27 14:37:01 2026 read: IOPS=46, BW=187KiB/s (192kB/s)(5048KiB/26943msec) slat (usec): min=35, max=56255, avg=12048.14, stdev=6176.12 clat (msec): min=27, max=305, avg=160.79, stdev=34.73 lat (msec): min=36, max=326, avg=172.84, stdev=35.99 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 113], 10.00th=[ 123], 20.00th=[ 134], | 30.00th=[ 144], 40.00th=[ 150], 50.00th=[ 159], 60.00th=[ 165], | 70.00th=[ 176], 80.00th=[ 186], 90.00th=[ 205], 95.00th=[ 222], | 99.00th=[ 266], 99.50th=[ 275], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 112, max= 253, per=25.14%, avg=186.04, stdev=31.85, samples=53 iops : min= 28, max= 63, avg=46.09, stdev= 7.96, samples=53 write: IOPS=48, BW=193KiB/s (197kB/s)(5192KiB/26943msec); 0 zone resets slat (usec): min=31, max=16539, avg=305.49, stdev=927.11 clat (msec): min=39, max=265, avg=163.36, stdev=31.01 lat (msec): min=39, max=265, avg=163.67, stdev=31.01 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 121], 10.00th=[ 127], 20.00th=[ 138], | 30.00th=[ 146], 40.00th=[ 155], 50.00th=[ 161], 60.00th=[ 169], | 70.00th=[ 178], 80.00th=[ 188], 90.00th=[ 205], 95.00th=[ 220], | 99.00th=[ 255], 99.50th=[ 259], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 104, max= 283, per=24.90%, avg=191.70, stdev=35.36, samples=53 iops : min= 26, max= 70, avg=47.49, stdev= 8.87, samples=53 lat (msec) : 50=0.23%, 100=1.41%, 250=96.84%, 500=1.52% cpu : usr=0.27%, sys=2.15%, ctx=4438, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180096: Thu Aug 27 14:37:01 2026 read: IOPS=45, BW=184KiB/s (188kB/s)(4976KiB/27103msec) slat (usec): min=42, max=43500, avg=11950.46, stdev=5885.48 clat (msec): min=29, max=279, avg=161.63, stdev=36.06 lat (msec): min=29, max=295, avg=173.59, stdev=37.03 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 113], 10.00th=[ 122], 20.00th=[ 133], | 30.00th=[ 142], 40.00th=[ 148], 50.00th=[ 157], 60.00th=[ 167], | 70.00th=[ 178], 80.00th=[ 190], 90.00th=[ 211], 95.00th=[ 230], | 99.00th=[ 262], 99.50th=[ 264], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 95, max= 248, per=24.54%, avg=181.57, stdev=35.79, samples=53 iops : min= 23, max= 62, avg=44.92, stdev= 9.01, samples=53 write: IOPS=48, BW=194KiB/s (199kB/s)(5264KiB/27103msec); 0 zone resets slat (usec): min=35, max=16354, avg=317.25, stdev=965.89 clat (msec): min=27, max=278, avg=163.37, stdev=30.76 lat (msec): min=29, max=278, avg=163.69, stdev=30.76 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 120], 10.00th=[ 127], 20.00th=[ 138], | 30.00th=[ 146], 40.00th=[ 155], 50.00th=[ 163], 60.00th=[ 169], | 70.00th=[ 178], 80.00th=[ 190], 90.00th=[ 203], 95.00th=[ 218], | 99.00th=[ 245], 99.50th=[ 253], 99.90th=[ 271], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 95, max= 303, per=24.94%, avg=192.00, stdev=42.89, samples=53 iops : min= 23, max= 75, avg=47.55, stdev=10.71, samples=53 lat (msec) : 50=0.20%, 100=1.02%, 250=97.34%, 500=1.45% cpu : usr=0.25%, sys=2.17%, ctx=4653, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180097: Thu Aug 27 14:37:01 2026 read: IOPS=46, BW=184KiB/s (189kB/s)(4984KiB/27015msec) slat (usec): min=2639, max=39075, avg=11758.94, stdev=5712.62 clat (msec): min=66, max=291, avg=161.17, stdev=34.59 lat (msec): min=77, max=303, avg=172.93, stdev=35.10 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 112], 10.00th=[ 121], 20.00th=[ 133], | 30.00th=[ 142], 40.00th=[ 150], 50.00th=[ 157], 60.00th=[ 167], | 70.00th=[ 176], 80.00th=[ 188], 90.00th=[ 207], 95.00th=[ 224], | 99.00th=[ 266], 99.50th=[ 275], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 88, max= 270, per=24.90%, avg=184.23, stdev=40.31, samples=53 iops : min= 22, max= 67, avg=45.62, stdev=10.09, samples=53 write: IOPS=48, BW=195KiB/s (199kB/s)(5256KiB/27015msec); 0 zone resets slat (usec): min=31, max=16707, avg=288.30, stdev=876.13 clat (msec): min=79, max=301, avg=163.11, stdev=30.23 lat (msec): min=79, max=301, avg=163.40, stdev=30.29 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 121], 10.00th=[ 128], 20.00th=[ 138], | 30.00th=[ 146], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 167], | 70.00th=[ 178], 80.00th=[ 188], 90.00th=[ 203], 95.00th=[ 215], | 99.00th=[ 247], 99.50th=[ 255], 99.90th=[ 288], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 103, max= 332, per=25.10%, avg=193.30, stdev=42.81, samples=53 iops : min= 25, max= 83, avg=47.91, stdev=10.76, samples=53 lat (msec) : 100=1.05%, 250=97.70%, 500=1.25% cpu : usr=0.26%, sys=2.04%, ctx=4524, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180098: Thu Aug 27 14:37:01 2026 read: IOPS=46, BW=188KiB/s (192kB/s)(5060KiB/26985msec) slat (usec): min=2903, max=52800, avg=11870.63, stdev=5840.51 clat (msec): min=17, max=272, avg=160.77, stdev=32.99 lat (msec): min=27, max=323, avg=172.66, stdev=34.21 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 113], 10.00th=[ 122], 20.00th=[ 134], | 30.00th=[ 142], 40.00th=[ 150], 50.00th=[ 159], 60.00th=[ 167], | 70.00th=[ 176], 80.00th=[ 188], 90.00th=[ 205], 95.00th=[ 220], | 99.00th=[ 259], 99.50th=[ 266], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 110, max= 262, per=25.27%, avg=187.02, stdev=34.62, samples=53 iops : min= 27, max= 65, avg=46.32, stdev= 8.68, samples=53 write: IOPS=47, BW=192KiB/s (197kB/s)(5180KiB/26985msec); 0 zone resets slat (usec): min=36, max=14097, avg=313.17, stdev=941.97 clat (msec): min=86, max=267, avg=162.72, stdev=29.24 lat (msec): min=86, max=267, avg=163.04, stdev=29.32 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 121], 10.00th=[ 128], 20.00th=[ 140], | 30.00th=[ 146], 40.00th=[ 155], 50.00th=[ 161], 60.00th=[ 167], | 70.00th=[ 176], 80.00th=[ 186], 90.00th=[ 201], 95.00th=[ 218], | 99.00th=[ 251], 99.50th=[ 257], 99.90th=[ 268], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 103, max= 279, per=24.80%, avg=190.92, stdev=39.53, samples=53 iops : min= 25, max= 69, avg=47.32, stdev= 9.95, samples=53 lat (msec) : 20=0.04%, 100=0.90%, 250=97.85%, 500=1.21% cpu : usr=0.30%, sys=2.13%, ctx=4544, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=740KiB/s (758kB/s), 184KiB/s-188KiB/s (188kB/s-192kB/s), io=19.6MiB (20.5MB), run=26943-27103msec WRITE: bw=771KiB/s (789kB/s), 192KiB/s-195KiB/s (197kB/s-199kB/s), io=20.4MiB (21.4MB), run=26943-27103msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180106: Thu Aug 27 14:37:11 2026 read: IOPS=1, BW=66.7MiB/s (69.9MB/s)(40.0MiB/600msec) slat (nsec): min=53798k, max=53798k, avg=53798151.00, stdev= 0.00 clat (nsec): min=501355k, max=501355k, avg=501354587.00, stdev= 0.00 lat (nsec): min=555158k, max=555158k, avg=555157959.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 502], 10.00th=[ 502], 20.00th=[ 502], | 30.00th=[ 502], 40.00th=[ 502], 50.00th=[ 502], 60.00th=[ 502], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.67%, ctx=10, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=66.7MiB/s (69.9MB/s), 66.7MiB/s-66.7MiB/s (69.9MB/s-69.9MB/s), io=40.0MiB (41.9MB), run=600-600msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout