== sanity test 398c: run fio to test AIO ================= 15:53:19 (1782762799) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.410589 s, 102 MB/s osc.lustre-OST0000-osc-ffff8e0b6cbbe800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179937: Mon Jun 29 15:53:56 2026 write: IOPS=92, BW=368KiB/s (377kB/s)(10.0MiB/27820msec); 0 zone resets slat (usec): min=49, max=21473, avg=320.42, stdev=995.50 clat (msec): min=17, max=328, avg=172.89, stdev=39.16 lat (msec): min=17, max=328, avg=173.22, stdev=39.19 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 106], 10.00th=[ 127], 20.00th=[ 140], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 180], | 70.00th=[ 192], 80.00th=[ 205], 90.00th=[ 222], 95.00th=[ 247], | 99.00th=[ 271], 99.50th=[ 284], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 255, max= 624, per=24.97%, avg=367.58, stdev=72.31, samples=55 iops : min= 63, max= 156, avg=91.55, stdev=18.10, samples=55 lat (msec) : 20=0.08%, 50=0.04%, 100=3.28%, 250=93.32%, 500=3.28% cpu : usr=0.18%, sys=1.27%, ctx=2679, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179938: Mon Jun 29 15:53:56 2026 write: IOPS=92, BW=368KiB/s (377kB/s)(10.0MiB/27800msec); 0 zone resets slat (usec): min=48, max=13896, avg=354.00, stdev=934.42 clat (msec): min=56, max=298, avg=173.19, stdev=39.63 lat (msec): min=57, max=298, avg=173.55, stdev=39.63 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 107], 10.00th=[ 124], 20.00th=[ 140], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 203], 90.00th=[ 228], 95.00th=[ 243], | 99.00th=[ 275], 99.50th=[ 288], 99.90th=[ 292], 99.95th=[ 296], | 99.99th=[ 300] bw ( KiB/s): min= 247, max= 512, per=24.96%, avg=367.36, stdev=73.84, samples=55 iops : min= 61, max= 128, avg=91.49, stdev=18.50, samples=55 lat (msec) : 100=3.20%, 250=92.93%, 500=3.87% cpu : usr=0.18%, sys=1.25%, ctx=2721, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179939: Mon Jun 29 15:53:56 2026 write: IOPS=92, BW=368KiB/s (377kB/s)(10.0MiB/27823msec); 0 zone resets slat (usec): min=52, max=28274, avg=398.04, stdev=1202.82 clat (msec): min=20, max=295, avg=172.86, stdev=39.76 lat (msec): min=21, max=295, avg=173.26, stdev=39.80 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 106], 10.00th=[ 125], 20.00th=[ 140], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 203], 90.00th=[ 226], 95.00th=[ 245], | 99.00th=[ 275], 99.50th=[ 284], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 223, max= 552, per=24.93%, avg=366.98, stdev=70.90, samples=55 iops : min= 55, max= 138, avg=91.36, stdev=17.75, samples=55 lat (msec) : 50=0.20%, 100=3.24%, 250=93.09%, 500=3.48% cpu : usr=0.19%, sys=1.34%, ctx=2929, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179940: Mon Jun 29 15:53:56 2026 write: IOPS=92, BW=369KiB/s (377kB/s)(10.0MiB/27785msec); 0 zone resets slat (usec): min=50, max=26013, avg=405.02, stdev=1231.18 clat (msec): min=22, max=337, avg=172.74, stdev=40.03 lat (msec): min=22, max=337, avg=173.15, stdev=40.04 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 107], 10.00th=[ 125], 20.00th=[ 140], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 180], | 70.00th=[ 190], 80.00th=[ 203], 90.00th=[ 226], 95.00th=[ 247], | 99.00th=[ 279], 99.50th=[ 288], 99.90th=[ 292], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 240, max= 576, per=24.96%, avg=367.38, stdev=68.32, samples=55 iops : min= 60, max= 144, avg=91.49, stdev=17.07, samples=55 lat (msec) : 50=0.08%, 100=2.85%, 250=93.09%, 500=3.98% cpu : usr=0.18%, sys=1.37%, ctx=2983, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1472KiB/s (1507kB/s), 368KiB/s-369KiB/s (377kB/s-377kB/s), io=40.0MiB (41.9MB), run=27785-27823msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179954: Mon Jun 29 15:54:35 2026 read: IOPS=41, BW=168KiB/s (172kB/s)(5048KiB/30127msec) slat (usec): min=61, max=74387, avg=13198.12, stdev=6939.58 clat (msec): min=52, max=375, avg=178.78, stdev=46.27 lat (msec): min=55, max=396, avg=191.98, stdev=48.09 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 114], 10.00th=[ 128], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 213], 90.00th=[ 239], 95.00th=[ 264], | 99.00th=[ 317], 99.50th=[ 347], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 103, max= 224, per=25.30%, avg=167.22, stdev=30.99, samples=59 iops : min= 25, max= 56, avg=41.41, stdev= 7.78, samples=59 write: IOPS=43, BW=172KiB/s (176kB/s)(5192KiB/30127msec); 0 zone resets slat (usec): min=58, max=11905, avg=333.08, stdev=708.55 clat (msec): min=24, max=362, avg=183.43, stdev=42.75 lat (msec): min=24, max=362, avg=183.76, stdev=42.77 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 123], 10.00th=[ 136], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 169], 50.00th=[ 180], 60.00th=[ 190], | 70.00th=[ 201], 80.00th=[ 213], 90.00th=[ 239], 95.00th=[ 259], | 99.00th=[ 317], 99.50th=[ 347], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 95, max= 272, per=25.12%, avg=172.80, stdev=44.04, samples=59 iops : min= 23, max= 68, avg=42.78, stdev=11.06, samples=59 lat (msec) : 50=0.08%, 100=1.41%, 250=91.80%, 500=6.72% cpu : usr=0.30%, sys=2.23%, ctx=4721, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179955: Mon Jun 29 15:54:35 2026 read: IOPS=41, BW=164KiB/s (168kB/s)(4976KiB/30273msec) slat (usec): min=2880, max=66131, avg=13377.86, stdev=7092.71 clat (msec): min=21, max=408, avg=177.67, stdev=44.98 lat (msec): min=28, max=418, avg=191.06, stdev=46.31 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 120], 10.00th=[ 128], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 209], 90.00th=[ 234], 95.00th=[ 259], | 99.00th=[ 313], 99.50th=[ 342], 99.90th=[ 393], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 71, max= 250, per=24.73%, avg=163.45, stdev=42.24, samples=60 iops : min= 17, max= 62, avg=40.45, stdev=10.62, samples=60 write: IOPS=43, BW=174KiB/s (178kB/s)(5264KiB/30273msec); 0 zone resets slat (usec): min=55, max=12108, avg=419.17, stdev=1038.46 clat (msec): min=82, max=375, avg=184.57, stdev=42.55 lat (msec): min=82, max=375, avg=184.99, stdev=42.59 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 125], 10.00th=[ 136], 20.00th=[ 150], | 30.00th=[ 161], 40.00th=[ 171], 50.00th=[ 182], 60.00th=[ 190], | 70.00th=[ 201], 80.00th=[ 215], 90.00th=[ 239], 95.00th=[ 264], | 99.00th=[ 317], 99.50th=[ 342], 99.90th=[ 363], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 104, max= 263, per=25.24%, avg=173.67, stdev=37.96, samples=60 iops : min= 26, max= 65, avg=43.02, stdev= 9.54, samples=60 lat (msec) : 50=0.08%, 100=1.02%, 250=92.11%, 500=6.80% cpu : usr=0.23%, sys=2.20%, ctx=4589, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179956: Mon Jun 29 15:54:35 2026 read: IOPS=41, BW=165KiB/s (169kB/s)(4984KiB/30283msec) slat (usec): min=2411, max=54618, avg=13211.44, stdev=6672.19 clat (msec): min=66, max=368, avg=177.05, stdev=43.12 lat (msec): min=77, max=387, avg=190.27, stdev=44.53 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 116], 10.00th=[ 126], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 209], 90.00th=[ 232], 95.00th=[ 255], | 99.00th=[ 305], 99.50th=[ 330], 99.90th=[ 363], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 7, max= 256, per=24.90%, avg=164.57, stdev=45.35, samples=60 iops : min= 1, max= 64, avg=40.73, stdev=11.42, samples=60 write: IOPS=43, BW=174KiB/s (178kB/s)(5256KiB/30283msec); 0 zone resets slat (usec): min=54, max=16003, avg=413.16, stdev=1009.69 clat (msec): min=87, max=354, avg=184.13, stdev=42.28 lat (msec): min=87, max=354, avg=184.54, stdev=42.30 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 124], 10.00th=[ 134], 20.00th=[ 148], | 30.00th=[ 161], 40.00th=[ 171], 50.00th=[ 180], 60.00th=[ 190], | 70.00th=[ 201], 80.00th=[ 215], 90.00th=[ 239], 95.00th=[ 264], | 99.00th=[ 305], 99.50th=[ 317], 99.90th=[ 347], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 37, max= 242, per=25.10%, avg=172.70, stdev=41.77, samples=60 iops : min= 9, max= 60, avg=42.77, stdev=10.48, samples=60 lat (msec) : 100=0.74%, 250=92.77%, 500=6.48% cpu : usr=0.22%, sys=2.12%, ctx=4717, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179957: Mon Jun 29 15:54:35 2026 read: IOPS=41, BW=167KiB/s (171kB/s)(5060KiB/30349msec) slat (usec): min=72, max=53322, avg=13228.55, stdev=6634.32 clat (msec): min=15, max=407, avg=177.26, stdev=44.68 lat (msec): min=20, max=432, avg=190.50, stdev=46.50 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 118], 10.00th=[ 129], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 211], 90.00th=[ 234], 95.00th=[ 259], | 99.00th=[ 317], 99.50th=[ 330], 99.90th=[ 393], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 52, max= 264, per=24.99%, avg=165.20, stdev=44.50, samples=60 iops : min= 13, max= 66, avg=40.90, stdev=11.15, samples=60 write: IOPS=42, BW=171KiB/s (175kB/s)(5180KiB/30349msec); 0 zone resets slat (usec): min=60, max=17122, avg=381.21, stdev=839.83 clat (msec): min=27, max=367, avg=185.17, stdev=44.29 lat (msec): min=27, max=367, avg=185.56, stdev=44.33 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 124], 10.00th=[ 136], 20.00th=[ 150], | 30.00th=[ 161], 40.00th=[ 171], 50.00th=[ 182], 60.00th=[ 192], | 70.00th=[ 203], 80.00th=[ 220], 90.00th=[ 243], 95.00th=[ 271], | 99.00th=[ 313], 99.50th=[ 330], 99.90th=[ 363], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 52, max= 303, per=24.51%, avg=168.65, stdev=41.41, samples=60 iops : min= 13, max= 75, avg=41.77, stdev=10.37, samples=60 lat (msec) : 20=0.04%, 50=0.39%, 100=0.70%, 250=91.91%, 500=6.95% cpu : usr=0.26%, sys=2.04%, ctx=4623, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=661KiB/s (677kB/s), 164KiB/s-168KiB/s (168kB/s-172kB/s), io=19.6MiB (20.5MB), run=30127-30349msec WRITE: bw=688KiB/s (705kB/s), 171KiB/s-174KiB/s (175kB/s-178kB/s), io=20.4MiB (21.4MB), run=30127-30349msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179967: Mon Jun 29 15:54:47 2026 read: IOPS=1, BW=79.2MiB/s (83.1MB/s)(40.0MiB/505msec) slat (nsec): min=40539k, max=40539k, avg=40538955.00, stdev= 0.00 clat (nsec): min=436319k, max=436319k, avg=436318670.00, stdev= 0.00 lat (nsec): min=476989k, max=476989k, avg=476989130.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 439], 10.00th=[ 439], 20.00th=[ 439], | 30.00th=[ 439], 40.00th=[ 439], 50.00th=[ 439], 60.00th=[ 439], | 70.00th=[ 439], 80.00th=[ 439], 90.00th=[ 439], 95.00th=[ 439], | 99.00th=[ 439], 99.50th=[ 439], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=1.59%, ctx=6, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=79.2MiB/s (83.1MB/s), 79.2MiB/s-79.2MiB/s (83.1MB/s-83.1MB/s), io=40.0MiB (41.9MB), run=505-505msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout