== sanity test 398c: run fio to test AIO ================= 10:13:17 (1758809597) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.138789 s, 302 MB/s osc.lustre-OST0000-osc-ffff984ec95fe000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=159487: Thu Sep 25 10:13:55 2025 write: IOPS=72, BW=292KiB/s (299kB/s)(10.0MiB/35090msec); 0 zone resets slat (usec): min=31, max=1569, avg=138.98, stdev=75.65 clat (msec): min=50, max=391, avg=218.84, stdev=41.60 lat (msec): min=50, max=391, avg=218.98, stdev=41.61 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 155], 10.00th=[ 176], 20.00th=[ 190], | 30.00th=[ 203], 40.00th=[ 211], 50.00th=[ 220], 60.00th=[ 228], | 70.00th=[ 236], 80.00th=[ 249], 90.00th=[ 268], 95.00th=[ 288], | 99.00th=[ 317], 99.50th=[ 334], 99.90th=[ 380], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 224, max= 536, per=24.95%, avg=291.13, stdev=44.16, samples=69 iops : min= 56, max= 134, avg=72.78, stdev=11.04, samples=69 lat (msec) : 100=1.48%, 250=78.95%, 500=19.57% cpu : usr=0.15%, sys=0.89%, ctx=2792, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=159488: Thu Sep 25 10:13:55 2025 write: IOPS=73, BW=292KiB/s (299kB/s)(10.0MiB/35047msec); 0 zone resets slat (usec): min=24, max=635, avg=140.85, stdev=67.85 clat (msec): min=50, max=367, avg=218.60, stdev=41.41 lat (msec): min=50, max=367, avg=218.74, stdev=41.42 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 155], 10.00th=[ 178], 20.00th=[ 190], | 30.00th=[ 201], 40.00th=[ 211], 50.00th=[ 220], 60.00th=[ 228], | 70.00th=[ 239], 80.00th=[ 249], 90.00th=[ 268], 95.00th=[ 288], | 99.00th=[ 321], 99.50th=[ 334], 99.90th=[ 355], 99.95th=[ 359], | 99.99th=[ 368] bw ( KiB/s): min= 224, max= 536, per=24.96%, avg=291.25, stdev=44.36, samples=69 iops : min= 56, max= 134, avg=72.81, stdev=11.09, samples=69 lat (msec) : 100=1.45%, 250=79.26%, 500=19.30% cpu : usr=0.15%, sys=0.93%, ctx=2765, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=159489: Thu Sep 25 10:13:55 2025 write: IOPS=72, BW=292KiB/s (299kB/s)(10.0MiB/35090msec); 0 zone resets slat (usec): min=30, max=1537, avg=139.55, stdev=77.85 clat (msec): min=50, max=353, avg=218.92, stdev=40.12 lat (msec): min=50, max=353, avg=219.06, stdev=40.12 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 157], 10.00th=[ 178], 20.00th=[ 192], | 30.00th=[ 203], 40.00th=[ 211], 50.00th=[ 220], 60.00th=[ 228], | 70.00th=[ 239], 80.00th=[ 249], 90.00th=[ 268], 95.00th=[ 284], | 99.00th=[ 309], 99.50th=[ 321], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 355] bw ( KiB/s): min= 200, max= 530, per=24.88%, avg=290.35, stdev=43.81, samples=69 iops : min= 50, max= 132, avg=72.58, stdev=10.91, samples=69 lat (msec) : 100=1.37%, 250=79.22%, 500=19.41% cpu : usr=0.12%, sys=0.92%, ctx=2896, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=159490: Thu Sep 25 10:13:55 2025 write: IOPS=73, BW=293KiB/s (300kB/s)(10.0MiB/34952msec); 0 zone resets slat (usec): min=31, max=793, avg=142.84, stdev=71.83 clat (msec): min=32, max=415, avg=218.17, stdev=42.82 lat (msec): min=32, max=415, avg=218.31, stdev=42.83 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 153], 10.00th=[ 174], 20.00th=[ 190], | 30.00th=[ 201], 40.00th=[ 211], 50.00th=[ 220], 60.00th=[ 228], | 70.00th=[ 236], 80.00th=[ 249], 90.00th=[ 271], 95.00th=[ 288], | 99.00th=[ 313], 99.50th=[ 330], 99.90th=[ 384], 99.95th=[ 397], | 99.99th=[ 418] bw ( KiB/s): min= 216, max= 587, per=25.04%, avg=292.22, stdev=48.22, samples=69 iops : min= 54, max= 146, avg=73.04, stdev=11.99, samples=69 lat (msec) : 50=0.43%, 100=1.09%, 250=78.95%, 500=19.53% cpu : usr=0.16%, sys=0.92%, ctx=2902, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1167KiB/s (1195kB/s), 292KiB/s-293KiB/s (299kB/s-300kB/s), io=40.0MiB (41.9MB), run=34952-35090msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=159505: Thu Sep 25 10:14:20 2025 read: IOPS=58, BW=235KiB/s (240kB/s)(5048KiB/21495msec) slat (usec): min=36, max=5642, avg=2450.47, stdev=785.02 clat (msec): min=4, max=197, avg=86.39, stdev=16.31 lat (msec): min=8, max=200, avg=88.84, stdev=16.40 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 75], | 30.00th=[ 80], 40.00th=[ 84], 50.00th=[ 87], 60.00th=[ 90], | 70.00th=[ 94], 80.00th=[ 99], 90.00th=[ 105], 95.00th=[ 112], | 99.00th=[ 125], 99.50th=[ 128], 99.90th=[ 167], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 144, max= 320, per=25.18%, avg=234.14, stdev=43.09, samples=42 iops : min= 36, max= 80, avg=58.50, stdev=10.77, samples=42 write: IOPS=60, BW=242KiB/s (247kB/s)(5192KiB/21495msec); 0 zone resets slat (usec): min=32, max=1285, avg=121.37, stdev=76.21 clat (msec): min=26, max=298, avg=178.25, stdev=39.30 lat (msec): min=26, max=298, avg=178.37, stdev=39.30 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 120], 10.00th=[ 129], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 169], 50.00th=[ 178], 60.00th=[ 186], | 70.00th=[ 199], 80.00th=[ 211], 90.00th=[ 230], 95.00th=[ 243], | 99.00th=[ 275], 99.50th=[ 292], 99.90th=[ 296], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 168, max= 304, per=24.91%, avg=241.12, stdev=30.28, samples=42 iops : min= 42, max= 76, avg=60.26, stdev= 7.57, samples=42 lat (msec) : 10=0.04%, 20=0.16%, 50=0.86%, 100=40.94%, 250=56.33% lat (msec) : 500=1.68% cpu : usr=0.20%, sys=1.71%, ctx=4812, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=159506: Thu Sep 25 10:14:20 2025 read: IOPS=57, BW=231KiB/s (237kB/s)(4976KiB/21539msec) slat (usec): min=1073, max=5761, avg=2486.14, stdev=764.79 clat (msec): min=2, max=138, avg=85.94, stdev=15.84 lat (msec): min=4, max=142, avg=88.43, stdev=15.88 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 75], | 30.00th=[ 79], 40.00th=[ 83], 50.00th=[ 86], 60.00th=[ 90], | 70.00th=[ 93], 80.00th=[ 99], 90.00th=[ 106], 95.00th=[ 113], | 99.00th=[ 123], 99.50th=[ 125], 99.90th=[ 126], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 128, max= 344, per=24.78%, avg=230.45, stdev=48.86, samples=42 iops : min= 32, max= 86, avg=57.60, stdev=12.25, samples=42 write: IOPS=61, BW=244KiB/s (250kB/s)(5264KiB/21539msec); 0 zone resets slat (usec): min=38, max=673, avg=120.44, stdev=69.94 clat (msec): min=32, max=305, avg=177.78, stdev=39.73 lat (msec): min=33, max=305, avg=177.90, stdev=39.73 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 116], 10.00th=[ 128], 20.00th=[ 144], | 30.00th=[ 157], 40.00th=[ 169], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 199], 80.00th=[ 211], 90.00th=[ 230], 95.00th=[ 245], | 99.00th=[ 275], 99.50th=[ 284], 99.90th=[ 296], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 184, max= 328, per=25.16%, avg=243.57, stdev=31.75, samples=42 iops : min= 46, max= 82, avg=60.86, stdev= 7.93, samples=42 lat (msec) : 4=0.12%, 10=0.04%, 20=0.20%, 50=0.35%, 100=40.35% lat (msec) : 250=57.34%, 500=1.60% cpu : usr=0.20%, sys=1.71%, ctx=4875, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=159507: Thu Sep 25 10:14:20 2025 read: IOPS=57, BW=231KiB/s (237kB/s)(4984KiB/21577msec) slat (usec): min=1051, max=6670, avg=2473.57, stdev=787.26 clat (msec): min=3, max=127, avg=86.07, stdev=15.51 lat (msec): min=5, max=131, avg=88.55, stdev=15.56 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 63], 10.00th=[ 69], 20.00th=[ 75], | 30.00th=[ 80], 40.00th=[ 83], 50.00th=[ 87], 60.00th=[ 90], | 70.00th=[ 94], 80.00th=[ 99], 90.00th=[ 105], 95.00th=[ 111], | 99.00th=[ 122], 99.50th=[ 124], 99.90th=[ 125], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 136, max= 328, per=24.79%, avg=230.57, stdev=46.14, samples=42 iops : min= 34, max= 82, avg=57.62, stdev=11.54, samples=42 write: IOPS=60, BW=244KiB/s (249kB/s)(5256KiB/21577msec); 0 zone resets slat (usec): min=27, max=1452, avg=122.60, stdev=80.78 clat (msec): min=51, max=288, avg=178.15, stdev=40.56 lat (msec): min=51, max=289, avg=178.27, stdev=40.57 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 114], 10.00th=[ 126], 20.00th=[ 144], | 30.00th=[ 157], 40.00th=[ 169], 50.00th=[ 178], 60.00th=[ 186], | 70.00th=[ 199], 80.00th=[ 211], 90.00th=[ 232], 95.00th=[ 249], | 99.00th=[ 275], 99.50th=[ 279], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 192, max= 296, per=24.85%, avg=240.60, stdev=27.86, samples=42 iops : min= 48, max= 74, avg=60.12, stdev= 6.98, samples=42 lat (msec) : 4=0.04%, 10=0.08%, 20=0.27%, 50=0.12%, 100=40.90% lat (msec) : 250=56.33%, 500=2.27% cpu : usr=0.19%, sys=1.71%, ctx=4733, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=159508: Thu Sep 25 10:14:20 2025 read: IOPS=59, BW=236KiB/s (242kB/s)(5060KiB/21417msec) slat (usec): min=951, max=6638, avg=2471.37, stdev=771.95 clat (msec): min=4, max=149, avg=86.13, stdev=15.59 lat (msec): min=8, max=152, avg=88.61, stdev=15.61 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 63], 10.00th=[ 69], 20.00th=[ 74], | 30.00th=[ 80], 40.00th=[ 83], 50.00th=[ 87], 60.00th=[ 90], | 70.00th=[ 94], 80.00th=[ 99], 90.00th=[ 105], 95.00th=[ 112], | 99.00th=[ 123], 99.50th=[ 126], 99.90th=[ 148], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 136, max= 320, per=25.53%, avg=237.45, stdev=43.06, samples=42 iops : min= 34, max= 80, avg=59.33, stdev=10.80, samples=42 write: IOPS=60, BW=242KiB/s (248kB/s)(5180KiB/21417msec); 0 zone resets slat (usec): min=39, max=501, avg=122.28, stdev=70.02 clat (msec): min=33, max=305, avg=177.46, stdev=38.85 lat (msec): min=33, max=305, avg=177.58, stdev=38.85 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 118], 10.00th=[ 129], 20.00th=[ 144], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 209], 90.00th=[ 228], 95.00th=[ 243], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 152, max= 296, per=24.94%, avg=241.40, stdev=30.37, samples=42 iops : min= 38, max= 74, avg=60.33, stdev= 7.59, samples=42 lat (msec) : 10=0.08%, 20=0.27%, 50=0.31%, 100=41.09%, 250=56.41% lat (msec) : 500=1.84% cpu : usr=0.22%, sys=1.71%, ctx=4953, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=930KiB/s (952kB/s), 231KiB/s-236KiB/s (237kB/s-242kB/s), io=19.6MiB (20.5MB), run=21417-21577msec WRITE: bw=968KiB/s (991kB/s), 242KiB/s-244KiB/s (247kB/s-250kB/s), io=20.4MiB (21.4MB), run=21417-21577msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=159516: Thu Sep 25 10:14:24 2025 read: IOPS=11, BW=444MiB/s (466MB/s)(40.0MiB/90msec) slat (nsec): min=10274k, max=10274k, avg=10273670.00, stdev= 0.00 clat (nsec): min=74749k, max=74749k, avg=74749310.00, stdev= 0.00 lat (nsec): min=85026k, max=85026k, avg=85025873.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[74974], 5.00th=[74974], 10.00th=[74974], 20.00th=[74974], | 30.00th=[74974], 40.00th=[74974], 50.00th=[74974], 60.00th=[74974], | 70.00th=[74974], 80.00th=[74974], 90.00th=[74974], 95.00th=[74974], | 99.00th=[74974], 99.50th=[74974], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=10.11%, ctx=43, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=444MiB/s (466MB/s), 444MiB/s-444MiB/s (466MB/s-466MB/s), io=40.0MiB (41.9MB), run=90-90msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout