== sanity test 398c: run fio to test AIO ================= 15:53:59 (1787860439) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.447094 s, 93.8 MB/s osc.lustre-OST0000-osc-ffff90fc6b30d000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179906: Thu Aug 27 15:54:36 2026 write: IOPS=91, BW=367KiB/s (376kB/s)(10.0MiB/27924msec); 0 zone resets slat (usec): min=27, max=11585, avg=233.22, stdev=639.93 clat (msec): min=42, max=315, avg=173.25, stdev=38.44 lat (msec): min=42, max=316, avg=173.48, stdev=38.43 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 112], 10.00th=[ 131], 20.00th=[ 144], | 30.00th=[ 157], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 180], | 70.00th=[ 190], 80.00th=[ 205], 90.00th=[ 222], 95.00th=[ 239], | 99.00th=[ 268], 99.50th=[ 284], 99.90th=[ 292], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 240, max= 544, per=24.83%, avg=364.07, stdev=74.57, samples=55 iops : min= 60, max= 136, avg=90.62, stdev=18.65, samples=55 lat (msec) : 50=0.08%, 100=3.09%, 250=93.71%, 500=3.12% cpu : usr=0.21%, sys=1.00%, ctx=2657, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179907: Thu Aug 27 15:54:36 2026 write: IOPS=92, BW=369KiB/s (378kB/s)(10.0MiB/27735msec); 0 zone resets slat (usec): min=30, max=14063, avg=257.03, stdev=790.70 clat (msec): min=15, max=318, avg=172.92, stdev=40.03 lat (msec): min=15, max=318, avg=173.17, stdev=40.01 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 103], 10.00th=[ 130], 20.00th=[ 144], | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 171], 60.00th=[ 180], | 70.00th=[ 192], 80.00th=[ 205], 90.00th=[ 224], 95.00th=[ 239], | 99.00th=[ 271], 99.50th=[ 284], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 248, max= 743, per=25.09%, avg=367.78, stdev=79.28, samples=55 iops : min= 62, max= 185, avg=91.58, stdev=19.77, samples=55 lat (msec) : 20=0.04%, 50=0.12%, 100=4.10%, 250=92.73%, 500=3.01% cpu : usr=0.21%, sys=1.12%, ctx=2592, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179908: Thu Aug 27 15:54:36 2026 write: IOPS=91, BW=367KiB/s (376kB/s)(10.0MiB/27905msec); 0 zone resets slat (usec): min=29, max=17670, avg=255.27, stdev=793.01 clat (msec): min=26, max=305, avg=173.18, stdev=37.97 lat (msec): min=26, max=305, avg=173.44, stdev=37.93 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 109], 10.00th=[ 130], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 180], | 70.00th=[ 190], 80.00th=[ 203], 90.00th=[ 224], 95.00th=[ 241], | 99.00th=[ 266], 99.50th=[ 284], 99.90th=[ 296], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 224, max= 520, per=24.82%, avg=363.82, stdev=59.98, samples=55 iops : min= 56, max= 130, avg=90.56, stdev=14.94, samples=55 lat (msec) : 50=0.16%, 100=2.89%, 250=93.91%, 500=3.05% cpu : usr=0.17%, sys=1.05%, ctx=2842, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179909: Thu Aug 27 15:54:36 2026 write: IOPS=91, BW=367KiB/s (376kB/s)(10.0MiB/27904msec); 0 zone resets slat (usec): min=29, max=15592, avg=198.88, stdev=691.45 clat (msec): min=69, max=311, avg=173.56, stdev=38.01 lat (msec): min=69, max=312, avg=173.76, stdev=38.00 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 109], 10.00th=[ 130], 20.00th=[ 146], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 192], 80.00th=[ 201], 90.00th=[ 226], 95.00th=[ 236], | 99.00th=[ 284], 99.50th=[ 292], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 232, max= 523, per=24.83%, avg=364.07, stdev=68.67, samples=55 iops : min= 58, max= 130, avg=90.64, stdev=17.18, samples=55 lat (msec) : 100=3.40%, 250=94.10%, 500=2.50% cpu : usr=0.14%, sys=0.94%, ctx=1940, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1467KiB/s (1502kB/s), 367KiB/s-369KiB/s (376kB/s-378kB/s), io=40.0MiB (41.9MB), run=27735-27924msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179924: Thu Aug 27 15:55:20 2026 read: IOPS=35, BW=143KiB/s (146kB/s)(5048KiB/35342msec) slat (usec): min=31, max=72412, avg=15213.71, stdev=8615.12 clat (msec): min=33, max=383, avg=207.85, stdev=52.05 lat (msec): min=47, max=409, avg=223.07, stdev=54.02 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 132], 10.00th=[ 146], 20.00th=[ 163], | 30.00th=[ 178], 40.00th=[ 192], 50.00th=[ 205], 60.00th=[ 218], | 70.00th=[ 232], 80.00th=[ 251], 90.00th=[ 275], 95.00th=[ 300], | 99.00th=[ 342], 99.50th=[ 351], 99.90th=[ 380], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 47, max= 247, per=25.05%, avg=142.03, stdev=37.90, samples=70 iops : min= 11, max= 61, avg=35.07, stdev= 9.53, samples=70 write: IOPS=36, BW=147KiB/s (150kB/s)(5192KiB/35342msec); 0 zone resets slat (usec): min=34, max=38584, avg=276.89, stdev=1356.56 clat (msec): min=66, max=395, avg=217.80, stdev=53.86 lat (msec): min=66, max=395, avg=218.08, stdev=53.89 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 140], 10.00th=[ 153], 20.00th=[ 176], | 30.00th=[ 190], 40.00th=[ 201], 50.00th=[ 211], 60.00th=[ 224], | 70.00th=[ 239], 80.00th=[ 259], 90.00th=[ 296], 95.00th=[ 317], | 99.00th=[ 372], 99.50th=[ 380], 99.90th=[ 388], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 70, max= 222, per=24.74%, avg=146.20, stdev=38.27, samples=70 iops : min= 17, max= 55, avg=36.10, stdev= 9.54, samples=70 lat (msec) : 50=0.04%, 100=0.82%, 250=77.30%, 500=21.84% cpu : usr=0.21%, sys=1.79%, ctx=4576, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179925: Thu Aug 27 15:55:20 2026 read: IOPS=35, BW=141KiB/s (144kB/s)(4976KiB/35319msec) slat (usec): min=2927, max=85445, avg=15116.87, stdev=8453.73 clat (msec): min=29, max=437, avg=208.95, stdev=54.94 lat (msec): min=39, max=458, avg=224.08, stdev=56.57 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 131], 10.00th=[ 146], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 215], | 70.00th=[ 230], 80.00th=[ 249], 90.00th=[ 279], 95.00th=[ 317], | 99.00th=[ 372], 99.50th=[ 384], 99.90th=[ 397], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 59, max= 247, per=24.66%, avg=139.80, stdev=35.42, samples=70 iops : min= 14, max= 61, avg=34.51, stdev= 8.90, samples=70 write: IOPS=37, BW=149KiB/s (153kB/s)(5264KiB/35319msec); 0 zone resets slat (usec): min=30, max=17647, avg=275.27, stdev=936.66 clat (msec): min=93, max=386, avg=214.83, stdev=51.33 lat (msec): min=93, max=388, avg=215.11, stdev=51.32 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 140], 10.00th=[ 155], 20.00th=[ 174], | 30.00th=[ 186], 40.00th=[ 201], 50.00th=[ 211], 60.00th=[ 220], | 70.00th=[ 234], 80.00th=[ 251], 90.00th=[ 284], 95.00th=[ 313], | 99.00th=[ 359], 99.50th=[ 372], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 55, max= 254, per=25.13%, avg=148.54, stdev=39.95, samples=70 iops : min= 13, max= 63, avg=36.71, stdev=10.02, samples=70 lat (msec) : 50=0.04%, 100=0.23%, 250=79.26%, 500=20.47% cpu : usr=0.25%, sys=1.56%, ctx=4469, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179926: Thu Aug 27 15:55:20 2026 read: IOPS=35, BW=141KiB/s (144kB/s)(4984KiB/35336msec) slat (usec): min=2761, max=61844, avg=15043.98, stdev=8223.28 clat (msec): min=52, max=436, avg=208.64, stdev=54.37 lat (msec): min=65, max=457, avg=223.69, stdev=56.03 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 128], 10.00th=[ 144], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 205], 60.00th=[ 218], | 70.00th=[ 230], 80.00th=[ 247], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 363], 99.50th=[ 380], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 71, max= 223, per=24.80%, avg=140.63, stdev=36.53, samples=70 iops : min= 17, max= 55, avg=34.71, stdev= 9.20, samples=70 write: IOPS=37, BW=149KiB/s (152kB/s)(5256KiB/35336msec); 0 zone resets slat (usec): min=30, max=16181, avg=220.13, stdev=718.26 clat (msec): min=84, max=405, avg=215.03, stdev=51.45 lat (msec): min=84, max=405, avg=215.26, stdev=51.48 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 138], 10.00th=[ 153], 20.00th=[ 171], | 30.00th=[ 186], 40.00th=[ 199], 50.00th=[ 211], 60.00th=[ 224], | 70.00th=[ 241], 80.00th=[ 259], 90.00th=[ 279], 95.00th=[ 309], | 99.00th=[ 351], 99.50th=[ 363], 99.90th=[ 388], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 55, max= 247, per=25.01%, avg=147.79, stdev=40.35, samples=70 iops : min= 13, max= 61, avg=36.53, stdev=10.10, samples=70 lat (msec) : 100=0.62%, 250=77.97%, 500=21.41% cpu : usr=0.22%, sys=1.53%, ctx=4178, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179927: Thu Aug 27 15:55:20 2026 read: IOPS=35, BW=143KiB/s (147kB/s)(5060KiB/35281msec) slat (usec): min=65, max=54946, avg=15118.23, stdev=8149.06 clat (msec): min=57, max=451, avg=207.63, stdev=53.02 lat (msec): min=69, max=463, avg=222.75, stdev=54.91 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 131], 10.00th=[ 144], 20.00th=[ 163], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 205], 60.00th=[ 218], | 70.00th=[ 232], 80.00th=[ 249], 90.00th=[ 275], 95.00th=[ 300], | 99.00th=[ 351], 99.50th=[ 376], 99.90th=[ 422], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 47, max= 239, per=25.13%, avg=142.50, stdev=35.92, samples=70 iops : min= 11, max= 59, avg=35.16, stdev= 8.94, samples=70 write: IOPS=36, BW=147KiB/s (150kB/s)(5180KiB/35281msec); 0 zone resets slat (usec): min=35, max=10375, avg=240.62, stdev=697.68 clat (msec): min=78, max=407, avg=215.85, stdev=51.89 lat (msec): min=78, max=407, avg=216.10, stdev=51.90 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 140], 10.00th=[ 155], 20.00th=[ 171], | 30.00th=[ 188], 40.00th=[ 199], 50.00th=[ 209], 60.00th=[ 224], | 70.00th=[ 241], 80.00th=[ 259], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 351], 99.50th=[ 363], 99.90th=[ 405], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 55, max= 220, per=24.68%, avg=145.89, stdev=34.76, samples=70 iops : min= 13, max= 55, avg=36.06, stdev= 8.77, samples=70 lat (msec) : 100=0.66%, 250=77.85%, 500=21.48% cpu : usr=0.27%, sys=1.62%, ctx=4519, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=568KiB/s (581kB/s), 141KiB/s-143KiB/s (144kB/s-147kB/s), io=19.6MiB (20.5MB), run=35281-35342msec WRITE: bw=591KiB/s (605kB/s), 147KiB/s-149KiB/s (150kB/s-153kB/s), io=20.4MiB (21.4MB), run=35281-35342msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179936: Thu Aug 27 15:55:28 2026 read: IOPS=1, BW=71.9MiB/s (75.4MB/s)(40.0MiB/556msec) slat (nsec): min=38290k, max=38290k, avg=38290086.00, stdev= 0.00 clat (nsec): min=490360k, max=490360k, avg=490360212.00, stdev= 0.00 lat (nsec): min=528656k, max=528656k, avg=528656177.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 489], 5.00th=[ 489], 10.00th=[ 489], 20.00th=[ 489], | 30.00th=[ 489], 40.00th=[ 489], 50.00th=[ 489], 60.00th=[ 489], | 70.00th=[ 489], 80.00th=[ 489], 90.00th=[ 489], 95.00th=[ 489], | 99.00th=[ 489], 99.50th=[ 489], 99.90th=[ 489], 99.95th=[ 489], | 99.99th=[ 489] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.70%, ctx=3, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=71.9MiB/s (75.4MB/s), 71.9MiB/s-71.9MiB/s (75.4MB/s-75.4MB/s), io=40.0MiB (41.9MB), run=556-556msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout