== sanity test 398c: run fio to test AIO ================= 08:42:51 (1787143371) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.578247 s, 72.5 MB/s osc.lustre-OST0000-osc-ffff8c5c482d1800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179700: Wed Aug 19 08:43:34 2026 write: IOPS=80, BW=320KiB/s (328kB/s)(10.0MiB/31967msec); 0 zone resets slat (usec): min=36, max=14697, avg=293.72, stdev=962.73 clat (msec): min=13, max=430, avg=198.40, stdev=56.51 lat (msec): min=14, max=430, avg=198.70, stdev=56.48 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 115], 10.00th=[ 136], 20.00th=[ 153], | 30.00th=[ 161], 40.00th=[ 176], 50.00th=[ 194], 60.00th=[ 211], | 70.00th=[ 226], 80.00th=[ 243], 90.00th=[ 271], 95.00th=[ 300], | 99.00th=[ 351], 99.50th=[ 372], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 183, max= 579, per=24.90%, avg=319.03, stdev=83.21, samples=63 iops : min= 45, max= 144, avg=79.33, stdev=20.82, samples=63 lat (msec) : 20=0.08%, 50=0.12%, 100=1.56%, 250=81.09%, 500=17.15% cpu : usr=0.13%, sys=0.91%, ctx=2190, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179701: Wed Aug 19 08:43:34 2026 write: IOPS=80, BW=320KiB/s (328kB/s)(10.0MiB/31969msec); 0 zone resets slat (usec): min=36, max=20674, avg=283.12, stdev=969.47 clat (msec): min=56, max=451, avg=198.71, stdev=57.15 lat (msec): min=56, max=451, avg=198.99, stdev=57.17 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 118], 10.00th=[ 133], 20.00th=[ 148], | 30.00th=[ 161], 40.00th=[ 174], 50.00th=[ 192], 60.00th=[ 213], | 70.00th=[ 234], 80.00th=[ 247], 90.00th=[ 275], 95.00th=[ 288], | 99.00th=[ 347], 99.50th=[ 447], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 127, max= 555, per=24.90%, avg=319.02, stdev=92.44, samples=63 iops : min= 31, max= 138, avg=79.32, stdev=23.12, samples=63 lat (msec) : 100=2.30%, 250=79.49%, 500=18.20% cpu : usr=0.15%, sys=1.01%, ctx=2071, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179702: Wed Aug 19 08:43:34 2026 write: IOPS=80, BW=321KiB/s (328kB/s)(10.0MiB/31941msec); 0 zone resets slat (usec): min=36, max=32291, avg=293.15, stdev=1107.26 clat (msec): min=25, max=431, avg=198.88, stdev=56.77 lat (msec): min=25, max=431, avg=199.17, stdev=56.74 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 117], 10.00th=[ 132], 20.00th=[ 148], | 30.00th=[ 161], 40.00th=[ 176], 50.00th=[ 194], 60.00th=[ 213], | 70.00th=[ 228], 80.00th=[ 247], 90.00th=[ 271], 95.00th=[ 296], | 99.00th=[ 342], 99.50th=[ 414], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 127, max= 576, per=24.81%, avg=317.81, stdev=84.49, samples=63 iops : min= 31, max= 144, avg=78.98, stdev=21.16, samples=63 lat (msec) : 50=0.08%, 100=1.29%, 250=80.43%, 500=18.20% cpu : usr=0.15%, sys=1.18%, ctx=2395, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179703: Wed Aug 19 08:43:34 2026 write: IOPS=80, BW=320KiB/s (328kB/s)(10.0MiB/31961msec); 0 zone resets slat (usec): min=35, max=21861, avg=348.96, stdev=1184.18 clat (msec): min=16, max=453, avg=198.26, stdev=56.45 lat (msec): min=16, max=453, avg=198.62, stdev=56.50 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 116], 10.00th=[ 133], 20.00th=[ 150], | 30.00th=[ 161], 40.00th=[ 174], 50.00th=[ 194], 60.00th=[ 213], | 70.00th=[ 228], 80.00th=[ 245], 90.00th=[ 275], 95.00th=[ 292], | 99.00th=[ 342], 99.50th=[ 388], 99.90th=[ 456], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 144, max= 579, per=24.91%, avg=319.08, stdev=82.76, samples=63 iops : min= 36, max= 144, avg=79.33, stdev=20.69, samples=63 lat (msec) : 20=0.04%, 50=0.23%, 100=1.80%, 250=80.90%, 500=17.03% cpu : usr=0.17%, sys=1.20%, ctx=2530, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1281KiB/s (1312kB/s), 320KiB/s-321KiB/s (328kB/s-328kB/s), io=40.0MiB (41.9MB), run=31941-31969msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179718: Wed Aug 19 08:44:18 2026 read: IOPS=36, BW=144KiB/s (148kB/s)(5048KiB/34945msec) slat (usec): min=3235, max=60693, avg=15539.11, stdev=7906.86 clat (msec): min=12, max=417, avg=207.26, stdev=45.08 lat (msec): min=28, max=454, avg=222.81, stdev=46.60 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 138], 10.00th=[ 153], 20.00th=[ 174], | 30.00th=[ 184], 40.00th=[ 194], 50.00th=[ 207], 60.00th=[ 218], | 70.00th=[ 230], 80.00th=[ 243], 90.00th=[ 264], 95.00th=[ 284], | 99.00th=[ 321], 99.50th=[ 334], 99.90th=[ 380], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 79, max= 199, per=24.99%, avg=143.42, stdev=25.44, samples=69 iops : min= 19, max= 49, avg=35.38, stdev= 6.41, samples=69 write: IOPS=37, BW=149KiB/s (152kB/s)(5192KiB/34945msec); 0 zone resets slat (usec): min=51, max=20432, avg=373.87, stdev=1316.20 clat (msec): min=77, max=382, avg=211.55, stdev=41.26 lat (msec): min=77, max=382, avg=211.92, stdev=41.26 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 146], 10.00th=[ 163], 20.00th=[ 182], | 30.00th=[ 194], 40.00th=[ 203], 50.00th=[ 211], 60.00th=[ 222], | 70.00th=[ 230], 80.00th=[ 243], 90.00th=[ 259], 95.00th=[ 279], | 99.00th=[ 338], 99.50th=[ 347], 99.90th=[ 363], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 63, max= 216, per=24.72%, avg=147.58, stdev=32.69, samples=69 iops : min= 15, max= 54, avg=36.41, stdev= 8.24, samples=69 lat (msec) : 20=0.04%, 100=0.86%, 250=84.53%, 500=14.57% cpu : usr=0.23%, sys=1.83%, ctx=4259, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179719: Wed Aug 19 08:44:18 2026 read: IOPS=35, BW=143KiB/s (146kB/s)(4976KiB/34785msec) slat (usec): min=52, max=61110, avg=15514.78, stdev=7947.24 clat (msec): min=17, max=376, avg=206.64, stdev=44.49 lat (msec): min=17, max=397, avg=222.16, stdev=46.15 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 136], 10.00th=[ 155], 20.00th=[ 176], | 30.00th=[ 188], 40.00th=[ 197], 50.00th=[ 207], 60.00th=[ 218], | 70.00th=[ 228], 80.00th=[ 239], 90.00th=[ 259], 95.00th=[ 279], | 99.00th=[ 326], 99.50th=[ 347], 99.90th=[ 372], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 71, max= 230, per=24.74%, avg=142.00, stdev=31.56, samples=68 iops : min= 17, max= 57, avg=35.00, stdev= 7.97, samples=68 write: IOPS=37, BW=151KiB/s (155kB/s)(5264KiB/34785msec); 0 zone resets slat (usec): min=53, max=14478, avg=309.35, stdev=896.46 clat (msec): min=43, max=381, avg=211.87, stdev=42.97 lat (msec): min=44, max=381, avg=212.18, stdev=42.94 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 142], 10.00th=[ 163], 20.00th=[ 182], | 30.00th=[ 194], 40.00th=[ 203], 50.00th=[ 213], 60.00th=[ 222], | 70.00th=[ 232], 80.00th=[ 245], 90.00th=[ 259], 95.00th=[ 275], | 99.00th=[ 326], 99.50th=[ 338], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 101, max= 283, per=25.30%, avg=151.01, stdev=34.39, samples=68 iops : min= 25, max= 70, avg=37.28, stdev= 8.63, samples=68 lat (msec) : 20=0.08%, 50=0.08%, 100=1.45%, 250=84.14%, 500=14.26% cpu : usr=0.24%, sys=1.85%, ctx=4280, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179720: Wed Aug 19 08:44:18 2026 read: IOPS=35, BW=143KiB/s (146kB/s)(4984KiB/34921msec) slat (usec): min=3505, max=62823, avg=15733.79, stdev=8309.21 clat (msec): min=73, max=376, avg=206.16, stdev=44.17 lat (msec): min=83, max=395, avg=221.90, stdev=45.94 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 138], 10.00th=[ 155], 20.00th=[ 174], | 30.00th=[ 184], 40.00th=[ 194], 50.00th=[ 203], 60.00th=[ 213], | 70.00th=[ 226], 80.00th=[ 241], 90.00th=[ 262], 95.00th=[ 284], | 99.00th=[ 330], 99.50th=[ 351], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 86, max= 215, per=24.74%, avg=142.03, stdev=30.23, samples=69 iops : min= 21, max= 53, avg=35.03, stdev= 7.56, samples=69 write: IOPS=37, BW=151KiB/s (154kB/s)(5256KiB/34921msec); 0 zone resets slat (usec): min=58, max=19576, avg=309.93, stdev=920.05 clat (msec): min=75, max=388, avg=212.49, stdev=43.95 lat (msec): min=75, max=388, avg=212.80, stdev=43.92 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 138], 10.00th=[ 161], 20.00th=[ 180], | 30.00th=[ 192], 40.00th=[ 205], 50.00th=[ 213], 60.00th=[ 222], | 70.00th=[ 232], 80.00th=[ 245], 90.00th=[ 266], 95.00th=[ 284], | 99.00th=[ 330], 99.50th=[ 351], 99.90th=[ 384], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 69, max= 236, per=24.96%, avg=149.00, stdev=36.35, samples=69 iops : min= 17, max= 59, avg=36.77, stdev= 9.19, samples=69 lat (msec) : 100=0.74%, 250=82.93%, 500=16.33% cpu : usr=0.21%, sys=1.74%, ctx=4304, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179721: Wed Aug 19 08:44:18 2026 read: IOPS=36, BW=145KiB/s (148kB/s)(5060KiB/34938msec) slat (usec): min=2355, max=59596, avg=15738.29, stdev=8345.76 clat (msec): min=43, max=469, avg=207.87, stdev=45.91 lat (msec): min=50, max=495, avg=223.61, stdev=47.74 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 136], 10.00th=[ 155], 20.00th=[ 171], | 30.00th=[ 186], 40.00th=[ 197], 50.00th=[ 207], 60.00th=[ 218], | 70.00th=[ 228], 80.00th=[ 243], 90.00th=[ 264], 95.00th=[ 284], | 99.00th=[ 330], 99.50th=[ 351], 99.90th=[ 435], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 84, max= 238, per=25.04%, avg=143.71, stdev=30.38, samples=69 iops : min= 21, max= 59, avg=35.43, stdev= 7.64, samples=69 write: IOPS=37, BW=148KiB/s (152kB/s)(5180KiB/34938msec); 0 zone resets slat (usec): min=40, max=26417, avg=337.62, stdev=1296.88 clat (msec): min=76, max=387, avg=211.41, stdev=42.95 lat (msec): min=76, max=387, avg=211.75, stdev=42.94 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 144], 10.00th=[ 163], 20.00th=[ 180], | 30.00th=[ 192], 40.00th=[ 203], 50.00th=[ 213], 60.00th=[ 220], | 70.00th=[ 230], 80.00th=[ 241], 90.00th=[ 259], 95.00th=[ 288], | 99.00th=[ 334], 99.50th=[ 342], 99.90th=[ 359], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 71, max= 215, per=24.67%, avg=147.26, stdev=32.56, samples=69 iops : min= 17, max= 53, avg=36.33, stdev= 8.18, samples=69 lat (msec) : 50=0.04%, 100=1.17%, 250=84.22%, 500=14.57% cpu : usr=0.21%, sys=1.75%, ctx=3923, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=574KiB/s (588kB/s), 143KiB/s-145KiB/s (146kB/s-148kB/s), io=19.6MiB (20.5MB), run=34785-34945msec WRITE: bw=598KiB/s (612kB/s), 148KiB/s-151KiB/s (152kB/s-155kB/s), io=20.4MiB (21.4MB), run=34785-34945msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179731: Wed Aug 19 08:44:33 2026 read: IOPS=1, BW=74.9MiB/s (78.5MB/s)(40.0MiB/534msec) slat (nsec): min=29799k, max=29799k, avg=29799476.00, stdev= 0.00 clat (nsec): min=479475k, max=479475k, avg=479474840.00, stdev= 0.00 lat (nsec): min=509280k, max=509280k, avg=509280416.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 481], 10.00th=[ 481], 20.00th=[ 481], | 30.00th=[ 481], 40.00th=[ 481], 50.00th=[ 481], 60.00th=[ 481], | 70.00th=[ 481], 80.00th=[ 481], 90.00th=[ 481], 95.00th=[ 481], | 99.00th=[ 481], 99.50th=[ 481], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 481] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.25%, ctx=6, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=74.9MiB/s (78.5MB/s), 74.9MiB/s-74.9MiB/s (78.5MB/s-78.5MB/s), io=40.0MiB (41.9MB), run=534-534msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout