== sanity test 398c: run fio to test AIO ================= 16:54:16 (1781211256) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.21463 s, 195 MB/s osc.lustre-OST0000-osc-ffff8ee1a9d5d000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=154842: Thu Jun 11 16:55:30 2026 write: IOPS=37, BW=149KiB/s (152kB/s)(10.0MiB/68851msec); 0 zone resets slat (usec): min=39, max=6007, avg=248.01, stdev=328.56 clat (msec): min=59, max=876, avg=429.83, stdev=133.42 lat (msec): min=59, max=876, avg=430.08, stdev=133.43 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 249], 10.00th=[ 288], 20.00th=[ 330], | 30.00th=[ 359], 40.00th=[ 380], 50.00th=[ 405], 60.00th=[ 435], | 70.00th=[ 477], 80.00th=[ 542], 90.00th=[ 617], 95.00th=[ 693], | 99.00th=[ 793], 99.50th=[ 827], 99.90th=[ 869], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 39, max= 365, per=24.90%, avg=147.63, stdev=48.41, samples=137 iops : min= 9, max= 91, avg=36.62, stdev=12.11, samples=137 lat (msec) : 100=0.66%, 250=4.73%, 500=68.59%, 750=23.98%, 1000=2.03% cpu : usr=0.06%, sys=0.59%, ctx=3149, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154843: Thu Jun 11 16:55:30 2026 write: IOPS=37, BW=148KiB/s (152kB/s)(10.0MiB/68957msec); 0 zone resets slat (usec): min=38, max=4835, avg=233.75, stdev=300.34 clat (msec): min=119, max=868, avg=430.45, stdev=130.74 lat (msec): min=119, max=869, avg=430.69, stdev=130.76 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 257], 10.00th=[ 284], 20.00th=[ 330], | 30.00th=[ 359], 40.00th=[ 380], 50.00th=[ 405], 60.00th=[ 435], | 70.00th=[ 472], 80.00th=[ 542], 90.00th=[ 617], 95.00th=[ 684], | 99.00th=[ 760], 99.50th=[ 802], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 869] bw ( KiB/s): min= 63, max= 312, per=24.89%, avg=147.60, stdev=47.26, samples=137 iops : min= 15, max= 78, avg=36.61, stdev=11.88, samples=137 lat (msec) : 250=4.61%, 500=69.96%, 750=24.06%, 1000=1.37% cpu : usr=0.07%, sys=0.54%, ctx=3093, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154844: Thu Jun 11 16:55:30 2026 write: IOPS=37, BW=149KiB/s (152kB/s)(10.0MiB/68917msec); 0 zone resets slat (usec): min=41, max=5879, avg=243.45, stdev=344.74 clat (msec): min=92, max=864, avg=430.04, stdev=132.38 lat (msec): min=92, max=864, avg=430.28, stdev=132.39 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 249], 10.00th=[ 288], 20.00th=[ 330], | 30.00th=[ 359], 40.00th=[ 380], 50.00th=[ 405], 60.00th=[ 435], | 70.00th=[ 477], 80.00th=[ 542], 90.00th=[ 625], 95.00th=[ 693], | 99.00th=[ 785], 99.50th=[ 818], 99.90th=[ 860], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 47, max= 381, per=24.90%, avg=147.64, stdev=47.89, samples=137 iops : min= 11, max= 95, avg=36.63, stdev=12.01, samples=137 lat (msec) : 100=0.62%, 250=4.53%, 500=69.14%, 750=24.22%, 1000=1.48% cpu : usr=0.07%, sys=0.56%, ctx=3093, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154845: Thu Jun 11 16:55:30 2026 write: IOPS=37, BW=149KiB/s (152kB/s)(10.0MiB/68918msec); 0 zone resets slat (usec): min=44, max=6865, avg=232.48, stdev=308.83 clat (msec): min=119, max=868, avg=430.00, stdev=130.16 lat (msec): min=119, max=869, avg=430.24, stdev=130.17 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 257], 10.00th=[ 288], 20.00th=[ 330], | 30.00th=[ 359], 40.00th=[ 380], 50.00th=[ 405], 60.00th=[ 435], | 70.00th=[ 472], 80.00th=[ 542], 90.00th=[ 625], 95.00th=[ 693], | 99.00th=[ 760], 99.50th=[ 793], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 55, max= 312, per=24.90%, avg=147.66, stdev=44.33, samples=137 iops : min= 13, max= 78, avg=36.63, stdev=11.13, samples=137 lat (msec) : 250=4.34%, 500=70.55%, 750=23.79%, 1000=1.33% cpu : usr=0.05%, sys=0.60%, ctx=2999, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=594KiB/s (608kB/s), 148KiB/s-149KiB/s (152kB/s-152kB/s), io=40.0MiB (41.9MB), run=68851-68957msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=154865: Thu Jun 11 16:56:14 2026 read: IOPS=31, BW=125KiB/s (128kB/s)(5048KiB/40466msec) slat (usec): min=1760, max=16143, avg=5025.10, stdev=2114.06 clat (msec): min=5, max=302, avg=166.25, stdev=39.14 lat (msec): min=10, max=308, avg=171.28, stdev=39.57 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 113], 10.00th=[ 125], 20.00th=[ 136], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 165], 60.00th=[ 174], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 215], 95.00th=[ 232], | 99.00th=[ 279], 99.50th=[ 288], 99.90th=[ 296], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 71, max= 240, per=25.26%, avg=124.79, stdev=31.83, samples=80 iops : min= 17, max= 60, avg=30.91, stdev= 8.02, samples=80 write: IOPS=32, BW=128KiB/s (131kB/s)(5192KiB/40466msec); 0 zone resets slat (usec): min=61, max=6702, avg=248.90, stdev=320.04 clat (msec): min=133, max=626, avg=330.86, stdev=82.74 lat (msec): min=133, max=626, avg=331.11, stdev=82.75 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 201], 10.00th=[ 224], 20.00th=[ 259], | 30.00th=[ 288], 40.00th=[ 309], 50.00th=[ 330], 60.00th=[ 351], | 70.00th=[ 372], 80.00th=[ 397], 90.00th=[ 439], 95.00th=[ 477], | 99.00th=[ 542], 99.50th=[ 609], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 72, max= 214, per=24.86%, avg=127.79, stdev=26.44, samples=80 iops : min= 18, max= 53, avg=31.66, stdev= 6.57, samples=80 lat (msec) : 10=0.12%, 20=0.04%, 50=0.27%, 100=0.70%, 250=55.43% lat (msec) : 500=42.23%, 750=1.21% cpu : usr=0.11%, sys=1.29%, ctx=5211, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154866: Thu Jun 11 16:56:14 2026 read: IOPS=30, BW=123KiB/s (126kB/s)(4976KiB/40582msec) slat (usec): min=1654, max=18001, avg=4964.12, stdev=2043.02 clat (msec): min=4, max=286, avg=166.01, stdev=37.16 lat (msec): min=8, max=290, avg=170.98, stdev=37.53 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 113], 10.00th=[ 125], 20.00th=[ 138], | 30.00th=[ 146], 40.00th=[ 155], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 184], 80.00th=[ 194], 90.00th=[ 213], 95.00th=[ 232], | 99.00th=[ 259], 99.50th=[ 271], 99.90th=[ 284], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 48, max= 215, per=24.62%, avg=121.61, stdev=38.81, samples=80 iops : min= 12, max= 53, avg=30.12, stdev= 9.70, samples=80 write: IOPS=32, BW=130KiB/s (133kB/s)(5264KiB/40582msec); 0 zone resets slat (usec): min=54, max=4255, avg=248.79, stdev=289.50 clat (msec): min=105, max=639, avg=330.42, stdev=83.29 lat (msec): min=106, max=639, avg=330.67, stdev=83.31 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 192], 10.00th=[ 224], 20.00th=[ 259], | 30.00th=[ 284], 40.00th=[ 309], 50.00th=[ 330], 60.00th=[ 351], | 70.00th=[ 376], 80.00th=[ 401], 90.00th=[ 439], 95.00th=[ 468], | 99.00th=[ 527], 99.50th=[ 550], 99.90th=[ 634], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 79, max= 192, per=25.11%, avg=129.05, stdev=25.12, samples=80 iops : min= 19, max= 48, avg=31.98, stdev= 6.31, samples=80 lat (msec) : 10=0.16%, 20=0.04%, 50=0.08%, 100=0.86%, 250=55.39% lat (msec) : 500=42.27%, 750=1.21% cpu : usr=0.14%, sys=1.26%, ctx=5059, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154867: Thu Jun 11 16:56:14 2026 read: IOPS=30, BW=123KiB/s (126kB/s)(4984KiB/40529msec) slat (usec): min=1595, max=21198, avg=5080.79, stdev=2116.68 clat (msec): min=8, max=301, avg=166.27, stdev=37.58 lat (msec): min=11, max=309, avg=171.36, stdev=37.94 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 117], 10.00th=[ 126], 20.00th=[ 138], | 30.00th=[ 146], 40.00th=[ 155], 50.00th=[ 163], 60.00th=[ 171], | 70.00th=[ 184], 80.00th=[ 194], 90.00th=[ 218], 95.00th=[ 234], | 99.00th=[ 271], 99.50th=[ 284], 99.90th=[ 296], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 40, max= 216, per=24.84%, avg=122.69, stdev=35.71, samples=80 iops : min= 10, max= 54, avg=30.39, stdev= 9.01, samples=80 write: IOPS=32, BW=130KiB/s (133kB/s)(5256KiB/40529msec); 0 zone resets slat (usec): min=49, max=2823, avg=239.22, stdev=247.51 clat (msec): min=61, max=638, avg=329.72, stdev=82.93 lat (msec): min=61, max=638, avg=329.96, stdev=82.94 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 205], 10.00th=[ 226], 20.00th=[ 262], | 30.00th=[ 284], 40.00th=[ 309], 50.00th=[ 330], 60.00th=[ 351], | 70.00th=[ 372], 80.00th=[ 397], 90.00th=[ 435], 95.00th=[ 464], | 99.00th=[ 542], 99.50th=[ 592], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 64, max= 176, per=24.95%, avg=128.26, stdev=27.87, samples=80 iops : min= 16, max= 44, avg=31.79, stdev= 6.92, samples=80 lat (msec) : 10=0.04%, 20=0.04%, 50=0.23%, 100=0.55%, 250=54.92% lat (msec) : 500=43.01%, 750=1.21% cpu : usr=0.15%, sys=1.24%, ctx=5203, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154868: Thu Jun 11 16:56:14 2026 read: IOPS=31, BW=126KiB/s (129kB/s)(5060KiB/40315msec) slat (usec): min=95, max=15195, avg=5050.06, stdev=2112.73 clat (msec): min=6, max=389, avg=166.95, stdev=39.44 lat (msec): min=9, max=394, avg=172.01, stdev=39.89 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 113], 10.00th=[ 123], 20.00th=[ 138], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 215], 95.00th=[ 230], | 99.00th=[ 259], 99.50th=[ 279], 99.90th=[ 372], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 32, max= 199, per=25.35%, avg=125.25, stdev=36.85, samples=80 iops : min= 8, max= 49, avg=31.03, stdev= 9.23, samples=80 write: IOPS=32, BW=128KiB/s (132kB/s)(5180KiB/40315msec); 0 zone resets slat (usec): min=40, max=2925, avg=247.72, stdev=271.66 clat (msec): min=50, max=633, avg=329.34, stdev=86.21 lat (msec): min=50, max=633, avg=329.59, stdev=86.22 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 194], 10.00th=[ 222], 20.00th=[ 259], | 30.00th=[ 279], 40.00th=[ 305], 50.00th=[ 330], 60.00th=[ 351], | 70.00th=[ 372], 80.00th=[ 397], 90.00th=[ 435], 95.00th=[ 472], | 99.00th=[ 600], 99.50th=[ 625], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 72, max= 183, per=24.82%, avg=127.55, stdev=27.30, samples=80 iops : min= 18, max= 45, avg=31.60, stdev= 6.82, samples=80 lat (msec) : 10=0.08%, 20=0.04%, 50=0.39%, 100=0.82%, 250=56.05% lat (msec) : 500=41.25%, 750=1.37% cpu : usr=0.14%, sys=1.27%, ctx=5108, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=495KiB/s (506kB/s), 123KiB/s-126KiB/s (126kB/s-129kB/s), io=19.6MiB (20.5MB), run=40315-40582msec WRITE: bw=515KiB/s (527kB/s), 128KiB/s-130KiB/s (131kB/s-133kB/s), io=20.4MiB (21.4MB), run=40315-40582msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=154877: Thu Jun 11 16:56:19 2026 read: IOPS=4, BW=199MiB/s (209MB/s)(40.0MiB/201msec) slat (nsec): min=14770k, max=14770k, avg=14770210.00, stdev= 0.00 clat (nsec): min=179924k, max=179924k, avg=179924281.00, stdev= 0.00 lat (nsec): min=194699k, max=194699k, avg=194698631.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 180], 10.00th=[ 180], 20.00th=[ 180], | 30.00th=[ 180], 40.00th=[ 180], 50.00th=[ 180], 60.00th=[ 180], | 70.00th=[ 180], 80.00th=[ 180], 90.00th=[ 180], 95.00th=[ 180], | 99.00th=[ 180], 99.50th=[ 180], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=5.50%, ctx=42, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=199MiB/s (209MB/s), 199MiB/s-199MiB/s (209MB/s-209MB/s), io=40.0MiB (41.9MB), run=201-201msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout