== sanity test 398c: run fio to test AIO ================= 05:37:51 (1773740271) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.579538 s, 72.4 MB/s osc.lustre-OST0000-osc-ffff8fb392160000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=184204: Tue Mar 17 05:38:35 2026 write: IOPS=73, BW=295KiB/s (302kB/s)(10.0MiB/34750msec); 0 zone resets slat (usec): min=38, max=47246, avg=400.93, stdev=1440.47 clat (msec): min=13, max=376, avg=215.43, stdev=52.92 lat (msec): min=15, max=376, avg=215.83, stdev=52.85 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 138], 10.00th=[ 153], 20.00th=[ 169], | 30.00th=[ 186], 40.00th=[ 194], 50.00th=[ 209], 60.00th=[ 228], | 70.00th=[ 245], 80.00th=[ 264], 90.00th=[ 288], 95.00th=[ 313], | 99.00th=[ 338], 99.50th=[ 368], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 150, max= 432, per=24.99%, avg=294.40, stdev=71.71, samples=68 iops : min= 37, max= 108, avg=73.15, stdev=17.99, samples=68 lat (msec) : 20=0.04%, 50=0.04%, 100=0.78%, 250=72.77%, 500=26.37% cpu : usr=0.18%, sys=1.30%, ctx=2681, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184205: Tue Mar 17 05:38:35 2026 write: IOPS=73, BW=296KiB/s (303kB/s)(10.0MiB/34644msec); 0 zone resets slat (usec): min=39, max=25870, avg=402.23, stdev=1308.55 clat (msec): min=38, max=377, avg=215.09, stdev=54.87 lat (msec): min=42, max=377, avg=215.50, stdev=54.81 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 132], 10.00th=[ 146], 20.00th=[ 169], | 30.00th=[ 186], 40.00th=[ 199], 50.00th=[ 209], 60.00th=[ 228], | 70.00th=[ 245], 80.00th=[ 259], 90.00th=[ 292], 95.00th=[ 313], | 99.00th=[ 355], 99.50th=[ 368], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 199, max= 504, per=25.10%, avg=295.71, stdev=60.97, samples=68 iops : min= 49, max= 126, avg=73.47, stdev=15.30, samples=68 lat (msec) : 50=0.12%, 100=0.55%, 250=72.85%, 500=26.48% cpu : usr=0.18%, sys=1.16%, ctx=2865, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184206: Tue Mar 17 05:38:35 2026 write: IOPS=73, BW=296KiB/s (303kB/s)(10.0MiB/34631msec); 0 zone resets slat (usec): min=37, max=21382, avg=375.21, stdev=1083.68 clat (msec): min=47, max=463, avg=215.88, stdev=52.80 lat (msec): min=49, max=463, avg=216.26, stdev=52.77 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 138], 10.00th=[ 150], 20.00th=[ 171], | 30.00th=[ 188], 40.00th=[ 199], 50.00th=[ 209], 60.00th=[ 224], | 70.00th=[ 245], 80.00th=[ 262], 90.00th=[ 288], 95.00th=[ 313], | 99.00th=[ 342], 99.50th=[ 351], 99.90th=[ 368], 99.95th=[ 443], | 99.99th=[ 464] bw ( KiB/s): min= 167, max= 447, per=25.08%, avg=295.47, stdev=70.84, samples=68 iops : min= 41, max= 111, avg=73.41, stdev=17.77, samples=68 lat (msec) : 50=0.04%, 100=0.78%, 250=72.66%, 500=26.52% cpu : usr=0.17%, sys=1.33%, ctx=2761, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184207: Tue Mar 17 05:38:35 2026 write: IOPS=73, BW=295KiB/s (302kB/s)(10.0MiB/34738msec); 0 zone resets slat (usec): min=40, max=15246, avg=328.38, stdev=937.35 clat (msec): min=61, max=367, avg=216.65, stdev=53.34 lat (msec): min=61, max=367, avg=216.98, stdev=53.32 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 136], 10.00th=[ 153], 20.00th=[ 174], | 30.00th=[ 186], 40.00th=[ 199], 50.00th=[ 209], 60.00th=[ 226], | 70.00th=[ 241], 80.00th=[ 266], 90.00th=[ 292], 95.00th=[ 317], | 99.00th=[ 342], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 151, max= 512, per=24.96%, avg=294.06, stdev=63.57, samples=68 iops : min= 37, max= 128, avg=73.06, stdev=15.92, samples=68 lat (msec) : 100=0.43%, 250=73.67%, 500=25.90% cpu : usr=0.14%, sys=1.20%, ctx=2499, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1179KiB/s (1207kB/s), 295KiB/s-296KiB/s (302kB/s-303kB/s), io=40.0MiB (41.9MB), run=34631-34750msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=184223: Tue Mar 17 05:39:20 2026 read: IOPS=35, BW=143KiB/s (147kB/s)(5048KiB/35262msec) slat (usec): min=126, max=78672, avg=15381.63, stdev=9003.23 clat (msec): min=39, max=428, avg=206.77, stdev=57.71 lat (msec): min=39, max=450, avg=222.16, stdev=60.27 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 117], 10.00th=[ 134], 20.00th=[ 157], | 30.00th=[ 176], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 218], | 70.00th=[ 234], 80.00th=[ 253], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 359], 99.50th=[ 372], 99.90th=[ 426], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 45, max= 271, per=24.90%, avg=141.70, stdev=43.80, samples=69 iops : min= 11, max= 67, avg=34.94, stdev=11.02, samples=69 write: IOPS=36, BW=147KiB/s (151kB/s)(5192KiB/35262msec); 0 zone resets slat (usec): min=56, max=17941, avg=408.56, stdev=1111.98 clat (msec): min=39, max=398, avg=214.26, stdev=55.92 lat (msec): min=39, max=399, avg=214.67, stdev=56.00 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 127], 10.00th=[ 140], 20.00th=[ 167], | 30.00th=[ 186], 40.00th=[ 199], 50.00th=[ 211], 60.00th=[ 228], | 70.00th=[ 243], 80.00th=[ 262], 90.00th=[ 288], 95.00th=[ 305], | 99.00th=[ 355], 99.50th=[ 376], 99.90th=[ 397], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 38, max= 253, per=24.73%, avg=146.39, stdev=40.66, samples=69 iops : min= 9, max= 63, avg=36.06, stdev=10.25, samples=69 lat (msec) : 50=0.23%, 100=0.55%, 250=75.62%, 500=23.59% cpu : usr=0.28%, sys=2.16%, ctx=4642, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184224: Tue Mar 17 05:39:20 2026 read: IOPS=35, BW=141KiB/s (145kB/s)(4976KiB/35209msec) slat (usec): min=2858, max=75145, avg=15231.69, stdev=8413.91 clat (msec): min=55, max=388, avg=207.51, stdev=57.50 lat (msec): min=65, max=416, avg=222.76, stdev=60.08 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 116], 10.00th=[ 132], 20.00th=[ 159], | 30.00th=[ 178], 40.00th=[ 192], 50.00th=[ 205], 60.00th=[ 218], | 70.00th=[ 232], 80.00th=[ 255], 90.00th=[ 288], 95.00th=[ 313], | 99.00th=[ 355], 99.50th=[ 363], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 45, max= 279, per=24.77%, avg=140.93, stdev=47.06, samples=69 iops : min= 11, max= 69, avg=34.72, stdev=11.82, samples=69 write: IOPS=37, BW=150KiB/s (153kB/s)(5264KiB/35209msec); 0 zone resets slat (usec): min=50, max=12620, avg=374.17, stdev=924.25 clat (msec): min=95, max=387, avg=214.65, stdev=54.24 lat (msec): min=95, max=388, avg=215.03, stdev=54.28 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 129], 10.00th=[ 140], 20.00th=[ 167], | 30.00th=[ 186], 40.00th=[ 201], 50.00th=[ 213], 60.00th=[ 226], | 70.00th=[ 243], 80.00th=[ 262], 90.00th=[ 288], 95.00th=[ 309], | 99.00th=[ 338], 99.50th=[ 359], 99.90th=[ 363], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 87, max= 238, per=25.27%, avg=149.59, stdev=38.55, samples=69 iops : min= 21, max= 59, avg=36.88, stdev= 9.73, samples=69 lat (msec) : 100=0.70%, 250=75.39%, 500=23.91% cpu : usr=0.28%, sys=2.01%, ctx=4516, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184225: Tue Mar 17 05:39:20 2026 read: IOPS=35, BW=142KiB/s (145kB/s)(4984KiB/35104msec) slat (usec): min=62, max=54943, avg=15261.96, stdev=8457.03 clat (msec): min=66, max=422, avg=209.07, stdev=56.46 lat (msec): min=85, max=433, avg=224.34, stdev=59.00 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 122], 10.00th=[ 138], 20.00th=[ 159], | 30.00th=[ 178], 40.00th=[ 192], 50.00th=[ 205], 60.00th=[ 220], | 70.00th=[ 236], 80.00th=[ 257], 90.00th=[ 288], 95.00th=[ 309], | 99.00th=[ 342], 99.50th=[ 359], 99.90th=[ 380], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 77, max= 280, per=24.93%, avg=141.87, stdev=40.51, samples=69 iops : min= 19, max= 70, avg=34.96, stdev=10.21, samples=69 write: IOPS=37, BW=150KiB/s (153kB/s)(5256KiB/35104msec); 0 zone resets slat (usec): min=38, max=33873, avg=406.28, stdev=1584.66 clat (msec): min=57, max=409, avg=213.53, stdev=54.98 lat (msec): min=57, max=409, avg=213.94, stdev=54.96 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 127], 10.00th=[ 140], 20.00th=[ 165], | 30.00th=[ 184], 40.00th=[ 199], 50.00th=[ 211], 60.00th=[ 224], | 70.00th=[ 241], 80.00th=[ 259], 90.00th=[ 288], 95.00th=[ 309], | 99.00th=[ 351], 99.50th=[ 363], 99.90th=[ 401], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 71, max= 277, per=25.20%, avg=149.20, stdev=44.95, samples=69 iops : min= 17, max= 69, avg=36.80, stdev=11.31, samples=69 lat (msec) : 100=0.59%, 250=75.90%, 500=23.52% cpu : usr=0.26%, sys=1.99%, ctx=4396, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184226: Tue Mar 17 05:39:20 2026 read: IOPS=35, BW=144KiB/s (147kB/s)(5060KiB/35214msec) slat (usec): min=2603, max=62385, avg=15335.19, stdev=8774.16 clat (msec): min=80, max=408, avg=208.26, stdev=57.46 lat (msec): min=86, max=426, avg=223.60, stdev=60.63 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 120], 10.00th=[ 136], 20.00th=[ 159], | 30.00th=[ 176], 40.00th=[ 192], 50.00th=[ 203], 60.00th=[ 218], | 70.00th=[ 234], 80.00th=[ 257], 90.00th=[ 288], 95.00th=[ 313], | 99.00th=[ 359], 99.50th=[ 372], 99.90th=[ 393], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 22, max= 285, per=25.17%, avg=143.20, stdev=43.90, samples=69 iops : min= 5, max= 71, avg=35.29, stdev=11.05, samples=69 write: IOPS=36, BW=147KiB/s (151kB/s)(5180KiB/35214msec); 0 zone resets slat (usec): min=45, max=7886, avg=300.76, stdev=645.11 clat (msec): min=93, max=388, avg=212.24, stdev=54.00 lat (msec): min=93, max=388, avg=212.54, stdev=54.05 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 127], 10.00th=[ 138], 20.00th=[ 167], | 30.00th=[ 184], 40.00th=[ 197], 50.00th=[ 211], 60.00th=[ 224], | 70.00th=[ 239], 80.00th=[ 259], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 338], 99.50th=[ 355], 99.90th=[ 368], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 30, max= 254, per=24.75%, avg=146.52, stdev=44.19, samples=69 iops : min= 7, max= 63, avg=36.13, stdev=11.12, samples=69 lat (msec) : 100=0.35%, 250=76.68%, 500=22.97% cpu : usr=0.25%, sys=2.02%, ctx=4309, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=569KiB/s (583kB/s), 141KiB/s-144KiB/s (145kB/s-147kB/s), io=19.6MiB (20.5MB), run=35104-35262msec WRITE: bw=592KiB/s (607kB/s), 147KiB/s-150KiB/s (151kB/s-153kB/s), io=20.4MiB (21.4MB), run=35104-35262msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=184235: Tue Mar 17 05:39:30 2026 read: IOPS=1, BW=70.8MiB/s (74.2MB/s)(40.0MiB/565msec) slat (nsec): min=39857k, max=39857k, avg=39856681.00, stdev= 0.00 clat (nsec): min=491247k, max=491247k, avg=491246664.00, stdev= 0.00 lat (nsec): min=531108k, max=531108k, avg=531107875.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 493], 10.00th=[ 493], 20.00th=[ 493], | 30.00th=[ 493], 40.00th=[ 493], 50.00th=[ 493], 60.00th=[ 493], | 70.00th=[ 493], 80.00th=[ 493], 90.00th=[ 493], 95.00th=[ 493], | 99.00th=[ 493], 99.50th=[ 493], 99.90th=[ 493], 99.95th=[ 493], | 99.99th=[ 493] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=1.60%, ctx=9, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=70.8MiB/s (74.2MB/s), 70.8MiB/s-70.8MiB/s (74.2MB/s-74.2MB/s), io=40.0MiB (41.9MB), run=565-565msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout