== sanity test 398c: run fio to test AIO ================= 03:52:52 (1778658772) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.519563 s, 80.7 MB/s osc.lustre-OST0000-osc-ffff90a2c5701800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=183621: Wed May 13 03:53:30 2026 write: IOPS=91, BW=368KiB/s (376kB/s)(10.0MiB/27862msec); 0 zone resets slat (usec): min=53, max=12098, avg=277.61, stdev=682.18 clat (msec): min=16, max=338, avg=173.73, stdev=42.45 lat (msec): min=18, max=338, avg=174.01, stdev=42.43 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 109], 10.00th=[ 126], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 161], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 192], 80.00th=[ 209], 90.00th=[ 232], 95.00th=[ 247], | 99.00th=[ 305], 99.50th=[ 317], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 245, max= 593, per=25.19%, avg=367.07, stdev=64.99, samples=55 iops : min= 61, max= 148, avg=91.31, stdev=16.27, samples=55 lat (msec) : 20=0.08%, 50=0.08%, 100=3.12%, 250=92.58%, 500=4.14% cpu : usr=0.19%, sys=1.25%, ctx=2992, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183622: Wed May 13 03:53:30 2026 write: IOPS=91, BW=364KiB/s (373kB/s)(10.0MiB/28097msec); 0 zone resets slat (usec): min=38, max=10989, avg=304.64, stdev=819.87 clat (msec): min=19, max=372, avg=174.44, stdev=41.39 lat (msec): min=20, max=372, avg=174.75, stdev=41.36 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 112], 10.00th=[ 130], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 178], | 70.00th=[ 190], 80.00th=[ 209], 90.00th=[ 230], 95.00th=[ 247], | 99.00th=[ 279], 99.50th=[ 330], 99.90th=[ 355], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 252, max= 510, per=25.00%, avg=364.25, stdev=63.56, samples=55 iops : min= 63, max= 127, avg=90.62, stdev=15.95, samples=55 lat (msec) : 20=0.04%, 50=0.04%, 100=2.97%, 250=93.71%, 500=3.24% cpu : usr=0.18%, sys=1.24%, ctx=2584, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183623: Wed May 13 03:53:30 2026 write: IOPS=91, BW=364KiB/s (373kB/s)(10.0MiB/28098msec); 0 zone resets slat (usec): min=40, max=16757, avg=292.08, stdev=808.49 clat (msec): min=42, max=350, avg=174.58, stdev=40.73 lat (msec): min=45, max=350, avg=174.87, stdev=40.70 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 117], 10.00th=[ 130], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 161], 50.00th=[ 167], 60.00th=[ 180], | 70.00th=[ 192], 80.00th=[ 207], 90.00th=[ 230], 95.00th=[ 249], | 99.00th=[ 288], 99.50th=[ 317], 99.90th=[ 326], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 252, max= 510, per=24.96%, avg=363.64, stdev=70.47, samples=55 iops : min= 63, max= 127, avg=90.47, stdev=17.65, samples=55 lat (msec) : 50=0.04%, 100=2.19%, 250=93.67%, 500=4.10% cpu : usr=0.18%, sys=1.10%, ctx=2823, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183624: Wed May 13 03:53:30 2026 write: IOPS=91, BW=367KiB/s (376kB/s)(10.0MiB/27882msec); 0 zone resets slat (usec): min=44, max=16134, avg=274.21, stdev=752.18 clat (msec): min=18, max=324, avg=173.38, stdev=40.47 lat (msec): min=18, max=324, avg=173.66, stdev=40.44 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 111], 10.00th=[ 129], 20.00th=[ 144], | 30.00th=[ 153], 40.00th=[ 161], 50.00th=[ 169], 60.00th=[ 178], | 70.00th=[ 190], 80.00th=[ 207], 90.00th=[ 226], 95.00th=[ 247], | 99.00th=[ 284], 99.50th=[ 288], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 245, max= 556, per=25.03%, avg=364.73, stdev=70.58, samples=55 iops : min= 61, max= 139, avg=90.75, stdev=17.74, samples=55 lat (msec) : 20=0.04%, 50=0.31%, 100=2.62%, 250=92.81%, 500=4.22% cpu : usr=0.19%, sys=1.20%, ctx=2260, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1458KiB/s (1493kB/s), 364KiB/s-368KiB/s (373kB/s-376kB/s), io=40.0MiB (41.9MB), run=27862-28098msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=183639: Wed May 13 03:54:21 2026 read: IOPS=29, BW=120KiB/s (123kB/s)(5048KiB/42159msec) slat (usec): min=2629, max=76908, avg=17933.80, stdev=11329.20 clat (msec): min=74, max=530, avg=252.00, stdev=72.49 lat (msec): min=77, max=576, avg=269.94, stdev=75.82 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 148], 10.00th=[ 169], 20.00th=[ 192], | 30.00th=[ 211], 40.00th=[ 226], 50.00th=[ 243], 60.00th=[ 262], | 70.00th=[ 284], 80.00th=[ 309], 90.00th=[ 347], 95.00th=[ 388], | 99.00th=[ 451], 99.50th=[ 468], 99.90th=[ 527], 99.95th=[ 531], | 99.99th=[ 531] bw ( KiB/s): min= 32, max= 207, per=24.96%, avg=118.82, stdev=34.62, samples=83 iops : min= 8, max= 51, avg=29.18, stdev= 8.66, samples=83 write: IOPS=30, BW=123KiB/s (126kB/s)(5192KiB/42159msec); 0 zone resets slat (usec): min=50, max=19652, avg=306.59, stdev=872.65 clat (msec): min=76, max=511, avg=253.81, stdev=71.34 lat (msec): min=76, max=511, avg=254.12, stdev=71.33 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 155], 10.00th=[ 176], 20.00th=[ 194], | 30.00th=[ 211], 40.00th=[ 226], 50.00th=[ 245], 60.00th=[ 264], | 70.00th=[ 288], 80.00th=[ 313], 90.00th=[ 351], 95.00th=[ 384], | 99.00th=[ 460], 99.50th=[ 481], 99.90th=[ 510], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 39, max= 280, per=24.77%, avg=122.59, stdev=44.23, samples=83 iops : min= 9, max= 70, avg=30.11, stdev=11.09, samples=83 lat (msec) : 100=0.86%, 250=52.58%, 500=46.25%, 750=0.31% cpu : usr=0.21%, sys=1.73%, ctx=4666, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183640: Wed May 13 03:54:21 2026 read: IOPS=29, BW=118KiB/s (121kB/s)(4976KiB/42119msec) slat (msec): min=2, max=162, avg=17.96, stdev=11.45 clat (msec): min=27, max=509, avg=249.02, stdev=72.48 lat (msec): min=42, max=543, avg=266.99, stdev=76.04 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 150], 10.00th=[ 167], 20.00th=[ 192], | 30.00th=[ 207], 40.00th=[ 222], 50.00th=[ 236], 60.00th=[ 257], | 70.00th=[ 279], 80.00th=[ 309], 90.00th=[ 355], 95.00th=[ 384], | 99.00th=[ 443], 99.50th=[ 460], 99.90th=[ 498], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 39, max= 223, per=24.66%, avg=117.39, stdev=40.25, samples=83 iops : min= 9, max= 55, avg=28.81, stdev=10.10, samples=83 write: IOPS=31, BW=125KiB/s (128kB/s)(5264KiB/42119msec); 0 zone resets slat (usec): min=55, max=13185, avg=299.78, stdev=853.26 clat (msec): min=66, max=500, avg=256.84, stdev=70.64 lat (msec): min=67, max=500, avg=257.14, stdev=70.62 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 157], 10.00th=[ 178], 20.00th=[ 201], | 30.00th=[ 218], 40.00th=[ 232], 50.00th=[ 245], 60.00th=[ 266], | 70.00th=[ 288], 80.00th=[ 313], 90.00th=[ 355], 95.00th=[ 393], | 99.00th=[ 460], 99.50th=[ 472], 99.90th=[ 489], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 55, max= 240, per=25.17%, avg=124.60, stdev=36.15, samples=83 iops : min= 13, max= 60, avg=30.63, stdev= 9.07, samples=83 lat (msec) : 50=0.04%, 100=0.82%, 250=53.44%, 500=45.62%, 750=0.08% cpu : usr=0.21%, sys=1.59%, ctx=4347, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183641: Wed May 13 03:54:21 2026 read: IOPS=29, BW=118KiB/s (121kB/s)(4984KiB/42075msec) slat (usec): min=3120, max=87435, avg=17971.52, stdev=11425.16 clat (msec): min=10, max=565, avg=249.69, stdev=71.42 lat (msec): min=24, max=578, avg=267.66, stdev=75.01 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 155], 10.00th=[ 176], 20.00th=[ 197], | 30.00th=[ 209], 40.00th=[ 224], 50.00th=[ 236], 60.00th=[ 253], | 70.00th=[ 279], 80.00th=[ 305], 90.00th=[ 351], 95.00th=[ 384], | 99.00th=[ 443], 99.50th=[ 472], 99.90th=[ 535], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 31, max= 200, per=24.87%, avg=118.40, stdev=36.53, samples=83 iops : min= 7, max= 50, avg=29.05, stdev= 9.17, samples=83 write: IOPS=31, BW=125KiB/s (128kB/s)(5256KiB/42075msec); 0 zone resets slat (usec): min=57, max=26094, avg=352.03, stdev=1171.03 clat (msec): min=76, max=521, avg=254.96, stdev=72.75 lat (msec): min=76, max=521, avg=255.31, stdev=72.70 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 157], 10.00th=[ 176], 20.00th=[ 194], | 30.00th=[ 211], 40.00th=[ 226], 50.00th=[ 243], 60.00th=[ 264], | 70.00th=[ 288], 80.00th=[ 317], 90.00th=[ 355], 95.00th=[ 388], | 99.00th=[ 456], 99.50th=[ 485], 99.90th=[ 502], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 32, max= 264, per=25.08%, avg=124.17, stdev=40.73, samples=83 iops : min= 8, max= 66, avg=30.51, stdev=10.18, samples=83 lat (msec) : 20=0.04%, 50=0.04%, 100=0.98%, 250=54.96%, 500=43.71% lat (msec) : 750=0.27% cpu : usr=0.21%, sys=1.73%, ctx=4452, majf=0, minf=19 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183642: Wed May 13 03:54:21 2026 read: IOPS=30, BW=121KiB/s (124kB/s)(5060KiB/41916msec) slat (usec): min=58, max=117267, avg=17576.31, stdev=11343.71 clat (msec): min=72, max=571, avg=251.72, stdev=72.19 lat (msec): min=82, max=600, avg=269.31, stdev=75.88 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 153], 10.00th=[ 171], 20.00th=[ 197], | 30.00th=[ 211], 40.00th=[ 228], 50.00th=[ 243], 60.00th=[ 257], | 70.00th=[ 279], 80.00th=[ 313], 90.00th=[ 351], 95.00th=[ 380], | 99.00th=[ 447], 99.50th=[ 489], 99.90th=[ 567], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 31, max= 240, per=25.32%, avg=120.50, stdev=33.04, samples=82 iops : min= 7, max= 60, avg=29.59, stdev= 8.27, samples=82 write: IOPS=30, BW=124KiB/s (127kB/s)(5180KiB/41916msec); 0 zone resets slat (usec): min=55, max=16187, avg=327.56, stdev=1019.09 clat (msec): min=31, max=564, avg=253.83, stdev=74.79 lat (msec): min=32, max=564, avg=254.16, stdev=74.80 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 155], 10.00th=[ 178], 20.00th=[ 194], | 30.00th=[ 211], 40.00th=[ 228], 50.00th=[ 243], 60.00th=[ 262], | 70.00th=[ 279], 80.00th=[ 313], 90.00th=[ 359], 95.00th=[ 388], | 99.00th=[ 493], 99.50th=[ 514], 99.90th=[ 535], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 39, max= 262, per=24.90%, avg=123.23, stdev=44.22, samples=82 iops : min= 9, max= 65, avg=30.28, stdev=11.10, samples=82 lat (msec) : 50=0.08%, 100=1.13%, 250=53.98%, 500=44.26%, 750=0.55% cpu : usr=0.23%, sys=1.80%, ctx=4393, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=476KiB/s (487kB/s), 118KiB/s-121KiB/s (121kB/s-124kB/s), io=19.6MiB (20.5MB), run=41916-42159msec WRITE: bw=496KiB/s (507kB/s), 123KiB/s-125KiB/s (126kB/s-128kB/s), io=20.4MiB (21.4MB), run=41916-42159msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=183653: Wed May 13 03:54:33 2026 read: IOPS=1, BW=66.1MiB/s (69.3MB/s)(40.0MiB/605msec) slat (nsec): min=40241k, max=40241k, avg=40241389.00, stdev= 0.00 clat (nsec): min=543207k, max=543207k, avg=543206566.00, stdev= 0.00 lat (nsec): min=583453k, max=583453k, avg=583452595.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 542], 10.00th=[ 542], 20.00th=[ 542], | 30.00th=[ 542], 40.00th=[ 542], 50.00th=[ 542], 60.00th=[ 542], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.15%, ctx=8, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=66.1MiB/s (69.3MB/s), 66.1MiB/s-66.1MiB/s (69.3MB/s-69.3MB/s), io=40.0MiB (41.9MB), run=605-605msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout