== sanity test 398c: run fio to test AIO ================= 09:54:40 (1761314080) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.235608 s, 178 MB/s osc.lustre-OST0000-osc-ffff94288349e800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=149588: Fri Oct 24 09:55:02 2025 write: IOPS=154, BW=619KiB/s (633kB/s)(10.0MiB/16555msec); 0 zone resets slat (usec): min=22, max=39223, avg=130.43, stdev=829.77 clat (msec): min=16, max=181, avg=102.93, stdev=25.20 lat (msec): min=16, max=181, avg=103.07, stdev=25.16 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 66], 10.00th=[ 72], 20.00th=[ 83], | 30.00th=[ 88], 40.00th=[ 95], 50.00th=[ 102], 60.00th=[ 109], | 70.00th=[ 115], 80.00th=[ 123], 90.00th=[ 140], 95.00th=[ 150], | 99.00th=[ 169], 99.50th=[ 174], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 488, max= 774, per=24.93%, avg=616.75, stdev=100.43, samples=32 iops : min= 122, max= 193, avg=153.91, stdev=25.11, samples=32 lat (msec) : 20=0.27%, 50=0.78%, 100=48.55%, 250=50.39% cpu : usr=0.22%, sys=1.38%, ctx=764, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149589: Fri Oct 24 09:55:02 2025 write: IOPS=155, BW=622KiB/s (637kB/s)(10.0MiB/16474msec); 0 zone resets slat (usec): min=24, max=5036, avg=146.39, stdev=342.16 clat (msec): min=9, max=246, avg=102.68, stdev=26.85 lat (msec): min=9, max=246, avg=102.82, stdev=26.85 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 67], 10.00th=[ 73], 20.00th=[ 81], | 30.00th=[ 88], 40.00th=[ 94], 50.00th=[ 102], 60.00th=[ 107], | 70.00th=[ 114], 80.00th=[ 125], 90.00th=[ 136], 95.00th=[ 150], | 99.00th=[ 176], 99.50th=[ 194], 99.90th=[ 247], 99.95th=[ 247], | 99.99th=[ 247] bw ( KiB/s): min= 416, max= 862, per=25.21%, avg=623.69, stdev=116.45, samples=32 iops : min= 104, max= 215, avg=155.66, stdev=29.08, samples=32 lat (msec) : 10=0.04%, 20=0.16%, 50=0.98%, 100=47.73%, 250=51.09% cpu : usr=0.25%, sys=1.54%, ctx=1254, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149590: Fri Oct 24 09:55:02 2025 write: IOPS=154, BW=619KiB/s (634kB/s)(10.0MiB/16536msec); 0 zone resets slat (usec): min=24, max=40561, avg=160.72, stdev=890.14 clat (msec): min=9, max=194, avg=102.77, stdev=25.28 lat (msec): min=9, max=194, avg=102.93, stdev=25.27 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 67], 10.00th=[ 73], 20.00th=[ 81], | 30.00th=[ 89], 40.00th=[ 94], 50.00th=[ 100], 60.00th=[ 107], | 70.00th=[ 115], 80.00th=[ 124], 90.00th=[ 138], 95.00th=[ 148], | 99.00th=[ 176], 99.50th=[ 184], 99.90th=[ 192], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 432, max= 896, per=25.05%, avg=619.81, stdev=109.89, samples=32 iops : min= 108, max= 224, avg=154.69, stdev=27.45, samples=32 lat (msec) : 10=0.04%, 50=0.82%, 100=50.12%, 250=49.02% cpu : usr=0.25%, sys=1.70%, ctx=1210, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149591: Fri Oct 24 09:55:02 2025 write: IOPS=155, BW=621KiB/s (636kB/s)(10.0MiB/16493msec); 0 zone resets slat (usec): min=21, max=8581, avg=138.52, stdev=324.05 clat (msec): min=11, max=216, avg=102.77, stdev=26.40 lat (msec): min=11, max=216, avg=102.91, stdev=26.41 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 67], 10.00th=[ 74], 20.00th=[ 82], | 30.00th=[ 88], 40.00th=[ 94], 50.00th=[ 101], 60.00th=[ 107], | 70.00th=[ 113], 80.00th=[ 124], 90.00th=[ 140], 95.00th=[ 153], | 99.00th=[ 174], 99.50th=[ 188], 99.90th=[ 205], 99.95th=[ 205], | 99.99th=[ 218] bw ( KiB/s): min= 408, max= 838, per=25.13%, avg=621.75, stdev=114.35, samples=32 iops : min= 102, max= 209, avg=155.16, stdev=28.55, samples=32 lat (msec) : 20=0.16%, 50=1.25%, 100=47.85%, 250=50.74% cpu : usr=0.28%, sys=1.56%, ctx=1233, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2474KiB/s (2534kB/s), 619KiB/s-622KiB/s (633kB/s-637kB/s), io=40.0MiB (41.9MB), run=16474-16555msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=149603: Fri Oct 24 09:55:27 2025 read: IOPS=59, BW=239KiB/s (245kB/s)(5048KiB/21114msec) slat (usec): min=1403, max=29521, avg=8281.68, stdev=4350.79 clat (msec): min=48, max=290, avg=125.75, stdev=39.97 lat (msec): min=55, max=302, avg=134.04, stdev=41.85 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 70], 10.00th=[ 79], 20.00th=[ 92], | 30.00th=[ 104], 40.00th=[ 112], 50.00th=[ 120], 60.00th=[ 131], | 70.00th=[ 142], 80.00th=[ 157], 90.00th=[ 182], 95.00th=[ 203], | 99.00th=[ 239], 99.50th=[ 255], 99.90th=[ 284], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 125, max= 400, per=25.03%, avg=237.76, stdev=64.28, samples=41 iops : min= 31, max= 100, avg=59.05, stdev=16.15, samples=41 write: IOPS=61, BW=246KiB/s (252kB/s)(5192KiB/21114msec); 0 zone resets slat (usec): min=26, max=10813, avg=212.75, stdev=629.58 clat (msec): min=15, max=296, avg=128.41, stdev=37.68 lat (msec): min=16, max=296, avg=128.63, stdev=37.74 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 79], 10.00th=[ 86], 20.00th=[ 99], | 30.00th=[ 106], 40.00th=[ 114], 50.00th=[ 123], 60.00th=[ 133], | 70.00th=[ 142], 80.00th=[ 155], 90.00th=[ 182], 95.00th=[ 197], | 99.00th=[ 245], 99.50th=[ 255], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 149, max= 384, per=24.82%, avg=245.49, stdev=62.71, samples=41 iops : min= 37, max= 96, avg=60.95, stdev=15.83, samples=41 lat (msec) : 20=0.04%, 50=0.16%, 100=24.41%, 250=74.65%, 500=0.74% cpu : usr=0.31%, sys=2.41%, ctx=2651, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149604: Fri Oct 24 09:55:27 2025 read: IOPS=58, BW=236KiB/s (241kB/s)(4976KiB/21103msec) slat (usec): min=1675, max=49830, avg=8222.04, stdev=4433.70 clat (msec): min=20, max=282, avg=124.64, stdev=37.69 lat (msec): min=27, max=302, avg=132.87, stdev=39.32 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 74], 10.00th=[ 83], 20.00th=[ 92], | 30.00th=[ 102], 40.00th=[ 111], 50.00th=[ 121], 60.00th=[ 129], | 70.00th=[ 138], 80.00th=[ 155], 90.00th=[ 180], 95.00th=[ 197], | 99.00th=[ 228], 99.50th=[ 241], 99.90th=[ 268], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 110, max= 352, per=24.76%, avg=235.24, stdev=63.51, samples=41 iops : min= 27, max= 88, avg=58.39, stdev=15.95, samples=41 write: IOPS=62, BW=249KiB/s (255kB/s)(5264KiB/21103msec); 0 zone resets slat (usec): min=28, max=11684, avg=189.91, stdev=680.57 clat (msec): min=51, max=307, avg=129.94, stdev=39.14 lat (msec): min=51, max=307, avg=130.13, stdev=39.16 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 75], 10.00th=[ 84], 20.00th=[ 95], | 30.00th=[ 106], 40.00th=[ 116], 50.00th=[ 127], 60.00th=[ 136], | 70.00th=[ 146], 80.00th=[ 163], 90.00th=[ 186], 95.00th=[ 203], | 99.00th=[ 232], 99.50th=[ 249], 99.90th=[ 305], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 143, max= 456, per=25.18%, avg=249.00, stdev=66.61, samples=41 iops : min= 35, max= 114, avg=61.83, stdev=16.75, samples=41 lat (msec) : 50=0.08%, 100=26.48%, 250=73.12%, 500=0.31% cpu : usr=0.31%, sys=2.18%, ctx=2503, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149605: Fri Oct 24 09:55:27 2025 read: IOPS=59, BW=237KiB/s (243kB/s)(4984KiB/21013msec) slat (usec): min=1782, max=40908, avg=8318.43, stdev=4443.54 clat (msec): min=15, max=287, avg=124.08, stdev=39.39 lat (msec): min=22, max=300, avg=132.41, stdev=41.10 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 69], 10.00th=[ 79], 20.00th=[ 92], | 30.00th=[ 102], 40.00th=[ 111], 50.00th=[ 118], 60.00th=[ 128], | 70.00th=[ 138], 80.00th=[ 153], 90.00th=[ 178], 95.00th=[ 201], | 99.00th=[ 241], 99.50th=[ 253], 99.90th=[ 279], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 110, max= 440, per=25.00%, avg=237.49, stdev=62.36, samples=41 iops : min= 27, max= 110, avg=59.00, stdev=15.67, samples=41 write: IOPS=62, BW=250KiB/s (256kB/s)(5256KiB/21013msec); 0 zone resets slat (usec): min=27, max=11730, avg=198.45, stdev=601.64 clat (msec): min=50, max=322, avg=129.07, stdev=38.25 lat (msec): min=50, max=322, avg=129.27, stdev=38.28 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 79], 10.00th=[ 85], 20.00th=[ 96], | 30.00th=[ 108], 40.00th=[ 117], 50.00th=[ 124], 60.00th=[ 133], | 70.00th=[ 142], 80.00th=[ 157], 90.00th=[ 182], 95.00th=[ 201], | 99.00th=[ 239], 99.50th=[ 292], 99.90th=[ 309], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 157, max= 448, per=25.20%, avg=249.24, stdev=67.06, samples=41 iops : min= 39, max= 112, avg=61.90, stdev=16.87, samples=41 lat (msec) : 20=0.04%, 50=0.12%, 100=25.20%, 250=74.02%, 500=0.62% cpu : usr=0.29%, sys=2.26%, ctx=2585, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149606: Fri Oct 24 09:55:27 2025 read: IOPS=59, BW=240KiB/s (246kB/s)(5060KiB/21087msec) slat (usec): min=2047, max=30103, avg=8297.97, stdev=4476.38 clat (msec): min=12, max=276, avg=124.65, stdev=39.01 lat (msec): min=18, max=288, avg=132.95, stdev=40.97 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 69], 10.00th=[ 80], 20.00th=[ 92], | 30.00th=[ 102], 40.00th=[ 110], 50.00th=[ 120], 60.00th=[ 129], | 70.00th=[ 140], 80.00th=[ 157], 90.00th=[ 182], 95.00th=[ 199], | 99.00th=[ 230], 99.50th=[ 241], 99.90th=[ 275], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 140, max= 408, per=25.31%, avg=240.44, stdev=68.44, samples=41 iops : min= 35, max= 102, avg=59.73, stdev=17.18, samples=41 write: IOPS=61, BW=246KiB/s (252kB/s)(5180KiB/21087msec); 0 zone resets slat (usec): min=26, max=9001, avg=211.46, stdev=587.84 clat (msec): min=57, max=305, avg=129.08, stdev=36.45 lat (msec): min=57, max=305, avg=129.30, stdev=36.48 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 79], 10.00th=[ 84], 20.00th=[ 97], | 30.00th=[ 108], 40.00th=[ 117], 50.00th=[ 126], 60.00th=[ 134], | 70.00th=[ 144], 80.00th=[ 159], 90.00th=[ 178], 95.00th=[ 194], | 99.00th=[ 232], 99.50th=[ 249], 99.90th=[ 279], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 149, max= 376, per=24.78%, avg=245.07, stdev=57.74, samples=41 iops : min= 37, max= 94, avg=60.88, stdev=14.55, samples=41 lat (msec) : 20=0.04%, 50=0.04%, 100=25.55%, 250=73.95%, 500=0.43% cpu : usr=0.33%, sys=2.26%, ctx=2645, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=950KiB/s (973kB/s), 236KiB/s-240KiB/s (241kB/s-246kB/s), io=19.6MiB (20.5MB), run=21013-21114msec WRITE: bw=989KiB/s (1013kB/s), 246KiB/s-250KiB/s (252kB/s-256kB/s), io=20.4MiB (21.4MB), run=21013-21114msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=149613: Fri Oct 24 09:55:35 2025 read: IOPS=3, BW=150MiB/s (158MB/s)(40.0MiB/266msec) slat (nsec): min=22420k, max=22420k, avg=22419914.00, stdev= 0.00 clat (nsec): min=230984k, max=230984k, avg=230984307.00, stdev= 0.00 lat (nsec): min=253409k, max=253409k, avg=253409401.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 232], 10.00th=[ 232], 20.00th=[ 232], | 30.00th=[ 232], 40.00th=[ 232], 50.00th=[ 232], 60.00th=[ 232], | 70.00th=[ 232], 80.00th=[ 232], 90.00th=[ 232], 95.00th=[ 232], | 99.00th=[ 232], 99.50th=[ 232], 99.90th=[ 232], 99.95th=[ 232], | 99.99th=[ 232] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=5.28%, ctx=2, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=150MiB/s (158MB/s), 150MiB/s-150MiB/s (158MB/s-158MB/s), io=40.0MiB (41.9MB), run=266-266msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck