== sanity test 398c: run fio to test AIO ================= 00:48:34 (1782362914) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.1307 s, 321 MB/s osc.lustre-OST0000-osc-ffff9b7a67222800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=155567: Thu Jun 25 00:49:07 2026 write: IOPS=85, BW=344KiB/s (352kB/s)(10.0MiB/29771msec); 0 zone resets slat (usec): min=50, max=976, avg=255.55, stdev=106.22 clat (msec): min=44, max=323, avg=185.59, stdev=32.81 lat (msec): min=45, max=324, avg=185.85, stdev=32.80 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 134], 10.00th=[ 148], 20.00th=[ 161], | 30.00th=[ 169], 40.00th=[ 178], 50.00th=[ 186], 60.00th=[ 194], | 70.00th=[ 203], 80.00th=[ 213], 90.00th=[ 228], 95.00th=[ 241], | 99.00th=[ 262], 99.50th=[ 284], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 326] bw ( KiB/s): min= 232, max= 484, per=24.95%, avg=343.12, stdev=43.38, samples=59 iops : min= 58, max= 121, avg=85.78, stdev=10.85, samples=59 lat (msec) : 50=0.20%, 100=0.86%, 250=96.64%, 500=2.30% cpu : usr=0.16%, sys=1.88%, ctx=2816, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=155568: Thu Jun 25 00:49:07 2026 write: IOPS=86, BW=344KiB/s (352kB/s)(10.0MiB/29757msec); 0 zone resets slat (usec): min=42, max=828, avg=246.89, stdev=108.99 clat (msec): min=37, max=437, avg=185.59, stdev=35.32 lat (msec): min=37, max=437, avg=185.83, stdev=35.33 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 133], 10.00th=[ 146], 20.00th=[ 161], | 30.00th=[ 169], 40.00th=[ 178], 50.00th=[ 186], 60.00th=[ 192], | 70.00th=[ 203], 80.00th=[ 211], 90.00th=[ 228], 95.00th=[ 241], | 99.00th=[ 275], 99.50th=[ 313], 99.90th=[ 388], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 240, max= 530, per=24.95%, avg=343.08, stdev=49.91, samples=59 iops : min= 60, max= 132, avg=85.76, stdev=12.45, samples=59 lat (msec) : 50=0.39%, 100=0.78%, 250=96.45%, 500=2.38% cpu : usr=0.18%, sys=1.76%, ctx=2774, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=155569: Thu Jun 25 00:49:07 2026 write: IOPS=85, BW=344KiB/s (352kB/s)(10.0MiB/29771msec); 0 zone resets slat (usec): min=42, max=994, avg=249.14, stdev=105.81 clat (msec): min=45, max=351, avg=185.55, stdev=33.46 lat (msec): min=45, max=351, avg=185.80, stdev=33.46 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 132], 10.00th=[ 148], 20.00th=[ 161], | 30.00th=[ 169], 40.00th=[ 176], 50.00th=[ 186], 60.00th=[ 194], | 70.00th=[ 201], 80.00th=[ 213], 90.00th=[ 226], 95.00th=[ 241], | 99.00th=[ 268], 99.50th=[ 300], 99.90th=[ 338], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 240, max= 458, per=24.97%, avg=343.36, stdev=41.19, samples=59 iops : min= 60, max= 114, avg=85.83, stdev=10.27, samples=59 lat (msec) : 50=0.16%, 100=0.78%, 250=96.25%, 500=2.81% cpu : usr=0.21%, sys=1.79%, ctx=2815, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=155570: Thu Jun 25 00:49:07 2026 write: IOPS=85, BW=344KiB/s (352kB/s)(10.0MiB/29771msec); 0 zone resets slat (usec): min=44, max=835, avg=255.41, stdev=105.72 clat (msec): min=44, max=375, avg=185.58, stdev=33.43 lat (msec): min=44, max=375, avg=185.83, stdev=33.43 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 133], 10.00th=[ 148], 20.00th=[ 161], | 30.00th=[ 169], 40.00th=[ 178], 50.00th=[ 186], 60.00th=[ 194], | 70.00th=[ 201], 80.00th=[ 211], 90.00th=[ 228], 95.00th=[ 239], | 99.00th=[ 271], 99.50th=[ 292], 99.90th=[ 338], 99.95th=[ 351], | 99.99th=[ 376] bw ( KiB/s): min= 232, max= 504, per=24.96%, avg=343.19, stdev=45.88, samples=59 iops : min= 58, max= 126, avg=85.80, stdev=11.47, samples=59 lat (msec) : 50=0.20%, 100=1.05%, 250=96.05%, 500=2.70% cpu : usr=0.19%, sys=1.82%, ctx=2848, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1376KiB/s (1409kB/s), 344KiB/s-344KiB/s (352kB/s-352kB/s), io=40.0MiB (41.9MB), run=29757-29771msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=155585: Thu Jun 25 00:49:31 2026 read: IOPS=62, BW=250KiB/s (256kB/s)(5048KiB/20201msec) slat (usec): min=1097, max=6323, avg=2526.93, stdev=812.61 clat (msec): min=3, max=145, avg=79.07, stdev=14.32 lat (msec): min=6, max=147, avg=81.60, stdev=14.36 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 58], 10.00th=[ 64], 20.00th=[ 69], | 30.00th=[ 73], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 83], | 70.00th=[ 86], 80.00th=[ 90], 90.00th=[ 95], 95.00th=[ 102], | 99.00th=[ 116], 99.50th=[ 120], 99.90th=[ 144], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 160, max= 360, per=25.13%, avg=249.52, stdev=48.33, samples=40 iops : min= 40, max= 90, avg=62.37, stdev=12.09, samples=40 write: IOPS=64, BW=257KiB/s (263kB/s)(5192KiB/20201msec); 0 zone resets slat (usec): min=53, max=882, avg=201.47, stdev=109.01 clat (msec): min=67, max=283, avg=168.84, stdev=36.35 lat (msec): min=67, max=283, avg=169.05, stdev=36.37 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 114], 10.00th=[ 123], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 159], 50.00th=[ 169], 60.00th=[ 180], | 70.00th=[ 190], 80.00th=[ 201], 90.00th=[ 215], 95.00th=[ 230], | 99.00th=[ 262], 99.50th=[ 266], 99.90th=[ 279], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 179, max= 336, per=24.88%, avg=257.27, stdev=31.42, samples=40 iops : min= 44, max= 84, avg=64.30, stdev= 7.90, samples=40 lat (msec) : 4=0.08%, 20=0.27%, 50=0.47%, 100=46.41%, 250=51.80% lat (msec) : 500=0.98% cpu : usr=0.26%, sys=2.40%, ctx=4925, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=155586: Thu Jun 25 00:49:31 2026 read: IOPS=61, BW=246KiB/s (252kB/s)(4976KiB/20190msec) slat (usec): min=1064, max=6336, avg=2555.80, stdev=814.04 clat (msec): min=3, max=171, avg=78.94, stdev=14.98 lat (msec): min=4, max=174, avg=81.49, stdev=15.03 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 57], 10.00th=[ 62], 20.00th=[ 68], | 30.00th=[ 73], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 83], | 70.00th=[ 86], 80.00th=[ 90], 90.00th=[ 95], 95.00th=[ 100], | 99.00th=[ 120], 99.50th=[ 131], 99.90th=[ 148], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 136, max= 352, per=24.73%, avg=245.60, stdev=51.88, samples=40 iops : min= 34, max= 88, avg=61.40, stdev=12.97, samples=40 write: IOPS=65, BW=261KiB/s (267kB/s)(5264KiB/20190msec); 0 zone resets slat (usec): min=62, max=771, avg=203.15, stdev=105.22 clat (msec): min=80, max=287, avg=167.74, stdev=36.56 lat (msec): min=80, max=287, avg=167.95, stdev=36.56 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 110], 10.00th=[ 122], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 188], 80.00th=[ 201], 90.00th=[ 215], 95.00th=[ 228], | 99.00th=[ 253], 99.50th=[ 268], 99.90th=[ 279], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 184, max= 328, per=25.23%, avg=260.92, stdev=33.07, samples=40 iops : min= 46, max= 82, avg=65.22, stdev= 8.28, samples=40 lat (msec) : 4=0.04%, 10=0.04%, 20=0.20%, 50=0.74%, 100=46.25% lat (msec) : 250=52.15%, 500=0.59% cpu : usr=0.20%, sys=2.45%, ctx=4851, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=155587: Thu Jun 25 00:49:31 2026 read: IOPS=61, BW=247KiB/s (253kB/s)(4984KiB/20201msec) slat (usec): min=1044, max=6330, avg=2546.10, stdev=815.89 clat (msec): min=2, max=145, avg=79.42, stdev=14.04 lat (msec): min=4, max=147, avg=81.97, stdev=14.10 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 58], 10.00th=[ 63], 20.00th=[ 70], | 30.00th=[ 73], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 83], | 70.00th=[ 86], 80.00th=[ 91], 90.00th=[ 97], 95.00th=[ 102], | 99.00th=[ 115], 99.50th=[ 116], 99.90th=[ 126], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 136, max= 376, per=24.91%, avg=247.40, stdev=53.54, samples=40 iops : min= 34, max= 94, avg=61.85, stdev=13.38, samples=40 write: IOPS=65, BW=260KiB/s (266kB/s)(5256KiB/20201msec); 0 zone resets slat (usec): min=48, max=790, avg=198.73, stdev=97.46 clat (msec): min=35, max=284, avg=167.43, stdev=36.70 lat (msec): min=35, max=284, avg=167.63, stdev=36.71 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 110], 10.00th=[ 123], 20.00th=[ 134], | 30.00th=[ 146], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 188], 80.00th=[ 199], 90.00th=[ 213], 95.00th=[ 230], | 99.00th=[ 257], 99.50th=[ 264], 99.90th=[ 275], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 168, max= 336, per=25.09%, avg=259.40, stdev=42.48, samples=40 iops : min= 42, max= 84, avg=64.85, stdev=10.62, samples=40 lat (msec) : 4=0.08%, 20=0.20%, 50=0.70%, 100=46.17%, 250=52.15% lat (msec) : 500=0.70% cpu : usr=0.25%, sys=2.40%, ctx=4835, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=155588: Thu Jun 25 00:49:31 2026 read: IOPS=63, BW=252KiB/s (258kB/s)(5060KiB/20050msec) slat (usec): min=96, max=6326, avg=2547.12, stdev=844.36 clat (msec): min=2, max=188, avg=78.64, stdev=16.57 lat (msec): min=6, max=191, avg=81.19, stdev=16.65 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 56], 10.00th=[ 62], 20.00th=[ 68], | 30.00th=[ 72], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 83], | 70.00th=[ 86], 80.00th=[ 90], 90.00th=[ 95], 95.00th=[ 101], | 99.00th=[ 131], 99.50th=[ 136], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 144, max= 360, per=25.57%, avg=253.87, stdev=52.01, samples=39 iops : min= 36, max= 90, avg=63.46, stdev=13.01, samples=39 write: IOPS=64, BW=258KiB/s (265kB/s)(5180KiB/20050msec); 0 zone resets slat (usec): min=61, max=709, avg=199.69, stdev=95.58 clat (msec): min=28, max=286, avg=167.98, stdev=37.25 lat (msec): min=28, max=286, avg=168.18, stdev=37.25 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 112], 10.00th=[ 123], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 159], 50.00th=[ 169], 60.00th=[ 178], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 218], 95.00th=[ 232], | 99.00th=[ 257], 99.50th=[ 271], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 184, max= 328, per=24.93%, avg=257.77, stdev=34.16, samples=39 iops : min= 46, max= 82, avg=64.44, stdev= 8.54, samples=39 lat (msec) : 4=0.04%, 10=0.16%, 20=0.43%, 50=0.98%, 100=46.41% lat (msec) : 250=51.25%, 500=0.74% cpu : usr=0.19%, sys=2.49%, ctx=4860, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=993KiB/s (1017kB/s), 246KiB/s-252KiB/s (252kB/s-258kB/s), io=19.6MiB (20.5MB), run=20050-20201msec WRITE: bw=1034KiB/s (1059kB/s), 257KiB/s-261KiB/s (263kB/s-267kB/s), io=20.4MiB (21.4MB), run=20050-20201msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=155594: Thu Jun 25 00:49:34 2026 read: IOPS=6, BW=280MiB/s (293MB/s)(40.0MiB/143msec) slat (nsec): min=15376k, max=15376k, avg=15376326.00, stdev= 0.00 clat (nsec): min=121834k, max=121834k, avg=121834434.00, stdev= 0.00 lat (nsec): min=137216k, max=137216k, avg=137215741.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 123], 10.00th=[ 123], 20.00th=[ 123], | 30.00th=[ 123], 40.00th=[ 123], 50.00th=[ 123], 60.00th=[ 123], | 70.00th=[ 123], 80.00th=[ 123], 90.00th=[ 123], 95.00th=[ 123], | 99.00th=[ 123], 99.50th=[ 123], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=9.79%, ctx=30, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=280MiB/s (293MB/s), 280MiB/s-280MiB/s (293MB/s-293MB/s), io=40.0MiB (41.9MB), run=143-143msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout