== sanity test 398c: run fio to test AIO ================= 07:33:00 (1787830380) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.486226 s, 86.3 MB/s osc.lustre-OST0000-osc-ffff8ed343456800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=151238: Thu Aug 27 07:33:44 2026 write: IOPS=75, BW=301KiB/s (308kB/s)(10.0MiB/34049msec); 0 zone resets slat (usec): min=28, max=22771, avg=239.36, stdev=1021.37 clat (msec): min=41, max=414, avg=211.70, stdev=55.85 lat (msec): min=41, max=414, avg=211.94, stdev=55.88 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 110], 10.00th=[ 136], 20.00th=[ 169], | 30.00th=[ 186], 40.00th=[ 199], 50.00th=[ 211], 60.00th=[ 228], | 70.00th=[ 239], 80.00th=[ 255], 90.00th=[ 284], 95.00th=[ 305], | 99.00th=[ 347], 99.50th=[ 376], 99.90th=[ 380], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 168, max= 629, per=24.93%, avg=299.69, stdev=79.06, samples=67 iops : min= 42, max= 157, avg=74.49, stdev=19.72, samples=67 lat (msec) : 50=0.04%, 100=3.55%, 250=73.52%, 500=22.89% cpu : usr=0.14%, sys=0.88%, ctx=2659, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151239: Thu Aug 27 07:33:44 2026 write: IOPS=75, BW=302KiB/s (309kB/s)(10.0MiB/33891msec); 0 zone resets slat (usec): min=28, max=15208, avg=212.40, stdev=866.15 clat (msec): min=77, max=399, avg=211.39, stdev=57.04 lat (msec): min=77, max=399, avg=211.60, stdev=57.07 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 106], 10.00th=[ 133], 20.00th=[ 169], | 30.00th=[ 186], 40.00th=[ 201], 50.00th=[ 213], 60.00th=[ 226], | 70.00th=[ 241], 80.00th=[ 257], 90.00th=[ 279], 95.00th=[ 309], | 99.00th=[ 347], 99.50th=[ 363], 99.90th=[ 380], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 134, max= 637, per=25.06%, avg=301.21, stdev=84.33, samples=67 iops : min= 33, max= 159, avg=74.85, stdev=21.09, samples=67 lat (msec) : 100=3.05%, 250=71.80%, 500=25.16% cpu : usr=0.12%, sys=0.66%, ctx=2212, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151240: Thu Aug 27 07:33:44 2026 write: IOPS=75, BW=301KiB/s (308kB/s)(10.0MiB/34055msec); 0 zone resets slat (usec): min=29, max=15992, avg=196.76, stdev=805.20 clat (msec): min=22, max=399, avg=211.65, stdev=54.87 lat (msec): min=24, max=399, avg=211.85, stdev=54.89 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 111], 10.00th=[ 142], 20.00th=[ 167], | 30.00th=[ 184], 40.00th=[ 201], 50.00th=[ 213], 60.00th=[ 226], | 70.00th=[ 239], 80.00th=[ 257], 90.00th=[ 275], 95.00th=[ 296], | 99.00th=[ 351], 99.50th=[ 368], 99.90th=[ 380], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 152, max= 629, per=24.86%, avg=298.79, stdev=72.19, samples=67 iops : min= 38, max= 157, avg=74.25, stdev=18.08, samples=67 lat (msec) : 50=0.08%, 100=1.80%, 250=74.14%, 500=23.98% cpu : usr=0.11%, sys=0.71%, ctx=2374, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151241: Thu Aug 27 07:33:44 2026 write: IOPS=75, BW=302KiB/s (309kB/s)(10.0MiB/33928msec); 0 zone resets slat (usec): min=29, max=21823, avg=251.71, stdev=1092.34 clat (msec): min=19, max=403, avg=211.20, stdev=57.55 lat (msec): min=19, max=403, avg=211.46, stdev=57.53 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 107], 10.00th=[ 132], 20.00th=[ 171], | 30.00th=[ 186], 40.00th=[ 199], 50.00th=[ 213], 60.00th=[ 226], | 70.00th=[ 241], 80.00th=[ 257], 90.00th=[ 279], 95.00th=[ 296], | 99.00th=[ 363], 99.50th=[ 380], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 156, max= 606, per=25.02%, avg=300.69, stdev=74.80, samples=67 iops : min= 39, max= 151, avg=74.73, stdev=18.66, samples=67 lat (msec) : 20=0.04%, 50=0.43%, 100=3.32%, 250=71.68%, 500=24.53% cpu : usr=0.19%, sys=0.85%, ctx=2706, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1203KiB/s (1232kB/s), 301KiB/s-302KiB/s (308kB/s-309kB/s), io=40.0MiB (41.9MB), run=33891-34055msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=151256: Thu Aug 27 07:34:32 2026 read: IOPS=31, BW=126KiB/s (129kB/s)(5048KiB/40145msec) slat (usec): min=2964, max=97086, avg=17960.14, stdev=10934.59 clat (msec): min=71, max=558, avg=237.94, stdev=68.87 lat (msec): min=90, max=583, avg=255.90, stdev=71.81 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 144], 10.00th=[ 161], 20.00th=[ 184], | 30.00th=[ 201], 40.00th=[ 211], 50.00th=[ 226], 60.00th=[ 245], | 70.00th=[ 266], 80.00th=[ 292], 90.00th=[ 330], 95.00th=[ 368], | 99.00th=[ 451], 99.50th=[ 472], 99.90th=[ 518], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 15, max= 223, per=25.04%, avg=124.94, stdev=37.69, samples=79 iops : min= 3, max= 55, avg=30.67, stdev= 9.42, samples=79 write: IOPS=32, BW=129KiB/s (132kB/s)(5192KiB/40145msec); 0 zone resets slat (usec): min=34, max=16642, avg=282.20, stdev=940.57 clat (msec): min=25, max=624, avg=243.47, stdev=69.56 lat (msec): min=25, max=624, avg=243.76, stdev=69.64 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 146], 10.00th=[ 163], 20.00th=[ 188], | 30.00th=[ 207], 40.00th=[ 222], 50.00th=[ 234], 60.00th=[ 249], | 70.00th=[ 271], 80.00th=[ 296], 90.00th=[ 338], 95.00th=[ 376], | 99.00th=[ 447], 99.50th=[ 468], 99.90th=[ 584], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 55, max= 231, per=24.79%, avg=128.89, stdev=38.43, samples=79 iops : min= 13, max= 57, avg=31.68, stdev= 9.64, samples=79 lat (msec) : 50=0.04%, 100=0.27%, 250=61.17%, 500=38.24%, 750=0.27% cpu : usr=0.28%, sys=1.61%, ctx=4381, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151257: Thu Aug 27 07:34:32 2026 read: IOPS=31, BW=124KiB/s (127kB/s)(4976KiB/40115msec) slat (msec): min=3, max=160, avg=17.32, stdev=10.69 clat (msec): min=30, max=484, avg=235.77, stdev=64.97 lat (msec): min=55, max=499, avg=253.10, stdev=67.47 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 146], 10.00th=[ 163], 20.00th=[ 182], | 30.00th=[ 199], 40.00th=[ 211], 50.00th=[ 226], 60.00th=[ 245], | 70.00th=[ 264], 80.00th=[ 288], 90.00th=[ 330], 95.00th=[ 363], | 99.00th=[ 405], 99.50th=[ 418], 99.90th=[ 472], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 39, max= 224, per=24.70%, avg=123.25, stdev=39.51, samples=79 iops : min= 9, max= 56, avg=30.27, stdev= 9.92, samples=79 write: IOPS=32, BW=131KiB/s (134kB/s)(5264KiB/40115msec); 0 zone resets slat (usec): min=33, max=18832, avg=280.91, stdev=996.09 clat (msec): min=97, max=488, avg=246.01, stdev=64.97 lat (msec): min=97, max=488, avg=246.29, stdev=65.00 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 150], 10.00th=[ 171], 20.00th=[ 192], | 30.00th=[ 209], 40.00th=[ 224], 50.00th=[ 239], 60.00th=[ 255], | 70.00th=[ 275], 80.00th=[ 296], 90.00th=[ 338], 95.00th=[ 368], | 99.00th=[ 430], 99.50th=[ 447], 99.90th=[ 477], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 63, max= 224, per=25.19%, avg=130.99, stdev=37.23, samples=79 iops : min= 15, max= 56, avg=32.20, stdev= 9.36, samples=79 lat (msec) : 50=0.04%, 100=0.23%, 250=60.43%, 500=39.30% cpu : usr=0.21%, sys=1.43%, ctx=4154, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151258: Thu Aug 27 07:34:32 2026 read: IOPS=31, BW=124KiB/s (127kB/s)(4984KiB/40048msec) slat (usec): min=2991, max=75321, avg=17384.74, stdev=10388.59 clat (msec): min=29, max=482, avg=237.06, stdev=66.36 lat (msec): min=44, max=499, avg=254.45, stdev=68.78 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 140], 10.00th=[ 161], 20.00th=[ 182], | 30.00th=[ 199], 40.00th=[ 215], 50.00th=[ 228], 60.00th=[ 243], | 70.00th=[ 264], 80.00th=[ 292], 90.00th=[ 330], 95.00th=[ 355], | 99.00th=[ 409], 99.50th=[ 447], 99.90th=[ 481], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 63, max= 223, per=24.95%, avg=124.52, stdev=33.87, samples=79 iops : min= 15, max= 55, avg=30.58, stdev= 8.53, samples=79 write: IOPS=32, BW=131KiB/s (134kB/s)(5256KiB/40048msec); 0 zone resets slat (usec): min=31, max=22991, avg=274.96, stdev=1044.76 clat (msec): min=84, max=460, avg=243.92, stdev=65.34 lat (msec): min=84, max=460, avg=244.20, stdev=65.36 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 150], 10.00th=[ 167], 20.00th=[ 188], | 30.00th=[ 205], 40.00th=[ 220], 50.00th=[ 236], 60.00th=[ 253], | 70.00th=[ 271], 80.00th=[ 296], 90.00th=[ 338], 95.00th=[ 372], | 99.00th=[ 418], 99.50th=[ 422], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 31, max= 224, per=25.13%, avg=130.68, stdev=41.89, samples=79 iops : min= 7, max= 56, avg=32.11, stdev=10.48, samples=79 lat (msec) : 50=0.04%, 100=0.23%, 250=60.66%, 500=39.06% cpu : usr=0.22%, sys=1.53%, ctx=4157, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151259: Thu Aug 27 07:34:32 2026 read: IOPS=31, BW=126KiB/s (129kB/s)(5060KiB/40082msec) slat (usec): min=49, max=99315, avg=17920.21, stdev=11074.58 clat (msec): min=58, max=485, avg=238.87, stdev=69.08 lat (msec): min=58, max=514, avg=256.80, stdev=71.96 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 140], 10.00th=[ 161], 20.00th=[ 182], | 30.00th=[ 199], 40.00th=[ 213], 50.00th=[ 228], 60.00th=[ 247], | 70.00th=[ 271], 80.00th=[ 300], 90.00th=[ 334], 95.00th=[ 368], | 99.00th=[ 426], 99.50th=[ 439], 99.90th=[ 472], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 56, max= 239, per=25.24%, avg=125.95, stdev=34.35, samples=79 iops : min= 14, max= 59, avg=30.96, stdev= 8.59, samples=79 write: IOPS=32, BW=129KiB/s (132kB/s)(5180KiB/40082msec); 0 zone resets slat (usec): min=33, max=11885, avg=253.82, stdev=730.10 clat (msec): min=57, max=481, avg=243.70, stdev=67.05 lat (msec): min=57, max=481, avg=243.96, stdev=67.07 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 140], 10.00th=[ 161], 20.00th=[ 190], | 30.00th=[ 205], 40.00th=[ 222], 50.00th=[ 236], 60.00th=[ 253], | 70.00th=[ 279], 80.00th=[ 300], 90.00th=[ 334], 95.00th=[ 359], | 99.00th=[ 430], 99.50th=[ 456], 99.90th=[ 472], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 56, max= 232, per=24.75%, avg=128.70, stdev=39.40, samples=79 iops : min= 14, max= 58, avg=31.62, stdev= 9.92, samples=79 lat (msec) : 100=0.47%, 250=59.88%, 500=39.65% cpu : usr=0.25%, sys=1.59%, ctx=4170, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=500KiB/s (512kB/s), 124KiB/s-126KiB/s (127kB/s-129kB/s), io=19.6MiB (20.5MB), run=40048-40145msec WRITE: bw=520KiB/s (533kB/s), 129KiB/s-131KiB/s (132kB/s-134kB/s), io=20.4MiB (21.4MB), run=40048-40145msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=151270: Thu Aug 27 07:34:41 2026 read: IOPS=2, BW=85.7MiB/s (89.8MB/s)(40.0MiB/467msec) slat (nsec): min=37742k, max=37742k, avg=37741698.00, stdev= 0.00 clat (nsec): min=410268k, max=410268k, avg=410267503.00, stdev= 0.00 lat (nsec): min=448014k, max=448014k, avg=448014185.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 409], 10.00th=[ 409], 20.00th=[ 409], | 30.00th=[ 409], 40.00th=[ 409], 50.00th=[ 409], 60.00th=[ 409], | 70.00th=[ 409], 80.00th=[ 409], 90.00th=[ 409], 95.00th=[ 409], | 99.00th=[ 409], 99.50th=[ 409], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.58%, ctx=14, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=85.7MiB/s (89.8MB/s), 85.7MiB/s-85.7MiB/s (89.8MB/s-89.8MB/s), io=40.0MiB (41.9MB), run=467-467msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout