== sanity test 398c: run fio to test AIO ================= 23:11:01 (1787800261) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.525752 s, 79.8 MB/s osc.lustre-OST0000-osc-ffff8bac43565800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180673: Wed Aug 26 23:11:44 2026 write: IOPS=73, BW=295KiB/s (302kB/s)(10.0MiB/34667msec); 0 zone resets slat (usec): min=30, max=23217, avg=310.32, stdev=1288.78 clat (msec): min=15, max=458, avg=216.16, stdev=58.66 lat (msec): min=15, max=458, avg=216.47, stdev=58.70 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 136], 10.00th=[ 150], 20.00th=[ 167], | 30.00th=[ 184], 40.00th=[ 197], 50.00th=[ 211], 60.00th=[ 226], | 70.00th=[ 243], 80.00th=[ 262], 90.00th=[ 288], 95.00th=[ 321], | 99.00th=[ 393], 99.50th=[ 409], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 167, max= 486, per=25.01%, avg=295.16, stdev=66.87, samples=68 iops : min= 41, max= 121, avg=73.32, stdev=16.74, samples=68 lat (msec) : 20=0.08%, 50=0.12%, 100=0.98%, 250=72.93%, 500=25.90% cpu : usr=0.19%, sys=0.98%, ctx=2906, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180674: Wed Aug 26 23:11:44 2026 write: IOPS=73, BW=296KiB/s (303kB/s)(10.0MiB/34610msec); 0 zone resets slat (usec): min=30, max=17797, avg=278.26, stdev=1058.31 clat (msec): min=33, max=417, avg=215.45, stdev=59.07 lat (msec): min=34, max=417, avg=215.74, stdev=59.09 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 133], 10.00th=[ 148], 20.00th=[ 167], | 30.00th=[ 184], 40.00th=[ 197], 50.00th=[ 211], 60.00th=[ 226], | 70.00th=[ 243], 80.00th=[ 264], 90.00th=[ 288], 95.00th=[ 321], | 99.00th=[ 384], 99.50th=[ 397], 99.90th=[ 418], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 167, max= 508, per=25.01%, avg=295.10, stdev=70.03, samples=68 iops : min= 41, max= 127, avg=73.31, stdev=17.58, samples=68 lat (msec) : 50=0.08%, 100=1.88%, 250=72.34%, 500=25.70% cpu : usr=0.15%, sys=0.93%, ctx=2667, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180675: Wed Aug 26 23:11:44 2026 write: IOPS=73, BW=296KiB/s (303kB/s)(10.0MiB/34651msec); 0 zone resets slat (usec): min=23, max=26065, avg=263.68, stdev=1082.67 clat (msec): min=46, max=416, avg=215.16, stdev=57.61 lat (msec): min=46, max=416, avg=215.43, stdev=57.57 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 133], 10.00th=[ 148], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 197], 50.00th=[ 209], 60.00th=[ 228], | 70.00th=[ 245], 80.00th=[ 259], 90.00th=[ 292], 95.00th=[ 317], | 99.00th=[ 368], 99.50th=[ 409], 99.90th=[ 418], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 150, max= 429, per=24.91%, avg=293.96, stdev=64.10, samples=68 iops : min= 37, max= 107, avg=73.04, stdev=16.05, samples=68 lat (msec) : 50=0.04%, 100=0.86%, 250=72.93%, 500=26.17% cpu : usr=0.18%, sys=0.85%, ctx=1888, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180676: Wed Aug 26 23:11:44 2026 write: IOPS=73, BW=295KiB/s (302kB/s)(10.0MiB/34705msec); 0 zone resets slat (usec): min=28, max=18661, avg=233.66, stdev=889.42 clat (msec): min=33, max=415, avg=215.70, stdev=57.66 lat (msec): min=33, max=415, avg=215.94, stdev=57.68 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 136], 10.00th=[ 150], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 197], 50.00th=[ 207], 60.00th=[ 230], | 70.00th=[ 247], 80.00th=[ 262], 90.00th=[ 288], 95.00th=[ 317], | 99.00th=[ 384], 99.50th=[ 388], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 126, max= 463, per=24.84%, avg=293.15, stdev=68.34, samples=68 iops : min= 31, max= 115, avg=72.84, stdev=17.08, samples=68 lat (msec) : 50=0.12%, 100=1.02%, 250=70.62%, 500=28.24% cpu : usr=0.13%, sys=0.78%, ctx=1881, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1180KiB/s (1209kB/s), 295KiB/s-296KiB/s (302kB/s-303kB/s), io=40.0MiB (41.9MB), run=34610-34705msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180691: Wed Aug 26 23:12:32 2026 read: IOPS=32, BW=132KiB/s (135kB/s)(5048KiB/38379msec) slat (usec): min=3109, max=94526, avg=16969.88, stdev=9858.54 clat (msec): min=18, max=448, avg=225.79, stdev=53.59 lat (msec): min=37, max=503, avg=242.77, stdev=56.00 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 153], 10.00th=[ 165], 20.00th=[ 180], | 30.00th=[ 194], 40.00th=[ 207], 50.00th=[ 222], 60.00th=[ 234], | 70.00th=[ 249], 80.00th=[ 268], 90.00th=[ 296], 95.00th=[ 321], | 99.00th=[ 376], 99.50th=[ 397], 99.90th=[ 435], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 55, max= 223, per=25.05%, avg=130.49, stdev=30.58, samples=76 iops : min= 13, max= 55, avg=32.13, stdev= 7.68, samples=76 write: IOPS=33, BW=135KiB/s (139kB/s)(5192KiB/38379msec); 0 zone resets slat (usec): min=36, max=40190, avg=363.72, stdev=1577.06 clat (msec): min=84, max=453, avg=234.70, stdev=52.30 lat (msec): min=85, max=453, avg=235.07, stdev=52.33 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 163], 10.00th=[ 174], 20.00th=[ 188], | 30.00th=[ 203], 40.00th=[ 215], 50.00th=[ 230], 60.00th=[ 243], | 70.00th=[ 257], 80.00th=[ 279], 90.00th=[ 305], 95.00th=[ 330], | 99.00th=[ 388], 99.50th=[ 409], 99.90th=[ 451], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 44, max= 207, per=24.84%, avg=134.64, stdev=34.00, samples=76 iops : min= 11, max= 51, avg=33.17, stdev= 8.51, samples=76 lat (msec) : 20=0.04%, 50=0.04%, 100=0.20%, 250=67.85%, 500=31.88% cpu : usr=0.25%, sys=1.83%, ctx=4196, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180692: Wed Aug 26 23:12:32 2026 read: IOPS=32, BW=130KiB/s (133kB/s)(4976KiB/38319msec) slat (usec): min=3037, max=67409, avg=17208.38, stdev=9988.33 clat (msec): min=36, max=470, avg=228.29, stdev=54.63 lat (msec): min=44, max=489, avg=245.51, stdev=56.86 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 153], 10.00th=[ 165], 20.00th=[ 184], | 30.00th=[ 197], 40.00th=[ 209], 50.00th=[ 224], 60.00th=[ 236], | 70.00th=[ 251], 80.00th=[ 271], 90.00th=[ 305], 95.00th=[ 330], | 99.00th=[ 376], 99.50th=[ 405], 99.90th=[ 468], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 59, max= 207, per=24.70%, avg=128.68, stdev=31.17, samples=75 iops : min= 14, max= 51, avg=31.65, stdev= 7.82, samples=75 write: IOPS=34, BW=137KiB/s (141kB/s)(5264KiB/38319msec); 0 zone resets slat (usec): min=30, max=18330, avg=332.68, stdev=1145.20 clat (msec): min=36, max=444, avg=231.71, stdev=52.62 lat (msec): min=36, max=444, avg=232.04, stdev=52.59 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 161], 10.00th=[ 169], 20.00th=[ 188], | 30.00th=[ 201], 40.00th=[ 211], 50.00th=[ 226], 60.00th=[ 241], | 70.00th=[ 255], 80.00th=[ 275], 90.00th=[ 305], 95.00th=[ 330], | 99.00th=[ 368], 99.50th=[ 376], 99.90th=[ 401], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 71, max= 223, per=25.25%, avg=136.84, stdev=34.27, samples=75 iops : min= 17, max= 55, avg=33.71, stdev= 8.58, samples=75 lat (msec) : 50=0.08%, 100=0.23%, 250=67.97%, 500=31.72% cpu : usr=0.28%, sys=1.94%, ctx=4314, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180693: Wed Aug 26 23:12:32 2026 read: IOPS=32, BW=131KiB/s (134kB/s)(4984KiB/38007msec) slat (usec): min=52, max=69964, avg=17123.46, stdev=9830.53 clat (msec): min=15, max=442, avg=227.27, stdev=56.91 lat (msec): min=19, max=463, avg=244.41, stdev=59.29 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 148], 10.00th=[ 161], 20.00th=[ 182], | 30.00th=[ 194], 40.00th=[ 209], 50.00th=[ 222], 60.00th=[ 239], | 70.00th=[ 253], 80.00th=[ 275], 90.00th=[ 300], 95.00th=[ 330], | 99.00th=[ 376], 99.50th=[ 401], 99.90th=[ 430], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 71, max= 198, per=25.08%, avg=130.67, stdev=31.12, samples=75 iops : min= 17, max= 49, avg=32.17, stdev= 7.74, samples=75 write: IOPS=34, BW=138KiB/s (142kB/s)(5256KiB/38007msec); 0 zone resets slat (usec): min=33, max=22072, avg=327.59, stdev=1266.68 clat (msec): min=46, max=438, avg=230.10, stdev=50.92 lat (msec): min=46, max=438, avg=230.43, stdev=50.88 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 159], 10.00th=[ 171], 20.00th=[ 186], | 30.00th=[ 201], 40.00th=[ 213], 50.00th=[ 226], 60.00th=[ 241], | 70.00th=[ 259], 80.00th=[ 275], 90.00th=[ 300], 95.00th=[ 317], | 99.00th=[ 355], 99.50th=[ 359], 99.90th=[ 435], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 47, max= 207, per=25.36%, avg=137.43, stdev=36.43, samples=75 iops : min= 11, max= 51, avg=33.84, stdev= 9.17, samples=75 lat (msec) : 20=0.12%, 50=0.16%, 100=0.55%, 250=66.13%, 500=33.05% cpu : usr=0.24%, sys=1.79%, ctx=4334, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180694: Wed Aug 26 23:12:32 2026 read: IOPS=32, BW=131KiB/s (135kB/s)(5060KiB/38517msec) slat (usec): min=58, max=354010, avg=17296.34, stdev=15412.93 clat (msec): min=16, max=787, avg=223.97, stdev=61.46 lat (msec): min=30, max=801, avg=241.28, stdev=64.49 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 146], 10.00th=[ 161], 20.00th=[ 180], | 30.00th=[ 192], 40.00th=[ 207], 50.00th=[ 220], 60.00th=[ 232], | 70.00th=[ 245], 80.00th=[ 268], 90.00th=[ 296], 95.00th=[ 317], | 99.00th=[ 380], 99.50th=[ 426], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 31, max= 223, per=24.93%, avg=129.88, stdev=36.76, samples=76 iops : min= 7, max= 55, avg=31.95, stdev= 9.20, samples=76 write: IOPS=33, BW=134KiB/s (138kB/s)(5180KiB/38517msec); 0 zone resets slat (usec): min=31, max=11669, avg=267.43, stdev=737.28 clat (msec): min=54, max=824, avg=237.90, stdev=75.88 lat (msec): min=54, max=825, avg=238.17, stdev=75.87 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 157], 10.00th=[ 171], 20.00th=[ 190], | 30.00th=[ 201], 40.00th=[ 213], 50.00th=[ 228], 60.00th=[ 243], | 70.00th=[ 257], 80.00th=[ 279], 90.00th=[ 309], 95.00th=[ 338], | 99.00th=[ 439], 99.50th=[ 802], 99.90th=[ 818], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 51, max= 216, per=24.49%, avg=132.74, stdev=33.86, samples=76 iops : min= 12, max= 54, avg=32.68, stdev= 8.53, samples=76 lat (msec) : 20=0.04%, 50=0.08%, 100=1.09%, 250=68.16%, 500=30.04% lat (msec) : 750=0.12%, 1000=0.47% cpu : usr=0.30%, sys=1.82%, ctx=4294, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=521KiB/s (534kB/s), 130KiB/s-132KiB/s (133kB/s-135kB/s), io=19.6MiB (20.5MB), run=38007-38517msec WRITE: bw=542KiB/s (555kB/s), 134KiB/s-138KiB/s (138kB/s-142kB/s), io=20.4MiB (21.4MB), run=38007-38517msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180705: Wed Aug 26 23:12:44 2026 read: IOPS=1, BW=64.1MiB/s (67.2MB/s)(40.0MiB/624msec) slat (nsec): min=73454k, max=73454k, avg=73453781.00, stdev= 0.00 clat (nsec): min=522578k, max=522578k, avg=522577825.00, stdev= 0.00 lat (nsec): min=596038k, max=596038k, avg=596038227.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 523], 5.00th=[ 523], 10.00th=[ 523], 20.00th=[ 523], | 30.00th=[ 523], 40.00th=[ 523], 50.00th=[ 523], 60.00th=[ 523], | 70.00th=[ 523], 80.00th=[ 523], 90.00th=[ 523], 95.00th=[ 523], | 99.00th=[ 523], 99.50th=[ 523], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 523] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.61%, ctx=5, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=64.1MiB/s (67.2MB/s), 64.1MiB/s-64.1MiB/s (67.2MB/s-67.2MB/s), io=40.0MiB (41.9MB), run=624-624msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout