== sanity test 398c: run fio to test AIO ================= 22:45:17 (1786675517) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.544429 s, 77.0 MB/s osc.lustre-OST0000-osc-ffff8fc14899b800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179654: Thu Aug 13 22:46:02 2026 write: IOPS=75, BW=301KiB/s (308kB/s)(10.0MiB/34031msec); 0 zone resets slat (usec): min=50, max=13584, avg=355.90, stdev=907.68 clat (msec): min=38, max=379, avg=211.24, stdev=45.67 lat (msec): min=39, max=379, avg=211.60, stdev=45.66 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 142], 10.00th=[ 155], 20.00th=[ 174], | 30.00th=[ 186], 40.00th=[ 199], 50.00th=[ 211], 60.00th=[ 220], | 70.00th=[ 232], 80.00th=[ 247], 90.00th=[ 279], 95.00th=[ 292], | 99.00th=[ 326], 99.50th=[ 363], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 167, max= 499, per=24.86%, avg=299.10, stdev=61.50, samples=67 iops : min= 41, max= 124, avg=74.39, stdev=15.41, samples=67 lat (msec) : 50=0.04%, 100=0.04%, 250=81.45%, 500=18.48% cpu : usr=0.13%, sys=1.01%, ctx=2279, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179655: Thu Aug 13 22:46:02 2026 write: IOPS=76, BW=305KiB/s (313kB/s)(10.0MiB/33541msec); 0 zone resets slat (usec): min=52, max=32766, avg=416.48, stdev=1236.62 clat (msec): min=30, max=458, avg=209.00, stdev=50.72 lat (msec): min=30, max=458, avg=209.42, stdev=50.72 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 131], 10.00th=[ 150], 20.00th=[ 169], | 30.00th=[ 186], 40.00th=[ 194], 50.00th=[ 205], 60.00th=[ 218], | 70.00th=[ 234], 80.00th=[ 249], 90.00th=[ 271], 95.00th=[ 296], | 99.00th=[ 347], 99.50th=[ 363], 99.90th=[ 393], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 183, max= 556, per=25.34%, avg=304.85, stdev=64.03, samples=66 iops : min= 45, max= 139, avg=75.85, stdev=16.03, samples=66 lat (msec) : 50=0.20%, 100=1.91%, 250=79.02%, 500=18.87% cpu : usr=0.13%, sys=1.31%, ctx=2648, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179656: Thu Aug 13 22:46:02 2026 write: IOPS=75, BW=302KiB/s (310kB/s)(10.0MiB/33879msec); 0 zone resets slat (usec): min=48, max=14678, avg=392.57, stdev=989.46 clat (msec): min=21, max=380, avg=210.54, stdev=48.13 lat (msec): min=21, max=380, avg=210.94, stdev=48.08 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 142], 10.00th=[ 157], 20.00th=[ 174], | 30.00th=[ 186], 40.00th=[ 197], 50.00th=[ 207], 60.00th=[ 218], | 70.00th=[ 230], 80.00th=[ 251], 90.00th=[ 275], 95.00th=[ 296], | 99.00th=[ 334], 99.50th=[ 342], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 380] bw ( KiB/s): min= 176, max= 491, per=25.02%, avg=301.04, stdev=56.31, samples=67 iops : min= 44, max= 122, avg=74.88, stdev=14.08, samples=67 lat (msec) : 50=0.35%, 100=0.70%, 250=79.61%, 500=19.34% cpu : usr=0.21%, sys=1.25%, ctx=2465, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179657: Thu Aug 13 22:46:02 2026 write: IOPS=75, BW=302KiB/s (309kB/s)(10.0MiB/33923msec); 0 zone resets slat (usec): min=54, max=24699, avg=396.31, stdev=1068.45 clat (msec): min=30, max=379, avg=210.62, stdev=47.31 lat (msec): min=31, max=379, avg=211.02, stdev=47.30 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 144], 10.00th=[ 159], 20.00th=[ 174], | 30.00th=[ 186], 40.00th=[ 197], 50.00th=[ 209], 60.00th=[ 220], | 70.00th=[ 232], 80.00th=[ 247], 90.00th=[ 275], 95.00th=[ 292], | 99.00th=[ 338], 99.50th=[ 372], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 191, max= 491, per=25.03%, avg=301.13, stdev=58.35, samples=67 iops : min= 47, max= 122, avg=74.91, stdev=14.61, samples=67 lat (msec) : 50=0.16%, 100=0.62%, 250=80.82%, 500=18.40% cpu : usr=0.19%, sys=1.12%, ctx=2717, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1204KiB/s (1232kB/s), 301KiB/s-305KiB/s (308kB/s-313kB/s), io=40.0MiB (41.9MB), run=33541-34031msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179672: Thu Aug 13 22:46:50 2026 read: IOPS=33, BW=132KiB/s (136kB/s)(5048KiB/38109msec) slat (usec): min=60, max=74989, avg=16658.33, stdev=8955.04 clat (msec): min=90, max=431, avg=227.90, stdev=51.39 lat (msec): min=90, max=457, avg=244.57, stdev=53.21 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 157], 10.00th=[ 169], 20.00th=[ 184], | 30.00th=[ 199], 40.00th=[ 213], 50.00th=[ 224], 60.00th=[ 232], | 70.00th=[ 249], 80.00th=[ 271], 90.00th=[ 296], 95.00th=[ 317], | 99.00th=[ 376], 99.50th=[ 388], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 71, max= 247, per=25.22%, avg=131.65, stdev=30.98, samples=75 iops : min= 17, max= 61, avg=32.36, stdev= 7.83, samples=75 write: IOPS=34, BW=136KiB/s (140kB/s)(5192KiB/38109msec); 0 zone resets slat (usec): min=54, max=26792, avg=446.82, stdev=1338.34 clat (msec): min=26, max=417, avg=230.67, stdev=48.33 lat (msec): min=26, max=419, avg=231.12, stdev=48.40 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 159], 10.00th=[ 176], 20.00th=[ 192], | 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 226], 60.00th=[ 239], | 70.00th=[ 251], 80.00th=[ 268], 90.00th=[ 292], 95.00th=[ 317], | 99.00th=[ 368], 99.50th=[ 388], 99.90th=[ 409], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 56, max= 231, per=24.98%, avg=135.91, stdev=31.31, samples=75 iops : min= 14, max= 57, avg=33.41, stdev= 7.77, samples=75 lat (msec) : 50=0.04%, 100=0.20%, 250=69.57%, 500=30.20% cpu : usr=0.29%, sys=1.78%, ctx=4334, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179673: Thu Aug 13 22:46:50 2026 read: IOPS=32, BW=130KiB/s (133kB/s)(4976KiB/38339msec) slat (usec): min=89, max=113333, avg=16832.34, stdev=8993.10 clat (msec): min=20, max=496, avg=228.57, stdev=51.01 lat (msec): min=27, max=528, avg=245.41, stdev=52.81 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 159], 10.00th=[ 169], 20.00th=[ 186], | 30.00th=[ 199], 40.00th=[ 211], 50.00th=[ 224], 60.00th=[ 239], | 70.00th=[ 251], 80.00th=[ 271], 90.00th=[ 296], 95.00th=[ 317], | 99.00th=[ 372], 99.50th=[ 388], 99.90th=[ 472], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 63, max= 183, per=24.66%, avg=128.74, stdev=26.99, samples=76 iops : min= 15, max= 45, avg=31.63, stdev= 6.79, samples=76 write: IOPS=34, BW=137KiB/s (141kB/s)(5264KiB/38339msec); 0 zone resets slat (usec): min=58, max=18398, avg=426.01, stdev=1228.32 clat (msec): min=38, max=461, avg=231.63, stdev=47.03 lat (msec): min=38, max=461, avg=232.05, stdev=47.10 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 163], 10.00th=[ 178], 20.00th=[ 197], | 30.00th=[ 207], 40.00th=[ 218], 50.00th=[ 228], 60.00th=[ 241], | 70.00th=[ 253], 80.00th=[ 271], 90.00th=[ 292], 95.00th=[ 317], | 99.00th=[ 342], 99.50th=[ 384], 99.90th=[ 451], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 79, max= 247, per=25.15%, avg=136.84, stdev=32.64, samples=76 iops : min= 19, max= 61, avg=33.67, stdev= 8.14, samples=76 lat (msec) : 50=0.12%, 100=0.20%, 250=67.93%, 500=31.76% cpu : usr=0.28%, sys=2.03%, ctx=4418, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179674: Thu Aug 13 22:46:50 2026 read: IOPS=32, BW=130KiB/s (133kB/s)(4984KiB/38393msec) slat (usec): min=60, max=56285, avg=16972.30, stdev=8702.50 clat (msec): min=49, max=404, avg=228.37, stdev=49.67 lat (msec): min=68, max=438, avg=245.35, stdev=51.08 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 155], 10.00th=[ 167], 20.00th=[ 184], | 30.00th=[ 199], 40.00th=[ 213], 50.00th=[ 226], 60.00th=[ 239], | 70.00th=[ 253], 80.00th=[ 271], 90.00th=[ 296], 95.00th=[ 317], | 99.00th=[ 355], 99.50th=[ 372], 99.90th=[ 393], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 55, max= 199, per=24.80%, avg=129.43, stdev=29.64, samples=76 iops : min= 13, max= 49, avg=31.79, stdev= 7.41, samples=76 write: IOPS=34, BW=137KiB/s (140kB/s)(5256KiB/38393msec); 0 zone resets slat (usec): min=53, max=22638, avg=401.64, stdev=1215.84 clat (msec): min=120, max=391, avg=232.41, stdev=46.03 lat (msec): min=120, max=391, avg=232.81, stdev=46.05 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 163], 10.00th=[ 176], 20.00th=[ 192], | 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 228], 60.00th=[ 241], | 70.00th=[ 255], 80.00th=[ 275], 90.00th=[ 296], 95.00th=[ 309], | 99.00th=[ 347], 99.50th=[ 372], 99.90th=[ 388], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 64, max= 207, per=24.98%, avg=135.88, stdev=34.45, samples=76 iops : min= 16, max= 51, avg=33.42, stdev= 8.64, samples=76 lat (msec) : 50=0.04%, 250=67.42%, 500=32.54% cpu : usr=0.23%, sys=1.95%, ctx=4407, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179675: Thu Aug 13 22:46:50 2026 read: IOPS=32, BW=132KiB/s (135kB/s)(5060KiB/38399msec) slat (usec): min=2927, max=73419, avg=17016.61, stdev=9174.96 clat (msec): min=108, max=404, avg=226.64, stdev=50.01 lat (msec): min=125, max=430, avg=243.67, stdev=51.64 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 157], 10.00th=[ 167], 20.00th=[ 184], | 30.00th=[ 197], 40.00th=[ 209], 50.00th=[ 220], 60.00th=[ 234], | 70.00th=[ 249], 80.00th=[ 271], 90.00th=[ 296], 95.00th=[ 317], | 99.00th=[ 359], 99.50th=[ 372], 99.90th=[ 393], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 55, max= 223, per=25.13%, avg=131.18, stdev=29.03, samples=76 iops : min= 13, max= 55, avg=32.24, stdev= 7.28, samples=76 write: IOPS=33, BW=135KiB/s (138kB/s)(5180KiB/38399msec); 0 zone resets slat (usec): min=56, max=36055, avg=464.06, stdev=1568.17 clat (msec): min=129, max=400, avg=233.02, stdev=46.60 lat (msec): min=130, max=400, avg=233.48, stdev=46.56 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 165], 10.00th=[ 176], 20.00th=[ 192], | 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 230], 60.00th=[ 243], | 70.00th=[ 255], 80.00th=[ 271], 90.00th=[ 296], 95.00th=[ 313], | 99.00th=[ 368], 99.50th=[ 380], 99.90th=[ 397], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 71, max= 192, per=24.62%, avg=133.93, stdev=33.61, samples=76 iops : min= 17, max= 48, avg=32.92, stdev= 8.44, samples=76 lat (msec) : 250=68.71%, 500=31.29% cpu : usr=0.24%, sys=2.07%, ctx=4459, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=523KiB/s (535kB/s), 130KiB/s-132KiB/s (133kB/s-136kB/s), io=19.6MiB (20.5MB), run=38109-38399msec WRITE: bw=544KiB/s (557kB/s), 135KiB/s-137KiB/s (138kB/s-141kB/s), io=20.4MiB (21.4MB), run=38109-38399msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179685: Thu Aug 13 22:47:01 2026 read: IOPS=1, BW=76.3MiB/s (80.0MB/s)(40.0MiB/524msec) slat (nsec): min=54709k, max=54709k, avg=54709115.00, stdev= 0.00 clat (nsec): min=427575k, max=427575k, avg=427575078.00, stdev= 0.00 lat (nsec): min=482290k, max=482290k, avg=482289600.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 426], 10.00th=[ 426], 20.00th=[ 426], | 30.00th=[ 426], 40.00th=[ 426], 50.00th=[ 426], 60.00th=[ 426], | 70.00th=[ 426], 80.00th=[ 426], 90.00th=[ 426], 95.00th=[ 426], | 99.00th=[ 426], 99.50th=[ 426], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 426] lat (msec) : 500=100.00% cpu : usr=0.19%, sys=2.87%, ctx=3, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=76.3MiB/s (80.0MB/s), 76.3MiB/s-76.3MiB/s (80.0MB/s-80.0MB/s), io=40.0MiB (41.9MB), run=524-524msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout