== sanity test 398c: run fio to test AIO ================= 03:00:13 (1786172413) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.499047 s, 84.0 MB/s osc.lustre-OST0000-osc-ffff9b9e08c52000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179676: Sat Aug 8 03:00:49 2026 write: IOPS=98, BW=395KiB/s (404kB/s)(10.0MiB/25946msec); 0 zone resets slat (usec): min=34, max=18314, avg=207.82, stdev=722.06 clat (msec): min=13, max=406, avg=161.82, stdev=43.70 lat (msec): min=13, max=406, avg=162.03, stdev=43.70 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 107], 10.00th=[ 118], 20.00th=[ 128], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 165], | 70.00th=[ 182], 80.00th=[ 199], 90.00th=[ 222], 95.00th=[ 243], | 99.00th=[ 275], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 252, max= 630, per=25.09%, avg=394.63, stdev=99.64, samples=51 iops : min= 63, max= 157, avg=98.22, stdev=24.94, samples=51 lat (msec) : 20=0.04%, 100=4.22%, 250=91.88%, 500=3.87% cpu : usr=0.14%, sys=0.93%, ctx=1863, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179677: Sat Aug 8 03:00:49 2026 write: IOPS=98, BW=393KiB/s (403kB/s)(10.0MiB/26029msec); 0 zone resets slat (usec): min=36, max=19775, avg=256.16, stdev=955.00 clat (msec): min=24, max=303, avg=161.24, stdev=42.03 lat (msec): min=24, max=303, avg=161.49, stdev=42.05 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 106], 10.00th=[ 118], 20.00th=[ 128], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 155], 60.00th=[ 165], | 70.00th=[ 180], 80.00th=[ 197], 90.00th=[ 222], 95.00th=[ 247], | 99.00th=[ 268], 99.50th=[ 275], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 247, max= 606, per=24.99%, avg=393.16, stdev=87.45, samples=51 iops : min= 61, max= 151, avg=97.82, stdev=21.89, samples=51 lat (msec) : 50=0.12%, 100=4.22%, 250=92.11%, 500=3.55% cpu : usr=0.10%, sys=1.31%, ctx=2455, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179678: Sat Aug 8 03:00:49 2026 write: IOPS=98, BW=394KiB/s (403kB/s)(10.0MiB/26001msec); 0 zone resets slat (usec): min=35, max=26771, avg=251.66, stdev=872.66 clat (msec): min=44, max=280, avg=161.53, stdev=41.45 lat (msec): min=45, max=280, avg=161.78, stdev=41.45 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 106], 10.00th=[ 117], 20.00th=[ 128], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 155], 60.00th=[ 167], | 70.00th=[ 178], 80.00th=[ 197], 90.00th=[ 224], 95.00th=[ 241], | 99.00th=[ 271], 99.50th=[ 275], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 223, max= 574, per=25.00%, avg=393.25, stdev=89.14, samples=51 iops : min= 55, max= 143, avg=97.84, stdev=22.36, samples=51 lat (msec) : 50=0.04%, 100=4.02%, 250=93.24%, 500=2.70% cpu : usr=0.15%, sys=1.30%, ctx=2595, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179679: Sat Aug 8 03:00:49 2026 write: IOPS=98, BW=393KiB/s (403kB/s)(10.0MiB/26035msec); 0 zone resets slat (usec): min=36, max=16558, avg=255.91, stdev=892.25 clat (msec): min=26, max=276, avg=161.58, stdev=42.82 lat (msec): min=26, max=276, avg=161.83, stdev=42.84 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 101], 10.00th=[ 118], 20.00th=[ 129], | 30.00th=[ 136], 40.00th=[ 146], 50.00th=[ 155], 60.00th=[ 165], | 70.00th=[ 180], 80.00th=[ 194], 90.00th=[ 228], 95.00th=[ 247], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 254, max= 550, per=24.91%, avg=391.76, stdev=85.57, samples=51 iops : min= 63, max= 137, avg=97.49, stdev=21.46, samples=51 lat (msec) : 50=0.16%, 100=4.57%, 250=90.62%, 500=4.65% cpu : usr=0.15%, sys=1.39%, ctx=2542, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1573KiB/s (1611kB/s), 393KiB/s-395KiB/s (403kB/s-404kB/s), io=40.0MiB (41.9MB), run=25946-26035msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179694: Sat Aug 8 03:01:33 2026 read: IOPS=37, BW=149KiB/s (153kB/s)(5048KiB/33812msec) slat (usec): min=2271, max=93526, avg=15308.22, stdev=9630.42 clat (msec): min=26, max=482, avg=199.51, stdev=58.12 lat (msec): min=43, max=503, avg=214.83, stdev=60.81 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 117], 10.00th=[ 136], 20.00th=[ 155], | 30.00th=[ 169], 40.00th=[ 182], 50.00th=[ 194], 60.00th=[ 205], | 70.00th=[ 218], 80.00th=[ 239], 90.00th=[ 279], 95.00th=[ 305], | 99.00th=[ 372], 99.50th=[ 418], 99.90th=[ 477], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 62, max= 259, per=24.88%, avg=147.55, stdev=37.47, samples=67 iops : min= 15, max= 64, avg=36.39, stdev= 9.40, samples=67 write: IOPS=38, BW=154KiB/s (157kB/s)(5192KiB/33812msec); 0 zone resets slat (usec): min=43, max=9788, avg=227.69, stdev=464.88 clat (msec): min=50, max=407, avg=204.33, stdev=52.62 lat (msec): min=50, max=407, avg=204.56, stdev=52.62 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 125], 10.00th=[ 140], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 188], 50.00th=[ 199], 60.00th=[ 211], | 70.00th=[ 226], 80.00th=[ 245], 90.00th=[ 275], 95.00th=[ 300], | 99.00th=[ 351], 99.50th=[ 368], 99.90th=[ 405], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 39, max= 246, per=24.72%, avg=152.54, stdev=44.88, samples=67 iops : min= 9, max= 61, avg=37.63, stdev=11.21, samples=67 lat (msec) : 50=0.04%, 100=1.48%, 250=81.45%, 500=17.03% cpu : usr=0.23%, sys=1.65%, ctx=4286, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179695: Sat Aug 8 03:01:33 2026 read: IOPS=36, BW=148KiB/s (151kB/s)(4976KiB/33691msec) slat (usec): min=106, max=76957, avg=14717.24, stdev=9063.55 clat (msec): min=13, max=443, avg=198.47, stdev=55.66 lat (msec): min=24, max=465, avg=213.20, stdev=58.24 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 116], 10.00th=[ 132], 20.00th=[ 153], | 30.00th=[ 169], 40.00th=[ 184], 50.00th=[ 194], 60.00th=[ 209], | 70.00th=[ 222], 80.00th=[ 245], 90.00th=[ 271], 95.00th=[ 296], | 99.00th=[ 347], 99.50th=[ 380], 99.90th=[ 435], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 62, max= 267, per=24.70%, avg=146.47, stdev=41.42, samples=66 iops : min= 15, max= 66, avg=36.12, stdev=10.29, samples=66 write: IOPS=39, BW=156KiB/s (160kB/s)(5264KiB/33691msec); 0 zone resets slat (usec): min=39, max=16246, avg=305.05, stdev=937.25 clat (msec): min=87, max=446, avg=207.15, stdev=51.78 lat (msec): min=88, max=446, avg=207.46, stdev=51.78 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 130], 10.00th=[ 146], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 213], | 70.00th=[ 230], 80.00th=[ 247], 90.00th=[ 271], 95.00th=[ 296], | 99.00th=[ 355], 99.50th=[ 376], 99.90th=[ 439], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 79, max= 264, per=25.19%, avg=155.42, stdev=37.94, samples=66 iops : min= 19, max= 66, avg=38.36, stdev= 9.56, samples=66 lat (msec) : 20=0.08%, 50=0.12%, 100=1.02%, 250=80.66%, 500=18.12% cpu : usr=0.25%, sys=1.68%, ctx=4227, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179696: Sat Aug 8 03:01:33 2026 read: IOPS=36, BW=147KiB/s (151kB/s)(4984KiB/33811msec) slat (usec): min=2592, max=74661, avg=14947.38, stdev=8751.85 clat (msec): min=62, max=475, avg=198.05, stdev=54.31 lat (msec): min=70, max=489, avg=213.00, stdev=56.87 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 117], 10.00th=[ 134], 20.00th=[ 155], | 30.00th=[ 169], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 205], | 70.00th=[ 220], 80.00th=[ 239], 90.00th=[ 266], 95.00th=[ 296], | 99.00th=[ 355], 99.50th=[ 384], 99.90th=[ 472], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 38, max= 271, per=24.71%, avg=146.52, stdev=43.69, samples=67 iops : min= 9, max= 67, avg=36.13, stdev=10.90, samples=67 write: IOPS=38, BW=155KiB/s (159kB/s)(5256KiB/33811msec); 0 zone resets slat (usec): min=40, max=24393, avg=323.96, stdev=1070.26 clat (msec): min=50, max=409, avg=206.31, stdev=53.63 lat (msec): min=50, max=410, avg=206.64, stdev=53.65 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 128], 10.00th=[ 144], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 228], 80.00th=[ 247], 90.00th=[ 279], 95.00th=[ 309], | 99.00th=[ 363], 99.50th=[ 384], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 46, max= 263, per=24.94%, avg=153.87, stdev=40.36, samples=67 iops : min= 11, max= 65, avg=37.97, stdev=10.13, samples=67 lat (msec) : 100=1.37%, 250=82.50%, 500=16.13% cpu : usr=0.26%, sys=1.99%, ctx=4610, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179697: Sat Aug 8 03:01:33 2026 read: IOPS=37, BW=150KiB/s (153kB/s)(5060KiB/33780msec) slat (usec): min=3208, max=94096, avg=14890.82, stdev=9073.91 clat (msec): min=25, max=432, avg=197.22, stdev=54.83 lat (msec): min=38, max=497, avg=212.13, stdev=57.51 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 117], 10.00th=[ 134], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 192], 60.00th=[ 203], | 70.00th=[ 218], 80.00th=[ 239], 90.00th=[ 268], 95.00th=[ 296], | 99.00th=[ 359], 99.50th=[ 393], 99.90th=[ 422], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 47, max= 232, per=25.09%, avg=148.81, stdev=39.74, samples=67 iops : min= 11, max= 58, avg=36.70, stdev= 9.98, samples=67 write: IOPS=38, BW=153KiB/s (157kB/s)(5180KiB/33780msec); 0 zone resets slat (usec): min=43, max=13909, avg=306.78, stdev=981.42 clat (msec): min=61, max=426, avg=206.60, stdev=53.31 lat (msec): min=62, max=426, avg=206.90, stdev=53.37 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 127], 10.00th=[ 144], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 213], | 70.00th=[ 226], 80.00th=[ 249], 90.00th=[ 275], 95.00th=[ 305], | 99.00th=[ 359], 99.50th=[ 397], 99.90th=[ 422], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 55, max= 251, per=24.65%, avg=152.09, stdev=37.41, samples=67 iops : min= 13, max= 62, avg=37.52, stdev= 9.33, samples=67 lat (msec) : 50=0.04%, 100=1.02%, 250=81.56%, 500=17.38% cpu : usr=0.25%, sys=1.79%, ctx=4303, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=594KiB/s (608kB/s), 147KiB/s-150KiB/s (151kB/s-153kB/s), io=19.6MiB (20.5MB), run=33691-33812msec WRITE: bw=618KiB/s (633kB/s), 153KiB/s-156KiB/s (157kB/s-160kB/s), io=20.4MiB (21.4MB), run=33691-33812msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179717: Sat Aug 8 03:01:42 2026 read: IOPS=1, BW=77.5MiB/s (81.3MB/s)(40.0MiB/516msec) slat (nsec): min=37146k, max=37146k, avg=37145771.00, stdev= 0.00 clat (nsec): min=453271k, max=453271k, avg=453270608.00, stdev= 0.00 lat (nsec): min=490423k, max=490423k, avg=490422529.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 456], 10.00th=[ 456], 20.00th=[ 456], | 30.00th=[ 456], 40.00th=[ 456], 50.00th=[ 456], 60.00th=[ 456], | 70.00th=[ 456], 80.00th=[ 456], 90.00th=[ 456], 95.00th=[ 456], | 99.00th=[ 456], 99.50th=[ 456], 99.90th=[ 456], 99.95th=[ 456], | 99.99th=[ 456] lat (msec) : 500=100.00% cpu : usr=0.19%, sys=3.11%, ctx=4, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=77.5MiB/s (81.3MB/s), 77.5MiB/s-77.5MiB/s (81.3MB/s-81.3MB/s), io=40.0MiB (41.9MB), run=516-516msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout