== sanity test 398c: run fio to test AIO ================= 06:39:01 (1789641541) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.461777 s, 90.8 MB/s osc.lustre-OST0000-osc-ffff95fca87d6000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179697: Thu Sep 17 06:39:37 2026 write: IOPS=98, BW=394KiB/s (404kB/s)(10.0MiB/25974msec); 0 zone resets slat (usec): min=28, max=16858, avg=213.23, stdev=727.76 clat (msec): min=21, max=394, avg=161.90, stdev=46.62 lat (msec): min=21, max=394, avg=162.12, stdev=46.63 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 102], 10.00th=[ 112], 20.00th=[ 125], | 30.00th=[ 134], 40.00th=[ 142], 50.00th=[ 153], 60.00th=[ 165], | 70.00th=[ 180], 80.00th=[ 201], 90.00th=[ 226], 95.00th=[ 253], | 99.00th=[ 296], 99.50th=[ 326], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 258, max= 624, per=25.27%, avg=395.41, stdev=86.10, samples=51 iops : min= 64, max= 156, avg=98.45, stdev=21.61, samples=51 lat (msec) : 50=0.16%, 100=4.06%, 250=90.20%, 500=5.59% cpu : usr=0.23%, sys=1.01%, ctx=2713, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179698: Thu Sep 17 06:39:37 2026 write: IOPS=97, BW=392KiB/s (401kB/s)(10.0MiB/26153msec); 0 zone resets slat (usec): min=29, max=11171, avg=222.92, stdev=638.99 clat (msec): min=18, max=364, avg=162.24, stdev=44.55 lat (msec): min=19, max=364, avg=162.46, stdev=44.56 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 102], 10.00th=[ 113], 20.00th=[ 126], | 30.00th=[ 136], 40.00th=[ 146], 50.00th=[ 155], 60.00th=[ 161], | 70.00th=[ 178], 80.00th=[ 203], 90.00th=[ 228], 95.00th=[ 247], | 99.00th=[ 279], 99.50th=[ 292], 99.90th=[ 330], 99.95th=[ 351], | 99.99th=[ 363] bw ( KiB/s): min= 252, max= 640, per=25.10%, avg=392.84, stdev=86.48, samples=51 iops : min= 63, max= 160, avg=97.82, stdev=21.64, samples=51 lat (msec) : 20=0.04%, 50=0.12%, 100=3.95%, 250=91.64%, 500=4.26% cpu : usr=0.17%, sys=1.04%, ctx=2799, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179699: Thu Sep 17 06:39:37 2026 write: IOPS=98, BW=393KiB/s (403kB/s)(10.0MiB/26045msec); 0 zone resets slat (usec): min=30, max=15346, avg=232.70, stdev=811.32 clat (msec): min=22, max=328, avg=161.83, stdev=44.55 lat (msec): min=23, max=328, avg=162.07, stdev=44.59 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 103], 10.00th=[ 112], 20.00th=[ 127], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 163], | 70.00th=[ 178], 80.00th=[ 203], 90.00th=[ 230], 95.00th=[ 251], | 99.00th=[ 271], 99.50th=[ 296], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 330] bw ( KiB/s): min= 254, max= 558, per=25.12%, avg=393.08, stdev=83.03, samples=51 iops : min= 63, max= 139, avg=97.88, stdev=20.77, samples=51 lat (msec) : 50=0.20%, 100=3.67%, 250=90.90%, 500=5.23% cpu : usr=0.18%, sys=0.98%, ctx=2885, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179700: Thu Sep 17 06:39:37 2026 write: IOPS=97, BW=391KiB/s (401kB/s)(10.0MiB/26167msec); 0 zone resets slat (usec): min=24, max=18626, avg=232.28, stdev=882.49 clat (msec): min=12, max=351, avg=162.17, stdev=43.73 lat (msec): min=12, max=351, avg=162.41, stdev=43.75 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 105], 10.00th=[ 113], 20.00th=[ 128], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 155], 60.00th=[ 165], | 70.00th=[ 180], 80.00th=[ 201], 90.00th=[ 226], 95.00th=[ 249], | 99.00th=[ 279], 99.50th=[ 284], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 253, max= 640, per=24.99%, avg=391.16, stdev=83.92, samples=51 iops : min= 63, max= 160, avg=97.39, stdev=21.01, samples=51 lat (msec) : 20=0.04%, 50=0.08%, 100=2.73%, 250=92.38%, 500=4.77% cpu : usr=0.19%, sys=1.04%, ctx=2962, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1565KiB/s (1603kB/s), 391KiB/s-394KiB/s (401kB/s-404kB/s), io=40.0MiB (41.9MB), run=25974-26167msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179715: Thu Sep 17 06:40:15 2026 read: IOPS=42, BW=169KiB/s (173kB/s)(5048KiB/29848msec) slat (usec): min=40, max=203515, avg=12788.53, stdev=11697.63 clat (msec): min=38, max=839, avg=175.79, stdev=66.32 lat (msec): min=38, max=852, avg=188.59, stdev=69.51 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 107], 10.00th=[ 118], 20.00th=[ 133], | 30.00th=[ 148], 40.00th=[ 161], 50.00th=[ 169], 60.00th=[ 178], | 70.00th=[ 190], 80.00th=[ 207], 90.00th=[ 232], 95.00th=[ 253], | 99.00th=[ 330], 99.50th=[ 760], 99.90th=[ 818], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 40, max= 256, per=25.09%, avg=168.34, stdev=46.05, samples=59 iops : min= 10, max= 64, avg=41.61, stdev=11.51, samples=59 write: IOPS=43, BW=174KiB/s (178kB/s)(5192KiB/29848msec); 0 zone resets slat (usec): min=34, max=20209, avg=222.61, stdev=765.68 clat (msec): min=35, max=839, avg=183.72, stdev=60.59 lat (msec): min=35, max=839, avg=183.94, stdev=60.56 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 111], 10.00th=[ 124], 20.00th=[ 144], | 30.00th=[ 161], 40.00th=[ 171], 50.00th=[ 180], 60.00th=[ 190], | 70.00th=[ 203], 80.00th=[ 218], 90.00th=[ 239], 95.00th=[ 255], | 99.00th=[ 321], 99.50th=[ 531], 99.90th=[ 810], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 48, max= 295, per=24.84%, avg=173.64, stdev=43.22, samples=59 iops : min= 12, max= 73, avg=42.95, stdev=10.76, samples=59 lat (msec) : 50=0.16%, 100=2.70%, 250=90.78%, 500=5.78%, 750=0.12% lat (msec) : 1000=0.47% cpu : usr=0.24%, sys=1.80%, ctx=4811, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179716: Thu Sep 17 06:40:15 2026 read: IOPS=42, BW=169KiB/s (173kB/s)(4976KiB/29459msec) slat (usec): min=2594, max=75104, avg=12626.23, stdev=7058.32 clat (msec): min=65, max=349, avg=173.48, stdev=42.32 lat (msec): min=69, max=362, avg=186.12, stdev=44.08 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 106], 10.00th=[ 123], 20.00th=[ 140], | 30.00th=[ 150], 40.00th=[ 161], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 205], 90.00th=[ 226], 95.00th=[ 247], | 99.00th=[ 288], 99.50th=[ 300], 99.90th=[ 347], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 36, max= 295, per=25.06%, avg=168.14, stdev=42.57, samples=58 iops : min= 9, max= 73, avg=41.57, stdev=10.59, samples=58 write: IOPS=44, BW=179KiB/s (183kB/s)(5264KiB/29459msec); 0 zone resets slat (usec): min=33, max=11333, avg=239.22, stdev=736.43 clat (msec): min=36, max=358, avg=179.59, stdev=41.12 lat (msec): min=36, max=358, avg=179.84, stdev=41.13 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 112], 10.00th=[ 126], 20.00th=[ 146], | 30.00th=[ 161], 40.00th=[ 169], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 199], 80.00th=[ 213], 90.00th=[ 230], 95.00th=[ 249], | 99.00th=[ 288], 99.50th=[ 321], 99.90th=[ 347], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 88, max= 279, per=25.60%, avg=178.93, stdev=40.27, samples=58 iops : min= 22, max= 69, avg=44.26, stdev=10.08, samples=58 lat (msec) : 50=0.04%, 100=2.58%, 250=92.73%, 500=4.65% cpu : usr=0.24%, sys=1.75%, ctx=4697, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179717: Thu Sep 17 06:40:15 2026 read: IOPS=42, BW=169KiB/s (173kB/s)(4984KiB/29500msec) slat (usec): min=1873, max=66851, avg=13049.27, stdev=7289.75 clat (msec): min=68, max=336, avg=172.52, stdev=42.88 lat (msec): min=74, max=357, avg=185.57, stdev=44.91 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 105], 10.00th=[ 120], 20.00th=[ 136], | 30.00th=[ 148], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 207], 90.00th=[ 226], 95.00th=[ 247], | 99.00th=[ 288], 99.50th=[ 326], 99.90th=[ 334], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 79, max= 256, per=25.17%, avg=168.88, stdev=41.44, samples=58 iops : min= 19, max= 64, avg=41.74, stdev=10.40, samples=58 write: IOPS=44, BW=178KiB/s (182kB/s)(5256KiB/29500msec); 0 zone resets slat (usec): min=31, max=8305, avg=200.81, stdev=534.69 clat (msec): min=80, max=347, avg=180.78, stdev=41.84 lat (msec): min=81, max=347, avg=180.98, stdev=41.84 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 113], 10.00th=[ 125], 20.00th=[ 146], | 30.00th=[ 161], 40.00th=[ 171], 50.00th=[ 180], 60.00th=[ 188], | 70.00th=[ 201], 80.00th=[ 213], 90.00th=[ 236], 95.00th=[ 253], | 99.00th=[ 292], 99.50th=[ 305], 99.90th=[ 317], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 86, max= 287, per=25.42%, avg=177.66, stdev=40.61, samples=58 iops : min= 21, max= 71, avg=43.95, stdev=10.13, samples=58 lat (msec) : 100=2.77%, 250=92.07%, 500=5.16% cpu : usr=0.28%, sys=1.61%, ctx=4664, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179718: Thu Sep 17 06:40:15 2026 read: IOPS=42, BW=169KiB/s (173kB/s)(5060KiB/29874msec) slat (msec): min=2, max=220, avg=12.98, stdev=12.95 clat (msec): min=33, max=1003, avg=173.35, stdev=63.13 lat (msec): min=47, max=1007, avg=186.33, stdev=67.93 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 102], 10.00th=[ 116], 20.00th=[ 136], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 169], 60.00th=[ 178], | 70.00th=[ 192], 80.00th=[ 203], 90.00th=[ 226], 95.00th=[ 247], | 99.00th=[ 313], 99.50th=[ 347], 99.90th=[ 969], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 72, max= 296, per=25.56%, avg=171.50, stdev=43.48, samples=58 iops : min= 18, max= 74, avg=42.41, stdev=10.86, samples=58 write: IOPS=43, BW=173KiB/s (178kB/s)(5180KiB/29874msec); 0 zone resets slat (usec): min=29, max=17758, avg=241.30, stdev=883.59 clat (msec): min=50, max=1021, avg=184.57, stdev=80.24 lat (msec): min=51, max=1021, avg=184.81, stdev=80.23 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 113], 10.00th=[ 123], 20.00th=[ 144], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 201], 80.00th=[ 211], 90.00th=[ 234], 95.00th=[ 253], | 99.00th=[ 342], 99.50th=[ 978], 99.90th=[ 1020], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 23, max= 287, per=24.60%, avg=171.97, stdev=46.99, samples=59 iops : min= 5, max= 71, avg=42.53, stdev=11.77, samples=59 lat (msec) : 50=0.16%, 100=3.28%, 250=91.37%, 500=4.61%, 750=0.08% lat (msec) : 1000=0.27%, 2000=0.23% cpu : usr=0.20%, sys=1.73%, ctx=4483, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=672KiB/s (688kB/s), 169KiB/s-169KiB/s (173kB/s-173kB/s), io=19.6MiB (20.5MB), run=29459-29874msec WRITE: bw=699KiB/s (716kB/s), 173KiB/s-179KiB/s (178kB/s-183kB/s), io=20.4MiB (21.4MB), run=29459-29874msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179728: Thu Sep 17 06:40:26 2026 read: IOPS=1, BW=78.3MiB/s (82.1MB/s)(40.0MiB/511msec) slat (nsec): min=58291k, max=58291k, avg=58290744.00, stdev= 0.00 clat (nsec): min=412594k, max=412594k, avg=412594377.00, stdev= 0.00 lat (nsec): min=470890k, max=470890k, avg=470889909.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 414], 10.00th=[ 414], 20.00th=[ 414], | 30.00th=[ 414], 40.00th=[ 414], 50.00th=[ 414], 60.00th=[ 414], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=1.76%, ctx=12, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=78.3MiB/s (82.1MB/s), 78.3MiB/s-78.3MiB/s (82.1MB/s-82.1MB/s), io=40.0MiB (41.9MB), run=511-511msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout