== sanity test 398c: run fio to test AIO ================= 22:39:52 (1788835192) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.51953 s, 80.7 MB/s osc.lustre-OST0000-osc-ffff935ae48cb000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=183188: Mon Sep 7 22:40:29 2026 write: IOPS=95, BW=382KiB/s (392kB/s)(10.0MiB/26779msec); 0 zone resets slat (usec): min=22, max=17257, avg=224.01, stdev=970.47 clat (msec): min=24, max=321, avg=165.91, stdev=44.27 lat (msec): min=24, max=321, avg=166.14, stdev=44.29 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 107], 10.00th=[ 116], 20.00th=[ 129], | 30.00th=[ 138], 40.00th=[ 150], 50.00th=[ 159], 60.00th=[ 171], | 70.00th=[ 186], 80.00th=[ 203], 90.00th=[ 222], 95.00th=[ 249], | 99.00th=[ 305], 99.50th=[ 313], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 173, max= 640, per=24.95%, avg=381.30, stdev=80.86, samples=53 iops : min= 43, max= 160, avg=94.89, stdev=20.29, samples=53 lat (msec) : 50=0.23%, 100=3.16%, 250=91.84%, 500=4.77% cpu : usr=0.23%, sys=1.00%, ctx=2651, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183189: Mon Sep 7 22:40:29 2026 write: IOPS=96, BW=387KiB/s (396kB/s)(10.0MiB/26492msec); 0 zone resets slat (usec): min=22, max=12449, avg=165.58, stdev=668.17 clat (msec): min=18, max=388, avg=165.21, stdev=47.12 lat (msec): min=19, max=388, avg=165.38, stdev=47.11 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 101], 10.00th=[ 113], 20.00th=[ 129], | 30.00th=[ 138], 40.00th=[ 150], 50.00th=[ 159], 60.00th=[ 171], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 232], 95.00th=[ 251], | 99.00th=[ 317], 99.50th=[ 330], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 249, max= 640, per=25.37%, avg=387.60, stdev=85.22, samples=52 iops : min= 62, max= 160, avg=96.44, stdev=21.38, samples=52 lat (msec) : 20=0.04%, 50=0.47%, 100=4.49%, 250=89.45%, 500=5.55% cpu : usr=0.13%, sys=0.88%, ctx=2193, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183190: Mon Sep 7 22:40:29 2026 write: IOPS=95, BW=383KiB/s (392kB/s)(10.0MiB/26739msec); 0 zone resets slat (usec): min=22, max=15114, avg=181.04, stdev=742.34 clat (msec): min=32, max=324, avg=166.13, stdev=44.09 lat (msec): min=32, max=324, avg=166.31, stdev=44.09 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 106], 10.00th=[ 115], 20.00th=[ 130], | 30.00th=[ 140], 40.00th=[ 150], 50.00th=[ 161], 60.00th=[ 171], | 70.00th=[ 186], 80.00th=[ 201], 90.00th=[ 228], 95.00th=[ 247], | 99.00th=[ 300], 99.50th=[ 313], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 213, max= 512, per=24.94%, avg=381.13, stdev=74.45, samples=53 iops : min= 53, max= 128, avg=94.89, stdev=18.70, samples=53 lat (msec) : 50=0.04%, 100=2.62%, 250=93.09%, 500=4.26% cpu : usr=0.12%, sys=0.93%, ctx=1926, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183191: Mon Sep 7 22:40:29 2026 write: IOPS=95, BW=382KiB/s (391kB/s)(10.0MiB/26793msec); 0 zone resets slat (usec): min=22, max=16809, avg=181.84, stdev=738.22 clat (msec): min=21, max=327, avg=166.22, stdev=44.47 lat (msec): min=22, max=327, avg=166.41, stdev=44.44 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 106], 10.00th=[ 117], 20.00th=[ 131], | 30.00th=[ 140], 40.00th=[ 150], 50.00th=[ 163], 60.00th=[ 171], | 70.00th=[ 184], 80.00th=[ 201], 90.00th=[ 224], 95.00th=[ 247], | 99.00th=[ 300], 99.50th=[ 317], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 251, max= 568, per=24.86%, avg=379.91, stdev=70.79, samples=53 iops : min= 62, max= 142, avg=94.55, stdev=17.82, samples=53 lat (msec) : 50=0.16%, 100=3.36%, 250=92.38%, 500=4.10% cpu : usr=0.21%, sys=1.07%, ctx=2478, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1529KiB/s (1565kB/s), 382KiB/s-387KiB/s (391kB/s-396kB/s), io=40.0MiB (41.9MB), run=26492-26793msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=183206: Mon Sep 7 22:41:15 2026 read: IOPS=37, BW=150KiB/s (153kB/s)(5048KiB/33765msec) slat (usec): min=2353, max=75180, avg=14395.71, stdev=8346.54 clat (msec): min=77, max=400, avg=200.68, stdev=59.46 lat (msec): min=82, max=412, avg=215.08, stdev=62.03 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 114], 10.00th=[ 129], 20.00th=[ 150], | 30.00th=[ 165], 40.00th=[ 180], 50.00th=[ 194], 60.00th=[ 209], | 70.00th=[ 228], 80.00th=[ 251], 90.00th=[ 284], 95.00th=[ 313], | 99.00th=[ 355], 99.50th=[ 363], 99.90th=[ 388], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 23, max= 271, per=25.34%, avg=148.52, stdev=44.72, samples=66 iops : min= 5, max= 67, avg=36.64, stdev=11.23, samples=66 write: IOPS=38, BW=154KiB/s (157kB/s)(5192KiB/33765msec); 0 zone resets slat (usec): min=27, max=16779, avg=228.31, stdev=856.03 clat (msec): min=94, max=375, avg=202.93, stdev=53.17 lat (msec): min=94, max=375, avg=203.16, stdev=53.23 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 126], 10.00th=[ 140], 20.00th=[ 157], | 30.00th=[ 171], 40.00th=[ 182], 50.00th=[ 197], 60.00th=[ 211], | 70.00th=[ 228], 80.00th=[ 251], 90.00th=[ 275], 95.00th=[ 300], | 99.00th=[ 342], 99.50th=[ 351], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 7, max= 294, per=25.16%, avg=153.73, stdev=51.08, samples=66 iops : min= 1, max= 73, avg=37.97, stdev=12.76, samples=66 lat (msec) : 100=1.13%, 250=78.95%, 500=19.92% cpu : usr=0.29%, sys=1.65%, ctx=4333, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183207: Mon Sep 7 22:41:15 2026 read: IOPS=37, BW=148KiB/s (152kB/s)(4976KiB/33571msec) slat (usec): min=50, max=80016, avg=14740.26, stdev=9440.21 clat (msec): min=17, max=380, avg=199.49, stdev=57.96 lat (msec): min=22, max=409, avg=214.24, stdev=60.63 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 116], 10.00th=[ 128], 20.00th=[ 148], | 30.00th=[ 165], 40.00th=[ 180], 50.00th=[ 192], 60.00th=[ 209], | 70.00th=[ 228], 80.00th=[ 249], 90.00th=[ 279], 95.00th=[ 305], | 99.00th=[ 351], 99.50th=[ 368], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 70, max= 294, per=25.17%, avg=147.47, stdev=42.64, samples=66 iops : min= 17, max= 73, avg=36.39, stdev=10.66, samples=66 write: IOPS=39, BW=157KiB/s (161kB/s)(5264KiB/33571msec); 0 zone resets slat (usec): min=22, max=13462, avg=215.34, stdev=719.87 clat (msec): min=93, max=391, avg=204.70, stdev=56.13 lat (msec): min=93, max=391, avg=204.92, stdev=56.15 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 124], 10.00th=[ 136], 20.00th=[ 157], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 197], 60.00th=[ 211], | 70.00th=[ 228], 80.00th=[ 255], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 351], 99.50th=[ 368], 99.90th=[ 384], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 71, max= 282, per=25.66%, avg=156.77, stdev=46.19, samples=66 iops : min= 17, max= 70, avg=38.73, stdev=11.56, samples=66 lat (msec) : 20=0.04%, 50=0.04%, 100=0.86%, 250=78.28%, 500=20.78% cpu : usr=0.29%, sys=1.86%, ctx=4517, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183208: Mon Sep 7 22:41:15 2026 read: IOPS=36, BW=146KiB/s (149kB/s)(4984KiB/34178msec) slat (msec): min=2, max=223, avg=14.81, stdev=12.33 clat (msec): min=35, max=750, avg=201.37, stdev=66.80 lat (msec): min=43, max=765, avg=216.19, stdev=69.77 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 114], 10.00th=[ 129], 20.00th=[ 150], | 30.00th=[ 167], 40.00th=[ 182], 50.00th=[ 194], 60.00th=[ 207], | 70.00th=[ 224], 80.00th=[ 243], 90.00th=[ 279], 95.00th=[ 313], | 99.00th=[ 384], 99.50th=[ 435], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 23, max= 232, per=24.79%, avg=145.30, stdev=46.06, samples=67 iops : min= 5, max= 58, avg=35.82, stdev=11.56, samples=67 write: IOPS=38, BW=154KiB/s (157kB/s)(5256KiB/34178msec); 0 zone resets slat (usec): min=24, max=29222, avg=242.85, stdev=1030.39 clat (msec): min=94, max=734, avg=207.68, stdev=68.43 lat (msec): min=94, max=734, avg=207.92, stdev=68.44 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 124], 10.00th=[ 138], 20.00th=[ 157], | 30.00th=[ 171], 40.00th=[ 184], 50.00th=[ 197], 60.00th=[ 213], | 70.00th=[ 230], 80.00th=[ 253], 90.00th=[ 292], 95.00th=[ 313], | 99.00th=[ 363], 99.50th=[ 684], 99.90th=[ 735], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 31, max= 270, per=25.03%, avg=152.96, stdev=50.48, samples=67 iops : min= 7, max= 67, avg=37.75, stdev=12.59, samples=67 lat (msec) : 50=0.08%, 100=0.51%, 250=79.96%, 500=18.91%, 750=0.51% lat (msec) : 1000=0.04% cpu : usr=0.25%, sys=1.74%, ctx=4342, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183209: Mon Sep 7 22:41:15 2026 read: IOPS=37, BW=148KiB/s (152kB/s)(5060KiB/34188msec) slat (msec): min=2, max=241, avg=15.00, stdev=12.76 clat (msec): min=12, max=840, avg=201.35, stdev=72.24 lat (msec): min=33, max=858, avg=216.36, stdev=75.30 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 112], 10.00th=[ 126], 20.00th=[ 148], | 30.00th=[ 165], 40.00th=[ 180], 50.00th=[ 190], 60.00th=[ 207], | 70.00th=[ 224], 80.00th=[ 249], 90.00th=[ 284], 95.00th=[ 313], | 99.00th=[ 372], 99.50th=[ 531], 99.90th=[ 785], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 15, max= 271, per=25.40%, avg=148.83, stdev=45.77, samples=66 iops : min= 3, max= 67, avg=36.74, stdev=11.48, samples=66 write: IOPS=37, BW=152KiB/s (155kB/s)(5180KiB/34188msec); 0 zone resets slat (usec): min=29, max=17642, avg=255.97, stdev=953.84 clat (msec): min=36, max=829, avg=207.12, stdev=72.51 lat (msec): min=36, max=830, avg=207.38, stdev=72.53 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 122], 10.00th=[ 138], 20.00th=[ 155], | 30.00th=[ 171], 40.00th=[ 186], 50.00th=[ 199], 60.00th=[ 209], | 70.00th=[ 226], 80.00th=[ 251], 90.00th=[ 288], 95.00th=[ 321], | 99.00th=[ 418], 99.50th=[ 718], 99.90th=[ 802], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 15, max= 263, per=24.48%, avg=149.58, stdev=50.34, samples=67 iops : min= 3, max= 65, avg=36.91, stdev=12.63, samples=67 lat (msec) : 20=0.04%, 50=0.23%, 100=1.09%, 250=78.91%, 500=19.14% lat (msec) : 750=0.27%, 1000=0.31% cpu : usr=0.24%, sys=1.76%, ctx=4450, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=587KiB/s (601kB/s), 146KiB/s-150KiB/s (149kB/s-153kB/s), io=19.6MiB (20.5MB), run=33571-34188msec WRITE: bw=611KiB/s (626kB/s), 152KiB/s-157KiB/s (155kB/s-161kB/s), io=20.4MiB (21.4MB), run=33571-34188msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=183218: Mon Sep 7 22:41:28 2026 read: IOPS=1, BW=63.4MiB/s (66.5MB/s)(40.0MiB/631msec) slat (nsec): min=30467k, max=30467k, avg=30466534.00, stdev= 0.00 clat (nsec): min=564221k, max=564221k, avg=564221295.00, stdev= 0.00 lat (nsec): min=594717k, max=594717k, avg=594716620.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 567], 10.00th=[ 567], 20.00th=[ 567], | 30.00th=[ 567], 40.00th=[ 567], 50.00th=[ 567], 60.00th=[ 567], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.38%, ctx=14, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=63.4MiB/s (66.5MB/s), 63.4MiB/s-63.4MiB/s (66.5MB/s-66.5MB/s), io=40.0MiB (41.9MB), run=631-631msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout