== sanity test 398c: run fio to test AIO ================= 16:29:47 (1787603387) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.276338 s, 152 MB/s osc.lustre-OST0000-osc-ffff9200c7c9a800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145527: Mon Aug 24 16:32:05 2026 write: IOPS=20, BW=80.3KiB/s (82.3kB/s)(10.0MiB/127466msec); 0 zone resets slat (usec): min=20, max=24885, avg=179.14, stdev=795.54 clat (msec): min=195, max=1596, avg=795.50, stdev=200.53 lat (msec): min=195, max=1597, avg=795.68, stdev=200.55 clat percentiles (msec): | 1.00th=[ 251], 5.00th=[ 489], 10.00th=[ 584], 20.00th=[ 659], | 30.00th=[ 693], 40.00th=[ 743], 50.00th=[ 776], 60.00th=[ 818], | 70.00th=[ 877], 80.00th=[ 953], 90.00th=[ 1070], 95.00th=[ 1150], | 99.00th=[ 1318], 99.50th=[ 1385], 99.90th=[ 1502], 99.95th=[ 1603], | 99.99th=[ 1603] bw ( KiB/s): min= 7, max= 256, per=25.13%, avg=80.66, stdev=32.78, samples=250 iops : min= 1, max= 64, avg=19.74, stdev= 8.25, samples=250 lat (msec) : 250=0.90%, 500=4.18%, 750=37.73%, 1000=42.54%, 2000=14.65% cpu : usr=0.05%, sys=0.26%, ctx=1785, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145528: Mon Aug 24 16:32:05 2026 write: IOPS=20, BW=80.6KiB/s (82.5kB/s)(10.0MiB/127062msec); 0 zone resets slat (usec): min=20, max=21043, avg=149.82, stdev=654.17 clat (msec): min=142, max=1546, avg=793.65, stdev=206.01 lat (msec): min=142, max=1546, avg=793.80, stdev=205.99 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 481], 10.00th=[ 575], 20.00th=[ 659], | 30.00th=[ 701], 40.00th=[ 743], 50.00th=[ 776], 60.00th=[ 810], | 70.00th=[ 869], 80.00th=[ 944], 90.00th=[ 1062], 95.00th=[ 1167], | 99.00th=[ 1318], 99.50th=[ 1368], 99.90th=[ 1452], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 7, max= 256, per=24.85%, avg=79.76, stdev=31.90, samples=253 iops : min= 1, max= 64, avg=19.52, stdev= 8.01, samples=253 lat (msec) : 250=1.88%, 500=3.79%, 750=36.68%, 1000=42.66%, 2000=15.00% cpu : usr=0.06%, sys=0.20%, ctx=1363, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145529: Mon Aug 24 16:32:05 2026 write: IOPS=20, BW=80.5KiB/s (82.4kB/s)(10.0MiB/127237msec); 0 zone resets slat (usec): min=22, max=14458, avg=161.52, stdev=564.83 clat (msec): min=104, max=1585, avg=794.06, stdev=207.03 lat (msec): min=105, max=1585, avg=794.23, stdev=207.01 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 472], 10.00th=[ 567], 20.00th=[ 651], | 30.00th=[ 701], 40.00th=[ 743], 50.00th=[ 785], 60.00th=[ 818], | 70.00th=[ 877], 80.00th=[ 953], 90.00th=[ 1062], 95.00th=[ 1167], | 99.00th=[ 1351], 99.50th=[ 1368], 99.90th=[ 1469], 99.95th=[ 1469], | 99.99th=[ 1586] bw ( KiB/s): min= 15, max= 232, per=24.85%, avg=79.76, stdev=30.49, samples=253 iops : min= 3, max= 58, avg=19.52, stdev= 7.66, samples=253 lat (msec) : 250=1.41%, 500=4.80%, 750=35.23%, 1000=43.52%, 2000=15.04% cpu : usr=0.05%, sys=0.23%, ctx=1889, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145530: Mon Aug 24 16:32:05 2026 write: IOPS=20, BW=80.4KiB/s (82.4kB/s)(10.0MiB/127306msec); 0 zone resets slat (usec): min=22, max=12601, avg=167.24, stdev=569.07 clat (msec): min=170, max=1558, avg=794.46, stdev=205.20 lat (msec): min=170, max=1558, avg=794.63, stdev=205.19 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 506], 10.00th=[ 575], 20.00th=[ 651], | 30.00th=[ 701], 40.00th=[ 743], 50.00th=[ 785], 60.00th=[ 818], | 70.00th=[ 869], 80.00th=[ 944], 90.00th=[ 1053], 95.00th=[ 1150], | 99.00th=[ 1401], 99.50th=[ 1435], 99.90th=[ 1536], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 7, max= 215, per=24.85%, avg=79.76, stdev=32.10, samples=253 iops : min= 1, max= 53, avg=19.52, stdev= 8.04, samples=253 lat (msec) : 250=1.13%, 500=3.67%, 750=37.66%, 1000=42.66%, 2000=14.88% cpu : usr=0.05%, sys=0.26%, ctx=2010, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=321KiB/s (329kB/s), 80.3KiB/s-80.6KiB/s (82.3kB/s-82.5kB/s), io=40.0MiB (41.9MB), run=127062-127466msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145558: Mon Aug 24 16:34:03 2026 read: IOPS=11, BW=46.6KiB/s (47.8kB/s)(5048KiB/108232msec) slat (usec): min=20, max=61440, avg=14713.17, stdev=8682.30 clat (msec): min=27, max=1059, avg=418.51, stdev=133.36 lat (msec): min=31, max=1072, avg=433.23, stdev=134.85 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 241], 10.00th=[ 284], 20.00th=[ 321], | 30.00th=[ 351], 40.00th=[ 376], 50.00th=[ 401], 60.00th=[ 430], | 70.00th=[ 460], 80.00th=[ 502], 90.00th=[ 575], 95.00th=[ 659], | 99.00th=[ 885], 99.50th=[ 919], 99.90th=[ 978], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 7, max= 143, per=25.02%, avg=46.04, stdev=22.39, samples=215 iops : min= 1, max= 35, avg=11.03, stdev= 5.63, samples=215 write: IOPS=11, BW=47.0KiB/s (49.1kB/s)(5192KiB/108232msec); 0 zone resets slat (usec): min=20, max=31010, avg=296.19, stdev=1540.75 clat (msec): min=174, max=1703, avg=911.69, stdev=266.78 lat (msec): min=174, max=1703, avg=911.99, stdev=266.82 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 510], 10.00th=[ 584], 20.00th=[ 667], | 30.00th=[ 768], 40.00th=[ 827], 50.00th=[ 885], 60.00th=[ 969], | 70.00th=[ 1036], 80.00th=[ 1150], 90.00th=[ 1284], 95.00th=[ 1385], | 99.00th=[ 1552], 99.50th=[ 1586], 99.90th=[ 1670], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 120, per=24.82%, avg=47.66, stdev=21.06, samples=213 iops : min= 1, max= 30, avg=11.44, stdev= 5.28, samples=213 lat (msec) : 50=0.20%, 100=0.23%, 250=2.66%, 500=38.55%, 750=20.43% lat (msec) : 1000=19.88%, 2000=18.05% cpu : usr=0.09%, sys=0.64%, ctx=4007, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145559: Mon Aug 24 16:34:03 2026 read: IOPS=11, BW=45.9KiB/s (46.0kB/s)(4976KiB/108500msec) slat (usec): min=3015, max=54385, avg=14656.73, stdev=8426.31 clat (msec): min=30, max=980, avg=421.90, stdev=122.54 lat (msec): min=51, max=1002, avg=436.57, stdev=123.86 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 253], 10.00th=[ 288], 20.00th=[ 330], | 30.00th=[ 355], 40.00th=[ 380], 50.00th=[ 405], 60.00th=[ 435], | 70.00th=[ 464], 80.00th=[ 510], 90.00th=[ 575], 95.00th=[ 642], | 99.00th=[ 827], 99.50th=[ 894], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 7, max= 128, per=24.77%, avg=45.58, stdev=22.32, samples=213 iops : min= 1, max= 32, avg=10.91, stdev= 5.61, samples=213 write: IOPS=12, BW=48.5KiB/s (49.7kB/s)(5264KiB/108500msec); 0 zone resets slat (usec): min=25, max=25321, avg=299.83, stdev=1240.50 clat (msec): min=168, max=1671, avg=903.66, stdev=265.72 lat (msec): min=168, max=1672, avg=903.96, stdev=265.77 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 493], 10.00th=[ 558], 20.00th=[ 667], | 30.00th=[ 760], 40.00th=[ 827], 50.00th=[ 885], 60.00th=[ 944], | 70.00th=[ 1036], 80.00th=[ 1116], 90.00th=[ 1267], 95.00th=[ 1401], | 99.00th=[ 1536], 99.50th=[ 1603], 99.90th=[ 1653], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 7, max= 103, per=25.12%, avg=48.24, stdev=20.89, samples=213 iops : min= 1, max= 25, avg=11.57, stdev= 5.25, samples=213 lat (msec) : 50=0.08%, 100=0.12%, 250=2.15%, 500=38.52%, 750=21.95% lat (msec) : 1000=19.69%, 2000=17.50% cpu : usr=0.13%, sys=0.64%, ctx=4416, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145560: Mon Aug 24 16:34:03 2026 read: IOPS=11, BW=45.9KiB/s (47.0kB/s)(4984KiB/108501msec) slat (usec): min=3008, max=67570, avg=14729.80, stdev=8642.03 clat (msec): min=27, max=982, avg=418.00, stdev=124.89 lat (msec): min=32, max=997, avg=432.74, stdev=126.61 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 249], 10.00th=[ 288], 20.00th=[ 330], | 30.00th=[ 355], 40.00th=[ 376], 50.00th=[ 401], 60.00th=[ 426], | 70.00th=[ 460], 80.00th=[ 506], 90.00th=[ 567], 95.00th=[ 642], | 99.00th=[ 810], 99.50th=[ 844], 99.90th=[ 978], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 7, max= 119, per=25.01%, avg=46.01, stdev=21.90, samples=212 iops : min= 1, max= 29, avg=11.02, stdev= 5.53, samples=212 write: IOPS=12, BW=48.4KiB/s (49.6kB/s)(5256KiB/108501msec); 0 zone resets slat (usec): min=30, max=17968, avg=291.11, stdev=1100.00 clat (msec): min=209, max=1701, avg=908.00, stdev=269.82 lat (msec): min=216, max=1701, avg=908.30, stdev=269.85 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 489], 10.00th=[ 567], 20.00th=[ 659], | 30.00th=[ 751], 40.00th=[ 827], 50.00th=[ 894], 60.00th=[ 969], | 70.00th=[ 1045], 80.00th=[ 1133], 90.00th=[ 1267], 95.00th=[ 1418], | 99.00th=[ 1569], 99.50th=[ 1620], 99.90th=[ 1670], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 119, per=24.71%, avg=47.44, stdev=19.83, samples=215 iops : min= 1, max= 29, avg=11.38, stdev= 4.99, samples=215 lat (msec) : 50=0.23%, 100=0.20%, 250=2.23%, 500=38.87%, 750=21.88% lat (msec) : 1000=18.59%, 2000=18.01% cpu : usr=0.12%, sys=0.69%, ctx=4440, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145561: Mon Aug 24 16:34:03 2026 read: IOPS=11, BW=46.7KiB/s (47.9kB/s)(5060KiB/108273msec) slat (usec): min=2795, max=62357, avg=14983.47, stdev=8640.12 clat (msec): min=11, max=1037, avg=423.62, stdev=129.74 lat (msec): min=39, max=1059, avg=438.61, stdev=131.23 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 257], 10.00th=[ 292], 20.00th=[ 326], | 30.00th=[ 355], 40.00th=[ 380], 50.00th=[ 409], 60.00th=[ 435], | 70.00th=[ 464], 80.00th=[ 510], 90.00th=[ 584], 95.00th=[ 659], | 99.00th=[ 885], 99.50th=[ 902], 99.90th=[ 1020], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 7, max= 127, per=25.48%, avg=46.88, stdev=21.62, samples=212 iops : min= 1, max= 31, avg=11.23, stdev= 5.43, samples=212 write: IOPS=11, BW=47.8KiB/s (48.0kB/s)(5180KiB/108273msec); 0 zone resets slat (usec): min=29, max=37880, avg=277.28, stdev=1470.76 clat (msec): min=168, max=1708, avg=906.20, stdev=262.46 lat (msec): min=168, max=1708, avg=906.48, stdev=262.49 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 498], 10.00th=[ 567], 20.00th=[ 667], | 30.00th=[ 760], 40.00th=[ 827], 50.00th=[ 894], 60.00th=[ 969], | 70.00th=[ 1053], 80.00th=[ 1116], 90.00th=[ 1250], 95.00th=[ 1368], | 99.00th=[ 1552], 99.50th=[ 1586], 99.90th=[ 1670], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 111, per=24.75%, avg=47.53, stdev=20.64, samples=213 iops : min= 1, max= 27, avg=11.40, stdev= 5.19, samples=213 lat (msec) : 20=0.04%, 50=0.12%, 100=0.23%, 250=1.91%, 500=39.06% lat (msec) : 750=21.52%, 1000=18.48%, 2000=18.63% cpu : usr=0.11%, sys=0.57%, ctx=4038, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=185KiB/s (189kB/s), 45.9KiB/s-46.7KiB/s (46.0kB/s-47.9kB/s), io=19.6MiB (20.5MB), run=108232-108501msec WRITE: bw=193KiB/s (197kB/s), 47.8KiB/s-48.5KiB/s (48.0kB/s-49.7kB/s), io=20.4MiB (21.4MB), run=108232-108501msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145580: Mon Aug 24 16:34:12 2026 read: IOPS=1, BW=56.4MiB/s (59.2MB/s)(40.0MiB/709msec) slat (nsec): min=25682k, max=25682k, avg=25682463.00, stdev= 0.00 clat (nsec): min=622580k, max=622580k, avg=622580367.00, stdev= 0.00 lat (nsec): min=648268k, max=648268k, avg=648268033.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 625], 5.00th=[ 625], 10.00th=[ 625], 20.00th=[ 625], | 30.00th=[ 625], 40.00th=[ 625], 50.00th=[ 625], 60.00th=[ 625], | 70.00th=[ 625], 80.00th=[ 625], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.41%, ctx=43, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=56.4MiB/s (59.2MB/s), 56.4MiB/s-56.4MiB/s (59.2MB/s-59.2MB/s), io=40.0MiB (41.9MB), run=709-709msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout