== sanity test 398c: run fio to test AIO ================= 09:43:56 (1786023836) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.67424 s, 62.2 MB/s osc.lustre-OST0000-osc-ffff890fc9306800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179198: Thu Aug 6 09:44:44 2026 write: IOPS=76, BW=304KiB/s (312kB/s)(10.0MiB/33640msec); 0 zone resets slat (usec): min=45, max=20465, avg=230.80, stdev=924.06 clat (msec): min=49, max=320, avg=209.32, stdev=40.15 lat (msec): min=49, max=320, avg=209.55, stdev=40.15 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 148], 10.00th=[ 167], 20.00th=[ 178], | 30.00th=[ 188], 40.00th=[ 201], 50.00th=[ 211], 60.00th=[ 220], | 70.00th=[ 228], 80.00th=[ 245], 90.00th=[ 262], 95.00th=[ 279], | 99.00th=[ 300], 99.50th=[ 309], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 199, max= 388, per=24.93%, avg=303.15, stdev=43.57, samples=66 iops : min= 49, max= 97, avg=75.41, stdev=10.94, samples=66 lat (msec) : 50=0.04%, 100=0.90%, 250=84.22%, 500=14.84% cpu : usr=0.14%, sys=0.96%, ctx=2673, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179199: Thu Aug 6 09:44:44 2026 write: IOPS=76, BW=304KiB/s (311kB/s)(10.0MiB/33677msec); 0 zone resets slat (usec): min=47, max=12806, avg=189.61, stdev=593.05 clat (msec): min=46, max=318, avg=209.51, stdev=39.73 lat (msec): min=46, max=318, avg=209.71, stdev=39.69 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 148], 10.00th=[ 167], 20.00th=[ 182], | 30.00th=[ 190], 40.00th=[ 197], 50.00th=[ 207], 60.00th=[ 218], | 70.00th=[ 228], 80.00th=[ 245], 90.00th=[ 262], 95.00th=[ 275], | 99.00th=[ 300], 99.50th=[ 305], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 163, max= 444, per=24.89%, avg=302.62, stdev=56.34, samples=66 iops : min= 40, max= 111, avg=75.29, stdev=14.15, samples=66 lat (msec) : 50=0.04%, 100=1.52%, 250=82.58%, 500=15.86% cpu : usr=0.11%, sys=0.76%, ctx=1920, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179200: Thu Aug 6 09:44:44 2026 write: IOPS=76, BW=305KiB/s (312kB/s)(10.0MiB/33563msec); 0 zone resets slat (usec): min=48, max=16066, avg=230.12, stdev=799.33 clat (msec): min=37, max=384, avg=209.30, stdev=40.20 lat (msec): min=37, max=384, avg=209.53, stdev=40.16 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 150], 10.00th=[ 165], 20.00th=[ 178], | 30.00th=[ 190], 40.00th=[ 199], 50.00th=[ 209], 60.00th=[ 218], | 70.00th=[ 230], 80.00th=[ 243], 90.00th=[ 259], 95.00th=[ 275], | 99.00th=[ 305], 99.50th=[ 321], 99.90th=[ 359], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 189, max= 396, per=25.03%, avg=304.42, stdev=49.70, samples=66 iops : min= 47, max= 99, avg=75.71, stdev=12.47, samples=66 lat (msec) : 50=0.08%, 100=1.37%, 250=83.83%, 500=14.73% cpu : usr=0.15%, sys=0.83%, ctx=2596, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179201: Thu Aug 6 09:44:44 2026 write: IOPS=76, BW=305KiB/s (312kB/s)(10.0MiB/33617msec); 0 zone resets slat (usec): min=48, max=12518, avg=224.81, stdev=696.88 clat (msec): min=15, max=326, avg=209.36, stdev=41.08 lat (msec): min=15, max=326, avg=209.59, stdev=41.05 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 146], 10.00th=[ 163], 20.00th=[ 180], | 30.00th=[ 188], 40.00th=[ 199], 50.00th=[ 209], 60.00th=[ 220], | 70.00th=[ 230], 80.00th=[ 245], 90.00th=[ 262], 95.00th=[ 275], | 99.00th=[ 305], 99.50th=[ 317], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 223, max= 412, per=24.97%, avg=303.64, stdev=43.06, samples=66 iops : min= 55, max= 103, avg=75.52, stdev=10.82, samples=66 lat (msec) : 20=0.04%, 50=0.16%, 100=1.33%, 250=81.76%, 500=16.72% cpu : usr=0.17%, sys=0.96%, ctx=2918, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1216KiB/s (1245kB/s), 304KiB/s-305KiB/s (311kB/s-312kB/s), io=40.0MiB (41.9MB), run=33563-33677msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179217: Thu Aug 6 09:45:47 2026 read: IOPS=24, BW=99.5KiB/s (102kB/s)(5048KiB/50749msec) slat (msec): min=2, max=142, avg=23.37, stdev=16.35 clat (msec): min=113, max=955, avg=297.63, stdev=100.53 lat (msec): min=117, max=982, avg=321.01, stdev=104.96 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 182], 10.00th=[ 201], 20.00th=[ 224], | 30.00th=[ 241], 40.00th=[ 255], 50.00th=[ 275], 60.00th=[ 296], | 70.00th=[ 326], 80.00th=[ 359], 90.00th=[ 426], 95.00th=[ 493], | 99.00th=[ 634], 99.50th=[ 693], 99.90th=[ 944], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 15, max= 208, per=25.05%, avg=98.71, stdev=34.81, samples=100 iops : min= 3, max= 52, avg=24.15, stdev= 8.76, samples=100 write: IOPS=25, BW=102KiB/s (105kB/s)(5192KiB/50749msec); 0 zone resets slat (usec): min=48, max=34227, avg=296.76, stdev=1485.33 clat (msec): min=131, max=824, avg=310.42, stdev=100.83 lat (msec): min=131, max=824, avg=310.72, stdev=100.87 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 194], 10.00th=[ 211], 20.00th=[ 234], | 30.00th=[ 253], 40.00th=[ 271], 50.00th=[ 288], 60.00th=[ 313], | 70.00th=[ 338], 80.00th=[ 376], 90.00th=[ 430], 95.00th=[ 481], | 99.00th=[ 709], 99.50th=[ 760], 99.90th=[ 810], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 23, max= 176, per=24.70%, avg=101.50, stdev=33.34, samples=100 iops : min= 5, max= 44, avg=24.86, stdev= 8.37, samples=100 lat (msec) : 250=32.81%, 500=62.85%, 750=3.87%, 1000=0.47% cpu : usr=0.13%, sys=1.13%, ctx=3656, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179218: Thu Aug 6 09:45:47 2026 read: IOPS=24, BW=98.0KiB/s (100kB/s)(4976KiB/50750msec) slat (usec): min=87, max=162658, avg=23841.68, stdev=17485.89 clat (msec): min=40, max=969, avg=302.60, stdev=101.44 lat (msec): min=51, max=979, avg=326.44, stdev=106.21 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 186], 10.00th=[ 205], 20.00th=[ 226], | 30.00th=[ 245], 40.00th=[ 266], 50.00th=[ 279], 60.00th=[ 305], | 70.00th=[ 334], 80.00th=[ 368], 90.00th=[ 426], 95.00th=[ 489], | 99.00th=[ 642], 99.50th=[ 726], 99.90th=[ 944], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 32, max= 191, per=24.66%, avg=97.16, stdev=32.94, samples=100 iops : min= 8, max= 47, avg=23.76, stdev= 8.27, samples=100 write: IOPS=25, BW=104KiB/s (106kB/s)(5264KiB/50750msec); 0 zone resets slat (usec): min=49, max=21055, avg=246.93, stdev=1155.46 clat (msec): min=21, max=989, avg=307.75, stdev=99.36 lat (msec): min=21, max=989, avg=308.00, stdev=99.36 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 197], 10.00th=[ 211], 20.00th=[ 234], | 30.00th=[ 251], 40.00th=[ 271], 50.00th=[ 284], 60.00th=[ 305], | 70.00th=[ 334], 80.00th=[ 380], 90.00th=[ 430], 95.00th=[ 489], | 99.00th=[ 659], 99.50th=[ 718], 99.90th=[ 986], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 15, max= 182, per=25.11%, avg=103.21, stdev=33.54, samples=100 iops : min= 3, max= 45, avg=25.28, stdev= 8.44, samples=100 lat (msec) : 50=0.08%, 100=0.23%, 250=31.21%, 500=64.22%, 750=3.87% lat (msec) : 1000=0.39% cpu : usr=0.16%, sys=1.13%, ctx=3834, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179219: Thu Aug 6 09:45:47 2026 read: IOPS=24, BW=98.1KiB/s (100kB/s)(4984KiB/50810msec) slat (msec): min=2, max=162, avg=23.79, stdev=16.73 clat (msec): min=46, max=1021, avg=301.77, stdev=103.71 lat (msec): min=65, max=1032, avg=325.57, stdev=108.32 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 184], 10.00th=[ 203], 20.00th=[ 226], | 30.00th=[ 243], 40.00th=[ 257], 50.00th=[ 275], 60.00th=[ 300], | 70.00th=[ 334], 80.00th=[ 368], 90.00th=[ 435], 95.00th=[ 489], | 99.00th=[ 676], 99.50th=[ 743], 99.90th=[ 1003], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 31, max= 184, per=24.66%, avg=97.18, stdev=34.33, samples=100 iops : min= 7, max= 46, avg=23.77, stdev= 8.61, samples=100 write: IOPS=25, BW=103KiB/s (106kB/s)(5256KiB/50810msec); 0 zone resets slat (usec): min=51, max=24201, avg=285.41, stdev=1304.45 clat (msec): min=42, max=903, avg=306.93, stdev=97.20 lat (msec): min=43, max=903, avg=307.22, stdev=97.18 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 192], 10.00th=[ 209], 20.00th=[ 234], | 30.00th=[ 249], 40.00th=[ 266], 50.00th=[ 288], 60.00th=[ 309], | 70.00th=[ 334], 80.00th=[ 372], 90.00th=[ 430], 95.00th=[ 485], | 99.00th=[ 667], 99.50th=[ 726], 99.90th=[ 827], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 23, max= 199, per=24.82%, avg=102.01, stdev=35.63, samples=100 iops : min= 5, max= 49, avg=24.97, stdev= 8.97, samples=100 lat (msec) : 50=0.12%, 100=0.12%, 250=32.70%, 500=63.16%, 750=3.52% lat (msec) : 1000=0.31%, 2000=0.08% cpu : usr=0.16%, sys=1.17%, ctx=3750, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179220: Thu Aug 6 09:45:47 2026 read: IOPS=24, BW=99.9KiB/s (102kB/s)(5060KiB/50631msec) slat (msec): min=2, max=124, avg=23.16, stdev=15.43 clat (msec): min=112, max=904, avg=297.99, stdev=96.48 lat (msec): min=121, max=923, avg=321.16, stdev=100.36 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 184], 10.00th=[ 205], 20.00th=[ 224], | 30.00th=[ 243], 40.00th=[ 262], 50.00th=[ 279], 60.00th=[ 300], | 70.00th=[ 326], 80.00th=[ 363], 90.00th=[ 426], 95.00th=[ 477], | 99.00th=[ 617], 99.50th=[ 659], 99.90th=[ 877], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 38, max= 199, per=25.22%, avg=99.38, stdev=32.78, samples=100 iops : min= 9, max= 49, avg=24.32, stdev= 8.26, samples=100 write: IOPS=25, BW=102KiB/s (105kB/s)(5180KiB/50631msec); 0 zone resets slat (usec): min=47, max=19372, avg=300.14, stdev=1344.40 clat (msec): min=133, max=931, avg=309.24, stdev=100.32 lat (msec): min=134, max=931, avg=309.55, stdev=100.44 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 194], 10.00th=[ 209], 20.00th=[ 232], | 30.00th=[ 251], 40.00th=[ 271], 50.00th=[ 288], 60.00th=[ 309], | 70.00th=[ 338], 80.00th=[ 376], 90.00th=[ 426], 95.00th=[ 468], | 99.00th=[ 684], 99.50th=[ 860], 99.90th=[ 919], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 31, max= 188, per=24.71%, avg=101.57, stdev=33.23, samples=100 iops : min= 7, max= 47, avg=24.88, stdev= 8.35, samples=100 lat (msec) : 250=32.07%, 500=64.22%, 750=3.20%, 1000=0.51% cpu : usr=0.16%, sys=1.15%, ctx=4177, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=395KiB/s (404kB/s), 98.0KiB/s-99.9KiB/s (100kB/s-102kB/s), io=19.6MiB (20.5MB), run=50631-50810msec WRITE: bw=411KiB/s (421kB/s), 102KiB/s-104KiB/s (105kB/s-106kB/s), io=20.4MiB (21.4MB), run=50631-50810msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179233: Thu Aug 6 09:46:01 2026 read: IOPS=1, BW=48.7MiB/s (51.1MB/s)(40.0MiB/821msec) slat (nsec): min=41181k, max=41181k, avg=41180870.00, stdev= 0.00 clat (nsec): min=753573k, max=753573k, avg=753572646.00, stdev= 0.00 lat (nsec): min=794758k, max=794758k, avg=794758044.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 751], 5.00th=[ 751], 10.00th=[ 751], 20.00th=[ 751], | 30.00th=[ 751], 40.00th=[ 751], 50.00th=[ 751], 60.00th=[ 751], | 70.00th=[ 751], 80.00th=[ 751], 90.00th=[ 751], 95.00th=[ 751], | 99.00th=[ 751], 99.50th=[ 751], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] lat (msec) : 1000=100.00% cpu : usr=0.12%, sys=1.22%, ctx=12, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=48.7MiB/s (51.1MB/s), 48.7MiB/s-48.7MiB/s (51.1MB/s-51.1MB/s), io=40.0MiB (41.9MB), run=821-821msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout