== sanity test 398c: run fio to test AIO ================= 10:51:46 (1721227906) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.150221 s, 279 MB/s osc.lustre-OST0000-osc-ffff88012d822800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=27396: Wed Jul 17 10:52:17 2024 write: IOPS=87, BW=349KiB/s (357kB/s)(10.0MiB/29371msec) slat (usec): min=11, max=315, avg=47.02, stdev=27.44 clat (msec): min=13, max=351, avg=183.44, stdev=35.04 lat (msec): min=13, max=351, avg=183.49, stdev=35.04 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 126], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 184], 60.00th=[ 192], | 70.00th=[ 203], 80.00th=[ 211], 90.00th=[ 226], 95.00th=[ 236], | 99.00th=[ 262], 99.50th=[ 275], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 351] bw ( KiB/s): min= 256, max= 560, per=25.08%, avg=348.66, stdev=52.98, samples=58 iops : min= 64, max= 140, avg=87.14, stdev=13.25, samples=58 lat (msec) : 20=0.04%, 50=0.04%, 100=1.56%, 250=96.25%, 500=2.11% cpu : usr=0.08%, sys=0.58%, ctx=1804, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=27397: Wed Jul 17 10:52:17 2024 write: IOPS=86, BW=348KiB/s (356kB/s)(10.0MiB/29445msec) slat (usec): min=11, max=313, avg=49.31, stdev=30.37 clat (msec): min=65, max=299, avg=183.86, stdev=33.02 lat (msec): min=65, max=299, avg=183.91, stdev=33.02 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 131], 10.00th=[ 144], 20.00th=[ 159], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 186], 60.00th=[ 194], | 70.00th=[ 201], 80.00th=[ 211], 90.00th=[ 226], 95.00th=[ 236], | 99.00th=[ 257], 99.50th=[ 271], 99.90th=[ 288], 99.95th=[ 296], | 99.99th=[ 300] bw ( KiB/s): min= 248, max= 512, per=24.96%, avg=346.90, stdev=46.89, samples=58 iops : min= 62, max= 128, avg=86.62, stdev=11.77, samples=58 lat (msec) : 100=1.29%, 250=96.91%, 500=1.80% cpu : usr=0.10%, sys=0.55%, ctx=2011, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=27398: Wed Jul 17 10:52:17 2024 write: IOPS=86, BW=348KiB/s (356kB/s)(10.0MiB/29455msec) slat (usec): min=12, max=386, avg=49.01, stdev=32.51 clat (msec): min=73, max=285, avg=183.93, stdev=33.39 lat (msec): min=73, max=285, avg=183.98, stdev=33.39 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 128], 10.00th=[ 142], 20.00th=[ 159], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 186], 60.00th=[ 194], | 70.00th=[ 203], 80.00th=[ 211], 90.00th=[ 226], 95.00th=[ 234], | 99.00th=[ 262], 99.50th=[ 275], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 288] bw ( KiB/s): min= 248, max= 520, per=24.95%, avg=346.86, stdev=45.68, samples=58 iops : min= 62, max= 130, avg=86.69, stdev=11.41, samples=58 lat (msec) : 100=1.29%, 250=96.56%, 500=2.15% cpu : usr=0.12%, sys=0.54%, ctx=1871, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=27399: Wed Jul 17 10:52:17 2024 write: IOPS=86, BW=348KiB/s (356kB/s)(10.0MiB/29434msec) slat (usec): min=11, max=1639, avg=48.59, stdev=43.06 clat (msec): min=72, max=338, avg=183.80, stdev=33.57 lat (msec): min=73, max=338, avg=183.85, stdev=33.57 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 128], 10.00th=[ 144], 20.00th=[ 159], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 184], 60.00th=[ 192], | 70.00th=[ 203], 80.00th=[ 211], 90.00th=[ 224], 95.00th=[ 234], | 99.00th=[ 264], 99.50th=[ 275], 99.90th=[ 309], 99.95th=[ 321], | 99.99th=[ 338] bw ( KiB/s): min= 256, max= 512, per=25.03%, avg=347.86, stdev=45.36, samples=58 iops : min= 64, max= 128, avg=86.97, stdev=11.34, samples=58 lat (msec) : 100=1.05%, 250=96.68%, 500=2.27% cpu : usr=0.11%, sys=0.54%, ctx=1908, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1391KiB/s (1424kB/s), 348KiB/s-349KiB/s (356kB/s-357kB/s), io=40.0MiB (41.9MB), run=29371-29455msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27433: Wed Jul 17 10:52:37 2024 read: IOPS=67, BW=269KiB/s (275kB/s)(5048KiB/18788msec) slat (usec): min=460, max=2485, avg=872.19, stdev=211.36 clat (usec): min=1133, max=165791, avg=78620.65, stdev=16484.17 lat (usec): min=1789, max=167139, avg=79494.62, stdev=16536.98 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 56], 10.00th=[ 62], 20.00th=[ 67], | 30.00th=[ 70], 40.00th=[ 73], 50.00th=[ 79], 60.00th=[ 82], | 70.00th=[ 86], 80.00th=[ 90], 90.00th=[ 99], 95.00th=[ 104], | 99.00th=[ 138], 99.50th=[ 159], 99.90th=[ 165], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 192, max= 376, per=25.21%, avg=268.51, stdev=43.80, samples=37 iops : min= 48, max= 94, avg=67.11, stdev=10.95, samples=37 write: IOPS=69, BW=276KiB/s (283kB/s)(5192KiB/18788msec) slat (usec): min=12, max=1157, avg=43.28, stdev=36.33 clat (msec): min=66, max=269, avg=153.94, stdev=35.38 lat (msec): min=66, max=269, avg=153.98, stdev=35.38 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 101], 10.00th=[ 109], 20.00th=[ 123], | 30.00th=[ 134], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 161], | 70.00th=[ 171], 80.00th=[ 184], 90.00th=[ 203], 95.00th=[ 218], | 99.00th=[ 243], 99.50th=[ 249], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 184, max= 376, per=24.90%, avg=275.86, stdev=43.55, samples=37 iops : min= 46, max= 94, avg=68.95, stdev=10.91, samples=37 lat (msec) : 2=0.04%, 20=0.16%, 50=0.86%, 100=46.76%, 250=51.95% lat (msec) : 500=0.23% cpu : usr=0.16%, sys=1.11%, ctx=3159, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27434: Wed Jul 17 10:52:37 2024 read: IOPS=66, BW=264KiB/s (271kB/s)(4976KiB/18817msec) slat (usec): min=23, max=2483, avg=868.53, stdev=208.65 clat (usec): min=1347, max=168791, avg=78406.33, stdev=16427.48 lat (usec): min=1804, max=169863, avg=79276.60, stdev=16466.88 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 56], 10.00th=[ 63], 20.00th=[ 67], | 30.00th=[ 71], 40.00th=[ 74], 50.00th=[ 80], 60.00th=[ 83], | 70.00th=[ 86], 80.00th=[ 90], 90.00th=[ 96], 95.00th=[ 103], | 99.00th=[ 123], 99.50th=[ 133], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 152, max= 384, per=24.76%, avg=263.68, stdev=53.17, samples=37 iops : min= 38, max= 96, avg=65.84, stdev=13.32, samples=37 write: IOPS=69, BW=280KiB/s (286kB/s)(5264KiB/18817msec) slat (usec): min=12, max=184, avg=41.26, stdev=17.95 clat (msec): min=32, max=272, avg=153.68, stdev=35.71 lat (msec): min=32, max=272, avg=153.72, stdev=35.71 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 101], 10.00th=[ 110], 20.00th=[ 123], | 30.00th=[ 133], 40.00th=[ 144], 50.00th=[ 150], 60.00th=[ 161], | 70.00th=[ 171], 80.00th=[ 184], 90.00th=[ 203], 95.00th=[ 215], | 99.00th=[ 239], 99.50th=[ 247], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 200, max= 408, per=25.18%, avg=279.03, stdev=41.11, samples=37 iops : min= 50, max= 102, avg=69.68, stdev=10.30, samples=37 lat (msec) : 2=0.08%, 4=0.12%, 20=0.39%, 50=0.82%, 100=46.60% lat (msec) : 250=51.76%, 500=0.23% cpu : usr=0.19%, sys=1.10%, ctx=2766, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27435: Wed Jul 17 10:52:37 2024 read: IOPS=66, BW=265KiB/s (271kB/s)(4984KiB/18842msec) slat (usec): min=426, max=2478, avg=873.40, stdev=211.93 clat (usec): min=983, max=171470, avg=77880.84, stdev=15949.34 lat (usec): min=1712, max=172971, avg=78755.97, stdev=16000.47 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 55], 10.00th=[ 62], 20.00th=[ 67], | 30.00th=[ 70], 40.00th=[ 74], 50.00th=[ 79], 60.00th=[ 82], | 70.00th=[ 86], 80.00th=[ 90], 90.00th=[ 95], 95.00th=[ 101], | 99.00th=[ 117], 99.50th=[ 165], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 144, max= 408, per=24.79%, avg=264.00, stdev=62.99, samples=37 iops : min= 36, max= 102, avg=66.00, stdev=15.75, samples=37 write: IOPS=69, BW=279KiB/s (286kB/s)(5256KiB/18842msec) slat (usec): min=12, max=174, avg=42.29, stdev=18.94 clat (msec): min=25, max=273, avg=154.53, stdev=36.33 lat (msec): min=25, max=273, avg=154.58, stdev=36.33 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 100], 10.00th=[ 111], 20.00th=[ 125], | 30.00th=[ 134], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 161], | 70.00th=[ 174], 80.00th=[ 184], 90.00th=[ 203], 95.00th=[ 220], | 99.00th=[ 243], 99.50th=[ 249], 99.90th=[ 271], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 184, max= 344, per=24.90%, avg=275.89, stdev=42.94, samples=37 iops : min= 46, max= 86, avg=68.97, stdev=10.74, samples=37 lat (usec) : 1000=0.04% lat (msec) : 20=0.23%, 50=1.21%, 100=47.66%, 250=50.62%, 500=0.23% cpu : usr=0.13%, sys=1.16%, ctx=2909, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27436: Wed Jul 17 10:52:37 2024 read: IOPS=67, BW=270KiB/s (276kB/s)(5060KiB/18775msec) slat (usec): min=457, max=2569, avg=880.55, stdev=219.32 clat (usec): min=915, max=167773, avg=78175.28, stdev=17031.31 lat (usec): min=1710, max=169518, avg=79057.61, stdev=17080.89 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[ 67], | 30.00th=[ 70], 40.00th=[ 74], 50.00th=[ 79], 60.00th=[ 82], | 70.00th=[ 86], 80.00th=[ 90], 90.00th=[ 96], 95.00th=[ 104], | 99.00th=[ 126], 99.50th=[ 161], 99.90th=[ 167], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 183, max= 360, per=25.37%, avg=270.16, stdev=46.61, samples=37 iops : min= 45, max= 90, avg=67.46, stdev=11.68, samples=37 write: IOPS=68, BW=276KiB/s (283kB/s)(5180KiB/18775msec) slat (usec): min=12, max=911, avg=42.88, stdev=40.19 clat (msec): min=74, max=272, avg=154.41, stdev=35.58 lat (msec): min=74, max=272, avg=154.45, stdev=35.58 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 101], 10.00th=[ 111], 20.00th=[ 122], | 30.00th=[ 133], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 161], | 70.00th=[ 174], 80.00th=[ 184], 90.00th=[ 205], 95.00th=[ 218], | 99.00th=[ 239], 99.50th=[ 251], 99.90th=[ 271], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 192, max= 368, per=24.77%, avg=274.49, stdev=36.71, samples=37 iops : min= 48, max= 92, avg=68.54, stdev= 9.19, samples=37 lat (usec) : 1000=0.04% lat (msec) : 10=0.04%, 20=0.47%, 50=0.62%, 100=47.34%, 250=51.21% lat (msec) : 500=0.27% cpu : usr=0.15%, sys=1.13%, ctx=2801, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1065KiB/s (1091kB/s), 264KiB/s-270KiB/s (271kB/s-276kB/s), io=19.6MiB (20.5MB), run=18775-18842msec WRITE: bw=1109KiB/s (1135kB/s), 276KiB/s-280KiB/s (283kB/s-286kB/s), io=20.4MiB (21.4MB), run=18775-18842msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=27454: Wed Jul 17 10:52:38 2024 read: IOPS=8, BW=328MiB/s (344MB/s)(40.0MiB/122msec) slat (nsec): min=6835.0k, max=6835.0k, avg=6835048.00, stdev= 0.00 clat (nsec): min=109996k, max=109996k, avg=109995587.00, stdev= 0.00 lat (nsec): min=116840k, max=116840k, avg=116839599.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 110], 10.00th=[ 110], 20.00th=[ 110], | 30.00th=[ 110], 40.00th=[ 110], 50.00th=[ 110], 60.00th=[ 110], | 70.00th=[ 110], 80.00th=[ 110], 90.00th=[ 110], 95.00th=[ 110], | 99.00th=[ 110], 99.50th=[ 110], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.96%, ctx=16, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=328MiB/s (344MB/s), 328MiB/s-328MiB/s (344MB/s-344MB/s), io=40.0MiB (41.9MB), run=122-122msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck