== sanity test 123ad: Verify batching statahead works correctly ========================================================== 12:07:35 (1783958855) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8acc988d0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8acc988d0800.batch_stats=0 llite.lustre-ffff8acc988d0800.statahead_max=32 llite.lustre-ffff8acc988d0800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8acc988d0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8acc988d0800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 1.88 seconds: 53.11 ops/second llite.lustre-ffff8acc988d0800.statahead_max=0 101 real 0m2.220s user 0m0.005s sys 0m0.334s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8acc988d0800.statahead_max=32 llite.lustre-ffff8acc988d0800.statahead_max=32 101 real 0m1.025s user 0m0.003s sys 0m0.161s 'ls -l' 100 files with statahead: 1 sec statahead total: 137 statahead wrong: 2 agl total: 131 list_total: 32 fname_total: 98 hit_total: 4011 miss_total: 1397 swrong=2 ewrong=2 total: 900 open/close in 9.78 seconds: 92.01 ops/second llite.lustre-ffff8acc988d0800.statahead_max=0 1001 real 0m21.024s user 0m0.028s sys 0m3.181s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff8acc988d0800.statahead_max=32 llite.lustre-ffff8acc988d0800.statahead_max=32 1001 real 0m10.658s user 0m0.028s sys 0m1.577s 'ls -l' 1000 files with statahead: 11 sec statahead total: 138 statahead wrong: 2 agl total: 132 list_total: 33 fname_total: 98 hit_total: 5007 miss_total: 1401 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 17 seconds llite.lustre-ffff8acc988d0800.statahead_stats= statahead total: 138 statahead wrong: 2 agl total: 132 list_total: 33 fname_total: 98 hit_total: 5007 miss_total: 1401 mdc.lustre-MDT0000-mdc-ffff8acc988d0800.batch_stats= snapshot_time: 1783958942.418953262 secs.nsecs start_time: 1783958860.395994783 secs.nsecs elapsed_time: 82.022958479 secs.nsecs subreqs per batch batches %% cum %% 1: 3 7 7 2: 2 5 13 4: 1 2 15 8: 1 2 18 16: 27 71 89 32: 4 10 100 mdc.lustre-MDT0001-mdc-ffff8acc988d0800.batch_stats= snapshot_time: 1783958942.419358770 secs.nsecs start_time: 1783958860.396148534 secs.nsecs elapsed_time: 82.023210236 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 5 5 8: 3 7 12 16: 11 28 41 32: 23 58 100 - open/close 1087 (time 1783958954.87 total 10.00 last 108.70) - open/close 2006 (time 1783958964.88 total 20.01 last 91.79) - open/close 2844 (time 1783958974.91 total 30.04 last 83.60) - open/close 3800 (time 1783958984.92 total 40.04 last 95.52) - open/close 4731 (time 1783958994.92 total 50.05 last 93.07) - open/close 5668 (time 1783959004.93 total 60.06 last 93.62) - open/close 6344 (time 1783959014.95 total 70.08 last 67.44) - open/close 7281 (time 1783959024.96 total 80.08 last 93.66) - open/close 8168 (time 1783959034.98 total 90.11 last 88.51) - open/close 9034 (time 1783959044.99 total 100.11 last 86.53) - open/close 9898 (time 1783959054.99 total 110.11 last 86.40) total: 10000 open/close in 111.27 seconds: 89.87 ops/second llite.lustre-ffff8acc988d0800.statahead_batch_max=0 llite.lustre-ffff8acc988d0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8acc988d0800.stats=clear mdc.lustre-MDT0001-mdc-ffff8acc988d0800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m0.647s user 0m0.323s sys 0m15.994s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8acc988d0800.statahead_batch_max=32 llite.lustre-ffff8acc988d0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8acc988d0800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8acc988d0800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8acc988d0800.stats=clear mdc.lustre-MDT0001-mdc-ffff8acc988d0800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m47.785s user 0m0.328s sys 0m15.398s batched statahead RPC= hit count=9985 mdc.lustre-MDT0000-mdc-ffff8acc988d0800.batch_stats= snapshot_time: 1783959251.125221042 secs.nsecs start_time: 1783959127.805962377 secs.nsecs elapsed_time: 123.319258665 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8acc988d0800.batch_stats= snapshot_time: 1783959251.125285545 secs.nsecs start_time: 1783959127.806101647 secs.nsecs elapsed_time: 123.319183898 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8acc988d0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8acc988d0800.batch_stats=0 llite.lustre-ffff8acc988d0800.statahead_max=2048 llite.lustre-ffff8acc988d0800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8acc988d0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8acc988d0800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 1.92 seconds: 52.07 ops/second llite.lustre-ffff8acc988d0800.statahead_max=0 101 real 0m1.998s user 0m0.004s sys 0m0.337s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8acc988d0800.statahead_max=2048 llite.lustre-ffff8acc988d0800.statahead_max=2048 101 real 0m0.985s user 0m0.004s sys 0m0.194s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10084 miss_total: 16 swrong=0 ewrong=0 - open/close 888 (time 1783959548.33 total 10.00 last 88.80) total: 900 open/close in 10.13 seconds: 88.85 ops/second llite.lustre-ffff8acc988d0800.statahead_max=0 1001 real 0m18.149s user 0m0.032s sys 0m3.291s 'ls -l' 1000 files without statahead: 18 sec llite.lustre-ffff8acc988d0800.statahead_max=2048 llite.lustre-ffff8acc988d0800.statahead_max=2048 1001 real 0m7.666s user 0m0.030s sys 0m1.672s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11083 miss_total: 17 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff8acc988d0800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11083 miss_total: 17 mdc.lustre-MDT0000-mdc-ffff8acc988d0800.batch_stats= snapshot_time: 1783959606.124519783 secs.nsecs start_time: 1783959253.732662778 secs.nsecs elapsed_time: 352.391857005 secs.nsecs subreqs per batch batches %% cum %% 1: 1 16 16 2: 0 0 16 4: 2 33 50 8: 1 16 66 16: 0 0 66 32: 0 0 66 64: 0 0 66 128: 0 0 66 256: 2 33 100 mdc.lustre-MDT0001-mdc-ffff8acc988d0800.batch_stats= snapshot_time: 1783959606.124592777 secs.nsecs start_time: 1783959253.732813714 secs.nsecs elapsed_time: 352.391779063 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 22 22 8: 1 11 33 16: 2 22 55 32: 0 0 55 64: 2 22 77 128: 0 0 77 256: 2 22 100 - open/close 773 (time 1783959619.33 total 10.00 last 77.27) - open/close 1632 (time 1783959629.34 total 20.01 last 85.87) - open/close 2517 (time 1783959639.34 total 30.01 last 88.48) - open/close 3286 (time 1783959649.34 total 40.01 last 76.89) - open/close 4133 (time 1783959659.34 total 50.01 last 84.69) - open/close 4931 (time 1783959669.34 total 60.01 last 79.79) - open/close 5665 (time 1783959679.35 total 70.02 last 73.37) - open/close 6410 (time 1783959689.35 total 80.03 last 74.45) - open/close 7188 (time 1783959699.36 total 90.04 last 77.72) - open/close 8051 (time 1783959709.38 total 100.05 last 86.18) - open/close 8899 (time 1783959719.38 total 110.05 last 84.80) - open/close 9712 (time 1783959729.38 total 120.05 last 81.27) total: 10000 open/close in 123.48 seconds: 80.98 ops/second llite.lustre-ffff8acc988d0800.statahead_batch_max=0 llite.lustre-ffff8acc988d0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8acc988d0800.stats=clear mdc.lustre-MDT0001-mdc-ffff8acc988d0800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m5.855s user 0m0.331s sys 0m16.619s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8acc988d0800.statahead_batch_max=256 llite.lustre-ffff8acc988d0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8acc988d0800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8acc988d0800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8acc988d0800.stats=clear mdc.lustre-MDT0001-mdc-ffff8acc988d0800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m3.066s user 0m0.308s sys 0m16.850s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8acc988d0800.batch_stats= snapshot_time: 1783959886.709202975 secs.nsecs start_time: 1783959809.693985920 secs.nsecs elapsed_time: 77.015217055 secs.nsecs subreqs per batch batches %% cum %% 1: 213 56 56 2: 49 12 69 4: 29 7 76 8: 24 6 83 16: 9 2 85 32: 13 3 89 64: 6 1 90 128: 1 0 91 256: 34 8 100 mdc.lustre-MDT0001-mdc-ffff8acc988d0800.batch_stats= snapshot_time: 1783959886.709276145 secs.nsecs start_time: 1783959809.694051711 secs.nsecs elapsed_time: 77.015224434 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8acc988d0800.statahead_batch_max=64 llite.lustre-ffff8acc988d0800.statahead_max=128