== sanity test 123ad: Verify batching statahead works correctly ========================================================== 12:13:41 (1787847221) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8a3788647000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a3788647000.batch_stats=0 llite.lustre-ffff8a3788647000.statahead_max=32 llite.lustre-ffff8a3788647000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8a3788647000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a3788647000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.30 seconds: 43.41 ops/second llite.lustre-ffff8a3788647000.statahead_max=0 101 real 0m1.660s user 0m0.004s sys 0m0.337s 'ls -l' 100 files without statahead: 1 sec llite.lustre-ffff8a3788647000.statahead_max=32 llite.lustre-ffff8a3788647000.statahead_max=32 101 real 0m1.357s user 0m0.007s sys 0m0.199s 'ls -l' 100 files with statahead: 1 sec statahead total: 150 statahead wrong: 2 agl total: 144 list_total: 32 fname_total: 117 hit_total: 3735 miss_total: 1494 swrong=2 ewrong=2 - open/close 726 (time 1787847249.04 total 10.00 last 72.58) total: 900 open/close in 12.77 seconds: 70.46 ops/second llite.lustre-ffff8a3788647000.statahead_max=0 1001 real 0m30.171s user 0m0.039s sys 0m3.098s 'ls -l' 1000 files without statahead: 30 sec llite.lustre-ffff8a3788647000.statahead_max=32 llite.lustre-ffff8a3788647000.statahead_max=32 1001 real 0m9.649s user 0m0.024s sys 0m1.405s 'ls -l' 1000 files with statahead: 9 sec statahead total: 151 statahead wrong: 2 agl total: 145 list_total: 33 fname_total: 117 hit_total: 4734 miss_total: 1495 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff8a3788647000.statahead_stats= statahead total: 151 statahead wrong: 2 agl total: 145 list_total: 33 fname_total: 117 hit_total: 4734 miss_total: 1495 mdc.lustre-MDT0000-mdc-ffff8a3788647000.batch_stats= snapshot_time: 1787847326.517984874 secs.nsecs start_time: 1787847226.222205067 secs.nsecs elapsed_time: 100.295779807 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 5 5 8: 2 5 10 16: 21 55 65 32: 13 34 100 mdc.lustre-MDT0001-mdc-ffff8a3788647000.batch_stats= snapshot_time: 1787847326.518066091 secs.nsecs start_time: 1787847226.222395934 secs.nsecs elapsed_time: 100.295670157 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 2 5 7 8: 1 2 10 16: 20 52 63 32: 14 36 100 - open/close 809 (time 1787847339.59 total 10.00 last 80.89) - open/close 1662 (time 1787847349.59 total 20.01 last 85.24) - open/close 2556 (time 1787847359.60 total 30.01 last 89.35) - open/close 3489 (time 1787847369.61 total 40.03 last 93.20) - open/close 4283 (time 1787847379.62 total 50.04 last 79.32) - open/close 5076 (time 1787847389.62 total 60.04 last 79.27) - open/close 5846 (time 1787847399.62 total 70.04 last 77.00) - open/close 6585 (time 1787847409.63 total 80.05 last 73.85) - open/close 7302 (time 1787847419.64 total 90.05 last 71.66) - open/close 7997 (time 1787847429.64 total 100.06 last 69.45) - open/close 8794 (time 1787847439.66 total 110.07 last 79.61) - open/close 9664 (time 1787847449.67 total 120.08 last 86.90) total: 10000 open/close in 124.27 seconds: 80.47 ops/second llite.lustre-ffff8a3788647000.statahead_batch_max=0 llite.lustre-ffff8a3788647000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a3788647000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a3788647000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m59.456s user 0m0.333s sys 0m15.651s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8a3788647000.statahead_batch_max=32 llite.lustre-ffff8a3788647000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a3788647000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8a3788647000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8a3788647000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a3788647000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m38.290s user 0m0.302s sys 0m14.306s batched statahead RPC= hit count=9985 mdc.lustre-MDT0000-mdc-ffff8a3788647000.batch_stats= snapshot_time: 1787847630.228018103 secs.nsecs start_time: 1787847521.202334300 secs.nsecs elapsed_time: 109.025683803 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8a3788647000.batch_stats= snapshot_time: 1787847630.228418768 secs.nsecs start_time: 1787847521.202489795 secs.nsecs elapsed_time: 109.025928973 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8a3788647000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a3788647000.batch_stats=0 llite.lustre-ffff8a3788647000.statahead_max=2048 llite.lustre-ffff8a3788647000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8a3788647000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a3788647000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.97 seconds: 33.62 ops/second llite.lustre-ffff8a3788647000.statahead_max=0 101 real 0m2.586s user 0m0.005s sys 0m0.295s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8a3788647000.statahead_max=2048 llite.lustre-ffff8a3788647000.statahead_max=2048 101 real 0m0.915s user 0m0.001s sys 0m0.164s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10084 miss_total: 16 swrong=0 ewrong=0 - open/close 567 (time 1787847927.27 total 10.01 last 56.67) total: 900 open/close in 15.13 seconds: 59.50 ops/second llite.lustre-ffff8a3788647000.statahead_max=0 1001 real 0m23.979s user 0m0.033s sys 0m3.028s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff8a3788647000.statahead_max=2048 llite.lustre-ffff8a3788647000.statahead_max=2048 1001 real 0m8.299s user 0m0.035s sys 0m1.308s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11083 miss_total: 17 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 23 seconds llite.lustre-ffff8a3788647000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11083 miss_total: 17 mdc.lustre-MDT0000-mdc-ffff8a3788647000.batch_stats= snapshot_time: 1787848002.987941220 secs.nsecs start_time: 1787847633.008801421 secs.nsecs elapsed_time: 369.979139799 secs.nsecs subreqs per batch batches %% cum %% 1: 1 16 16 2: 0 0 16 4: 3 50 66 8: 0 0 66 16: 0 0 66 32: 0 0 66 64: 0 0 66 128: 0 0 66 256: 2 33 100 mdc.lustre-MDT0001-mdc-ffff8a3788647000.batch_stats= snapshot_time: 1787848002.988018497 secs.nsecs start_time: 1787847633.010273749 secs.nsecs elapsed_time: 369.977744748 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 3 37 37 16: 1 12 50 32: 0 0 50 64: 1 12 62 128: 1 12 75 256: 2 25 100 - open/close 772 (time 1787848016.37 total 10.01 last 77.16) - open/close 1477 (time 1787848026.38 total 20.02 last 70.41) - open/close 2250 (time 1787848036.38 total 30.02 last 77.26) - open/close 3058 (time 1787848046.38 total 40.02 last 80.80) - open/close 3635 (time 1787848056.42 total 50.05 last 57.52) - open/close 4206 (time 1787848066.45 total 60.08 last 56.94) - open/close 4826 (time 1787848076.47 total 70.11 last 61.82) - open/close 5394 (time 1787848086.48 total 80.12 last 56.78) - open/close 6044 (time 1787848096.49 total 90.12 last 64.95) - open/close 6745 (time 1787848106.49 total 100.13 last 70.04) - open/close 7398 (time 1787848116.50 total 110.14 last 65.25) - open/close 7894 (time 1787848126.51 total 120.15 last 49.56) - open/close 8570 (time 1787848136.52 total 130.16 last 67.54) - open/close 9274 (time 1787848146.52 total 140.16 last 70.38) total: 10000 open/close in 149.82 seconds: 66.75 ops/second llite.lustre-ffff8a3788647000.statahead_batch_max=0 llite.lustre-ffff8a3788647000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a3788647000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a3788647000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m12.105s user 0m0.418s sys 0m16.086s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8a3788647000.statahead_batch_max=256 llite.lustre-ffff8a3788647000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a3788647000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8a3788647000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8a3788647000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a3788647000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m7.581s user 0m0.338s sys 0m14.876s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8a3788647000.batch_stats= snapshot_time: 1787848322.478534571 secs.nsecs start_time: 1787848240.076002554 secs.nsecs elapsed_time: 82.402532017 secs.nsecs subreqs per batch batches %% cum %% 1: 84 40 40 2: 31 14 54 4: 16 7 62 8: 11 5 67 16: 11 5 72 32: 15 7 80 64: 7 3 83 128: 0 0 83 256: 35 16 100 mdc.lustre-MDT0001-mdc-ffff8a3788647000.batch_stats= snapshot_time: 1787848322.478613441 secs.nsecs start_time: 1787848240.076811109 secs.nsecs elapsed_time: 82.401802332 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8a3788647000.statahead_batch_max=64 llite.lustre-ffff8a3788647000.statahead_max=128