== sanity test 123ad: Verify batching statahead works correctly ========================================================== 08:20:57 (1787833257) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9f4b7b477000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f4b7b477000.batch_stats=0 llite.lustre-ffff9f4b7b477000.statahead_max=32 llite.lustre-ffff9f4b7b477000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9f4b7b477000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f4b7b477000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.32 seconds: 30.09 ops/second llite.lustre-ffff9f4b7b477000.statahead_max=0 101 real 0m2.785s user 0m0.007s sys 0m0.369s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9f4b7b477000.statahead_max=32 llite.lustre-ffff9f4b7b477000.statahead_max=32 101 real 0m1.216s user 0m0.005s sys 0m0.186s 'ls -l' 100 files with statahead: 1 sec statahead total: 115 statahead wrong: 2 agl total: 110 list_total: 31 fname_total: 77 hit_total: 4610 miss_total: 780 swrong=2 ewrong=2 - open/close 677 (time 1787833287.94 total 10.01 last 67.66) total: 900 open/close in 13.77 seconds: 65.35 ops/second llite.lustre-ffff9f4b7b477000.statahead_max=0 1001 real 0m28.089s user 0m0.030s sys 0m2.982s 'ls -l' 1000 files without statahead: 28 sec llite.lustre-ffff9f4b7b477000.statahead_max=32 llite.lustre-ffff9f4b7b477000.statahead_max=32 1001 real 0m10.468s user 0m0.024s sys 0m1.460s 'ls -l' 1000 files with statahead: 10 sec statahead total: 116 statahead wrong: 2 agl total: 111 list_total: 32 fname_total: 77 hit_total: 5608 miss_total: 782 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 29 seconds llite.lustre-ffff9f4b7b477000.statahead_stats= statahead total: 116 statahead wrong: 2 agl total: 111 list_total: 32 fname_total: 77 hit_total: 5608 miss_total: 782 mdc.lustre-MDT0000-mdc-ffff9f4b7b477000.batch_stats= snapshot_time: 1787833374.714805695 secs.nsecs start_time: 1787833263.021786058 secs.nsecs elapsed_time: 111.693019637 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 3 8 13 4: 0 0 13 8: 2 5 18 16: 23 62 81 32: 7 18 100 mdc.lustre-MDT0001-mdc-ffff9f4b7b477000.batch_stats= snapshot_time: 1787833374.714876453 secs.nsecs start_time: 1787833263.021930257 secs.nsecs elapsed_time: 111.692946196 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 1 2 5 8: 4 10 15 16: 11 28 43 32: 22 56 100 - open/close 608 (time 1787833387.42 total 10.01 last 60.77) - open/close 1230 (time 1787833397.44 total 20.02 last 62.10) - open/close 1912 (time 1787833407.44 total 30.02 last 68.19) - open/close 2590 (time 1787833417.45 total 40.03 last 67.75) - open/close 3251 (time 1787833427.45 total 50.03 last 66.08) - open/close 3845 (time 1787833437.47 total 60.05 last 59.31) - open/close 4453 (time 1787833447.47 total 70.05 last 60.79) - open/close 5236 (time 1787833457.48 total 80.06 last 78.22) - open/close 5927 (time 1787833467.49 total 90.07 last 69.05) - open/close 6601 (time 1787833477.49 total 100.07 last 67.40) - open/close 7265 (time 1787833487.50 total 110.08 last 66.33) - open/close 7896 (time 1787833497.51 total 120.09 last 63.05) - open/close 8561 (time 1787833507.51 total 130.09 last 66.49) - open/close 9304 (time 1787833517.52 total 140.10 last 74.21) - open/close 9959 (time 1787833527.53 total 150.11 last 65.44) total: 10000 open/close in 150.77 seconds: 66.33 ops/second llite.lustre-ffff9f4b7b477000.statahead_batch_max=0 llite.lustre-ffff9f4b7b477000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f4b7b477000.stats=clear mdc.lustre-MDT0001-mdc-ffff9f4b7b477000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m19.640s user 0m0.327s sys 0m14.798s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9f4b7b477000.statahead_batch_max=32 llite.lustre-ffff9f4b7b477000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f4b7b477000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9f4b7b477000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9f4b7b477000.stats=clear mdc.lustre-MDT0001-mdc-ffff9f4b7b477000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m55.796s user 0m0.312s sys 0m13.626s batched statahead RPC= hit count=9980 mdc.lustre-MDT0000-mdc-ffff9f4b7b477000.batch_stats= snapshot_time: 1787833747.045600739 secs.nsecs start_time: 1787833619.592151814 secs.nsecs elapsed_time: 127.453448925 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9f4b7b477000.batch_stats= snapshot_time: 1787833747.045680331 secs.nsecs start_time: 1787833619.592234495 secs.nsecs elapsed_time: 127.453445836 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9f4b7b477000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f4b7b477000.batch_stats=0 llite.lustre-ffff9f4b7b477000.statahead_max=2048 llite.lustre-ffff9f4b7b477000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9f4b7b477000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f4b7b477000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.09 seconds: 32.38 ops/second llite.lustre-ffff9f4b7b477000.statahead_max=0 101 real 0m2.742s user 0m0.003s sys 0m0.330s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9f4b7b477000.statahead_max=2048 llite.lustre-ffff9f4b7b477000.statahead_max=2048 101 real 0m0.964s user 0m0.007s sys 0m0.196s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10079 miss_total: 21 swrong=0 ewrong=0 - open/close 786 (time 1787834073.48 total 10.01 last 78.52) total: 900 open/close in 11.91 seconds: 75.60 ops/second llite.lustre-ffff9f4b7b477000.statahead_max=0 1001 real 0m25.622s user 0m0.036s sys 0m3.171s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff9f4b7b477000.statahead_max=2048 llite.lustre-ffff9f4b7b477000.statahead_max=2048 1001 real 0m6.391s user 0m0.021s sys 0m1.605s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11078 miss_total: 22 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff9f4b7b477000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11078 miss_total: 22 mdc.lustre-MDT0000-mdc-ffff9f4b7b477000.batch_stats= snapshot_time: 1787834141.505627341 secs.nsecs start_time: 1787833750.274530791 secs.nsecs elapsed_time: 391.231096550 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 25 25 8: 3 37 62 16: 0 0 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff9f4b7b477000.batch_stats= snapshot_time: 1787834141.505804851 secs.nsecs start_time: 1787833750.274669524 secs.nsecs elapsed_time: 391.231135327 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 25 25 8: 2 25 50 16: 1 12 62 32: 1 12 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 - open/close 790 (time 1787834154.25 total 10.00 last 79.00) - open/close 1439 (time 1787834164.28 total 20.02 last 64.75) - open/close 2166 (time 1787834174.29 total 30.04 last 72.61) - open/close 2895 (time 1787834184.30 total 40.05 last 72.80) - open/close 3685 (time 1787834194.31 total 50.05 last 78.98) - open/close 4499 (time 1787834204.31 total 60.06 last 81.37) - open/close 5263 (time 1787834214.31 total 70.06 last 76.38) - open/close 5926 (time 1787834224.34 total 80.09 last 66.11) - open/close 6636 (time 1787834234.35 total 90.09 last 70.96) - open/close 7427 (time 1787834244.35 total 100.10 last 79.08) - open/close 8169 (time 1787834254.36 total 110.11 last 74.13) - open/close 8889 (time 1787834264.36 total 120.11 last 71.99) - open/close 9580 (time 1787834274.36 total 130.11 last 69.08) total: 10000 open/close in 137.14 seconds: 72.92 ops/second llite.lustre-ffff9f4b7b477000.statahead_batch_max=0 llite.lustre-ffff9f4b7b477000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f4b7b477000.stats=clear mdc.lustre-MDT0001-mdc-ffff9f4b7b477000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m1.665s user 0m0.351s sys 0m15.972s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9f4b7b477000.statahead_batch_max=256 llite.lustre-ffff9f4b7b477000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f4b7b477000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9f4b7b477000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9f4b7b477000.stats=clear mdc.lustre-MDT0001-mdc-ffff9f4b7b477000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m48.654s user 0m0.294s sys 0m15.372s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9f4b7b477000.batch_stats= snapshot_time: 1787834411.217482352 secs.nsecs start_time: 1787834352.590975668 secs.nsecs elapsed_time: 58.626506684 secs.nsecs subreqs per batch batches %% cum %% 1: 157 48 48 2: 37 11 59 4: 41 12 72 8: 25 7 80 16: 8 2 82 32: 16 4 87 64: 4 1 88 128: 2 0 89 256: 34 10 100 mdc.lustre-MDT0001-mdc-ffff9f4b7b477000.batch_stats= snapshot_time: 1787834411.217556608 secs.nsecs start_time: 1787834352.591713140 secs.nsecs elapsed_time: 58.625843468 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9f4b7b477000.statahead_batch_max=64 llite.lustre-ffff9f4b7b477000.statahead_max=128