== sanity test 123ad: Verify batching statahead works correctly ========================================================== 16:55:08 (1788468908) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff944e36cb1000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff944e36cb1000.batch_stats=0 llite.lustre-ffff944e36cb1000.statahead_max=32 llite.lustre-ffff944e36cb1000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff944e36cb1000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff944e36cb1000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.78 seconds: 35.94 ops/second llite.lustre-ffff944e36cb1000.statahead_max=0 101 real 0m3.170s user 0m0.006s sys 0m0.318s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff944e36cb1000.statahead_max=32 llite.lustre-ffff944e36cb1000.statahead_max=32 101 real 0m0.857s user 0m0.006s sys 0m0.151s 'ls -l' 100 files with statahead: 1 sec statahead total: 159 statahead wrong: 2 agl total: 153 list_total: 31 fname_total: 128 hit_total: 3600 miss_total: 1229 swrong=2 ewrong=2 - open/close 833 (time 1788468937.60 total 10.01 last 83.25) total: 900 open/close in 10.95 seconds: 82.17 ops/second llite.lustre-ffff944e36cb1000.statahead_max=0 1001 real 0m18.311s user 0m0.023s sys 0m2.814s 'ls -l' 1000 files without statahead: 19 sec llite.lustre-ffff944e36cb1000.statahead_max=32 llite.lustre-ffff944e36cb1000.statahead_max=32 1001 real 0m6.811s user 0m0.023s sys 0m1.332s 'ls -l' 1000 files with statahead: 6 sec statahead total: 160 statahead wrong: 2 agl total: 154 list_total: 32 fname_total: 128 hit_total: 4598 miss_total: 1231 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff944e36cb1000.statahead_stats= statahead total: 160 statahead wrong: 2 agl total: 154 list_total: 32 fname_total: 128 hit_total: 4598 miss_total: 1231 mdc.lustre-MDT0000-mdc-ffff944e36cb1000.batch_stats= snapshot_time: 1788468994.531645098 secs.nsecs start_time: 1788468913.939797132 secs.nsecs elapsed_time: 80.591847966 secs.nsecs subreqs per batch batches %% cum %% 1: 4 10 10 2: 1 2 12 4: 3 7 20 8: 2 5 25 16: 22 56 82 32: 7 17 100 mdc.lustre-MDT0001-mdc-ffff944e36cb1000.batch_stats= snapshot_time: 1788468994.531710088 secs.nsecs start_time: 1788468913.939868009 secs.nsecs elapsed_time: 80.591842079 secs.nsecs subreqs per batch batches %% cum %% 1: 3 7 7 2: 0 0 7 4: 1 2 9 8: 3 7 17 16: 12 29 46 32: 22 53 100 - open/close 593 (time 1788469007.73 total 10.01 last 59.23) - open/close 1191 (time 1788469017.74 total 20.02 last 59.77) - open/close 1847 (time 1788469027.75 total 30.03 last 65.52) - open/close 2525 (time 1788469037.75 total 40.03 last 67.79) - open/close 3321 (time 1788469047.78 total 50.06 last 79.39) - open/close 4100 (time 1788469057.78 total 60.06 last 77.89) - open/close 4963 (time 1788469067.78 total 70.06 last 86.27) - open/close 5722 (time 1788469077.80 total 80.08 last 75.79) - open/close 6413 (time 1788469087.81 total 90.09 last 69.03) - open/close 7026 (time 1788469097.82 total 100.10 last 61.24) - open/close 7843 (time 1788469107.84 total 110.12 last 81.55) - open/close 8596 (time 1788469117.84 total 120.12 last 75.25) - open/close 9292 (time 1788469127.87 total 130.15 last 69.44) total: 10000 open/close in 139.52 seconds: 71.67 ops/second llite.lustre-ffff944e36cb1000.statahead_batch_max=0 llite.lustre-ffff944e36cb1000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff944e36cb1000.stats=clear mdc.lustre-MDT0001-mdc-ffff944e36cb1000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m19.011s user 0m0.293s sys 0m14.638s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff944e36cb1000.statahead_batch_max=32 llite.lustre-ffff944e36cb1000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff944e36cb1000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff944e36cb1000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff944e36cb1000.stats=clear mdc.lustre-MDT0001-mdc-ffff944e36cb1000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m52.061s user 0m0.293s sys 0m13.735s batched statahead RPC= hit count=9976 mdc.lustre-MDT0000-mdc-ffff944e36cb1000.batch_stats= snapshot_time: 1788469347.222540467 secs.nsecs start_time: 1788469224.475988174 secs.nsecs elapsed_time: 122.746552293 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff944e36cb1000.batch_stats= snapshot_time: 1788469347.222610117 secs.nsecs start_time: 1788469224.476086061 secs.nsecs elapsed_time: 122.746524056 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff944e36cb1000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff944e36cb1000.batch_stats=0 llite.lustre-ffff944e36cb1000.statahead_max=2048 llite.lustre-ffff944e36cb1000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff944e36cb1000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff944e36cb1000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 3.82 seconds: 26.20 ops/second llite.lustre-ffff944e36cb1000.statahead_max=0 101 real 0m3.924s user 0m0.007s sys 0m0.336s 'ls -l' 100 files without statahead: 4 sec llite.lustre-ffff944e36cb1000.statahead_max=2048 llite.lustre-ffff944e36cb1000.statahead_max=2048 101 real 0m0.833s user 0m0.002s sys 0m0.175s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10075 miss_total: 25 swrong=0 ewrong=0 - open/close 750 (time 1788469664.25 total 10.00 last 74.97) total: 900 open/close in 12.05 seconds: 74.70 ops/second llite.lustre-ffff944e36cb1000.statahead_max=0 1001 real 0m19.319s user 0m0.031s sys 0m2.783s 'ls -l' 1000 files without statahead: 19 sec llite.lustre-ffff944e36cb1000.statahead_max=2048 llite.lustre-ffff944e36cb1000.statahead_max=2048 1001 real 0m8.742s user 0m0.018s sys 0m1.273s 'ls -l' 1000 files with statahead: 9 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11074 miss_total: 26 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff944e36cb1000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11074 miss_total: 26 mdc.lustre-MDT0000-mdc-ffff944e36cb1000.batch_stats= snapshot_time: 1788469727.128395817 secs.nsecs start_time: 1788469350.235678139 secs.nsecs elapsed_time: 376.892717678 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 5 62 62 16: 0 0 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff944e36cb1000.batch_stats= snapshot_time: 1788469727.128471207 secs.nsecs start_time: 1788469350.235842024 secs.nsecs elapsed_time: 376.892629183 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 12 12 4: 3 37 50 8: 0 0 50 16: 1 12 62 32: 1 12 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 - open/close 730 (time 1788469741.25 total 10.00 last 72.99) - open/close 1352 (time 1788469751.26 total 20.01 last 62.12) - open/close 1964 (time 1788469761.29 total 30.05 last 61.00) - open/close 2801 (time 1788469771.29 total 40.05 last 83.70) - open/close 3643 (time 1788469781.30 total 50.05 last 84.17) - open/close 4602 (time 1788469791.30 total 60.05 last 95.90) - open/close 5392 (time 1788469801.30 total 70.06 last 78.95) - open/close 6079 (time 1788469811.31 total 80.06 last 68.66) - open/close 6535 (time 1788469821.33 total 90.08 last 45.50) - open/close 7029 (time 1788469831.33 total 100.09 last 49.39) - open/close 7757 (time 1788469841.34 total 110.10 last 72.73) - open/close 8441 (time 1788469851.35 total 120.11 last 68.32) - open/close 9327 (time 1788469861.38 total 130.13 last 88.40) - open/close 9869 (time 1788469871.38 total 140.14 last 54.17) total: 10000 open/close in 142.10 seconds: 70.37 ops/second llite.lustre-ffff944e36cb1000.statahead_batch_max=0 llite.lustre-ffff944e36cb1000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff944e36cb1000.stats=clear mdc.lustre-MDT0001-mdc-ffff944e36cb1000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m14.269s user 0m0.327s sys 0m14.581s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff944e36cb1000.statahead_batch_max=256 llite.lustre-ffff944e36cb1000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff944e36cb1000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff944e36cb1000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff944e36cb1000.stats=clear mdc.lustre-MDT0001-mdc-ffff944e36cb1000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m55.587s user 0m0.259s sys 0m14.728s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff944e36cb1000.batch_stats= snapshot_time: 1788470021.943003789 secs.nsecs start_time: 1788469957.191010861 secs.nsecs elapsed_time: 64.751992928 secs.nsecs subreqs per batch batches %% cum %% 1: 122 46 46 2: 36 13 59 4: 25 9 69 8: 12 4 73 16: 8 3 76 32: 20 7 84 64: 7 2 86 128: 1 0 87 256: 34 12 100 mdc.lustre-MDT0001-mdc-ffff944e36cb1000.batch_stats= snapshot_time: 1788470021.943072421 secs.nsecs start_time: 1788469957.191095294 secs.nsecs elapsed_time: 64.751977127 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff944e36cb1000.statahead_batch_max=64 llite.lustre-ffff944e36cb1000.statahead_max=128