== sanity test 123ad: Verify batching statahead works correctly ========================================================== 08:57:20 (1787749040) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9752013f9000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9752013f9000.batch_stats=0 llite.lustre-ffff9752013f9000.statahead_max=32 llite.lustre-ffff9752013f9000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9752013f9000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9752013f9000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.97 seconds: 33.66 ops/second llite.lustre-ffff9752013f9000.statahead_max=0 101 real 0m2.869s user 0m0.008s sys 0m0.324s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9752013f9000.statahead_max=32 llite.lustre-ffff9752013f9000.statahead_max=32 101 real 0m1.400s user 0m0.006s sys 0m0.166s 'ls -l' 100 files with statahead: 1 sec statahead total: 133 statahead wrong: 2 agl total: 127 list_total: 31 fname_total: 98 hit_total: 3690 miss_total: 1577 swrong=2 ewrong=2 - open/close 665 (time 1787749071.05 total 10.00 last 66.50) total: 900 open/close in 13.42 seconds: 67.06 ops/second llite.lustre-ffff9752013f9000.statahead_max=0 1001 real 0m24.907s user 0m0.037s sys 0m2.994s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff9752013f9000.statahead_max=32 llite.lustre-ffff9752013f9000.statahead_max=32 1001 real 0m10.390s user 0m0.036s sys 0m1.480s 'ls -l' 1000 files with statahead: 10 sec statahead total: 134 statahead wrong: 2 agl total: 128 list_total: 32 fname_total: 98 hit_total: 4684 miss_total: 1583 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff9752013f9000.statahead_stats= statahead total: 134 statahead wrong: 2 agl total: 128 list_total: 32 fname_total: 98 hit_total: 4684 miss_total: 1583 mdc.lustre-MDT0000-mdc-ffff9752013f9000.batch_stats= snapshot_time: 1787749143.574698625 secs.nsecs start_time: 1787749045.777684143 secs.nsecs elapsed_time: 97.797014482 secs.nsecs subreqs per batch batches %% cum %% 1: 5 13 13 2: 0 0 13 4: 1 2 15 8: 3 7 23 16: 24 63 86 32: 5 13 100 mdc.lustre-MDT0001-mdc-ffff9752013f9000.batch_stats= snapshot_time: 1787749143.574775386 secs.nsecs start_time: 1787749045.777806653 secs.nsecs elapsed_time: 97.796968733 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 2 2 8: 3 7 10 16: 10 26 36 32: 24 63 100 - open/close 748 (time 1787749156.25 total 10.00 last 74.78) - open/close 1473 (time 1787749166.26 total 20.02 last 72.39) - open/close 2177 (time 1787749176.26 total 30.02 last 70.38) - open/close 2937 (time 1787749186.29 total 40.05 last 75.81) - open/close 3655 (time 1787749196.31 total 50.07 last 71.66) - open/close 4278 (time 1787749206.32 total 60.07 last 62.25) - open/close 4946 (time 1787749216.33 total 70.08 last 66.74) - open/close 5743 (time 1787749226.34 total 80.10 last 79.60) - open/close 6478 (time 1787749236.36 total 90.11 last 73.38) - open/close 7116 (time 1787749246.36 total 100.12 last 63.78) - open/close 7701 (time 1787749256.37 total 110.12 last 58.46) - open/close 8358 (time 1787749266.40 total 120.16 last 65.49) - open/close 9111 (time 1787749276.41 total 130.17 last 75.20) - open/close 9836 (time 1787749286.42 total 140.18 last 72.42) total: 10000 open/close in 142.31 seconds: 70.27 ops/second llite.lustre-ffff9752013f9000.statahead_batch_max=0 llite.lustre-ffff9752013f9000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9752013f9000.stats=clear mdc.lustre-MDT0001-mdc-ffff9752013f9000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m8.876s user 0m0.397s sys 0m16.352s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9752013f9000.statahead_batch_max=32 llite.lustre-ffff9752013f9000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9752013f9000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9752013f9000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9752013f9000.stats=clear mdc.lustre-MDT0001-mdc-ffff9752013f9000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m59.571s user 0m0.357s sys 0m14.708s batched statahead RPC= hit count=9980 mdc.lustre-MDT0000-mdc-ffff9752013f9000.batch_stats= snapshot_time: 1787749499.369618628 secs.nsecs start_time: 1787749367.386080377 secs.nsecs elapsed_time: 131.983538251 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9752013f9000.batch_stats= snapshot_time: 1787749499.370331223 secs.nsecs start_time: 1787749367.386410992 secs.nsecs elapsed_time: 131.983920231 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9752013f9000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9752013f9000.batch_stats=0 llite.lustre-ffff9752013f9000.statahead_max=2048 llite.lustre-ffff9752013f9000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9752013f9000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9752013f9000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.54 seconds: 39.34 ops/second llite.lustre-ffff9752013f9000.statahead_max=0 101 real 0m2.882s user 0m0.005s sys 0m0.329s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9752013f9000.statahead_max=2048 llite.lustre-ffff9752013f9000.statahead_max=2048 101 real 0m1.531s user 0m0.007s sys 0m0.168s 'ls -l' 100 files with statahead: 2 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10079 miss_total: 21 swrong=0 ewrong=0 - open/close 738 (time 1787749841.89 total 10.03 last 73.58) total: 900 open/close in 12.47 seconds: 72.16 ops/second llite.lustre-ffff9752013f9000.statahead_max=0 1001 real 0m29.896s user 0m0.055s sys 0m3.098s 'ls -l' 1000 files without statahead: 29 sec llite.lustre-ffff9752013f9000.statahead_max=2048 llite.lustre-ffff9752013f9000.statahead_max=2048 1001 real 0m7.385s user 0m0.029s sys 0m1.427s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11078 miss_total: 22 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff9752013f9000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11078 miss_total: 22 mdc.lustre-MDT0000-mdc-ffff9752013f9000.batch_stats= snapshot_time: 1787749918.442495152 secs.nsecs start_time: 1787749502.665198771 secs.nsecs elapsed_time: 415.777296381 secs.nsecs subreqs per batch batches %% cum %% 1: 3 37 37 2: 0 0 37 4: 1 12 50 8: 1 12 62 16: 0 0 62 32: 1 12 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff9752013f9000.batch_stats= snapshot_time: 1787749918.442563672 secs.nsecs start_time: 1787749502.665366311 secs.nsecs elapsed_time: 415.777197361 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 22 22 8: 3 33 55 16: 0 0 55 32: 1 11 66 64: 0 0 66 128: 1 11 77 256: 2 22 100 - open/close 707 (time 1787749931.47 total 10.01 last 70.63) - open/close 1393 (time 1787749941.48 total 20.02 last 68.56) - open/close 2071 (time 1787749951.48 total 30.02 last 67.79) - open/close 2834 (time 1787749961.48 total 40.02 last 76.27) - open/close 3514 (time 1787749971.51 total 50.05 last 67.79) - open/close 4201 (time 1787749981.52 total 60.06 last 68.65) - open/close 4766 (time 1787749991.53 total 70.07 last 56.45) - open/close 5362 (time 1787750001.54 total 80.08 last 59.53) - open/close 5962 (time 1787750011.55 total 90.09 last 59.95) - open/close 6622 (time 1787750021.56 total 100.10 last 65.90) - open/close 7315 (time 1787750031.57 total 110.11 last 69.25) - open/close 7978 (time 1787750041.59 total 120.13 last 66.17) - open/close 8614 (time 1787750051.59 total 130.13 last 63.60) - open/close 9230 (time 1787750061.60 total 140.14 last 61.54) - open/close 9861 (time 1787750071.61 total 150.15 last 63.06) total: 10000 open/close in 151.94 seconds: 65.81 ops/second llite.lustre-ffff9752013f9000.statahead_batch_max=0 llite.lustre-ffff9752013f9000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9752013f9000.stats=clear mdc.lustre-MDT0001-mdc-ffff9752013f9000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m15.975s user 0m0.391s sys 0m16.564s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9752013f9000.statahead_batch_max=256 llite.lustre-ffff9752013f9000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9752013f9000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9752013f9000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9752013f9000.stats=clear mdc.lustre-MDT0001-mdc-ffff9752013f9000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m51.994s user 0m0.293s sys 0m15.112s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9752013f9000.batch_stats= snapshot_time: 1787750223.597975793 secs.nsecs start_time: 1787750160.478381012 secs.nsecs elapsed_time: 63.119594781 secs.nsecs subreqs per batch batches %% cum %% 1: 51 32 32 2: 17 10 43 4: 17 10 54 8: 2 1 56 16: 8 5 61 32: 17 10 72 64: 8 5 77 128: 0 0 77 256: 35 22 100 mdc.lustre-MDT0001-mdc-ffff9752013f9000.batch_stats= snapshot_time: 1787750223.598122276 secs.nsecs start_time: 1787750160.478510810 secs.nsecs elapsed_time: 63.119611466 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9752013f9000.statahead_batch_max=64 llite.lustre-ffff9752013f9000.statahead_max=128