== sanity test 123ad: Verify batching statahead works correctly ========================================================== 21:56:12 (1789782972) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff977dd038c000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff977dd038c000.batch_stats=0 llite.lustre-ffff977dd038c000.statahead_max=32 llite.lustre-ffff977dd038c000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff977dd038c000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff977dd038c000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.79 seconds: 35.79 ops/second llite.lustre-ffff977dd038c000.statahead_max=0 101 real 0m3.513s user 0m0.005s sys 0m0.341s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff977dd038c000.statahead_max=32 llite.lustre-ffff977dd038c000.statahead_max=32 101 real 0m1.460s user 0m0.005s sys 0m0.156s 'ls -l' 100 files with statahead: 2 sec statahead total: 122 statahead wrong: 2 agl total: 115 list_total: 32 fname_total: 84 hit_total: 3561 miss_total: 1425 swrong=2 ewrong=2 - open/close 691 (time 1789783003.98 total 10.01 last 69.06) total: 900 open/close in 12.29 seconds: 73.26 ops/second llite.lustre-ffff977dd038c000.statahead_max=0 1001 real 0m21.408s user 0m0.032s sys 0m2.838s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff977dd038c000.statahead_max=32 llite.lustre-ffff977dd038c000.statahead_max=32 1001 real 0m7.552s user 0m0.022s sys 0m1.359s 'ls -l' 1000 files with statahead: 8 sec statahead total: 123 statahead wrong: 2 agl total: 116 list_total: 33 fname_total: 84 hit_total: 4559 miss_total: 1427 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff977dd038c000.statahead_stats= statahead total: 123 statahead wrong: 2 agl total: 116 list_total: 33 fname_total: 84 hit_total: 4559 miss_total: 1427 mdc.lustre-MDT0000-mdc-ffff977dd038c000.batch_stats= snapshot_time: 1789783068.576355151 secs.nsecs start_time: 1789782978.092154632 secs.nsecs elapsed_time: 90.484200519 secs.nsecs subreqs per batch batches %% cum %% 1: 3 7 7 2: 3 7 15 4: 0 0 15 8: 2 5 21 16: 24 63 84 32: 6 15 100 mdc.lustre-MDT0001-mdc-ffff977dd038c000.batch_stats= snapshot_time: 1789783068.576421851 secs.nsecs start_time: 1789782978.092363578 secs.nsecs elapsed_time: 90.484058273 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 5 5 8: 4 10 15 16: 10 25 41 32: 23 58 100 - open/close 841 (time 1789783081.36 total 10.01 last 84.00) - open/close 1700 (time 1789783091.37 total 20.02 last 85.84) - open/close 2488 (time 1789783101.38 total 30.02 last 78.76) - open/close 3328 (time 1789783111.39 total 40.04 last 83.90) - open/close 4125 (time 1789783121.40 total 50.05 last 79.62) - open/close 4951 (time 1789783131.41 total 60.05 last 82.53) - open/close 5844 (time 1789783141.41 total 70.06 last 89.23) - open/close 6653 (time 1789783151.42 total 80.06 last 80.88) - open/close 7603 (time 1789783161.43 total 90.08 last 94.87) - open/close 8387 (time 1789783171.44 total 100.09 last 78.30) - open/close 9251 (time 1789783181.45 total 110.10 last 86.34) total: 10000 open/close in 118.11 seconds: 84.67 ops/second llite.lustre-ffff977dd038c000.statahead_batch_max=0 llite.lustre-ffff977dd038c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff977dd038c000.stats=clear mdc.lustre-MDT0001-mdc-ffff977dd038c000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.389s user 0m0.309s sys 0m15.335s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff977dd038c000.statahead_batch_max=32 llite.lustre-ffff977dd038c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff977dd038c000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff977dd038c000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff977dd038c000.stats=clear mdc.lustre-MDT0001-mdc-ffff977dd038c000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m40.798s user 0m0.302s sys 0m13.721s batched statahead RPC= hit count=9979 mdc.lustre-MDT0000-mdc-ffff977dd038c000.batch_stats= snapshot_time: 1789783373.214903944 secs.nsecs start_time: 1789783260.577088052 secs.nsecs elapsed_time: 112.637815892 secs.nsecs subreqs per batch batches %% cum %% 1: 1 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff977dd038c000.batch_stats= snapshot_time: 1789783373.215273491 secs.nsecs start_time: 1789783260.577466647 secs.nsecs elapsed_time: 112.637806844 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff977dd038c000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff977dd038c000.batch_stats=0 llite.lustre-ffff977dd038c000.statahead_max=2048 llite.lustre-ffff977dd038c000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff977dd038c000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff977dd038c000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.16 seconds: 46.31 ops/second llite.lustre-ffff977dd038c000.statahead_max=0 101 real 0m1.949s user 0m0.004s sys 0m0.299s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff977dd038c000.statahead_max=2048 llite.lustre-ffff977dd038c000.statahead_max=2048 101 real 0m0.975s user 0m0.004s sys 0m0.169s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10078 miss_total: 22 swrong=0 ewrong=0 total: 900 open/close in 9.94 seconds: 90.52 ops/second llite.lustre-ffff977dd038c000.statahead_max=0 1001 real 0m19.681s user 0m0.037s sys 0m2.776s 'ls -l' 1000 files without statahead: 19 sec llite.lustre-ffff977dd038c000.statahead_max=2048 llite.lustre-ffff977dd038c000.statahead_max=2048 1001 real 0m5.859s user 0m0.026s sys 0m1.431s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11077 miss_total: 23 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 25 seconds llite.lustre-ffff977dd038c000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11077 miss_total: 23 mdc.lustre-MDT0000-mdc-ffff977dd038c000.batch_stats= snapshot_time: 1789783729.592772706 secs.nsecs start_time: 1789783375.508094849 secs.nsecs elapsed_time: 354.084677857 secs.nsecs subreqs per batch batches %% cum %% 1: 1 14 14 2: 1 14 28 4: 1 14 42 8: 1 14 57 16: 1 14 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff977dd038c000.batch_stats= snapshot_time: 1789783729.592870651 secs.nsecs start_time: 1789783375.508272865 secs.nsecs elapsed_time: 354.084597786 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 11 11 8: 4 44 55 16: 0 0 55 32: 1 11 66 64: 0 0 66 128: 1 11 77 256: 2 22 100 - open/close 805 (time 1789783742.33 total 10.02 last 80.34) - open/close 1380 (time 1789783752.37 total 20.06 last 57.29) - open/close 2175 (time 1789783762.38 total 30.06 last 79.43) - open/close 2926 (time 1789783772.38 total 40.07 last 75.09) - open/close 3700 (time 1789783782.39 total 50.08 last 77.31) - open/close 4562 (time 1789783792.40 total 60.08 last 86.17) - open/close 5485 (time 1789783802.40 total 70.09 last 92.24) - open/close 6362 (time 1789783812.41 total 80.09 last 87.66) - open/close 7292 (time 1789783822.42 total 90.10 last 92.91) - open/close 8197 (time 1789783832.42 total 100.10 last 90.50) - open/close 9121 (time 1789783842.42 total 110.11 last 92.36) total: 10000 open/close in 119.70 seconds: 83.54 ops/second llite.lustre-ffff977dd038c000.statahead_batch_max=0 llite.lustre-ffff977dd038c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff977dd038c000.stats=clear mdc.lustre-MDT0001-mdc-ffff977dd038c000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m8.603s user 0m0.307s sys 0m14.836s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff977dd038c000.statahead_batch_max=256 llite.lustre-ffff977dd038c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff977dd038c000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff977dd038c000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff977dd038c000.stats=clear mdc.lustre-MDT0001-mdc-ffff977dd038c000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m51.520s user 0m0.294s sys 0m14.762s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff977dd038c000.batch_stats= snapshot_time: 1789783991.229896162 secs.nsecs start_time: 1789783929.155591036 secs.nsecs elapsed_time: 62.074305126 secs.nsecs subreqs per batch batches %% cum %% 1: 102 44 44 2: 33 14 59 4: 15 6 65 8: 14 6 71 16: 6 2 74 32: 14 6 80 64: 8 3 84 128: 1 0 84 256: 35 15 100 mdc.lustre-MDT0001-mdc-ffff977dd038c000.batch_stats= snapshot_time: 1789783991.229967899 secs.nsecs start_time: 1789783929.155714487 secs.nsecs elapsed_time: 62.074253412 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff977dd038c000.statahead_batch_max=64 llite.lustre-ffff977dd038c000.statahead_max=128