== sanity test 123ad: Verify batching statahead works correctly ========================================================== 17:19:33 (1790716773) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8b8e1b447000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8b8e1b447000.batch_stats=0 llite.lustre-ffff8b8e1b447000.statahead_max=32 llite.lustre-ffff8b8e1b447000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8b8e1b447000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8b8e1b447000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.40 seconds: 41.59 ops/second llite.lustre-ffff8b8e1b447000.statahead_max=0 101 real 0m2.151s user 0m0.004s sys 0m0.294s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8b8e1b447000.statahead_max=32 llite.lustre-ffff8b8e1b447000.statahead_max=32 101 real 0m1.248s user 0m0.007s sys 0m0.164s 'ls -l' 100 files with statahead: 2 sec statahead total: 140 statahead wrong: 2 agl total: 134 list_total: 31 fname_total: 108 hit_total: 4112 miss_total: 1185 swrong=2 ewrong=2 total: 900 open/close in 8.84 seconds: 101.76 ops/second llite.lustre-ffff8b8e1b447000.statahead_max=0 1001 real 0m20.560s user 0m0.037s sys 0m2.840s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff8b8e1b447000.statahead_max=32 llite.lustre-ffff8b8e1b447000.statahead_max=32 1001 real 0m8.121s user 0m0.033s sys 0m1.323s 'ls -l' 1000 files with statahead: 8 sec statahead total: 141 statahead wrong: 2 agl total: 135 list_total: 32 fname_total: 108 hit_total: 5111 miss_total: 1186 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff8b8e1b447000.statahead_stats= statahead total: 141 statahead wrong: 2 agl total: 135 list_total: 32 fname_total: 108 hit_total: 5111 miss_total: 1186 mdc.lustre-MDT0000-mdc-ffff8b8e1b447000.batch_stats= snapshot_time: 1790716863.126347102 secs.nsecs start_time: 1790716778.865060694 secs.nsecs elapsed_time: 84.261286408 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 2 5 8 4: 2 5 13 8: 1 2 16 16: 25 67 83 32: 6 16 100 mdc.lustre-MDT0001-mdc-ffff8b8e1b447000.batch_stats= snapshot_time: 1790716863.126417565 secs.nsecs start_time: 1790716778.865150228 secs.nsecs elapsed_time: 84.261267337 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 1 2 5 8: 4 10 15 16: 11 28 43 32: 22 56 100 - open/close 1033 (time 1790716876.42 total 10.00 last 103.26) - open/close 2018 (time 1790716886.44 total 20.02 last 98.34) - open/close 2732 (time 1790716896.44 total 30.03 last 71.35) - open/close 3509 (time 1790716906.45 total 40.03 last 77.66) - open/close 4392 (time 1790716916.45 total 50.04 last 88.24) - open/close 5194 (time 1790716926.46 total 60.05 last 80.13) - open/close 6138 (time 1790716936.47 total 70.05 last 94.36) - open/close 7123 (time 1790716946.47 total 80.06 last 98.44) - open/close 7819 (time 1790716956.49 total 90.07 last 69.51) - open/close 8580 (time 1790716966.49 total 100.08 last 76.07) - open/close 9241 (time 1790716976.50 total 110.08 last 66.07) - open/close 9979 (time 1790716986.50 total 120.08 last 73.80) total: 10000 open/close in 120.25 seconds: 83.16 ops/second llite.lustre-ffff8b8e1b447000.statahead_batch_max=0 llite.lustre-ffff8b8e1b447000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8b8e1b447000.stats=clear mdc.lustre-MDT0001-mdc-ffff8b8e1b447000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m55.722s user 0m0.296s sys 0m14.264s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8b8e1b447000.statahead_batch_max=32 llite.lustre-ffff8b8e1b447000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8b8e1b447000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8b8e1b447000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8b8e1b447000.stats=clear mdc.lustre-MDT0001-mdc-ffff8b8e1b447000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m34.036s user 0m0.278s sys 0m13.056s batched statahead RPC= hit count=9983 mdc.lustre-MDT0000-mdc-ffff8b8e1b447000.batch_stats= snapshot_time: 1790717157.027294575 secs.nsecs start_time: 1790717052.262814145 secs.nsecs elapsed_time: 104.764480430 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8b8e1b447000.batch_stats= snapshot_time: 1790717157.027360398 secs.nsecs start_time: 1790717052.263033280 secs.nsecs elapsed_time: 104.764327118 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8b8e1b447000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8b8e1b447000.batch_stats=0 llite.lustre-ffff8b8e1b447000.statahead_max=2048 llite.lustre-ffff8b8e1b447000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8b8e1b447000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8b8e1b447000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.95 seconds: 33.88 ops/second llite.lustre-ffff8b8e1b447000.statahead_max=0 101 real 0m2.625s user 0m0.007s sys 0m0.333s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8b8e1b447000.statahead_max=2048 llite.lustre-ffff8b8e1b447000.statahead_max=2048 101 real 0m0.852s user 0m0.005s sys 0m0.169s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10082 miss_total: 18 swrong=0 ewrong=0 - open/close 819 (time 1790717428.07 total 10.01 last 81.86) total: 900 open/close in 11.02 seconds: 81.65 ops/second llite.lustre-ffff8b8e1b447000.statahead_max=0 1001 real 0m21.631s user 0m0.029s sys 0m2.799s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff8b8e1b447000.statahead_max=2048 llite.lustre-ffff8b8e1b447000.statahead_max=2048 1001 real 0m6.152s user 0m0.031s sys 0m1.347s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff8b8e1b447000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 mdc.lustre-MDT0000-mdc-ffff8b8e1b447000.batch_stats= snapshot_time: 1790717490.080354703 secs.nsecs start_time: 1790717159.240096447 secs.nsecs elapsed_time: 330.840258256 secs.nsecs subreqs per batch batches %% cum %% 1: 1 14 14 2: 2 28 42 4: 1 14 57 8: 1 14 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff8b8e1b447000.batch_stats= snapshot_time: 1790717490.080427786 secs.nsecs start_time: 1790717159.240246065 secs.nsecs elapsed_time: 330.840181721 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 10 10 8: 3 30 40 16: 0 0 40 32: 3 30 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 - open/close 994 (time 1790717502.81 total 10.00 last 99.36) - open/close 1909 (time 1790717512.83 total 20.02 last 91.38) - open/close 2852 (time 1790717522.84 total 30.03 last 94.22) - open/close 3769 (time 1790717532.84 total 40.03 last 91.66) - open/close 4783 (time 1790717542.85 total 50.04 last 101.27) - open/close 5613 (time 1790717552.86 total 60.05 last 82.96) - open/close 6496 (time 1790717562.87 total 70.06 last 88.22) - open/close 7366 (time 1790717572.88 total 80.07 last 86.91) - open/close 8327 (time 1790717582.89 total 90.08 last 96.00) - open/close 9232 (time 1790717592.89 total 100.08 last 90.44) total: 10000 open/close in 109.16 seconds: 91.61 ops/second llite.lustre-ffff8b8e1b447000.statahead_batch_max=0 llite.lustre-ffff8b8e1b447000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8b8e1b447000.stats=clear mdc.lustre-MDT0001-mdc-ffff8b8e1b447000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m51.562s user 0m0.282s sys 0m13.843s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8b8e1b447000.statahead_batch_max=256 llite.lustre-ffff8b8e1b447000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8b8e1b447000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8b8e1b447000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8b8e1b447000.stats=clear mdc.lustre-MDT0001-mdc-ffff8b8e1b447000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m45.232s user 0m0.260s sys 0m14.027s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8b8e1b447000.batch_stats= snapshot_time: 1790717715.903471215 secs.nsecs start_time: 1790717660.787455567 secs.nsecs elapsed_time: 55.116015648 secs.nsecs subreqs per batch batches %% cum %% 1: 95 47 47 2: 22 10 58 4: 15 7 65 8: 7 3 69 16: 2 0 70 32: 12 5 76 64: 10 4 81 128: 4 1 83 256: 34 16 100 mdc.lustre-MDT0001-mdc-ffff8b8e1b447000.batch_stats= snapshot_time: 1790717715.903540220 secs.nsecs start_time: 1790717660.787769351 secs.nsecs elapsed_time: 55.115770869 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8b8e1b447000.statahead_batch_max=64 llite.lustre-ffff8b8e1b447000.statahead_max=128