== sanity test 123ad: Verify batching statahead works correctly ========================================================== 00:58:46 (1783141126) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff984d46e49000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff984d46e49000.batch_stats=0 llite.lustre-ffff984d46e49000.statahead_max=32 llite.lustre-ffff984d46e49000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff984d46e49000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff984d46e49000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.77 seconds: 36.04 ops/second llite.lustre-ffff984d46e49000.statahead_max=0 101 real 0m2.273s user 0m0.007s sys 0m0.359s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff984d46e49000.statahead_max=32 llite.lustre-ffff984d46e49000.statahead_max=32 101 real 0m1.188s user 0m0.006s sys 0m0.213s 'ls -l' 100 files with statahead: 1 sec statahead total: 116 statahead wrong: 2 agl total: 110 list_total: 32 fname_total: 84 hit_total: 4132 miss_total: 1673 swrong=2 ewrong=2 - open/close 657 (time 1783141153.83 total 10.01 last 65.64) total: 900 open/close in 13.74 seconds: 65.53 ops/second llite.lustre-ffff984d46e49000.statahead_max=0 1001 real 0m23.893s user 0m0.037s sys 0m3.587s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff984d46e49000.statahead_max=32 llite.lustre-ffff984d46e49000.statahead_max=32 1001 real 0m8.233s user 0m0.035s sys 0m1.664s 'ls -l' 1000 files with statahead: 8 sec statahead total: 117 statahead wrong: 2 agl total: 111 list_total: 33 fname_total: 84 hit_total: 5129 miss_total: 1676 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 15 seconds llite.lustre-ffff984d46e49000.statahead_stats= statahead total: 117 statahead wrong: 2 agl total: 111 list_total: 33 fname_total: 84 hit_total: 5129 miss_total: 1676 mdc.lustre-MDT0000-mdc-ffff984d46e49000.batch_stats= snapshot_time: 1783141214.520236620 secs.nsecs start_time: 1783141130.572469914 secs.nsecs elapsed_time: 83.947766706 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 3 8 8 4: 0 0 8 8: 2 5 13 16: 27 75 88 32: 4 11 100 mdc.lustre-MDT0001-mdc-ffff984d46e49000.batch_stats= snapshot_time: 1783141214.520581506 secs.nsecs start_time: 1783141130.572572256 secs.nsecs elapsed_time: 83.948009250 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 1 2 7 8: 3 7 15 16: 11 27 42 32: 23 57 100 - open/close 740 (time 1783141226.96 total 10.01 last 73.93) - open/close 1483 (time 1783141236.97 total 20.02 last 74.24) - open/close 2299 (time 1783141246.98 total 30.03 last 81.51) - open/close 3125 (time 1783141256.99 total 40.04 last 82.52) - open/close 3940 (time 1783141266.99 total 50.04 last 81.48) - open/close 4732 (time 1783141277.00 total 60.05 last 79.14) - open/close 5582 (time 1783141287.00 total 70.05 last 84.95) - open/close 6444 (time 1783141297.01 total 80.06 last 86.14) - open/close 7270 (time 1783141307.02 total 90.07 last 82.55) - open/close 8080 (time 1783141317.03 total 100.08 last 80.90) - open/close 8905 (time 1783141327.03 total 110.08 last 82.48) - open/close 9730 (time 1783141337.04 total 120.09 last 82.46) total: 10000 open/close in 123.35 seconds: 81.07 ops/second llite.lustre-ffff984d46e49000.statahead_batch_max=0 llite.lustre-ffff984d46e49000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff984d46e49000.stats=clear mdc.lustre-MDT0001-mdc-ffff984d46e49000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m48.296s user 0m0.364s sys 0m21.799s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff984d46e49000.statahead_batch_max=32 llite.lustre-ffff984d46e49000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff984d46e49000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff984d46e49000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff984d46e49000.stats=clear mdc.lustre-MDT0001-mdc-ffff984d46e49000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m51.794s user 0m0.347s sys 0m16.927s batched statahead RPC= hit count=9983 mdc.lustre-MDT0000-mdc-ffff984d46e49000.batch_stats= snapshot_time: 1783141517.639871350 secs.nsecs start_time: 1783141396.118126740 secs.nsecs elapsed_time: 121.521744610 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff984d46e49000.batch_stats= snapshot_time: 1783141517.639944497 secs.nsecs start_time: 1783141396.118310465 secs.nsecs elapsed_time: 121.521634032 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff984d46e49000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff984d46e49000.batch_stats=0 llite.lustre-ffff984d46e49000.statahead_max=2048 llite.lustre-ffff984d46e49000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff984d46e49000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff984d46e49000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.38 seconds: 29.56 ops/second llite.lustre-ffff984d46e49000.statahead_max=0 101 real 0m3.588s user 0m0.007s sys 0m0.419s 'ls -l' 100 files without statahead: 4 sec llite.lustre-ffff984d46e49000.statahead_max=2048 llite.lustre-ffff984d46e49000.statahead_max=2048 101 real 0m1.153s user 0m0.004s sys 0m0.221s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10082 miss_total: 18 swrong=0 ewrong=0 - open/close 599 (time 1783141891.65 total 10.00 last 59.87) total: 900 open/close in 14.30 seconds: 62.92 ops/second llite.lustre-ffff984d46e49000.statahead_max=0 1001 real 0m28.899s user 0m0.061s sys 0m3.799s 'ls -l' 1000 files without statahead: 29 sec llite.lustre-ffff984d46e49000.statahead_max=2048 llite.lustre-ffff984d46e49000.statahead_max=2048 1001 real 0m8.910s user 0m0.038s sys 0m1.529s 'ls -l' 1000 files with statahead: 9 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 25 seconds llite.lustre-ffff984d46e49000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 mdc.lustre-MDT0000-mdc-ffff984d46e49000.batch_stats= snapshot_time: 1783141974.890092683 secs.nsecs start_time: 1783141521.285370131 secs.nsecs elapsed_time: 453.604722552 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 20 20 8: 4 40 60 16: 1 10 70 32: 1 10 80 64: 0 0 80 128: 0 0 80 256: 2 20 100 mdc.lustre-MDT0001-mdc-ffff984d46e49000.batch_stats= snapshot_time: 1783141974.890190059 secs.nsecs start_time: 1783141521.285525243 secs.nsecs elapsed_time: 453.604664816 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 20 20 8: 2 20 40 16: 3 30 70 32: 1 10 80 64: 0 0 80 128: 0 0 80 256: 2 20 100 - open/close 620 (time 1783141987.73 total 10.01 last 61.95) - open/close 1234 (time 1783141997.73 total 20.01 last 61.40) - open/close 1900 (time 1783142007.74 total 30.02 last 66.53) - open/close 2521 (time 1783142017.76 total 40.04 last 61.96) - open/close 3167 (time 1783142027.79 total 50.06 last 64.46) - open/close 3791 (time 1783142037.80 total 60.08 last 62.31) - open/close 4455 (time 1783142047.82 total 70.09 last 66.30) - open/close 5092 (time 1783142057.83 total 80.10 last 63.62) - open/close 5693 (time 1783142067.84 total 90.12 last 60.02) - open/close 6296 (time 1783142077.85 total 100.12 last 60.26) - open/close 6955 (time 1783142087.86 total 110.13 last 65.83) - open/close 7625 (time 1783142097.86 total 120.14 last 66.97) - open/close 8314 (time 1783142107.87 total 130.15 last 68.86) - open/close 9026 (time 1783142117.89 total 140.17 last 71.03) - open/close 9663 (time 1783142127.91 total 150.18 last 63.61) total: 10000 open/close in 155.76 seconds: 64.20 ops/second llite.lustre-ffff984d46e49000.statahead_batch_max=0 llite.lustre-ffff984d46e49000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff984d46e49000.stats=clear mdc.lustre-MDT0001-mdc-ffff984d46e49000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m8.181s user 0m0.395s sys 0m17.959s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff984d46e49000.statahead_batch_max=256 llite.lustre-ffff984d46e49000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff984d46e49000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff984d46e49000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff984d46e49000.stats=clear mdc.lustre-MDT0001-mdc-ffff984d46e49000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m58.843s user 0m0.360s sys 0m18.223s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff984d46e49000.batch_stats= snapshot_time: 1783142285.524844016 secs.nsecs start_time: 1783142212.058741138 secs.nsecs elapsed_time: 73.466102878 secs.nsecs subreqs per batch batches %% cum %% 1: 277 56 56 2: 68 13 69 4: 51 10 80 8: 30 6 86 16: 12 2 88 32: 17 3 92 64: 6 1 93 128: 0 0 93 256: 33 6 100 mdc.lustre-MDT0001-mdc-ffff984d46e49000.batch_stats= snapshot_time: 1783142285.524915574 secs.nsecs start_time: 1783142212.058823408 secs.nsecs elapsed_time: 73.466092166 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff984d46e49000.statahead_batch_max=64 llite.lustre-ffff984d46e49000.statahead_max=128