== sanity test 123ad: Verify batching statahead works correctly ========================================================== 04:46:14 (1781685974) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8aefc5f35000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8aefc5f35000.batch_stats=0 llite.lustre-ffff8aefc5f35000.statahead_max=32 llite.lustre-ffff8aefc5f35000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8aefc5f35000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8aefc5f35000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.98 seconds: 33.57 ops/second llite.lustre-ffff8aefc5f35000.statahead_max=0 101 real 0m2.720s user 0m0.006s sys 0m0.402s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8aefc5f35000.statahead_max=32 llite.lustre-ffff8aefc5f35000.statahead_max=32 101 real 0m0.943s user 0m0.003s sys 0m0.177s 'ls -l' 100 files with statahead: 1 sec statahead total: 60 statahead wrong: 2 agl total: 56 list_total: 25 fname_total: 35 hit_total: 13707 miss_total: 268 swrong=2 ewrong=2 - open/close 843 (time 1781686002.53 total 10.01 last 84.24) total: 900 open/close in 10.71 seconds: 84.07 ops/second llite.lustre-ffff8aefc5f35000.statahead_max=0 1001 real 0m17.836s user 0m0.033s sys 0m3.058s 'ls -l' 1000 files without statahead: 18 sec llite.lustre-ffff8aefc5f35000.statahead_max=32 llite.lustre-ffff8aefc5f35000.statahead_max=32 1001 real 0m8.755s user 0m0.025s sys 0m1.498s 'ls -l' 1000 files with statahead: 9 sec statahead total: 61 statahead wrong: 2 agl total: 57 list_total: 26 fname_total: 35 hit_total: 14705 miss_total: 270 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 17 seconds llite.lustre-ffff8aefc5f35000.statahead_stats= statahead total: 61 statahead wrong: 2 agl total: 57 list_total: 26 fname_total: 35 hit_total: 14705 miss_total: 270 mdc.lustre-MDT0000-mdc-ffff8aefc5f35000.batch_stats= snapshot_time: 1781686057.572025148 secs.nsecs start_time: 1781685979.011727873 secs.nsecs elapsed_time: 78.560297275 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 2 5 7 8: 2 5 13 16: 20 52 65 32: 13 34 100 mdc.lustre-MDT0001-mdc-ffff8aefc5f35000.batch_stats= snapshot_time: 1781686057.572104331 secs.nsecs start_time: 1781685979.011909496 secs.nsecs elapsed_time: 78.560194835 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 0 0 5 8: 2 5 10 16: 19 50 60 32: 15 39 100 - open/close 945 (time 1781686069.76 total 10.01 last 94.42) - open/close 1849 (time 1781686079.76 total 20.01 last 90.37) - open/close 2795 (time 1781686089.77 total 30.02 last 94.56) - open/close 3777 (time 1781686099.77 total 40.02 last 98.16) - open/close 4732 (time 1781686109.78 total 50.03 last 95.44) - open/close 5666 (time 1781686119.78 total 60.03 last 93.40) - open/close 6565 (time 1781686129.78 total 70.03 last 89.84) - open/close 7655 (time 1781686139.79 total 80.04 last 108.93) - open/close 8670 (time 1781686149.80 total 90.05 last 101.42) - open/close 9773 (time 1781686159.80 total 100.05 last 110.26) total: 10000 open/close in 102.24 seconds: 97.81 ops/second llite.lustre-ffff8aefc5f35000.statahead_batch_max=0 llite.lustre-ffff8aefc5f35000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8aefc5f35000.stats=clear mdc.lustre-MDT0001-mdc-ffff8aefc5f35000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m53.405s user 0m0.304s sys 0m16.325s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8aefc5f35000.statahead_batch_max=32 llite.lustre-ffff8aefc5f35000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8aefc5f35000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8aefc5f35000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8aefc5f35000.stats=clear mdc.lustre-MDT0001-mdc-ffff8aefc5f35000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m25.213s user 0m0.277s sys 0m15.257s batched statahead RPC= hit count=9990 mdc.lustre-MDT0000-mdc-ffff8aefc5f35000.batch_stats= snapshot_time: 1781686318.999023547 secs.nsecs start_time: 1781686223.086486057 secs.nsecs elapsed_time: 95.912537490 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8aefc5f35000.batch_stats= snapshot_time: 1781686318.999187234 secs.nsecs start_time: 1781686223.086569222 secs.nsecs elapsed_time: 95.912618012 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8aefc5f35000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8aefc5f35000.batch_stats=0 llite.lustre-ffff8aefc5f35000.statahead_max=2048 llite.lustre-ffff8aefc5f35000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8aefc5f35000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8aefc5f35000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 1.75 seconds: 57.24 ops/second llite.lustre-ffff8aefc5f35000.statahead_max=0 101 real 0m1.917s user 0m0.004s sys 0m0.333s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8aefc5f35000.statahead_max=2048 llite.lustre-ffff8aefc5f35000.statahead_max=2048 101 real 0m0.705s user 0m0.004s sys 0m0.174s 'ls -l' 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10089 miss_total: 11 swrong=0 ewrong=0 total: 900 open/close in 8.03 seconds: 112.15 ops/second llite.lustre-ffff8aefc5f35000.statahead_max=0 1001 real 0m15.656s user 0m0.032s sys 0m2.905s 'ls -l' 1000 files without statahead: 16 sec llite.lustre-ffff8aefc5f35000.statahead_max=2048 llite.lustre-ffff8aefc5f35000.statahead_max=2048 1001 real 0m4.663s user 0m0.020s sys 0m1.571s 'ls -l' 1000 files with statahead: 5 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11088 miss_total: 12 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 14 seconds llite.lustre-ffff8aefc5f35000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11088 miss_total: 12 mdc.lustre-MDT0000-mdc-ffff8aefc5f35000.batch_stats= snapshot_time: 1781686595.679360540 secs.nsecs start_time: 1781686320.688232793 secs.nsecs elapsed_time: 274.991127747 secs.nsecs subreqs per batch batches %% cum %% 1: 3 33 33 2: 3 33 66 4: 0 0 66 8: 0 0 66 16: 1 11 77 32: 0 0 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff8aefc5f35000.batch_stats= snapshot_time: 1781686595.679446331 secs.nsecs start_time: 1781686320.688879773 secs.nsecs elapsed_time: 274.990566558 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 5 50 50 16: 0 0 50 32: 2 20 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 - open/close 1254 (time 1781686607.46 total 10.00 last 125.38) - open/close 2478 (time 1781686617.47 total 20.01 last 122.34) - open/close 3675 (time 1781686627.47 total 30.01 last 119.67) - open/close 4863 (time 1781686637.47 total 40.01 last 118.78) - open/close 5925 (time 1781686647.47 total 50.01 last 106.16) - open/close 7186 (time 1781686657.48 total 60.02 last 126.06) - open/close 8337 (time 1781686667.48 total 70.02 last 115.08) - open/close 9459 (time 1781686677.49 total 80.03 last 112.11) total: 10000 open/close in 84.89 seconds: 117.80 ops/second llite.lustre-ffff8aefc5f35000.statahead_batch_max=0 llite.lustre-ffff8aefc5f35000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8aefc5f35000.stats=clear mdc.lustre-MDT0001-mdc-ffff8aefc5f35000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m42.892s user 0m0.280s sys 0m18.122s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8aefc5f35000.statahead_batch_max=256 llite.lustre-ffff8aefc5f35000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8aefc5f35000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8aefc5f35000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8aefc5f35000.stats=clear mdc.lustre-MDT0001-mdc-ffff8aefc5f35000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m38.884s user 0m0.261s sys 0m17.122s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8aefc5f35000.batch_stats= snapshot_time: 1781686780.844540392 secs.nsecs start_time: 1781686731.924440710 secs.nsecs elapsed_time: 48.920099682 secs.nsecs subreqs per batch batches %% cum %% 1: 268 59 59 2: 59 13 72 4: 43 9 81 8: 18 3 85 16: 11 2 88 32: 13 2 90 64: 7 1 92 128: 0 0 92 256: 34 7 100 mdc.lustre-MDT0001-mdc-ffff8aefc5f35000.batch_stats= snapshot_time: 1781686780.844603355 secs.nsecs start_time: 1781686731.924503055 secs.nsecs elapsed_time: 48.920100300 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8aefc5f35000.statahead_batch_max=64 llite.lustre-ffff8aefc5f35000.statahead_max=128