== sanity test 123ad: Verify batching statahead works correctly ========================================================== 05:43:54 (1788515034) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9d232ec32000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d232ec32000.batch_stats=0 llite.lustre-ffff9d232ec32000.statahead_max=32 llite.lustre-ffff9d232ec32000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9d232ec32000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d232ec32000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 3.12 seconds: 32.08 ops/second llite.lustre-ffff9d232ec32000.statahead_max=0 101 real 0m3.028s user 0m0.004s sys 0m0.396s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9d232ec32000.statahead_max=32 llite.lustre-ffff9d232ec32000.statahead_max=32 101 real 0m1.440s user 0m0.005s sys 0m0.204s 'ls -l' 100 files with statahead: 1 sec statahead total: 130 statahead wrong: 2 agl total: 125 list_total: 30 fname_total: 86 hit_total: 4274 miss_total: 1270 swrong=2 ewrong=2 - open/close 600 (time 1788515065.50 total 10.00 last 60.00) total: 900 open/close in 14.52 seconds: 61.99 ops/second llite.lustre-ffff9d232ec32000.statahead_max=0 1001 real 0m30.765s user 0m0.039s sys 0m3.496s 'ls -l' 1000 files without statahead: 31 sec llite.lustre-ffff9d232ec32000.statahead_max=32 llite.lustre-ffff9d232ec32000.statahead_max=32 1001 real 0m11.628s user 0m0.034s sys 0m1.610s 'ls -l' 1000 files with statahead: 12 sec statahead total: 131 statahead wrong: 2 agl total: 126 list_total: 31 fname_total: 86 hit_total: 5271 miss_total: 1273 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff9d232ec32000.statahead_stats= statahead total: 131 statahead wrong: 2 agl total: 126 list_total: 31 fname_total: 86 hit_total: 5271 miss_total: 1273 mdc.lustre-MDT0000-mdc-ffff9d232ec32000.batch_stats= snapshot_time: 1788515149.174432396 secs.nsecs start_time: 1788515040.128586264 secs.nsecs elapsed_time: 109.045846132 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 3 8 13 4: 1 2 16 8: 2 5 21 16: 21 56 78 32: 8 21 100 mdc.lustre-MDT0001-mdc-ffff9d232ec32000.batch_stats= snapshot_time: 1788515149.174726662 secs.nsecs start_time: 1788515040.128683539 secs.nsecs elapsed_time: 109.046043123 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 4 10 10 16: 15 38 48 32: 20 51 100 - open/close 761 (time 1788515161.76 total 10.00 last 76.09) - open/close 1508 (time 1788515171.76 total 20.00 last 74.69) - open/close 2267 (time 1788515181.76 total 30.01 last 75.88) - open/close 3033 (time 1788515191.78 total 40.02 last 76.49) - open/close 3800 (time 1788515201.79 total 50.03 last 76.61) - open/close 4545 (time 1788515211.80 total 60.05 last 74.39) - open/close 5277 (time 1788515221.81 total 70.05 last 73.15) - open/close 6011 (time 1788515231.81 total 80.06 last 73.37) - open/close 6723 (time 1788515241.82 total 90.06 last 71.17) - open/close 7483 (time 1788515251.83 total 100.07 last 75.90) - open/close 8186 (time 1788515261.83 total 110.08 last 70.28) - open/close 8919 (time 1788515271.84 total 120.08 last 73.27) - open/close 9599 (time 1788515281.85 total 130.09 last 67.93) total: 10000 open/close in 135.44 seconds: 73.84 ops/second llite.lustre-ffff9d232ec32000.statahead_batch_max=0 llite.lustre-ffff9d232ec32000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d232ec32000.stats=clear mdc.lustre-MDT0001-mdc-ffff9d232ec32000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m56.874s user 0m0.317s sys 0m18.447s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9d232ec32000.statahead_batch_max=32 llite.lustre-ffff9d232ec32000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d232ec32000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9d232ec32000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9d232ec32000.stats=clear mdc.lustre-MDT0001-mdc-ffff9d232ec32000.stats=clear ls -l files in d123ad.sanity: 10001 real 2m1.377s user 0m0.312s sys 0m15.864s batched statahead RPC= hit count=9987 mdc.lustre-MDT0000-mdc-ffff9d232ec32000.batch_stats= snapshot_time: 1788515486.764299385 secs.nsecs start_time: 1788515352.392884999 secs.nsecs elapsed_time: 134.371414386 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9d232ec32000.batch_stats= snapshot_time: 1788515486.764364853 secs.nsecs start_time: 1788515352.392969029 secs.nsecs elapsed_time: 134.371395824 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9d232ec32000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d232ec32000.batch_stats=0 llite.lustre-ffff9d232ec32000.statahead_max=2048 llite.lustre-ffff9d232ec32000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9d232ec32000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d232ec32000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.64 seconds: 37.82 ops/second llite.lustre-ffff9d232ec32000.statahead_max=0 101 real 0m2.824s user 0m0.003s sys 0m0.357s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9d232ec32000.statahead_max=2048 llite.lustre-ffff9d232ec32000.statahead_max=2048 101 real 0m1.004s user 0m0.005s sys 0m0.217s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10086 miss_total: 14 swrong=0 ewrong=0 - open/close 701 (time 1788515829.77 total 10.00 last 70.07) total: 900 open/close in 13.19 seconds: 68.21 ops/second llite.lustre-ffff9d232ec32000.statahead_max=0 1001 real 0m27.546s user 0m0.042s sys 0m3.612s 'ls -l' 1000 files without statahead: 27 sec llite.lustre-ffff9d232ec32000.statahead_max=2048 llite.lustre-ffff9d232ec32000.statahead_max=2048 1001 real 0m7.174s user 0m0.037s sys 0m2.005s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11085 miss_total: 15 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff9d232ec32000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11085 miss_total: 15 mdc.lustre-MDT0000-mdc-ffff9d232ec32000.batch_stats= snapshot_time: 1788515900.889990526 secs.nsecs start_time: 1788515489.184807936 secs.nsecs elapsed_time: 411.705182590 secs.nsecs subreqs per batch batches %% cum %% 1: 2 28 28 2: 2 28 57 4: 1 14 71 8: 0 0 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff9d232ec32000.batch_stats= snapshot_time: 1788515900.890066553 secs.nsecs start_time: 1788515489.184939683 secs.nsecs elapsed_time: 411.705126870 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 3 30 30 16: 2 20 50 32: 3 30 80 64: 0 0 80 128: 0 0 80 256: 2 20 100 - open/close 738 (time 1788515912.89 total 10.01 last 73.72) - open/close 1477 (time 1788515922.90 total 20.01 last 73.88) - open/close 2089 (time 1788515932.93 total 30.04 last 61.02) - open/close 2768 (time 1788515942.93 total 40.05 last 67.86) - open/close 3487 (time 1788515952.93 total 50.05 last 71.88) - open/close 4135 (time 1788515962.94 total 60.06 last 64.78) - open/close 4832 (time 1788515972.95 total 70.07 last 69.62) - open/close 5575 (time 1788515982.95 total 80.07 last 74.27) - open/close 6323 (time 1788515992.96 total 90.08 last 74.75) - open/close 6960 (time 1788516002.97 total 100.09 last 63.62) - open/close 7658 (time 1788516012.98 total 110.10 last 69.76) - open/close 8370 (time 1788516022.98 total 120.10 last 71.18) - open/close 9116 (time 1788516033.00 total 130.12 last 74.45) - open/close 9798 (time 1788516043.01 total 140.13 last 68.16) total: 10000 open/close in 142.77 seconds: 70.04 ops/second llite.lustre-ffff9d232ec32000.statahead_batch_max=0 llite.lustre-ffff9d232ec32000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d232ec32000.stats=clear mdc.lustre-MDT0001-mdc-ffff9d232ec32000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m9.227s user 0m0.317s sys 0m16.623s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9d232ec32000.statahead_batch_max=256 llite.lustre-ffff9d232ec32000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d232ec32000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9d232ec32000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9d232ec32000.stats=clear mdc.lustre-MDT0001-mdc-ffff9d232ec32000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m4.197s user 0m0.298s sys 0m16.224s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9d232ec32000.batch_stats= snapshot_time: 1788516204.648277399 secs.nsecs start_time: 1788516125.344499715 secs.nsecs elapsed_time: 79.303777684 secs.nsecs subreqs per batch batches %% cum %% 1: 178 51 51 2: 50 14 65 4: 31 8 74 8: 16 4 79 16: 12 3 82 32: 21 6 89 64: 3 0 89 128: 1 0 90 256: 34 9 100 mdc.lustre-MDT0001-mdc-ffff9d232ec32000.batch_stats= snapshot_time: 1788516204.648373512 secs.nsecs start_time: 1788516125.348418630 secs.nsecs elapsed_time: 79.299954882 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9d232ec32000.statahead_batch_max=64 llite.lustre-ffff9d232ec32000.statahead_max=128