== sanity test 123ad: Verify batching statahead works correctly ========================================================== 23:55:14 (1788839714) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8fbc93f84000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8fbc93f84000.batch_stats=0 llite.lustre-ffff8fbc93f84000.statahead_max=32 llite.lustre-ffff8fbc93f84000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8fbc93f84000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8fbc93f84000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.44 seconds: 40.97 ops/second llite.lustre-ffff8fbc93f84000.statahead_max=0 101 real 0m3.446s user 0m0.007s sys 0m0.324s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8fbc93f84000.statahead_max=32 llite.lustre-ffff8fbc93f84000.statahead_max=32 101 real 0m1.413s user 0m0.003s sys 0m0.202s 'ls -l' 100 files with statahead: 1 sec statahead total: 122 statahead wrong: 2 agl total: 117 list_total: 31 fname_total: 91 hit_total: 4503 miss_total: 1292 swrong=2 ewrong=2 - open/close 851 (time 1788839745.34 total 10.01 last 85.03) total: 900 open/close in 10.89 seconds: 82.62 ops/second llite.lustre-ffff8fbc93f84000.statahead_max=0 1001 real 0m20.611s user 0m0.029s sys 0m2.878s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff8fbc93f84000.statahead_max=32 llite.lustre-ffff8fbc93f84000.statahead_max=32 1001 real 0m9.191s user 0m0.029s sys 0m1.387s 'ls -l' 1000 files with statahead: 9 sec statahead total: 123 statahead wrong: 2 agl total: 118 list_total: 32 fname_total: 91 hit_total: 5500 miss_total: 1295 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 23 seconds llite.lustre-ffff8fbc93f84000.statahead_stats= statahead total: 123 statahead wrong: 2 agl total: 118 list_total: 32 fname_total: 91 hit_total: 5500 miss_total: 1295 mdc.lustre-MDT0000-mdc-ffff8fbc93f84000.batch_stats= snapshot_time: 1788839813.159330672 secs.nsecs start_time: 1788839719.921977100 secs.nsecs elapsed_time: 93.237353572 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 2 5 7 8: 3 7 15 16: 20 51 66 32: 13 33 100 mdc.lustre-MDT0001-mdc-ffff8fbc93f84000.batch_stats= snapshot_time: 1788839813.159412246 secs.nsecs start_time: 1788839719.922260963 secs.nsecs elapsed_time: 93.237151283 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 3 7 10 8: 1 2 12 16: 19 48 61 32: 15 38 100 - open/close 512 (time 1788839826.32 total 10.01 last 51.17) - open/close 1155 (time 1788839836.34 total 20.02 last 64.21) - open/close 2066 (time 1788839846.34 total 30.03 last 91.04) - open/close 2969 (time 1788839856.35 total 40.04 last 90.21) - open/close 3735 (time 1788839866.36 total 50.04 last 76.58) - open/close 4524 (time 1788839876.36 total 60.04 last 78.86) - open/close 5340 (time 1788839886.36 total 70.05 last 81.57) - open/close 6113 (time 1788839896.39 total 80.07 last 77.13) - open/close 6977 (time 1788839906.39 total 90.07 last 86.36) - open/close 7911 (time 1788839916.40 total 100.08 last 93.35) - open/close 8723 (time 1788839926.42 total 110.10 last 80.99) - open/close 9603 (time 1788839936.43 total 120.11 last 87.96) total: 10000 open/close in 124.84 seconds: 80.10 ops/second llite.lustre-ffff8fbc93f84000.statahead_batch_max=0 llite.lustre-ffff8fbc93f84000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8fbc93f84000.stats=clear mdc.lustre-MDT0001-mdc-ffff8fbc93f84000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.200s user 0m0.323s sys 0m14.061s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8fbc93f84000.statahead_batch_max=32 llite.lustre-ffff8fbc93f84000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8fbc93f84000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8fbc93f84000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8fbc93f84000.stats=clear mdc.lustre-MDT0001-mdc-ffff8fbc93f84000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m47.082s user 0m0.284s sys 0m13.316s batched statahead RPC= hit count=9974 mdc.lustre-MDT0000-mdc-ffff8fbc93f84000.batch_stats= snapshot_time: 1788840129.995329470 secs.nsecs start_time: 1788840011.517682723 secs.nsecs elapsed_time: 118.477646747 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 2 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8fbc93f84000.batch_stats= snapshot_time: 1788840129.995414120 secs.nsecs start_time: 1788840011.517756530 secs.nsecs elapsed_time: 118.477657590 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8fbc93f84000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8fbc93f84000.batch_stats=0 llite.lustre-ffff8fbc93f84000.statahead_max=2048 llite.lustre-ffff8fbc93f84000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8fbc93f84000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8fbc93f84000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.34 seconds: 42.72 ops/second llite.lustre-ffff8fbc93f84000.statahead_max=0 101 real 0m1.930s user 0m0.005s sys 0m0.291s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8fbc93f84000.statahead_max=2048 llite.lustre-ffff8fbc93f84000.statahead_max=2048 101 real 0m0.969s user 0m0.007s sys 0m0.177s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10073 miss_total: 27 swrong=0 ewrong=0 total: 900 open/close in 9.19 seconds: 97.89 ops/second llite.lustre-ffff8fbc93f84000.statahead_max=0 1001 real 0m18.671s user 0m0.037s sys 0m2.829s 'ls -l' 1000 files without statahead: 18 sec llite.lustre-ffff8fbc93f84000.statahead_max=2048 llite.lustre-ffff8fbc93f84000.statahead_max=2048 1001 real 0m5.760s user 0m0.032s sys 0m1.270s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11072 miss_total: 28 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff8fbc93f84000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11072 miss_total: 28 mdc.lustre-MDT0000-mdc-ffff8fbc93f84000.batch_stats= snapshot_time: 1788840470.547194835 secs.nsecs start_time: 1788840132.845412520 secs.nsecs elapsed_time: 337.701782315 secs.nsecs subreqs per batch batches %% cum %% 1: 2 25 25 2: 0 0 25 4: 2 25 50 8: 1 12 62 16: 1 12 75 32: 0 0 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff8fbc93f84000.batch_stats= snapshot_time: 1788840470.547291090 secs.nsecs start_time: 1788840132.845575130 secs.nsecs elapsed_time: 337.701715960 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 10 10 8: 3 30 40 16: 2 20 60 32: 1 10 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 - open/close 717 (time 1788840483.38 total 10.00 last 71.69) - open/close 1514 (time 1788840493.38 total 20.01 last 79.67) - open/close 2498 (time 1788840503.39 total 30.01 last 98.32) - open/close 3363 (time 1788840513.39 total 40.02 last 86.48) - open/close 4218 (time 1788840523.40 total 50.03 last 85.42) - open/close 4977 (time 1788840533.42 total 60.05 last 75.75) - open/close 5605 (time 1788840543.43 total 70.06 last 62.74) - open/close 6408 (time 1788840553.44 total 80.06 last 80.25) - open/close 7127 (time 1788840563.46 total 90.08 last 71.77) - open/close 7892 (time 1788840573.47 total 100.09 last 76.41) - open/close 8766 (time 1788840583.47 total 110.10 last 87.37) - open/close 9498 (time 1788840593.48 total 120.10 last 73.17) total: 10000 open/close in 125.45 seconds: 79.71 ops/second llite.lustre-ffff8fbc93f84000.statahead_batch_max=0 llite.lustre-ffff8fbc93f84000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8fbc93f84000.stats=clear mdc.lustre-MDT0001-mdc-ffff8fbc93f84000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m0.135s user 0m0.315s sys 0m13.672s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8fbc93f84000.statahead_batch_max=256 llite.lustre-ffff8fbc93f84000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8fbc93f84000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8fbc93f84000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8fbc93f84000.stats=clear mdc.lustre-MDT0001-mdc-ffff8fbc93f84000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m47.243s user 0m0.289s sys 0m13.448s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8fbc93f84000.batch_stats= snapshot_time: 1788840729.163852756 secs.nsecs start_time: 1788840668.512610638 secs.nsecs elapsed_time: 60.651242118 secs.nsecs subreqs per batch batches %% cum %% 1: 93 41 41 2: 31 13 55 4: 19 8 64 8: 11 4 69 16: 9 4 73 32: 16 7 80 64: 8 3 84 128: 0 0 84 256: 35 15 100 mdc.lustre-MDT0001-mdc-ffff8fbc93f84000.batch_stats= snapshot_time: 1788840729.163944411 secs.nsecs start_time: 1788840668.512759828 secs.nsecs elapsed_time: 60.651184583 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8fbc93f84000.statahead_batch_max=64 llite.lustre-ffff8fbc93f84000.statahead_max=128