== sanity test 123ad: Verify batching statahead works correctly ========================================================== 17:03:25 (1784235805) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9fcf90c33000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9fcf90c33000.batch_stats=0 llite.lustre-ffff9fcf90c33000.statahead_max=32 llite.lustre-ffff9fcf90c33000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9fcf90c33000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9fcf90c33000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 1.85 seconds: 54.10 ops/second llite.lustre-ffff9fcf90c33000.statahead_max=0 101 real 0m1.813s user 0m0.006s sys 0m0.317s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9fcf90c33000.statahead_max=32 llite.lustre-ffff9fcf90c33000.statahead_max=32 101 real 0m0.824s user 0m0.006s sys 0m0.242s 'ls -l' 100 files with statahead: 1 sec statahead total: 73 statahead wrong: 1 agl total: 69 list_total: 29 fname_total: 41 hit_total: 3573 miss_total: 245 swrong=1 ewrong=1 total: 900 open/close in 8.28 seconds: 108.73 ops/second llite.lustre-ffff9fcf90c33000.statahead_max=0 1001 real 0m15.663s user 0m0.036s sys 0m2.894s 'ls -l' 1000 files without statahead: 16 sec llite.lustre-ffff9fcf90c33000.statahead_max=32 llite.lustre-ffff9fcf90c33000.statahead_max=32 1001 real 0m6.325s user 0m0.025s sys 0m1.422s 'ls -l' 1000 files with statahead: 6 sec statahead total: 74 statahead wrong: 1 agl total: 70 list_total: 30 fname_total: 41 hit_total: 4569 miss_total: 249 swrong=1 ewrong=1 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 12 seconds llite.lustre-ffff9fcf90c33000.statahead_stats= statahead total: 74 statahead wrong: 1 agl total: 70 list_total: 30 fname_total: 41 hit_total: 4569 miss_total: 249 mdc.lustre-MDT0000-mdc-ffff9fcf90c33000.batch_stats= snapshot_time: 1784235867.372695582 secs.nsecs start_time: 1784235808.527260679 secs.nsecs elapsed_time: 58.845434903 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 3 8 10 4: 1 2 13 8: 2 5 18 16: 25 67 86 32: 5 13 100 mdc.lustre-MDT0001-mdc-ffff9fcf90c33000.batch_stats= snapshot_time: 1784235867.372763279 secs.nsecs start_time: 1784235808.527333927 secs.nsecs elapsed_time: 58.845429352 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 2 2 8: 4 10 12 16: 9 23 35 32: 25 64 100 - open/close 1109 (time 1784235879.13 total 10.01 last 110.83) - open/close 2212 (time 1784235889.13 total 20.01 last 110.25) - open/close 3289 (time 1784235899.15 total 30.03 last 107.50) - open/close 4387 (time 1784235909.16 total 40.03 last 109.75) - open/close 5441 (time 1784235919.16 total 50.03 last 105.40) - open/close 7197 (time 1784235929.16 total 60.04 last 175.57) - open/close 9364 (time 1784235939.16 total 70.04 last 216.67) total: 10000 open/close in 75.59 seconds: 132.30 ops/second llite.lustre-ffff9fcf90c33000.statahead_batch_max=0 llite.lustre-ffff9fcf90c33000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9fcf90c33000.stats=clear mdc.lustre-MDT0001-mdc-ffff9fcf90c33000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m41.835s user 0m0.305s sys 0m19.336s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9fcf90c33000.statahead_batch_max=32 llite.lustre-ffff9fcf90c33000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9fcf90c33000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9fcf90c33000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9fcf90c33000.stats=clear mdc.lustre-MDT0001-mdc-ffff9fcf90c33000.stats=clear ls -l files in d123ad.sanity: 10001 real 2m4.640s user 0m0.293s sys 0m15.831s batched statahead RPC= hit count=9976 mdc.lustre-MDT0000-mdc-ffff9fcf90c33000.batch_stats= snapshot_time: 1784236124.812083380 secs.nsecs start_time: 1784235991.796609424 secs.nsecs elapsed_time: 133.015473956 secs.nsecs subreqs per batch batches %% cum %% 1: 1 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9fcf90c33000.batch_stats= snapshot_time: 1784236124.812151740 secs.nsecs start_time: 1784235991.796744871 secs.nsecs elapsed_time: 133.015406869 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9fcf90c33000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9fcf90c33000.batch_stats=0 llite.lustre-ffff9fcf90c33000.statahead_max=2048 llite.lustre-ffff9fcf90c33000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9fcf90c33000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9fcf90c33000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.31 seconds: 43.26 ops/second llite.lustre-ffff9fcf90c33000.statahead_max=0 101 real 0m2.550s user 0m0.002s sys 0m0.336s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9fcf90c33000.statahead_max=2048 llite.lustre-ffff9fcf90c33000.statahead_max=2048 101 real 0m0.653s user 0m0.006s sys 0m0.187s 'ls -l' 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10075 miss_total: 25 swrong=0 ewrong=0 - open/close 874 (time 1784236410.06 total 10.01 last 87.30) total: 900 open/close in 10.24 seconds: 87.88 ops/second llite.lustre-ffff9fcf90c33000.statahead_max=0 1001 real 0m20.983s user 0m0.049s sys 0m3.088s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff9fcf90c33000.statahead_max=2048 llite.lustre-ffff9fcf90c33000.statahead_max=2048 1001 real 0m7.300s user 0m0.041s sys 0m1.523s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11074 miss_total: 26 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff9fcf90c33000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11074 miss_total: 26 mdc.lustre-MDT0000-mdc-ffff9fcf90c33000.batch_stats= snapshot_time: 1784236472.152987644 secs.nsecs start_time: 1784236127.555476907 secs.nsecs elapsed_time: 344.597510737 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 37 37 8: 2 25 62 16: 0 0 62 32: 1 12 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff9fcf90c33000.batch_stats= snapshot_time: 1784236472.153062784 secs.nsecs start_time: 1784236127.555703338 secs.nsecs elapsed_time: 344.597359446 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 37 37 8: 1 12 50 16: 1 12 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 - open/close 889 (time 1784236484.76 total 10.01 last 88.80) - open/close 1721 (time 1784236494.76 total 20.01 last 83.19) - open/close 2666 (time 1784236504.76 total 30.01 last 94.48) - open/close 3625 (time 1784236514.76 total 40.02 last 95.90) - open/close 4574 (time 1784236524.76 total 50.02 last 94.88) - open/close 5542 (time 1784236534.77 total 60.03 last 96.68) - open/close 6325 (time 1784236544.78 total 70.04 last 78.22) - open/close 7131 (time 1784236554.80 total 80.06 last 80.45) - open/close 8049 (time 1784236564.82 total 90.07 last 91.66) - open/close 8966 (time 1784236574.84 total 100.09 last 91.51) - open/close 9838 (time 1784236584.84 total 110.10 last 87.15) total: 10000 open/close in 111.78 seconds: 89.46 ops/second llite.lustre-ffff9fcf90c33000.statahead_batch_max=0 llite.lustre-ffff9fcf90c33000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9fcf90c33000.stats=clear mdc.lustre-MDT0001-mdc-ffff9fcf90c33000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m54.798s user 0m0.312s sys 0m15.402s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9fcf90c33000.statahead_batch_max=256 llite.lustre-ffff9fcf90c33000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9fcf90c33000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9fcf90c33000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9fcf90c33000.stats=clear mdc.lustre-MDT0001-mdc-ffff9fcf90c33000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m55.848s user 0m0.326s sys 0m15.572s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9fcf90c33000.batch_stats= snapshot_time: 1784236720.933545402 secs.nsecs start_time: 1784236649.841074248 secs.nsecs elapsed_time: 71.092471154 secs.nsecs subreqs per batch batches %% cum %% 1: 140 44 44 2: 55 17 62 4: 24 7 70 8: 26 8 78 16: 10 3 81 32: 15 4 86 64: 6 1 88 128: 2 0 89 256: 34 10 100 mdc.lustre-MDT0001-mdc-ffff9fcf90c33000.batch_stats= snapshot_time: 1784236720.933610889 secs.nsecs start_time: 1784236649.841155253 secs.nsecs elapsed_time: 71.092455636 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9fcf90c33000.statahead_batch_max=64 llite.lustre-ffff9fcf90c33000.statahead_max=128