== sanity test 123ad: Verify batching statahead works correctly ========================================================== 10:35:34 (1790951734) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8d89581ce000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8d89581ce000.batch_stats=0 llite.lustre-ffff8d89581ce000.statahead_max=32 llite.lustre-ffff8d89581ce000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8d89581ce000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8d89581ce000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.72 seconds: 36.71 ops/second llite.lustre-ffff8d89581ce000.statahead_max=0 101 real 0m2.088s user 0m0.004s sys 0m0.299s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8d89581ce000.statahead_max=32 llite.lustre-ffff8d89581ce000.statahead_max=32 101 real 0m1.183s user 0m0.003s sys 0m0.176s 'ls -l' 100 files with statahead: 1 sec statahead total: 130 statahead wrong: 2 agl total: 124 list_total: 30 fname_total: 63 hit_total: 3809 miss_total: 1619 swrong=2 ewrong=2 - open/close 840 (time 1790951763.04 total 10.00 last 83.97) total: 900 open/close in 10.79 seconds: 83.41 ops/second llite.lustre-ffff8d89581ce000.statahead_max=0 1001 real 0m19.652s user 0m0.038s sys 0m2.821s 'ls -l' 1000 files without statahead: 19 sec llite.lustre-ffff8d89581ce000.statahead_max=32 llite.lustre-ffff8d89581ce000.statahead_max=32 1001 real 0m8.228s user 0m0.025s sys 0m1.340s 'ls -l' 1000 files with statahead: 9 sec statahead total: 131 statahead wrong: 2 agl total: 125 list_total: 31 fname_total: 63 hit_total: 4807 miss_total: 1621 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff8d89581ce000.statahead_stats= statahead total: 131 statahead wrong: 2 agl total: 125 list_total: 31 fname_total: 63 hit_total: 4807 miss_total: 1621 mdc.lustre-MDT0000-mdc-ffff8d89581ce000.batch_stats= snapshot_time: 1790951820.917381404 secs.nsecs start_time: 1790951740.044878137 secs.nsecs elapsed_time: 80.872503267 secs.nsecs subreqs per batch batches %% cum %% 1: 3 7 7 2: 2 5 13 4: 2 5 18 8: 1 2 21 16: 25 65 86 32: 5 13 100 mdc.lustre-MDT0001-mdc-ffff8d89581ce000.batch_stats= snapshot_time: 1790951820.917440187 secs.nsecs start_time: 1790951740.044962361 secs.nsecs elapsed_time: 80.872477826 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 2 2 8: 4 10 12 16: 10 25 38 32: 24 61 100 - open/close 888 (time 1790951833.23 total 10.00 last 88.78) - open/close 1618 (time 1790951843.24 total 20.01 last 72.93) - open/close 2246 (time 1790951853.24 total 30.02 last 62.76) - open/close 3057 (time 1790951863.25 total 40.03 last 81.02) - open/close 3967 (time 1790951873.26 total 50.03 last 90.98) - open/close 4887 (time 1790951883.27 total 60.04 last 91.92) - open/close 5730 (time 1790951893.27 total 70.05 last 84.26) - open/close 6516 (time 1790951903.27 total 80.05 last 78.56) - open/close 7489 (time 1790951913.28 total 90.05 last 97.28) - open/close 8390 (time 1790951923.28 total 100.06 last 90.04) - open/close 9248 (time 1790951933.29 total 110.06 last 85.78) total: 10000 open/close in 118.58 seconds: 84.33 ops/second llite.lustre-ffff8d89581ce000.statahead_batch_max=0 llite.lustre-ffff8d89581ce000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8d89581ce000.stats=clear mdc.lustre-MDT0001-mdc-ffff8d89581ce000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m1.226s user 0m0.306s sys 0m15.952s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8d89581ce000.statahead_batch_max=32 llite.lustre-ffff8d89581ce000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8d89581ce000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8d89581ce000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8d89581ce000.stats=clear mdc.lustre-MDT0001-mdc-ffff8d89581ce000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m36.567s user 0m0.293s sys 0m13.923s batched statahead RPC= hit count=9978 mdc.lustre-MDT0000-mdc-ffff8d89581ce000.batch_stats= snapshot_time: 1790952117.459807971 secs.nsecs start_time: 1790952011.618318102 secs.nsecs elapsed_time: 105.841489869 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8d89581ce000.batch_stats= snapshot_time: 1790952117.459876482 secs.nsecs start_time: 1790952011.618419029 secs.nsecs elapsed_time: 105.841457453 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8d89581ce000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8d89581ce000.batch_stats=0 llite.lustre-ffff8d89581ce000.statahead_max=2048 llite.lustre-ffff8d89581ce000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8d89581ce000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8d89581ce000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.29 seconds: 43.68 ops/second llite.lustre-ffff8d89581ce000.statahead_max=0 101 real 0m2.060s user 0m0.010s sys 0m0.286s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8d89581ce000.statahead_max=2048 llite.lustre-ffff8d89581ce000.statahead_max=2048 101 real 0m0.838s user 0m0.003s sys 0m0.153s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10077 miss_total: 23 swrong=0 ewrong=0 - open/close 802 (time 1790952404.78 total 10.01 last 80.13) total: 900 open/close in 11.32 seconds: 79.48 ops/second llite.lustre-ffff8d89581ce000.statahead_max=0 1001 real 0m19.892s user 0m0.041s sys 0m2.805s 'ls -l' 1000 files without statahead: 20 sec llite.lustre-ffff8d89581ce000.statahead_max=2048 llite.lustre-ffff8d89581ce000.statahead_max=2048 1001 real 0m6.200s user 0m0.029s sys 0m1.375s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 16 seconds llite.lustre-ffff8d89581ce000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 mdc.lustre-MDT0000-mdc-ffff8d89581ce000.batch_stats= snapshot_time: 1790952458.387127042 secs.nsecs start_time: 1790952120.180753541 secs.nsecs elapsed_time: 338.206373501 secs.nsecs subreqs per batch batches %% cum %% 1: 1 11 11 2: 0 0 11 4: 2 22 33 8: 1 11 44 16: 2 22 66 32: 0 0 66 64: 1 11 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff8d89581ce000.batch_stats= snapshot_time: 1790952458.387205584 secs.nsecs start_time: 1790952120.189074744 secs.nsecs elapsed_time: 338.198130840 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 22 22 8: 4 44 66 16: 0 0 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 - open/close 925 (time 1790952470.74 total 10.01 last 92.44) - open/close 1845 (time 1790952480.75 total 20.02 last 91.85) - open/close 2581 (time 1790952490.77 total 30.04 last 73.50) - open/close 3145 (time 1790952500.78 total 40.05 last 56.32) - open/close 3754 (time 1790952510.80 total 50.07 last 60.80) - open/close 4367 (time 1790952520.80 total 60.07 last 61.26) - open/close 5134 (time 1790952530.83 total 70.10 last 76.50) - open/close 5930 (time 1790952540.83 total 80.10 last 79.59) - open/close 6749 (time 1790952550.84 total 90.11 last 81.84) - open/close 7451 (time 1790952560.85 total 100.12 last 70.15) - open/close 8238 (time 1790952570.85 total 110.12 last 78.64) - open/close 8997 (time 1790952580.87 total 120.14 last 75.79) - open/close 9836 (time 1790952590.88 total 130.15 last 83.81) total: 10000 open/close in 131.93 seconds: 75.80 ops/second llite.lustre-ffff8d89581ce000.statahead_batch_max=0 llite.lustre-ffff8d89581ce000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8d89581ce000.stats=clear mdc.lustre-MDT0001-mdc-ffff8d89581ce000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m12.380s user 0m0.303s sys 0m14.930s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8d89581ce000.statahead_batch_max=256 llite.lustre-ffff8d89581ce000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8d89581ce000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8d89581ce000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8d89581ce000.stats=clear mdc.lustre-MDT0001-mdc-ffff8d89581ce000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m57.360s user 0m0.299s sys 0m14.769s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8d89581ce000.batch_stats= snapshot_time: 1790952748.471121015 secs.nsecs start_time: 1790952673.875059959 secs.nsecs elapsed_time: 74.596061056 secs.nsecs subreqs per batch batches %% cum %% 1: 84 42 42 2: 21 10 53 4: 17 8 61 8: 13 6 68 16: 6 3 71 32: 12 6 77 64: 6 3 80 128: 5 2 82 256: 34 17 100 mdc.lustre-MDT0001-mdc-ffff8d89581ce000.batch_stats= snapshot_time: 1790952748.471200583 secs.nsecs start_time: 1790952673.875174254 secs.nsecs elapsed_time: 74.596026329 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8d89581ce000.statahead_batch_max=64 llite.lustre-ffff8d89581ce000.statahead_max=128