== sanity test 123ad: Verify batching statahead works correctly ========================================================== 16:12:43 (1788466363) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff89c1f0a31000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff89c1f0a31000.batch_stats=0 llite.lustre-ffff89c1f0a31000.statahead_max=32 llite.lustre-ffff89c1f0a31000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff89c1f0a31000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff89c1f0a31000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.99 seconds: 33.48 ops/second llite.lustre-ffff89c1f0a31000.statahead_max=0 101 real 0m2.319s user 0m0.005s sys 0m0.302s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff89c1f0a31000.statahead_max=32 llite.lustre-ffff89c1f0a31000.statahead_max=32 101 real 0m1.167s user 0m0.006s sys 0m0.160s 'ls -l' 100 files with statahead: 1 sec statahead total: 138 statahead wrong: 2 agl total: 132 list_total: 30 fname_total: 97 hit_total: 4043 miss_total: 1486 swrong=2 ewrong=2 - open/close 815 (time 1788466391.53 total 10.01 last 81.44) total: 900 open/close in 11.12 seconds: 80.96 ops/second llite.lustre-ffff89c1f0a31000.statahead_max=0 1001 real 0m21.038s user 0m0.039s sys 0m3.040s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff89c1f0a31000.statahead_max=32 llite.lustre-ffff89c1f0a31000.statahead_max=32 1001 real 0m8.622s user 0m0.037s sys 0m1.468s 'ls -l' 1000 files with statahead: 9 sec statahead total: 139 statahead wrong: 2 agl total: 133 list_total: 31 fname_total: 97 hit_total: 5040 miss_total: 1489 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff89c1f0a31000.statahead_stats= statahead total: 139 statahead wrong: 2 agl total: 133 list_total: 31 fname_total: 97 hit_total: 5040 miss_total: 1489 mdc.lustre-MDT0000-mdc-ffff89c1f0a31000.batch_stats= snapshot_time: 1788466452.670263531 secs.nsecs start_time: 1788466368.676202428 secs.nsecs elapsed_time: 83.994061103 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 1 2 8 4: 3 8 16 8: 1 2 18 16: 23 62 81 32: 7 18 100 mdc.lustre-MDT0001-mdc-ffff89c1f0a31000.batch_stats= snapshot_time: 1788466452.670339253 secs.nsecs start_time: 1788466368.676363213 secs.nsecs elapsed_time: 83.993976040 secs.nsecs subreqs per batch batches %% cum %% 1: 2 4 4 2: 0 0 4 4: 1 2 7 8: 3 7 14 16: 11 26 41 32: 24 58 100 - open/close 792 (time 1788466464.98 total 10.01 last 79.12) - open/close 1591 (time 1788466474.98 total 20.01 last 79.87) - open/close 2457 (time 1788466484.99 total 30.02 last 86.55) - open/close 3240 (time 1788466495.00 total 40.03 last 78.25) - open/close 4030 (time 1788466505.01 total 50.04 last 78.93) - open/close 4862 (time 1788466515.01 total 60.04 last 83.19) - open/close 5646 (time 1788466525.01 total 70.04 last 78.38) - open/close 6444 (time 1788466535.02 total 80.05 last 79.76) - open/close 7202 (time 1788466545.03 total 90.06 last 75.67) - open/close 7995 (time 1788466555.05 total 100.08 last 79.14) - open/close 8758 (time 1788466565.06 total 110.09 last 76.27) - open/close 9563 (time 1788466575.07 total 120.10 last 80.37) total: 10000 open/close in 125.08 seconds: 79.95 ops/second llite.lustre-ffff89c1f0a31000.statahead_batch_max=0 llite.lustre-ffff89c1f0a31000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff89c1f0a31000.stats=clear mdc.lustre-MDT0001-mdc-ffff89c1f0a31000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m57.711s user 0m0.310s sys 0m16.857s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff89c1f0a31000.statahead_batch_max=32 llite.lustre-ffff89c1f0a31000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff89c1f0a31000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff89c1f0a31000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff89c1f0a31000.stats=clear mdc.lustre-MDT0001-mdc-ffff89c1f0a31000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m27.503s user 0m0.317s sys 0m14.810s batched statahead RPC= hit count=9982 mdc.lustre-MDT0000-mdc-ffff89c1f0a31000.batch_stats= snapshot_time: 1788466741.783849870 secs.nsecs start_time: 1788466645.077416290 secs.nsecs elapsed_time: 96.706433580 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff89c1f0a31000.batch_stats= snapshot_time: 1788466741.783919305 secs.nsecs start_time: 1788466645.077507466 secs.nsecs elapsed_time: 96.706411839 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff89c1f0a31000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff89c1f0a31000.batch_stats=0 llite.lustre-ffff89c1f0a31000.statahead_max=2048 llite.lustre-ffff89c1f0a31000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff89c1f0a31000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff89c1f0a31000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.36 seconds: 42.38 ops/second llite.lustre-ffff89c1f0a31000.statahead_max=0 101 real 0m2.070s user 0m0.004s sys 0m0.310s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff89c1f0a31000.statahead_max=2048 llite.lustre-ffff89c1f0a31000.statahead_max=2048 101 real 0m0.726s user 0m0.003s sys 0m0.155s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10081 miss_total: 19 swrong=0 ewrong=0 total: 900 open/close in 8.77 seconds: 102.59 ops/second llite.lustre-ffff89c1f0a31000.statahead_max=0 1001 real 0m17.651s user 0m0.028s sys 0m2.759s 'ls -l' 1000 files without statahead: 18 sec llite.lustre-ffff89c1f0a31000.statahead_max=2048 llite.lustre-ffff89c1f0a31000.statahead_max=2048 1001 real 0m6.018s user 0m0.026s sys 0m1.381s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 16 seconds llite.lustre-ffff89c1f0a31000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 mdc.lustre-MDT0000-mdc-ffff89c1f0a31000.batch_stats= snapshot_time: 1788467056.804232777 secs.nsecs start_time: 1788466743.843369188 secs.nsecs elapsed_time: 312.960863589 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 11 11 4: 1 11 22 8: 3 33 55 16: 1 11 66 32: 0 0 66 64: 1 11 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff89c1f0a31000.batch_stats= snapshot_time: 1788467056.804305492 secs.nsecs start_time: 1788466743.843651283 secs.nsecs elapsed_time: 312.960654209 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 22 22 8: 3 33 55 16: 1 11 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 - open/close 788 (time 1788467069.01 total 10.00 last 78.76) - open/close 1563 (time 1788467079.02 total 20.01 last 77.43) - open/close 2361 (time 1788467089.02 total 30.02 last 79.78) - open/close 3236 (time 1788467099.03 total 40.02 last 87.45) - open/close 4144 (time 1788467109.04 total 50.03 last 90.69) - open/close 5117 (time 1788467119.04 total 60.04 last 97.28) - open/close 6070 (time 1788467129.05 total 70.04 last 95.26) - open/close 7032 (time 1788467139.05 total 80.05 last 96.16) - open/close 7966 (time 1788467149.07 total 90.07 last 93.21) - open/close 8927 (time 1788467159.07 total 100.07 last 96.09) - open/close 9877 (time 1788467169.08 total 110.08 last 94.92) total: 10000 open/close in 111.44 seconds: 89.74 ops/second llite.lustre-ffff89c1f0a31000.statahead_batch_max=0 llite.lustre-ffff89c1f0a31000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff89c1f0a31000.stats=clear mdc.lustre-MDT0001-mdc-ffff89c1f0a31000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m40.851s user 0m0.264s sys 0m19.235s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff89c1f0a31000.statahead_batch_max=256 llite.lustre-ffff89c1f0a31000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff89c1f0a31000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff89c1f0a31000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff89c1f0a31000.stats=clear mdc.lustre-MDT0001-mdc-ffff89c1f0a31000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m38.637s user 0m0.251s sys 0m16.813s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff89c1f0a31000.batch_stats= snapshot_time: 1788467263.260409204 secs.nsecs start_time: 1788467217.431331257 secs.nsecs elapsed_time: 45.829077947 secs.nsecs subreqs per batch batches %% cum %% 1: 225 57 57 2: 42 10 68 4: 41 10 78 8: 19 4 83 16: 3 0 84 32: 15 3 88 64: 9 2 90 128: 3 0 91 256: 33 8 100 mdc.lustre-MDT0001-mdc-ffff89c1f0a31000.batch_stats= snapshot_time: 1788467263.260474501 secs.nsecs start_time: 1788467217.431669831 secs.nsecs elapsed_time: 45.828804670 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff89c1f0a31000.statahead_batch_max=64 llite.lustre-ffff89c1f0a31000.statahead_max=128