== sanity test 123ad: Verify batching statahead works correctly ========================================================== 19:57:24 (1788566244) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff913c04f06000.batch_stats=0 llite.lustre-ffff913c04f06000.statahead_max=32 llite.lustre-ffff913c04f06000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff913c04f06000.batch_stats=0 total: 100 open/close in 3.43 seconds: 29.13 ops/second llite.lustre-ffff913c04f06000.statahead_max=0 101 real 0m3.255s user 0m0.007s sys 0m0.310s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff913c04f06000.statahead_max=32 llite.lustre-ffff913c04f06000.statahead_max=32 101 real 0m1.759s user 0m0.005s sys 0m0.159s 'ls -l' 100 files with statahead: 2 sec statahead total: 170 statahead wrong: 2 agl total: 163 list_total: 30 fname_total: 140 hit_total: 3742 miss_total: 1373 swrong=2 ewrong=2 - open/close 620 (time 1788566278.72 total 10.01 last 61.92) total: 900 open/close in 15.62 seconds: 57.60 ops/second llite.lustre-ffff913c04f06000.statahead_max=0 1001 real 0m25.308s user 0m0.035s sys 0m3.224s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff913c04f06000.statahead_max=32 llite.lustre-ffff913c04f06000.statahead_max=32 1001 real 0m12.605s user 0m0.034s sys 0m1.425s 'ls -l' 1000 files with statahead: 12 sec statahead total: 171 statahead wrong: 2 agl total: 164 list_total: 31 fname_total: 140 hit_total: 4740 miss_total: 1375 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 30 seconds llite.lustre-ffff913c04f06000.statahead_stats= statahead total: 171 statahead wrong: 2 agl total: 164 list_total: 31 fname_total: 140 hit_total: 4740 miss_total: 1375 mdc.lustre-MDT0000-mdc-ffff913c04f06000.batch_stats= snapshot_time: 1788566368.179070449 secs.nsecs start_time: 1788566250.524912307 secs.nsecs elapsed_time: 117.654158142 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 0 0 2 8: 3 7 10 16: 2 5 15 32: 34 85 100 - open/close 538 (time 1788566381.81 total 10.00 last 53.78) - open/close 929 (time 1788566391.82 total 20.01 last 39.08) - open/close 1490 (time 1788566401.82 total 30.01 last 56.09) - open/close 2130 (time 1788566411.83 total 40.02 last 63.94) - open/close 2776 (time 1788566421.84 total 50.03 last 64.53) - open/close 3448 (time 1788566431.86 total 60.05 last 67.07) - open/close 4070 (time 1788566441.87 total 70.06 last 62.14) - open/close 4550 (time 1788566451.89 total 80.07 last 47.93) - open/close 5053 (time 1788566461.89 total 90.08 last 50.30) - open/close 5661 (time 1788566471.90 total 100.09 last 60.73) - open/close 6315 (time 1788566481.90 total 110.09 last 65.37) - open/close 6862 (time 1788566491.91 total 120.10 last 54.68) - open/close 7510 (time 1788566501.92 total 130.10 last 64.74) - open/close 8256 (time 1788566511.92 total 140.10 last 74.60) - open/close 8950 (time 1788566521.92 total 150.11 last 69.34) - open/close 9557 (time 1788566531.93 total 160.12 last 60.68) total: 10000 open/close in 166.55 seconds: 60.04 ops/second llite.lustre-ffff913c04f06000.statahead_batch_max=0 llite.lustre-ffff913c04f06000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff913c04f06000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m17.754s user 0m0.328s sys 0m13.610s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff913c04f06000.statahead_batch_max=32 llite.lustre-ffff913c04f06000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff913c04f06000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff913c04f06000.stats=clear ls -l files in d123ad.sanity: 10001 real 2m11.188s user 0m0.334s sys 0m13.302s batched statahead RPC= hit count=9973 mdc.lustre-MDT0000-mdc-ffff913c04f06000.batch_stats= snapshot_time: 1788566770.031721326 secs.nsecs start_time: 1788566626.069210164 secs.nsecs elapsed_time: 143.962511162 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 2 0 0 16: 1 0 0 32: 313 99 100 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff913c04f06000.batch_stats=0 llite.lustre-ffff913c04f06000.statahead_max=2048 llite.lustre-ffff913c04f06000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff913c04f06000.batch_stats=0 total: 100 open/close in 2.60 seconds: 38.46 ops/second llite.lustre-ffff913c04f06000.statahead_max=0 101 real 0m2.752s user 0m0.005s sys 0m0.286s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff913c04f06000.statahead_max=2048 llite.lustre-ffff913c04f06000.statahead_max=2048 101 real 0m0.905s user 0m0.007s sys 0m0.148s 'ls -l' 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10072 miss_total: 28 swrong=0 ewrong=0 - open/close 607 (time 1788567140.69 total 10.01 last 60.64) total: 900 open/close in 15.66 seconds: 57.46 ops/second llite.lustre-ffff913c04f06000.statahead_max=0 1001 real 0m28.994s user 0m0.045s sys 0m2.990s 'ls -l' 1000 files without statahead: 29 sec llite.lustre-ffff913c04f06000.statahead_max=2048 llite.lustre-ffff913c04f06000.statahead_max=2048 1001 real 0m7.668s user 0m0.035s sys 0m1.302s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11071 miss_total: 29 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 29 seconds llite.lustre-ffff913c04f06000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11071 miss_total: 29 mdc.lustre-MDT0000-mdc-ffff913c04f06000.batch_stats= snapshot_time: 1788567233.371008152 secs.nsecs start_time: 1788566773.913659504 secs.nsecs elapsed_time: 459.457348648 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 2 18 18 16: 2 18 36 32: 2 18 54 64: 0 0 54 128: 1 9 63 256: 4 36 100 - open/close 611 (time 1788567246.57 total 10.01 last 61.03) - open/close 1175 (time 1788567256.57 total 20.02 last 56.36) - open/close 1659 (time 1788567266.58 total 30.02 last 48.37) - open/close 2030 (time 1788567276.59 total 40.04 last 37.06) - open/close 2673 (time 1788567286.60 total 50.04 last 64.25) - open/close 3113 (time 1788567296.62 total 60.06 last 43.91) - open/close 3737 (time 1788567306.62 total 70.06 last 62.40) - open/close 4451 (time 1788567316.63 total 80.07 last 71.34) - open/close 5121 (time 1788567326.63 total 90.07 last 66.99) - open/close 5796 (time 1788567336.64 total 100.08 last 67.45) - open/close 6611 (time 1788567346.65 total 110.09 last 81.42) - open/close 7467 (time 1788567356.65 total 120.09 last 85.58) - open/close 8254 (time 1788567366.65 total 130.10 last 78.67) - open/close 8871 (time 1788567376.66 total 140.10 last 61.67) - open/close 9316 (time 1788567386.66 total 150.11 last 44.48) - open/close 9832 (time 1788567396.66 total 160.11 last 51.59) total: 10000 open/close in 163.10 seconds: 61.31 ops/second llite.lustre-ffff913c04f06000.statahead_batch_max=0 llite.lustre-ffff913c04f06000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff913c04f06000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m6.761s user 0m0.308s sys 0m13.253s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff913c04f06000.statahead_batch_max=256 llite.lustre-ffff913c04f06000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff913c04f06000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff913c04f06000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m0.429s user 0m0.302s sys 0m12.740s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff913c04f06000.batch_stats= snapshot_time: 1788567550.961759108 secs.nsecs start_time: 1788567478.718263476 secs.nsecs elapsed_time: 72.243495632 secs.nsecs subreqs per batch batches %% cum %% 1: 103 39 39 2: 42 15 54 4: 27 10 65 8: 24 9 74 16: 14 5 79 32: 10 3 83 64: 6 2 85 128: 4 1 87 256: 34 12 100 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff913c04f06000.statahead_batch_max=64 llite.lustre-ffff913c04f06000.statahead_max=128