== sanity test 123ad: Verify batching statahead works correctly ========================================================== 00:21:41 (1789186901) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9f282e6c7000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f282e6c7000.batch_stats=0 llite.lustre-ffff9f282e6c7000.statahead_max=32 llite.lustre-ffff9f282e6c7000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9f282e6c7000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f282e6c7000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 1.88 seconds: 53.16 ops/second llite.lustre-ffff9f282e6c7000.statahead_max=0 101 real 0m1.375s user 0m0.003s sys 0m0.267s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9f282e6c7000.statahead_max=32 llite.lustre-ffff9f282e6c7000.statahead_max=32 101 real 0m0.656s user 0m0.006s sys 0m0.145s 'ls -l' 100 files with statahead: 1 sec statahead total: 140 statahead wrong: 2 agl total: 134 list_total: 31 fname_total: 81 hit_total: 3595 miss_total: 1626 swrong=2 ewrong=2 total: 900 open/close in 8.26 seconds: 108.90 ops/second llite.lustre-ffff9f282e6c7000.statahead_max=0 1001 real 0m14.240s user 0m0.026s sys 0m2.594s 'ls -l' 1000 files without statahead: 14 sec llite.lustre-ffff9f282e6c7000.statahead_max=32 llite.lustre-ffff9f282e6c7000.statahead_max=32 1001 real 0m6.093s user 0m0.018s sys 0m1.237s 'ls -l' 1000 files with statahead: 6 sec statahead total: 141 statahead wrong: 2 agl total: 135 list_total: 32 fname_total: 81 hit_total: 4593 miss_total: 1628 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 11 seconds llite.lustre-ffff9f282e6c7000.statahead_stats= statahead total: 141 statahead wrong: 2 agl total: 135 list_total: 32 fname_total: 81 hit_total: 4593 miss_total: 1628 mdc.lustre-MDT0000-mdc-ffff9f282e6c7000.batch_stats= snapshot_time: 1789186960.146989477 secs.nsecs start_time: 1789186904.740873117 secs.nsecs elapsed_time: 55.406116360 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 1 2 5 8: 4 10 15 16: 17 43 58 32: 16 41 100 mdc.lustre-MDT0001-mdc-ffff9f282e6c7000.batch_stats= snapshot_time: 1789186960.147050387 secs.nsecs start_time: 1789186904.741023546 secs.nsecs elapsed_time: 55.406026841 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 3 7 12 8: 2 5 17 16: 19 47 65 32: 14 35 100 - open/close 1118 (time 1789186972.19 total 10.00 last 111.76) - open/close 2251 (time 1789186982.20 total 20.01 last 113.25) - open/close 3540 (time 1789186992.20 total 30.01 last 128.89) - open/close 4640 (time 1789187002.21 total 40.02 last 109.93) - open/close 5830 (time 1789187012.21 total 50.02 last 118.91) - open/close 7104 (time 1789187022.22 total 60.03 last 127.36) - open/close 8163 (time 1789187032.22 total 70.03 last 105.90) - open/close 9224 (time 1789187042.22 total 80.03 last 106.08) total: 10000 open/close in 87.75 seconds: 113.96 ops/second llite.lustre-ffff9f282e6c7000.statahead_batch_max=0 llite.lustre-ffff9f282e6c7000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f282e6c7000.stats=clear mdc.lustre-MDT0001-mdc-ffff9f282e6c7000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m42.643s user 0m0.281s sys 0m15.363s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9f282e6c7000.statahead_batch_max=32 llite.lustre-ffff9f282e6c7000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f282e6c7000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9f282e6c7000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9f282e6c7000.stats=clear mdc.lustre-MDT0001-mdc-ffff9f282e6c7000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m3.249s user 0m0.225s sys 0m12.775s batched statahead RPC= hit count=9982 mdc.lustre-MDT0000-mdc-ffff9f282e6c7000.batch_stats= snapshot_time: 1789187169.317163902 secs.nsecs start_time: 1789187098.666614617 secs.nsecs elapsed_time: 70.650549285 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9f282e6c7000.batch_stats= snapshot_time: 1789187169.317233617 secs.nsecs start_time: 1789187098.666742661 secs.nsecs elapsed_time: 70.650490956 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9f282e6c7000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f282e6c7000.batch_stats=0 llite.lustre-ffff9f282e6c7000.statahead_max=2048 llite.lustre-ffff9f282e6c7000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9f282e6c7000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f282e6c7000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 1.62 seconds: 61.59 ops/second llite.lustre-ffff9f282e6c7000.statahead_max=0 101 real 0m1.482s user 0m0.007s sys 0m0.276s 'ls -l' 100 files without statahead: 1 sec llite.lustre-ffff9f282e6c7000.statahead_max=2048 llite.lustre-ffff9f282e6c7000.statahead_max=2048 101 real 0m0.429s user 0m0.003s sys 0m0.186s 'ls -l' 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10081 miss_total: 19 swrong=0 ewrong=0 total: 900 open/close in 7.03 seconds: 127.98 ops/second llite.lustre-ffff9f282e6c7000.statahead_max=0 1001 real 0m15.023s user 0m0.036s sys 0m2.645s 'ls -l' 1000 files without statahead: 15 sec llite.lustre-ffff9f282e6c7000.statahead_max=2048 llite.lustre-ffff9f282e6c7000.statahead_max=2048 1001 real 0m4.872s user 0m0.026s sys 0m1.627s 'ls -l' 1000 files with statahead: 5 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 13 seconds llite.lustre-ffff9f282e6c7000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 mdc.lustre-MDT0000-mdc-ffff9f282e6c7000.batch_stats= snapshot_time: 1789187413.157915752 secs.nsecs start_time: 1789187170.641566822 secs.nsecs elapsed_time: 242.516348930 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 10 10 4: 0 0 10 8: 3 30 40 16: 2 20 60 32: 2 20 80 64: 0 0 80 128: 0 0 80 256: 2 20 100 mdc.lustre-MDT0001-mdc-ffff9f282e6c7000.batch_stats= snapshot_time: 1789187413.157981737 secs.nsecs start_time: 1789187170.641723342 secs.nsecs elapsed_time: 242.516258395 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 10 10 4: 2 20 30 8: 2 20 50 16: 1 10 60 32: 2 20 80 64: 0 0 80 128: 0 0 80 256: 2 20 100 - open/close 1135 (time 1789187424.97 total 10.00 last 113.48) - open/close 2287 (time 1789187434.98 total 20.01 last 115.15) - open/close 3478 (time 1789187444.99 total 30.02 last 118.94) - open/close 4628 (time 1789187454.99 total 40.02 last 114.96) - open/close 5887 (time 1789187465.01 total 50.04 last 125.74) - open/close 7160 (time 1789187475.01 total 60.04 last 127.26) - open/close 8411 (time 1789187485.01 total 70.04 last 125.06) - open/close 9668 (time 1789187495.01 total 80.04 last 125.69) total: 10000 open/close in 81.97 seconds: 122.00 ops/second llite.lustre-ffff9f282e6c7000.statahead_batch_max=0 llite.lustre-ffff9f282e6c7000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f282e6c7000.stats=clear mdc.lustre-MDT0001-mdc-ffff9f282e6c7000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m35.429s user 0m0.247s sys 0m17.520s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9f282e6c7000.statahead_batch_max=256 llite.lustre-ffff9f282e6c7000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f282e6c7000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9f282e6c7000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9f282e6c7000.stats=clear mdc.lustre-MDT0001-mdc-ffff9f282e6c7000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m31.412s user 0m0.214s sys 0m15.768s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9f282e6c7000.batch_stats= snapshot_time: 1789187574.131196845 secs.nsecs start_time: 1789187536.704750696 secs.nsecs elapsed_time: 37.426446149 secs.nsecs subreqs per batch batches %% cum %% 1: 97 44 44 2: 20 9 53 4: 25 11 64 8: 13 5 70 16: 5 2 72 32: 16 7 80 64: 9 4 84 128: 0 0 84 256: 35 15 100 mdc.lustre-MDT0001-mdc-ffff9f282e6c7000.batch_stats= snapshot_time: 1789187574.131239589 secs.nsecs start_time: 1789187536.704874458 secs.nsecs elapsed_time: 37.426365131 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9f282e6c7000.statahead_batch_max=64 llite.lustre-ffff9f282e6c7000.statahead_max=128