== sanity test 123ad: Verify batching statahead works correctly ========================================================== 05:58:13 (1788515893) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff980f85487800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff980f85487800.batch_stats=0 llite.lustre-ffff980f85487800.statahead_max=32 llite.lustre-ffff980f85487800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff980f85487800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff980f85487800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.53 seconds: 39.60 ops/second llite.lustre-ffff980f85487800.statahead_max=0 101 real 0m2.490s user 0m0.005s sys 0m0.370s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff980f85487800.statahead_max=32 llite.lustre-ffff980f85487800.statahead_max=32 101 real 0m1.173s user 0m0.009s sys 0m0.192s 'ls -l' 100 files with statahead: 1 sec statahead total: 148 statahead wrong: 2 agl total: 141 list_total: 32 fname_total: 116 hit_total: 4550 miss_total: 1302 swrong=2 ewrong=2 - open/close 720 (time 1788515921.62 total 10.00 last 72.00) total: 900 open/close in 12.93 seconds: 69.63 ops/second llite.lustre-ffff980f85487800.statahead_max=0 1001 real 0m25.708s user 0m0.043s sys 0m3.103s 'ls -l' 1000 files without statahead: 26 sec llite.lustre-ffff980f85487800.statahead_max=32 llite.lustre-ffff980f85487800.statahead_max=32 1001 real 0m9.338s user 0m0.027s sys 0m1.558s 'ls -l' 1000 files with statahead: 10 sec statahead total: 149 statahead wrong: 2 agl total: 142 list_total: 33 fname_total: 116 hit_total: 5547 miss_total: 1305 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff980f85487800.statahead_stats= statahead total: 149 statahead wrong: 2 agl total: 142 list_total: 33 fname_total: 116 hit_total: 5547 miss_total: 1305 mdc.lustre-MDT0000-mdc-ffff980f85487800.batch_stats= snapshot_time: 1788515993.952693910 secs.nsecs start_time: 1788515899.052495508 secs.nsecs elapsed_time: 94.900198402 secs.nsecs subreqs per batch batches %% cum %% 1: 3 7 7 2: 2 5 13 4: 1 2 15 8: 1 2 18 16: 25 65 84 32: 6 15 100 mdc.lustre-MDT0001-mdc-ffff980f85487800.batch_stats= snapshot_time: 1788515993.952761933 secs.nsecs start_time: 1788515899.052673140 secs.nsecs elapsed_time: 94.900088793 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 1 2 7 8: 3 7 15 16: 12 30 45 32: 22 55 100 - open/close 679 (time 1788516007.07 total 10.01 last 67.84) - open/close 1354 (time 1788516017.07 total 20.01 last 67.47) - open/close 2091 (time 1788516027.08 total 30.02 last 73.62) - open/close 2806 (time 1788516037.10 total 40.05 last 71.34) - open/close 3669 (time 1788516047.13 total 50.07 last 86.08) - open/close 4436 (time 1788516057.15 total 60.09 last 76.54) - open/close 5089 (time 1788516067.15 total 70.09 last 65.29) - open/close 5902 (time 1788516077.16 total 80.10 last 81.23) - open/close 6741 (time 1788516087.16 total 90.10 last 83.90) - open/close 7420 (time 1788516097.17 total 100.12 last 67.80) - open/close 8045 (time 1788516107.18 total 110.13 last 62.43) - open/close 8849 (time 1788516117.19 total 120.13 last 80.37) - open/close 9416 (time 1788516127.19 total 130.14 last 56.67) total: 10000 open/close in 139.11 seconds: 71.89 ops/second llite.lustre-ffff980f85487800.statahead_batch_max=0 llite.lustre-ffff980f85487800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff980f85487800.stats=clear mdc.lustre-MDT0001-mdc-ffff980f85487800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m7.822s user 0m0.361s sys 0m16.483s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff980f85487800.statahead_batch_max=32 llite.lustre-ffff980f85487800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff980f85487800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff980f85487800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff980f85487800.stats=clear mdc.lustre-MDT0001-mdc-ffff980f85487800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m42.773s user 0m0.310s sys 0m15.074s batched statahead RPC= hit count=9981 mdc.lustre-MDT0000-mdc-ffff980f85487800.batch_stats= snapshot_time: 1788516324.841594158 secs.nsecs start_time: 1788516212.347673406 secs.nsecs elapsed_time: 112.493920752 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff980f85487800.batch_stats= snapshot_time: 1788516324.841664706 secs.nsecs start_time: 1788516212.347988728 secs.nsecs elapsed_time: 112.493675978 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff980f85487800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff980f85487800.batch_stats=0 llite.lustre-ffff980f85487800.statahead_max=2048 llite.lustre-ffff980f85487800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff980f85487800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff980f85487800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.41 seconds: 41.53 ops/second llite.lustre-ffff980f85487800.statahead_max=0 101 real 0m2.473s user 0m0.000s sys 0m0.319s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff980f85487800.statahead_max=2048 llite.lustre-ffff980f85487800.statahead_max=2048 101 real 0m1.067s user 0m0.004s sys 0m0.165s 'ls -l' 100 files with statahead: 2 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10080 miss_total: 20 swrong=0 ewrong=0 - open/close 804 (time 1788516634.31 total 10.01 last 80.33) total: 900 open/close in 11.58 seconds: 77.74 ops/second llite.lustre-ffff980f85487800.statahead_max=0 1001 real 0m21.236s user 0m0.028s sys 0m3.010s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff980f85487800.statahead_max=2048 llite.lustre-ffff980f85487800.statahead_max=2048 1001 real 0m7.303s user 0m0.029s sys 0m1.443s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff980f85487800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 mdc.lustre-MDT0000-mdc-ffff980f85487800.batch_stats= snapshot_time: 1788516697.454508919 secs.nsecs start_time: 1788516327.569377595 secs.nsecs elapsed_time: 369.885131324 secs.nsecs subreqs per batch batches %% cum %% 1: 1 14 14 2: 3 42 57 4: 0 0 57 8: 1 14 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff980f85487800.batch_stats= snapshot_time: 1788516697.454573122 secs.nsecs start_time: 1788516327.569541745 secs.nsecs elapsed_time: 369.885031377 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 12 12 8: 3 37 50 16: 0 0 50 32: 0 0 50 64: 1 12 62 128: 1 12 75 256: 2 25 100 - open/close 867 (time 1788516710.14 total 10.00 last 86.70) - open/close 1635 (time 1788516720.15 total 20.02 last 76.67) - open/close 2519 (time 1788516730.17 total 30.03 last 88.27) - open/close 3366 (time 1788516740.17 total 40.03 last 84.69) - open/close 4229 (time 1788516750.17 total 50.03 last 86.28) - open/close 5172 (time 1788516760.18 total 60.04 last 94.27) - open/close 5991 (time 1788516770.18 total 70.04 last 81.87) - open/close 6751 (time 1788516780.19 total 80.06 last 75.89) - open/close 7371 (time 1788516790.20 total 90.06 last 61.98) - open/close 7960 (time 1788516800.21 total 100.07 last 58.82) - open/close 8793 (time 1788516810.21 total 110.08 last 83.27) - open/close 9664 (time 1788516820.22 total 120.08 last 87.07) total: 10000 open/close in 124.25 seconds: 80.48 ops/second llite.lustre-ffff980f85487800.statahead_batch_max=0 llite.lustre-ffff980f85487800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff980f85487800.stats=clear mdc.lustre-MDT0001-mdc-ffff980f85487800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m4.007s user 0m0.312s sys 0m14.952s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff980f85487800.statahead_batch_max=256 llite.lustre-ffff980f85487800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff980f85487800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff980f85487800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff980f85487800.stats=clear mdc.lustre-MDT0001-mdc-ffff980f85487800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m51.359s user 0m0.283s sys 0m15.090s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff980f85487800.batch_stats= snapshot_time: 1788516958.449238126 secs.nsecs start_time: 1788516896.630839341 secs.nsecs elapsed_time: 61.818398785 secs.nsecs subreqs per batch batches %% cum %% 1: 131 45 45 2: 39 13 59 4: 31 10 70 8: 13 4 74 16: 9 3 77 32: 16 5 83 64: 10 3 87 128: 4 1 88 256: 33 11 100 mdc.lustre-MDT0001-mdc-ffff980f85487800.batch_stats= snapshot_time: 1788516958.449318738 secs.nsecs start_time: 1788516896.631198781 secs.nsecs elapsed_time: 61.818119957 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff980f85487800.statahead_batch_max=64 llite.lustre-ffff980f85487800.statahead_max=128