== sanity test 123ad: Verify batching statahead works correctly ========================================================== 09:36:36 (1788874596) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff89d047e75800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff89d047e75800.batch_stats=0 llite.lustre-ffff89d047e75800.statahead_max=32 llite.lustre-ffff89d047e75800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff89d047e75800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff89d047e75800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.34 seconds: 42.82 ops/second llite.lustre-ffff89d047e75800.statahead_max=0 101 real 0m2.173s user 0m0.004s sys 0m0.316s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff89d047e75800.statahead_max=32 llite.lustre-ffff89d047e75800.statahead_max=32 101 real 0m1.172s user 0m0.004s sys 0m0.194s 'ls -l' 100 files with statahead: 2 sec statahead total: 146 statahead wrong: 2 agl total: 140 list_total: 31 fname_total: 115 hit_total: 4080 miss_total: 1707 swrong=2 ewrong=2 total: 900 open/close in 9.40 seconds: 95.71 ops/second llite.lustre-ffff89d047e75800.statahead_max=0 1001 real 0m19.365s user 0m0.038s sys 0m2.910s 'ls -l' 1000 files without statahead: 19 sec llite.lustre-ffff89d047e75800.statahead_max=32 llite.lustre-ffff89d047e75800.statahead_max=32 1001 real 0m7.698s user 0m0.024s sys 0m1.322s 'ls -l' 1000 files with statahead: 8 sec statahead total: 147 statahead wrong: 2 agl total: 141 list_total: 32 fname_total: 115 hit_total: 5078 miss_total: 1709 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 15 seconds llite.lustre-ffff89d047e75800.statahead_stats= statahead total: 147 statahead wrong: 2 agl total: 141 list_total: 32 fname_total: 115 hit_total: 5078 miss_total: 1709 mdc.lustre-MDT0000-mdc-ffff89d047e75800.batch_stats= snapshot_time: 1788874679.770216575 secs.nsecs start_time: 1788874603.077246912 secs.nsecs elapsed_time: 76.692969663 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 2 5 8 4: 3 8 16 8: 3 8 24 16: 24 64 89 32: 4 10 100 mdc.lustre-MDT0001-mdc-ffff89d047e75800.batch_stats= snapshot_time: 1788874679.770279556 secs.nsecs start_time: 1788874603.077331996 secs.nsecs elapsed_time: 76.692947560 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 2 2 8: 3 7 10 16: 11 28 38 32: 24 61 100 - open/close 1003 (time 1788874692.26 total 10.01 last 100.22) - open/close 1652 (time 1788874702.26 total 20.01 last 64.88) - open/close 2449 (time 1788874712.27 total 30.02 last 79.65) - open/close 3326 (time 1788874722.28 total 40.03 last 87.61) - open/close 4166 (time 1788874732.28 total 50.03 last 83.98) - open/close 5160 (time 1788874742.29 total 60.03 last 99.37) - open/close 6029 (time 1788874752.29 total 70.04 last 86.85) - open/close 6902 (time 1788874762.31 total 80.05 last 87.16) - open/close 7846 (time 1788874772.32 total 90.06 last 94.32) - open/close 8879 (time 1788874782.32 total 100.07 last 103.25) - open/close 9848 (time 1788874792.33 total 110.08 last 96.81) total: 10000 open/close in 111.54 seconds: 89.66 ops/second llite.lustre-ffff89d047e75800.statahead_batch_max=0 llite.lustre-ffff89d047e75800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff89d047e75800.stats=clear mdc.lustre-MDT0001-mdc-ffff89d047e75800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m51.788s user 0m0.263s sys 0m13.742s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff89d047e75800.statahead_batch_max=32 llite.lustre-ffff89d047e75800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff89d047e75800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff89d047e75800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff89d047e75800.stats=clear mdc.lustre-MDT0001-mdc-ffff89d047e75800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m30.939s user 0m0.286s sys 0m13.121s batched statahead RPC= hit count=9981 mdc.lustre-MDT0000-mdc-ffff89d047e75800.batch_stats= snapshot_time: 1788874954.044699840 secs.nsecs start_time: 1788874852.357242862 secs.nsecs elapsed_time: 101.687456978 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff89d047e75800.batch_stats= snapshot_time: 1788874954.045007293 secs.nsecs start_time: 1788874852.357367599 secs.nsecs elapsed_time: 101.687639694 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff89d047e75800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff89d047e75800.batch_stats=0 llite.lustre-ffff89d047e75800.statahead_max=2048 llite.lustre-ffff89d047e75800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff89d047e75800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff89d047e75800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 1.75 seconds: 57.11 ops/second llite.lustre-ffff89d047e75800.statahead_max=0 101 real 0m1.498s user 0m0.004s sys 0m0.305s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff89d047e75800.statahead_max=2048 llite.lustre-ffff89d047e75800.statahead_max=2048 101 real 0m0.736s user 0m0.003s sys 0m0.158s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10080 miss_total: 20 swrong=0 ewrong=0 total: 900 open/close in 9.96 seconds: 90.40 ops/second llite.lustre-ffff89d047e75800.statahead_max=0 1001 real 0m18.675s user 0m0.028s sys 0m2.850s 'ls -l' 1000 files without statahead: 19 sec llite.lustre-ffff89d047e75800.statahead_max=2048 llite.lustre-ffff89d047e75800.statahead_max=2048 1001 real 0m4.754s user 0m0.024s sys 0m1.374s 'ls -l' 1000 files with statahead: 5 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff89d047e75800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 mdc.lustre-MDT0000-mdc-ffff89d047e75800.batch_stats= snapshot_time: 1788875278.796341300 secs.nsecs start_time: 1788874956.559924489 secs.nsecs elapsed_time: 322.236416811 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 14 14 4: 3 42 57 8: 1 14 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff89d047e75800.batch_stats= snapshot_time: 1788875278.796407228 secs.nsecs start_time: 1788874956.560067368 secs.nsecs elapsed_time: 322.236339860 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 3 30 30 16: 2 20 50 32: 2 20 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 - open/close 1000 (time 1788875291.18 total 10.00 last 99.96) - open/close 2065 (time 1788875301.18 total 20.01 last 106.48) - open/close 3028 (time 1788875311.19 total 30.02 last 96.20) - open/close 4083 (time 1788875321.20 total 40.02 last 105.45) - open/close 5054 (time 1788875331.20 total 50.03 last 97.04) - open/close 5939 (time 1788875341.21 total 60.03 last 88.47) - open/close 6799 (time 1788875351.21 total 70.03 last 85.97) - open/close 7596 (time 1788875361.23 total 80.05 last 79.54) - open/close 8386 (time 1788875371.23 total 90.05 last 78.99) - open/close 9315 (time 1788875381.23 total 100.05 last 92.89) - open/close 9917 (time 1788875391.23 total 110.06 last 60.18) total: 10000 open/close in 110.93 seconds: 90.15 ops/second llite.lustre-ffff89d047e75800.statahead_batch_max=0 llite.lustre-ffff89d047e75800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff89d047e75800.stats=clear mdc.lustre-MDT0001-mdc-ffff89d047e75800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m55.082s user 0m0.267s sys 0m13.493s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff89d047e75800.statahead_batch_max=256 llite.lustre-ffff89d047e75800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff89d047e75800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff89d047e75800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff89d047e75800.stats=clear mdc.lustre-MDT0001-mdc-ffff89d047e75800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m50.655s user 0m0.308s sys 0m13.528s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff89d047e75800.batch_stats= snapshot_time: 1788875517.015175290 secs.nsecs start_time: 1788875455.728458353 secs.nsecs elapsed_time: 61.286716937 secs.nsecs subreqs per batch batches %% cum %% 1: 133 44 44 2: 40 13 58 4: 27 9 67 8: 25 8 75 16: 12 4 79 32: 20 6 86 64: 6 2 88 128: 0 0 88 256: 34 11 100 mdc.lustre-MDT0001-mdc-ffff89d047e75800.batch_stats= snapshot_time: 1788875517.015242883 secs.nsecs start_time: 1788875455.728586450 secs.nsecs elapsed_time: 61.286656433 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff89d047e75800.statahead_batch_max=64 llite.lustre-ffff89d047e75800.statahead_max=128