== sanity test 123ad: Verify batching statahead works correctly ========================================================== 16:14:08 (1789503248) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff94da2036c000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff94da2036c000.batch_stats=0 llite.lustre-ffff94da2036c000.statahead_max=32 llite.lustre-ffff94da2036c000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff94da2036c000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff94da2036c000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 3.35 seconds: 29.85 ops/second llite.lustre-ffff94da2036c000.statahead_max=0 101 real 0m2.317s user 0m0.002s sys 0m0.310s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff94da2036c000.statahead_max=32 llite.lustre-ffff94da2036c000.statahead_max=32 101 real 0m1.178s user 0m0.005s sys 0m0.172s 'ls -l' 100 files with statahead: 2 sec statahead total: 158 statahead wrong: 2 agl total: 152 list_total: 31 fname_total: 101 hit_total: 3586 miss_total: 1311 swrong=2 ewrong=2 - open/close 885 (time 1789503278.48 total 10.00 last 88.49) total: 900 open/close in 10.17 seconds: 88.45 ops/second llite.lustre-ffff94da2036c000.statahead_max=0 1001 real 0m23.426s user 0m0.033s sys 0m2.826s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff94da2036c000.statahead_max=32 llite.lustre-ffff94da2036c000.statahead_max=32 1001 real 0m8.046s user 0m0.025s sys 0m1.305s 'ls -l' 1000 files with statahead: 8 sec statahead total: 159 statahead wrong: 2 agl total: 153 list_total: 32 fname_total: 101 hit_total: 4583 miss_total: 1314 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff94da2036c000.statahead_stats= statahead total: 159 statahead wrong: 2 agl total: 153 list_total: 32 fname_total: 101 hit_total: 4583 miss_total: 1314 mdc.lustre-MDT0000-mdc-ffff94da2036c000.batch_stats= snapshot_time: 1789503341.562906824 secs.nsecs start_time: 1789503254.108335804 secs.nsecs elapsed_time: 87.454571020 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 2 5 8 4: 1 2 10 8: 2 5 16 16: 24 64 81 32: 7 18 100 mdc.lustre-MDT0001-mdc-ffff94da2036c000.batch_stats= snapshot_time: 1789503341.563100827 secs.nsecs start_time: 1789503254.108862343 secs.nsecs elapsed_time: 87.454238484 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 2 5 7 8: 1 2 10 16: 13 33 43 32: 22 56 100 - open/close 596 (time 1789503354.83 total 10.01 last 59.54) - open/close 1497 (time 1789503364.83 total 20.02 last 90.03) - open/close 2283 (time 1789503374.84 total 30.03 last 78.54) - open/close 3060 (time 1789503384.84 total 40.03 last 77.69) - open/close 3727 (time 1789503394.86 total 50.04 last 66.59) - open/close 4426 (time 1789503404.86 total 60.04 last 69.89) - open/close 4997 (time 1789503414.89 total 70.07 last 56.96) - open/close 5775 (time 1789503424.89 total 80.07 last 77.77) - open/close 6615 (time 1789503434.89 total 90.07 last 83.98) - open/close 7501 (time 1789503444.94 total 100.13 last 88.15) - open/close 8147 (time 1789503454.97 total 110.15 last 64.44) - open/close 8884 (time 1789503464.97 total 120.16 last 73.66) - open/close 9651 (time 1789503474.99 total 130.17 last 76.57) total: 10000 open/close in 134.35 seconds: 74.43 ops/second llite.lustre-ffff94da2036c000.statahead_batch_max=0 llite.lustre-ffff94da2036c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff94da2036c000.stats=clear mdc.lustre-MDT0001-mdc-ffff94da2036c000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m4.883s user 0m0.305s sys 0m13.923s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff94da2036c000.statahead_batch_max=32 llite.lustre-ffff94da2036c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff94da2036c000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff94da2036c000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff94da2036c000.stats=clear mdc.lustre-MDT0001-mdc-ffff94da2036c000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m55.508s user 0m0.288s sys 0m13.546s batched statahead RPC= hit count=9972 mdc.lustre-MDT0000-mdc-ffff94da2036c000.batch_stats= snapshot_time: 1789503681.834747189 secs.nsecs start_time: 1789503555.098688064 secs.nsecs elapsed_time: 126.736059125 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 2 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff94da2036c000.batch_stats= snapshot_time: 1789503681.834828525 secs.nsecs start_time: 1789503555.098759945 secs.nsecs elapsed_time: 126.736068580 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff94da2036c000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff94da2036c000.batch_stats=0 llite.lustre-ffff94da2036c000.statahead_max=2048 llite.lustre-ffff94da2036c000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff94da2036c000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff94da2036c000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.03 seconds: 49.35 ops/second llite.lustre-ffff94da2036c000.statahead_max=0 101 real 0m2.316s user 0m0.005s sys 0m0.302s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff94da2036c000.statahead_max=2048 llite.lustre-ffff94da2036c000.statahead_max=2048 101 real 0m0.833s user 0m0.003s sys 0m0.169s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10071 miss_total: 29 swrong=0 ewrong=0 - open/close 892 (time 1789503952.36 total 10.00 last 89.19) total: 900 open/close in 10.07 seconds: 89.36 ops/second llite.lustre-ffff94da2036c000.statahead_max=0 1001 real 0m18.351s user 0m0.031s sys 0m2.814s 'ls -l' 1000 files without statahead: 19 sec llite.lustre-ffff94da2036c000.statahead_max=2048 llite.lustre-ffff94da2036c000.statahead_max=2048 1001 real 0m6.682s user 0m0.032s sys 0m1.328s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11070 miss_total: 30 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff94da2036c000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11070 miss_total: 30 mdc.lustre-MDT0000-mdc-ffff94da2036c000.batch_stats= snapshot_time: 1789504006.314675446 secs.nsecs start_time: 1789503684.457327111 secs.nsecs elapsed_time: 321.857348335 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 3 60 60 16: 0 0 60 32: 0 0 60 64: 0 0 60 128: 0 0 60 256: 2 40 100 mdc.lustre-MDT0001-mdc-ffff94da2036c000.batch_stats= snapshot_time: 1789504006.314742591 secs.nsecs start_time: 1789503684.457470410 secs.nsecs elapsed_time: 321.857272181 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 25 25 8: 1 12 37 16: 1 12 50 32: 0 0 50 64: 1 12 62 128: 1 12 75 256: 2 25 100 - open/close 703 (time 1789504019.05 total 10.00 last 70.30) - open/close 1524 (time 1789504029.06 total 20.00 last 82.08) - open/close 2357 (time 1789504039.07 total 30.02 last 83.19) - open/close 3157 (time 1789504049.08 total 40.02 last 79.94) - open/close 4030 (time 1789504059.09 total 50.03 last 87.23) - open/close 4898 (time 1789504069.09 total 60.04 last 86.72) - open/close 5849 (time 1789504079.09 total 70.04 last 95.10) - open/close 6871 (time 1789504089.10 total 80.04 last 102.18) - open/close 7857 (time 1789504099.10 total 90.05 last 98.57) - open/close 8774 (time 1789504109.10 total 100.05 last 91.66) - open/close 9730 (time 1789504119.11 total 110.05 last 95.57) total: 10000 open/close in 113.02 seconds: 88.48 ops/second llite.lustre-ffff94da2036c000.statahead_batch_max=0 llite.lustre-ffff94da2036c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff94da2036c000.stats=clear mdc.lustre-MDT0001-mdc-ffff94da2036c000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m9.159s user 0m0.322s sys 0m14.153s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff94da2036c000.statahead_batch_max=256 llite.lustre-ffff94da2036c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff94da2036c000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff94da2036c000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff94da2036c000.stats=clear mdc.lustre-MDT0001-mdc-ffff94da2036c000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m50.228s user 0m0.305s sys 0m13.984s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff94da2036c000.batch_stats= snapshot_time: 1789504260.365025139 secs.nsecs start_time: 1789504199.089825984 secs.nsecs elapsed_time: 61.275199155 secs.nsecs subreqs per batch batches %% cum %% 1: 151 44 44 2: 45 13 57 4: 48 14 71 8: 25 7 78 16: 12 3 82 32: 19 5 87 64: 6 1 89 128: 2 0 90 256: 33 9 100 mdc.lustre-MDT0001-mdc-ffff94da2036c000.batch_stats= snapshot_time: 1789504260.365097243 secs.nsecs start_time: 1789504199.094035670 secs.nsecs elapsed_time: 61.271061573 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff94da2036c000.statahead_batch_max=64 llite.lustre-ffff94da2036c000.statahead_max=128