== sanity test 123ad: Verify batching statahead works correctly ========================================================== 16:12:11 (1789071131) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8c1d473ea000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c1d473ea000.batch_stats=0 llite.lustre-ffff8c1d473ea000.statahead_max=32 llite.lustre-ffff8c1d473ea000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8c1d473ea000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c1d473ea000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.85 seconds: 35.07 ops/second llite.lustre-ffff8c1d473ea000.statahead_max=0 101 real 0m3.270s user 0m0.004s sys 0m0.345s 'ls -l' 100 files without statahead: 4 sec llite.lustre-ffff8c1d473ea000.statahead_max=32 llite.lustre-ffff8c1d473ea000.statahead_max=32 101 real 0m1.408s user 0m0.007s sys 0m0.193s 'ls -l' 100 files with statahead: 2 sec statahead total: 130 statahead wrong: 2 agl total: 124 list_total: 31 fname_total: 88 hit_total: 3708 miss_total: 1191 swrong=2 ewrong=2 - open/close 762 (time 1789071162.63 total 10.00 last 76.19) total: 900 open/close in 11.91 seconds: 75.57 ops/second llite.lustre-ffff8c1d473ea000.statahead_max=0 1001 real 0m20.876s user 0m0.027s sys 0m2.907s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff8c1d473ea000.statahead_max=32 llite.lustre-ffff8c1d473ea000.statahead_max=32 1001 real 0m9.630s user 0m0.023s sys 0m1.412s 'ls -l' 1000 files with statahead: 10 sec statahead total: 131 statahead wrong: 2 agl total: 125 list_total: 32 fname_total: 88 hit_total: 4705 miss_total: 1194 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff8c1d473ea000.statahead_stats= statahead total: 131 statahead wrong: 2 agl total: 125 list_total: 32 fname_total: 88 hit_total: 4705 miss_total: 1194 mdc.lustre-MDT0000-mdc-ffff8c1d473ea000.batch_stats= snapshot_time: 1789071229.509460712 secs.nsecs start_time: 1789071137.359663949 secs.nsecs elapsed_time: 92.149796763 secs.nsecs subreqs per batch batches %% cum %% 1: 3 7 7 2: 2 5 13 4: 2 5 18 8: 2 5 23 16: 25 65 89 32: 4 10 100 mdc.lustre-MDT0001-mdc-ffff8c1d473ea000.batch_stats= snapshot_time: 1789071229.509536788 secs.nsecs start_time: 1789071137.359808376 secs.nsecs elapsed_time: 92.149728412 secs.nsecs subreqs per batch batches %% cum %% 1: 2 4 4 2: 1 2 7 4: 1 2 9 8: 3 7 17 16: 9 21 39 32: 25 60 100 - open/close 874 (time 1789071242.22 total 10.01 last 87.34) - open/close 1669 (time 1789071252.23 total 20.01 last 79.44) - open/close 2413 (time 1789071262.24 total 30.03 last 74.29) - open/close 3088 (time 1789071272.25 total 40.03 last 67.47) - open/close 3820 (time 1789071282.26 total 50.04 last 73.13) - open/close 4630 (time 1789071292.26 total 60.05 last 80.96) - open/close 5443 (time 1789071302.27 total 70.05 last 81.26) - open/close 6237 (time 1789071312.27 total 80.06 last 79.38) - open/close 7016 (time 1789071322.28 total 90.06 last 77.86) - open/close 7720 (time 1789071332.29 total 100.08 last 70.27) - open/close 8484 (time 1789071342.31 total 110.09 last 76.29) - open/close 9256 (time 1789071352.34 total 120.12 last 76.98) - open/close 9867 (time 1789071362.35 total 130.13 last 61.04) total: 10000 open/close in 132.26 seconds: 75.61 ops/second llite.lustre-ffff8c1d473ea000.statahead_batch_max=0 llite.lustre-ffff8c1d473ea000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c1d473ea000.stats=clear mdc.lustre-MDT0001-mdc-ffff8c1d473ea000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m5.028s user 0m0.322s sys 0m15.292s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8c1d473ea000.statahead_batch_max=32 llite.lustre-ffff8c1d473ea000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c1d473ea000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8c1d473ea000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8c1d473ea000.stats=clear mdc.lustre-MDT0001-mdc-ffff8c1d473ea000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m47.016s user 0m0.322s sys 0m13.956s batched statahead RPC= hit count=9978 mdc.lustre-MDT0000-mdc-ffff8c1d473ea000.batch_stats= snapshot_time: 1789071555.434811079 secs.nsecs start_time: 1789071437.141497442 secs.nsecs elapsed_time: 118.293313637 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8c1d473ea000.batch_stats= snapshot_time: 1789071555.435025508 secs.nsecs start_time: 1789071437.141704905 secs.nsecs elapsed_time: 118.293320603 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8c1d473ea000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c1d473ea000.batch_stats=0 llite.lustre-ffff8c1d473ea000.statahead_max=2048 llite.lustre-ffff8c1d473ea000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8c1d473ea000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c1d473ea000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.68 seconds: 37.38 ops/second llite.lustre-ffff8c1d473ea000.statahead_max=0 101 real 0m2.478s user 0m0.007s sys 0m0.301s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8c1d473ea000.statahead_max=2048 llite.lustre-ffff8c1d473ea000.statahead_max=2048 101 real 0m0.982s user 0m0.003s sys 0m0.159s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10077 miss_total: 23 swrong=0 ewrong=0 - open/close 549 (time 1789071870.91 total 10.00 last 54.88) total: 900 open/close in 15.92 seconds: 56.55 ops/second llite.lustre-ffff8c1d473ea000.statahead_max=0 1001 real 0m22.196s user 0m0.036s sys 0m2.914s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff8c1d473ea000.statahead_max=2048 llite.lustre-ffff8c1d473ea000.statahead_max=2048 1001 real 0m7.866s user 0m0.026s sys 0m1.455s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff8c1d473ea000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 mdc.lustre-MDT0000-mdc-ffff8c1d473ea000.batch_stats= snapshot_time: 1789071936.521532831 secs.nsecs start_time: 1789071558.062207947 secs.nsecs elapsed_time: 378.459324884 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 22 22 8: 3 33 55 16: 1 11 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff8c1d473ea000.batch_stats= snapshot_time: 1789071936.521596710 secs.nsecs start_time: 1789071558.062341583 secs.nsecs elapsed_time: 378.459255127 secs.nsecs subreqs per batch batches %% cum %% 1: 1 11 11 2: 0 0 11 4: 1 11 22 8: 3 33 55 16: 1 11 66 32: 0 0 66 64: 1 11 77 128: 0 0 77 256: 2 22 100 - open/close 750 (time 1789071948.53 total 10.00 last 74.99) - open/close 1380 (time 1789071958.54 total 20.01 last 62.95) - open/close 2087 (time 1789071968.55 total 30.02 last 70.64) - open/close 2824 (time 1789071978.56 total 40.02 last 73.64) - open/close 3524 (time 1789071988.57 total 50.04 last 69.92) - open/close 4148 (time 1789071998.58 total 60.05 last 62.34) - open/close 5015 (time 1789072008.60 total 70.07 last 86.51) - open/close 5700 (time 1789072018.62 total 80.08 last 68.38) - open/close 6249 (time 1789072028.62 total 90.09 last 54.87) - open/close 6855 (time 1789072038.64 total 100.10 last 60.51) - open/close 7491 (time 1789072048.64 total 110.11 last 63.57) - open/close 8179 (time 1789072058.65 total 120.12 last 68.72) - open/close 8957 (time 1789072068.66 total 130.13 last 77.74) - open/close 9646 (time 1789072078.67 total 140.14 last 68.85) total: 10000 open/close in 145.76 seconds: 68.61 ops/second llite.lustre-ffff8c1d473ea000.statahead_batch_max=0 llite.lustre-ffff8c1d473ea000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c1d473ea000.stats=clear mdc.lustre-MDT0001-mdc-ffff8c1d473ea000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m12.786s user 0m0.305s sys 0m15.696s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8c1d473ea000.statahead_batch_max=256 llite.lustre-ffff8c1d473ea000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c1d473ea000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8c1d473ea000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8c1d473ea000.stats=clear mdc.lustre-MDT0001-mdc-ffff8c1d473ea000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m52.170s user 0m0.314s sys 0m15.740s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8c1d473ea000.batch_stats= snapshot_time: 1789072233.414201104 secs.nsecs start_time: 1789072166.785642472 secs.nsecs elapsed_time: 66.628558632 secs.nsecs subreqs per batch batches %% cum %% 1: 155 53 53 2: 34 11 65 4: 16 5 70 8: 21 7 77 16: 7 2 80 32: 13 4 84 64: 7 2 87 128: 3 1 88 256: 34 11 100 mdc.lustre-MDT0001-mdc-ffff8c1d473ea000.batch_stats= snapshot_time: 1789072233.414282928 secs.nsecs start_time: 1789072166.785728839 secs.nsecs elapsed_time: 66.628554089 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8c1d473ea000.statahead_batch_max=64 llite.lustre-ffff8c1d473ea000.statahead_max=128