== sanity test 123ad: Verify batching statahead works correctly ========================================================== 20:55:12 (1787619312) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffffa04f2bcdb800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa04f2bcdb800.batch_stats=0 llite.lustre-ffffa04f2bcdb800.statahead_max=32 llite.lustre-ffffa04f2bcdb800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffffa04f2bcdb800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa04f2bcdb800.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 3.00 seconds: 33.37 ops/second llite.lustre-ffffa04f2bcdb800.statahead_max=0 101 real 0m2.425s user 0m0.003s sys 0m0.320s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffffa04f2bcdb800.statahead_max=32 llite.lustre-ffffa04f2bcdb800.statahead_max=32 101 real 0m1.269s user 0m0.002s sys 0m0.161s 'ls -l' 100 files with statahead: 2 sec statahead total: 130 statahead wrong: 2 agl total: 124 list_total: 32 fname_total: 98 hit_total: 3839 miss_total: 1569 swrong=2 ewrong=2 - open/close 790 (time 1787619343.40 total 10.02 last 78.86) total: 900 open/close in 11.58 seconds: 77.74 ops/second llite.lustre-ffffa04f2bcdb800.statahead_max=0 1001 real 0m22.450s user 0m0.037s sys 0m2.881s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffffa04f2bcdb800.statahead_max=32 llite.lustre-ffffa04f2bcdb800.statahead_max=32 1001 real 0m7.706s user 0m0.033s sys 0m1.427s 'ls -l' 1000 files with statahead: 8 sec statahead total: 131 statahead wrong: 2 agl total: 125 list_total: 33 fname_total: 98 hit_total: 4835 miss_total: 1573 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 17 seconds llite.lustre-ffffa04f2bcdb800.statahead_stats= statahead total: 131 statahead wrong: 2 agl total: 125 list_total: 33 fname_total: 98 hit_total: 4835 miss_total: 1573 mdc.lustre-MDT0000-mdc-ffffa04f2bcdb800.batch_stats= snapshot_time: 1787619403.367570889 secs.nsecs start_time: 1787619318.739647999 secs.nsecs elapsed_time: 84.627922890 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 2 5 7 8: 1 2 10 16: 21 55 65 32: 13 34 100 mdc.lustre-MDT0001-mdc-ffffa04f2bcdb800.batch_stats= snapshot_time: 1787619403.367655227 secs.nsecs start_time: 1787619318.739903579 secs.nsecs elapsed_time: 84.627751648 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 2 5 7 4: 0 0 7 8: 2 5 12 16: 19 48 61 32: 15 38 100 - open/close 845 (time 1787619415.46 total 10.01 last 84.42) - open/close 1669 (time 1787619425.46 total 20.01 last 82.37) - open/close 2320 (time 1787619435.47 total 30.01 last 65.09) - open/close 2825 (time 1787619445.49 total 40.04 last 50.36) - open/close 3409 (time 1787619455.50 total 50.05 last 58.36) - open/close 4188 (time 1787619465.51 total 60.05 last 77.85) - open/close 5035 (time 1787619475.51 total 70.06 last 84.67) - open/close 5877 (time 1787619485.53 total 80.08 last 84.06) - open/close 6784 (time 1787619495.53 total 90.08 last 90.66) - open/close 7655 (time 1787619505.53 total 100.08 last 87.06) - open/close 8510 (time 1787619515.55 total 110.10 last 85.40) - open/close 9359 (time 1787619525.56 total 120.11 last 84.80) total: 10000 open/close in 127.56 seconds: 78.39 ops/second llite.lustre-ffffa04f2bcdb800.statahead_batch_max=0 llite.lustre-ffffa04f2bcdb800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa04f2bcdb800.stats=clear mdc.lustre-MDT0001-mdc-ffffa04f2bcdb800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m50.233s user 0m0.307s sys 0m16.965s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffffa04f2bcdb800.statahead_batch_max=32 llite.lustre-ffffa04f2bcdb800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa04f2bcdb800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffffa04f2bcdb800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffffa04f2bcdb800.stats=clear mdc.lustre-MDT0001-mdc-ffffa04f2bcdb800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m23.422s user 0m0.268s sys 0m14.023s batched statahead RPC= hit count=9982 mdc.lustre-MDT0000-mdc-ffffa04f2bcdb800.batch_stats= snapshot_time: 1787619681.344055525 secs.nsecs start_time: 1787619590.396758970 secs.nsecs elapsed_time: 90.947296555 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffffa04f2bcdb800.batch_stats= snapshot_time: 1787619681.344133434 secs.nsecs start_time: 1787619590.396905018 secs.nsecs elapsed_time: 90.947228416 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffffa04f2bcdb800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa04f2bcdb800.batch_stats=0 llite.lustre-ffffa04f2bcdb800.statahead_max=2048 llite.lustre-ffffa04f2bcdb800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffffa04f2bcdb800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa04f2bcdb800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.61 seconds: 38.36 ops/second llite.lustre-ffffa04f2bcdb800.statahead_max=0 101 real 0m2.522s user 0m0.006s sys 0m0.316s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffffa04f2bcdb800.statahead_max=2048 llite.lustre-ffffa04f2bcdb800.statahead_max=2048 101 real 0m0.825s user 0m0.001s sys 0m0.158s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10081 miss_total: 19 swrong=0 ewrong=0 - open/close 691 (time 1787619973.98 total 10.01 last 69.06) total: 900 open/close in 13.27 seconds: 67.80 ops/second llite.lustre-ffffa04f2bcdb800.statahead_max=0 1001 real 0m23.800s user 0m0.031s sys 0m2.898s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffffa04f2bcdb800.statahead_max=2048 llite.lustre-ffffa04f2bcdb800.statahead_max=2048 1001 real 0m5.719s user 0m0.021s sys 0m1.380s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffffa04f2bcdb800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 mdc.lustre-MDT0000-mdc-ffffa04f2bcdb800.batch_stats= snapshot_time: 1787620038.184029316 secs.nsecs start_time: 1787619684.046548007 secs.nsecs elapsed_time: 354.137481309 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 12 12 4: 0 0 12 8: 3 37 50 16: 1 12 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffffa04f2bcdb800.batch_stats= snapshot_time: 1787620038.184122498 secs.nsecs start_time: 1787619684.046689464 secs.nsecs elapsed_time: 354.137433034 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 37 37 8: 2 25 62 16: 0 0 62 32: 1 12 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 - open/close 806 (time 1787620050.91 total 10.00 last 80.59) - open/close 1628 (time 1787620060.92 total 20.01 last 82.12) - open/close 2469 (time 1787620070.93 total 30.02 last 83.99) - open/close 3194 (time 1787620080.93 total 40.03 last 72.50) - open/close 3840 (time 1787620090.94 total 50.03 last 64.54) - open/close 4497 (time 1787620100.95 total 60.04 last 65.69) - open/close 4958 (time 1787620110.95 total 70.04 last 46.07) - open/close 5416 (time 1787620120.97 total 80.07 last 45.70) - open/close 6090 (time 1787620130.98 total 90.07 last 67.37) - open/close 6812 (time 1787620140.98 total 100.07 last 72.19) - open/close 7488 (time 1787620150.98 total 110.07 last 67.60) - open/close 8057 (time 1787620160.99 total 120.08 last 56.87) - open/close 8557 (time 1787620171.00 total 130.09 last 49.92) - open/close 9105 (time 1787620181.01 total 140.10 last 54.74) - open/close 9914 (time 1787620191.02 total 150.11 last 80.88) total: 10000 open/close in 151.36 seconds: 66.07 ops/second llite.lustre-ffffa04f2bcdb800.statahead_batch_max=0 llite.lustre-ffffa04f2bcdb800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa04f2bcdb800.stats=clear mdc.lustre-MDT0001-mdc-ffffa04f2bcdb800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m15.552s user 0m0.321s sys 0m14.677s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffffa04f2bcdb800.statahead_batch_max=256 llite.lustre-ffffa04f2bcdb800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa04f2bcdb800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffffa04f2bcdb800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffffa04f2bcdb800.stats=clear mdc.lustre-MDT0001-mdc-ffffa04f2bcdb800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m0.096s user 0m0.305s sys 0m14.933s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffffa04f2bcdb800.batch_stats= snapshot_time: 1787620348.521847237 secs.nsecs start_time: 1787620275.886523496 secs.nsecs elapsed_time: 72.635323741 secs.nsecs subreqs per batch batches %% cum %% 1: 51 35 35 2: 17 11 46 4: 9 6 53 8: 3 2 55 16: 7 4 60 32: 13 8 68 64: 8 5 74 128: 1 0 75 256: 36 24 100 mdc.lustre-MDT0001-mdc-ffffa04f2bcdb800.batch_stats= snapshot_time: 1787620348.521932387 secs.nsecs start_time: 1787620275.886612176 secs.nsecs elapsed_time: 72.635320211 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffffa04f2bcdb800.statahead_batch_max=64 llite.lustre-ffffa04f2bcdb800.statahead_max=128