== sanity test 123ad: Verify batching statahead works correctly ========================================================== 18:37:33 (1789079853) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff980157161000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff980157161000.batch_stats=0 llite.lustre-ffff980157161000.statahead_max=32 llite.lustre-ffff980157161000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff980157161000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff980157161000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 3.11 seconds: 32.17 ops/second llite.lustre-ffff980157161000.statahead_max=0 101 real 0m2.343s user 0m0.004s sys 0m0.314s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff980157161000.statahead_max=32 llite.lustre-ffff980157161000.statahead_max=32 101 real 0m1.170s user 0m0.007s sys 0m0.204s 'ls -l' 100 files with statahead: 2 sec statahead total: 119 statahead wrong: 2 agl total: 113 list_total: 30 fname_total: 64 hit_total: 4052 miss_total: 1714 swrong=2 ewrong=2 - open/close 650 (time 1789079883.48 total 10.00 last 64.99) total: 900 open/close in 13.97 seconds: 64.42 ops/second llite.lustre-ffff980157161000.statahead_max=0 1001 real 0m23.054s user 0m0.031s sys 0m3.017s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff980157161000.statahead_max=32 llite.lustre-ffff980157161000.statahead_max=32 1001 real 0m8.766s user 0m0.020s sys 0m1.459s 'ls -l' 1000 files with statahead: 9 sec statahead total: 120 statahead wrong: 2 agl total: 114 list_total: 31 fname_total: 64 hit_total: 5049 miss_total: 1717 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 24 seconds llite.lustre-ffff980157161000.statahead_stats= statahead total: 120 statahead wrong: 2 agl total: 114 list_total: 31 fname_total: 64 hit_total: 5049 miss_total: 1717 mdc.lustre-MDT0000-mdc-ffff980157161000.batch_stats= snapshot_time: 1789079956.358490704 secs.nsecs start_time: 1789079859.716073197 secs.nsecs elapsed_time: 96.642417507 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 3 7 13 4: 2 5 18 8: 2 5 23 16: 22 57 81 32: 7 18 100 mdc.lustre-MDT0001-mdc-ffff980157161000.batch_stats= snapshot_time: 1789079956.358573532 secs.nsecs start_time: 1789079859.716217981 secs.nsecs elapsed_time: 96.642355551 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 1 2 5 8: 5 12 17 16: 12 30 47 32: 21 52 100 - open/close 703 (time 1789079968.94 total 10.00 last 70.27) - open/close 1388 (time 1789079978.95 total 20.02 last 68.43) - open/close 2092 (time 1789079988.96 total 30.02 last 70.37) - open/close 2842 (time 1789079998.96 total 40.03 last 74.94) - open/close 3562 (time 1789080008.98 total 50.04 last 71.90) - open/close 4242 (time 1789080018.99 total 60.05 last 67.95) - open/close 4963 (time 1789080028.99 total 70.05 last 72.06) - open/close 5688 (time 1789080039.00 total 80.06 last 72.43) - open/close 6369 (time 1789080049.01 total 90.07 last 68.05) - open/close 7030 (time 1789080059.04 total 100.10 last 65.92) - open/close 7704 (time 1789080069.05 total 110.12 last 67.29) - open/close 8342 (time 1789080079.06 total 120.12 last 63.74) - open/close 9046 (time 1789080089.07 total 130.13 last 70.33) - open/close 9740 (time 1789080099.08 total 140.14 last 69.36) total: 10000 open/close in 143.79 seconds: 69.55 ops/second llite.lustre-ffff980157161000.statahead_batch_max=0 llite.lustre-ffff980157161000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff980157161000.stats=clear mdc.lustre-MDT0001-mdc-ffff980157161000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m58.073s user 0m0.346s sys 0m17.056s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff980157161000.statahead_batch_max=32 llite.lustre-ffff980157161000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff980157161000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff980157161000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff980157161000.stats=clear mdc.lustre-MDT0001-mdc-ffff980157161000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m41.385s user 0m0.305s sys 0m14.857s batched statahead RPC= hit count=9982 mdc.lustre-MDT0000-mdc-ffff980157161000.batch_stats= snapshot_time: 1789080280.996001024 secs.nsecs start_time: 1789080169.143144963 secs.nsecs elapsed_time: 111.852856061 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff980157161000.batch_stats= snapshot_time: 1789080280.996076984 secs.nsecs start_time: 1789080169.143392616 secs.nsecs elapsed_time: 111.852684368 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff980157161000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff980157161000.batch_stats=0 llite.lustre-ffff980157161000.statahead_max=2048 llite.lustre-ffff980157161000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff980157161000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff980157161000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.33 seconds: 30.00 ops/second llite.lustre-ffff980157161000.statahead_max=0 101 real 0m2.947s user 0m0.005s sys 0m0.361s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff980157161000.statahead_max=2048 llite.lustre-ffff980157161000.statahead_max=2048 101 real 0m1.094s user 0m0.008s sys 0m0.222s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10081 miss_total: 19 swrong=0 ewrong=0 - open/close 650 (time 1789080620.43 total 10.01 last 64.94) total: 900 open/close in 13.92 seconds: 64.68 ops/second llite.lustre-ffff980157161000.statahead_max=0 1001 real 0m22.078s user 0m0.040s sys 0m3.032s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff980157161000.statahead_max=2048 llite.lustre-ffff980157161000.statahead_max=2048 1001 real 0m7.013s user 0m0.034s sys 0m1.426s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 24 seconds llite.lustre-ffff980157161000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 mdc.lustre-MDT0000-mdc-ffff980157161000.batch_stats= snapshot_time: 1789080690.776754922 secs.nsecs start_time: 1789080283.576122989 secs.nsecs elapsed_time: 407.200631933 secs.nsecs subreqs per batch batches %% cum %% 1: 1 16 16 2: 0 0 16 4: 3 50 66 8: 0 0 66 16: 0 0 66 32: 0 0 66 64: 0 0 66 128: 0 0 66 256: 2 33 100 mdc.lustre-MDT0001-mdc-ffff980157161000.batch_stats= snapshot_time: 1789080690.776852580 secs.nsecs start_time: 1789080283.576257456 secs.nsecs elapsed_time: 407.200595124 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 4 50 50 16: 0 0 50 32: 0 0 50 64: 1 12 62 128: 1 12 75 256: 2 25 100 - open/close 680 (time 1789080704.03 total 10.00 last 67.99) - open/close 1378 (time 1789080714.03 total 20.01 last 69.77) - open/close 2011 (time 1789080724.04 total 30.02 last 63.23) - open/close 2520 (time 1789080734.07 total 40.05 last 50.75) - open/close 3176 (time 1789080744.08 total 50.05 last 65.57) - open/close 3916 (time 1789080754.09 total 60.06 last 73.91) - open/close 4612 (time 1789080764.10 total 70.07 last 69.54) - open/close 5327 (time 1789080774.11 total 80.08 last 71.43) - open/close 6029 (time 1789080784.13 total 90.10 last 70.06) - open/close 6690 (time 1789080794.13 total 100.10 last 66.07) - open/close 7412 (time 1789080804.15 total 110.12 last 72.09) - open/close 8055 (time 1789080814.15 total 120.12 last 64.28) - open/close 8728 (time 1789080824.16 total 130.14 last 67.21) - open/close 9498 (time 1789080834.17 total 140.14 last 76.98) total: 10000 open/close in 146.75 seconds: 68.14 ops/second llite.lustre-ffff980157161000.statahead_batch_max=0 llite.lustre-ffff980157161000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff980157161000.stats=clear mdc.lustre-MDT0001-mdc-ffff980157161000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m3.810s user 0m0.342s sys 0m16.752s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff980157161000.statahead_batch_max=256 llite.lustre-ffff980157161000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff980157161000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff980157161000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff980157161000.stats=clear mdc.lustre-MDT0001-mdc-ffff980157161000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m49.343s user 0m0.326s sys 0m17.591s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff980157161000.batch_stats= snapshot_time: 1789080974.305649895 secs.nsecs start_time: 1789080912.167182607 secs.nsecs elapsed_time: 62.138467288 secs.nsecs subreqs per batch batches %% cum %% 1: 164 54 54 2: 31 10 65 4: 21 7 72 8: 12 4 76 16: 11 3 79 32: 16 5 85 64: 9 3 88 128: 1 0 88 256: 34 11 100 mdc.lustre-MDT0001-mdc-ffff980157161000.batch_stats= snapshot_time: 1789080974.305717948 secs.nsecs start_time: 1789080912.167275605 secs.nsecs elapsed_time: 62.138442343 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff980157161000.statahead_batch_max=64 llite.lustre-ffff980157161000.statahead_max=128