== sanity test 123ad: Verify batching statahead works correctly ========================================================== 03:40:47 (1788334847) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff95103de00800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff95103de00800.batch_stats=0 llite.lustre-ffff95103de00800.statahead_max=32 llite.lustre-ffff95103de00800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff95103de00800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff95103de00800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.38 seconds: 42.05 ops/second llite.lustre-ffff95103de00800.statahead_max=0 101 real 0m2.059s user 0m0.007s sys 0m0.291s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff95103de00800.statahead_max=32 llite.lustre-ffff95103de00800.statahead_max=32 101 real 0m0.982s user 0m0.001s sys 0m0.183s 'ls -l' 100 files with statahead: 1 sec statahead total: 138 statahead wrong: 2 agl total: 131 list_total: 31 fname_total: 91 hit_total: 3832 miss_total: 1515 swrong=2 ewrong=2 - open/close 860 (time 1788334873.56 total 10.00 last 85.98) total: 900 open/close in 10.52 seconds: 85.56 ops/second llite.lustre-ffff95103de00800.statahead_max=0 1001 real 0m19.464s user 0m0.035s sys 0m2.874s 'ls -l' 1000 files without statahead: 19 sec llite.lustre-ffff95103de00800.statahead_max=32 llite.lustre-ffff95103de00800.statahead_max=32 1001 real 0m8.883s user 0m0.027s sys 0m1.360s 'ls -l' 1000 files with statahead: 9 sec statahead total: 139 statahead wrong: 2 agl total: 132 list_total: 32 fname_total: 91 hit_total: 4827 miss_total: 1520 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff95103de00800.statahead_stats= statahead total: 139 statahead wrong: 2 agl total: 132 list_total: 32 fname_total: 91 hit_total: 4827 miss_total: 1520 mdc.lustre-MDT0000-mdc-ffff95103de00800.batch_stats= snapshot_time: 1788334931.963407648 secs.nsecs start_time: 1788334851.520475849 secs.nsecs elapsed_time: 80.442931799 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 2 5 7 8: 2 5 12 16: 21 53 66 32: 13 33 100 mdc.lustre-MDT0001-mdc-ffff95103de00800.batch_stats= snapshot_time: 1788334931.963479185 secs.nsecs start_time: 1788334851.520604552 secs.nsecs elapsed_time: 80.442874633 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 2 5 7 8: 5 12 20 16: 19 47 67 32: 13 32 100 - open/close 836 (time 1788334944.30 total 10.01 last 83.55) - open/close 1784 (time 1788334954.30 total 20.01 last 94.78) - open/close 2699 (time 1788334964.31 total 30.01 last 91.48) - open/close 3531 (time 1788334974.31 total 40.02 last 83.13) - open/close 4409 (time 1788334984.32 total 50.03 last 87.72) - open/close 5275 (time 1788334994.33 total 60.03 last 86.57) - open/close 6132 (time 1788335004.34 total 70.05 last 85.56) - open/close 6969 (time 1788335014.35 total 80.05 last 83.66) - open/close 7873 (time 1788335024.36 total 90.06 last 90.32) - open/close 8787 (time 1788335034.36 total 100.07 last 91.34) - open/close 9632 (time 1788335044.37 total 110.08 last 84.43) total: 10000 open/close in 114.10 seconds: 87.64 ops/second llite.lustre-ffff95103de00800.statahead_batch_max=0 llite.lustre-ffff95103de00800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff95103de00800.stats=clear mdc.lustre-MDT0001-mdc-ffff95103de00800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m47.040s user 0m0.315s sys 0m17.395s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff95103de00800.statahead_batch_max=32 llite.lustre-ffff95103de00800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff95103de00800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff95103de00800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff95103de00800.stats=clear mdc.lustre-MDT0001-mdc-ffff95103de00800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m24.902s user 0m0.309s sys 0m14.460s batched statahead RPC= hit count=9981 mdc.lustre-MDT0000-mdc-ffff95103de00800.batch_stats= snapshot_time: 1788335196.119379619 secs.nsecs start_time: 1788335102.332473404 secs.nsecs elapsed_time: 93.786906215 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff95103de00800.batch_stats= snapshot_time: 1788335196.119438666 secs.nsecs start_time: 1788335102.332595573 secs.nsecs elapsed_time: 93.786843093 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff95103de00800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff95103de00800.batch_stats=0 llite.lustre-ffff95103de00800.statahead_max=2048 llite.lustre-ffff95103de00800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff95103de00800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff95103de00800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.25 seconds: 44.44 ops/second llite.lustre-ffff95103de00800.statahead_max=0 101 real 0m1.802s user 0m0.006s sys 0m0.295s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff95103de00800.statahead_max=2048 llite.lustre-ffff95103de00800.statahead_max=2048 101 real 0m0.748s user 0m0.006s sys 0m0.186s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10080 miss_total: 20 swrong=0 ewrong=0 - open/close 878 (time 1788335463.55 total 10.00 last 87.79) total: 900 open/close in 10.37 seconds: 86.77 ops/second llite.lustre-ffff95103de00800.statahead_max=0 1001 real 0m17.592s user 0m0.037s sys 0m2.803s 'ls -l' 1000 files without statahead: 17 sec llite.lustre-ffff95103de00800.statahead_max=2048 llite.lustre-ffff95103de00800.statahead_max=2048 1001 real 0m5.156s user 0m0.026s sys 0m1.431s 'ls -l' 1000 files with statahead: 5 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 16 seconds llite.lustre-ffff95103de00800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 mdc.lustre-MDT0000-mdc-ffff95103de00800.batch_stats= snapshot_time: 1788335512.748619121 secs.nsecs start_time: 1788335198.364549143 secs.nsecs elapsed_time: 314.384069978 secs.nsecs subreqs per batch batches %% cum %% 1: 2 25 25 2: 0 0 25 4: 2 25 50 8: 2 25 75 16: 0 0 75 32: 0 0 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff95103de00800.batch_stats= snapshot_time: 1788335512.748687675 secs.nsecs start_time: 1788335198.366717422 secs.nsecs elapsed_time: 314.381970253 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 22 22 8: 2 22 44 16: 1 11 55 32: 1 11 66 64: 0 0 66 128: 1 11 77 256: 2 22 100 - open/close 897 (time 1788335524.71 total 10.01 last 89.57) - open/close 1794 (time 1788335534.73 total 20.03 last 89.53) - open/close 2739 (time 1788335544.74 total 30.04 last 94.46) - open/close 3590 (time 1788335554.74 total 40.04 last 85.07) - open/close 4506 (time 1788335564.75 total 50.05 last 91.54) - open/close 5405 (time 1788335574.75 total 60.05 last 89.89) - open/close 6312 (time 1788335584.75 total 70.05 last 90.66) - open/close 7156 (time 1788335594.76 total 80.06 last 84.35) - open/close 7955 (time 1788335604.77 total 90.07 last 79.80) - open/close 8758 (time 1788335614.78 total 100.08 last 80.21) - open/close 9601 (time 1788335624.79 total 110.09 last 84.22) total: 10000 open/close in 114.62 seconds: 87.25 ops/second llite.lustre-ffff95103de00800.statahead_batch_max=0 llite.lustre-ffff95103de00800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff95103de00800.stats=clear mdc.lustre-MDT0001-mdc-ffff95103de00800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m42.534s user 0m0.327s sys 0m19.078s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff95103de00800.statahead_batch_max=256 llite.lustre-ffff95103de00800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff95103de00800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff95103de00800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff95103de00800.stats=clear mdc.lustre-MDT0001-mdc-ffff95103de00800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m36.128s user 0m0.283s sys 0m17.071s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff95103de00800.batch_stats= snapshot_time: 1788335722.041443728 secs.nsecs start_time: 1788335678.382505400 secs.nsecs elapsed_time: 43.658938328 secs.nsecs subreqs per batch batches %% cum %% 1: 87 49 49 2: 10 5 55 4: 7 4 59 8: 7 4 63 16: 5 2 66 32: 13 7 73 64: 11 6 80 128: 0 0 80 256: 35 20 100 mdc.lustre-MDT0001-mdc-ffff95103de00800.batch_stats= snapshot_time: 1788335722.041501573 secs.nsecs start_time: 1788335678.382594064 secs.nsecs elapsed_time: 43.658907509 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff95103de00800.statahead_batch_max=64 llite.lustre-ffff95103de00800.statahead_max=128