== sanity test 123ad: Verify batching statahead works correctly ========================================================== 12:38:39 (1788539919) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8b1fd1a39000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8b1fd1a39000.batch_stats=0 llite.lustre-ffff8b1fd1a39000.statahead_max=32 llite.lustre-ffff8b1fd1a39000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8b1fd1a39000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8b1fd1a39000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.59 seconds: 38.68 ops/second llite.lustre-ffff8b1fd1a39000.statahead_max=0 101 real 0m2.412s user 0m0.004s sys 0m0.342s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8b1fd1a39000.statahead_max=32 llite.lustre-ffff8b1fd1a39000.statahead_max=32 101 real 0m1.059s user 0m0.003s sys 0m0.163s 'ls -l' 100 files with statahead: 1 sec statahead total: 146 statahead wrong: 2 agl total: 140 list_total: 31 fname_total: 101 hit_total: 3900 miss_total: 1353 swrong=2 ewrong=2 - open/close 749 (time 1788539946.97 total 10.01 last 74.85) total: 900 open/close in 12.13 seconds: 74.19 ops/second llite.lustre-ffff8b1fd1a39000.statahead_max=0 1001 real 0m21.810s user 0m0.029s sys 0m2.805s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff8b1fd1a39000.statahead_max=32 llite.lustre-ffff8b1fd1a39000.statahead_max=32 1001 real 0m9.263s user 0m0.030s sys 0m1.475s 'ls -l' 1000 files with statahead: 9 sec statahead total: 147 statahead wrong: 2 agl total: 141 list_total: 32 fname_total: 101 hit_total: 4898 miss_total: 1355 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff8b1fd1a39000.statahead_stats= statahead total: 147 statahead wrong: 2 agl total: 141 list_total: 32 fname_total: 101 hit_total: 4898 miss_total: 1355 mdc.lustre-MDT0000-mdc-ffff8b1fd1a39000.batch_stats= snapshot_time: 1788540014.031901192 secs.nsecs start_time: 1788539924.565281467 secs.nsecs elapsed_time: 89.466619725 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 2 5 8 4: 1 2 11 8: 1 2 13 16: 24 66 80 32: 7 19 100 mdc.lustre-MDT0001-mdc-ffff8b1fd1a39000.batch_stats= snapshot_time: 1788540014.031966242 secs.nsecs start_time: 1788539924.565454991 secs.nsecs elapsed_time: 89.466511251 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 1 2 5 8: 2 5 10 16: 9 23 34 32: 25 65 100 - open/close 770 (time 1788540026.56 total 10.01 last 76.95) - open/close 1493 (time 1788540036.58 total 20.02 last 72.19) - open/close 2115 (time 1788540046.59 total 30.04 last 62.11) - open/close 2783 (time 1788540056.59 total 40.04 last 66.80) - open/close 3467 (time 1788540066.60 total 50.05 last 68.34) - open/close 4107 (time 1788540076.61 total 60.05 last 63.96) - open/close 5004 (time 1788540086.61 total 70.05 last 89.70) - open/close 5836 (time 1788540096.61 total 80.06 last 83.18) - open/close 6575 (time 1788540106.64 total 90.09 last 73.67) - open/close 7294 (time 1788540116.65 total 100.09 last 71.87) - open/close 8005 (time 1788540126.66 total 110.10 last 71.04) - open/close 8808 (time 1788540136.66 total 120.10 last 80.27) - open/close 9603 (time 1788540146.67 total 130.11 last 79.42) total: 10000 open/close in 135.46 seconds: 73.82 ops/second llite.lustre-ffff8b1fd1a39000.statahead_batch_max=0 llite.lustre-ffff8b1fd1a39000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8b1fd1a39000.stats=clear mdc.lustre-MDT0001-mdc-ffff8b1fd1a39000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m0.300s user 0m0.302s sys 0m14.904s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8b1fd1a39000.statahead_batch_max=32 llite.lustre-ffff8b1fd1a39000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8b1fd1a39000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8b1fd1a39000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8b1fd1a39000.stats=clear mdc.lustre-MDT0001-mdc-ffff8b1fd1a39000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m49.934s user 0m0.326s sys 0m14.611s batched statahead RPC= hit count=9976 mdc.lustre-MDT0000-mdc-ffff8b1fd1a39000.batch_stats= snapshot_time: 1788540341.855541782 secs.nsecs start_time: 1788540220.060040343 secs.nsecs elapsed_time: 121.795501439 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8b1fd1a39000.batch_stats= snapshot_time: 1788540341.855614936 secs.nsecs start_time: 1788540220.060138997 secs.nsecs elapsed_time: 121.795475939 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8b1fd1a39000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8b1fd1a39000.batch_stats=0 llite.lustre-ffff8b1fd1a39000.statahead_max=2048 llite.lustre-ffff8b1fd1a39000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8b1fd1a39000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8b1fd1a39000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.55 seconds: 39.15 ops/second llite.lustre-ffff8b1fd1a39000.statahead_max=0 101 real 0m2.947s user 0m0.005s sys 0m0.341s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8b1fd1a39000.statahead_max=2048 llite.lustre-ffff8b1fd1a39000.statahead_max=2048 101 real 0m1.082s user 0m0.006s sys 0m0.187s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10074 miss_total: 26 swrong=0 ewrong=0 - open/close 744 (time 1788540644.18 total 10.00 last 74.37) total: 900 open/close in 12.33 seconds: 72.97 ops/second llite.lustre-ffff8b1fd1a39000.statahead_max=0 1001 real 0m21.761s user 0m0.033s sys 0m3.069s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff8b1fd1a39000.statahead_max=2048 llite.lustre-ffff8b1fd1a39000.statahead_max=2048 1001 real 0m5.007s user 0m0.045s sys 0m1.752s 'ls -l' 1000 files with statahead: 5 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11073 miss_total: 27 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff8b1fd1a39000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11073 miss_total: 27 mdc.lustre-MDT0000-mdc-ffff8b1fd1a39000.batch_stats= snapshot_time: 1788540702.921606309 secs.nsecs start_time: 1788540344.783079993 secs.nsecs elapsed_time: 358.138526316 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 12 12 8: 4 50 62 16: 0 0 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff8b1fd1a39000.batch_stats= snapshot_time: 1788540702.921694406 secs.nsecs start_time: 1788540344.783294879 secs.nsecs elapsed_time: 358.138399527 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 2 25 25 4: 2 25 50 8: 0 0 50 16: 1 12 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 - open/close 748 (time 1788540715.16 total 10.00 last 74.78) - open/close 1453 (time 1788540725.17 total 20.01 last 70.46) - open/close 1944 (time 1788540735.17 total 30.01 last 49.08) - open/close 2657 (time 1788540745.18 total 40.02 last 71.23) - open/close 3447 (time 1788540755.19 total 50.03 last 78.95) - open/close 4235 (time 1788540765.19 total 60.03 last 78.79) - open/close 4835 (time 1788540775.20 total 70.04 last 59.95) - open/close 5603 (time 1788540785.21 total 80.05 last 76.74) - open/close 6342 (time 1788540795.21 total 90.05 last 73.88) - open/close 7107 (time 1788540805.22 total 100.05 last 76.45) - open/close 7917 (time 1788540815.22 total 110.06 last 80.93) - open/close 8686 (time 1788540825.23 total 120.07 last 76.84) - open/close 9372 (time 1788540835.24 total 130.08 last 68.52) total: 10000 open/close in 139.73 seconds: 71.57 ops/second llite.lustre-ffff8b1fd1a39000.statahead_batch_max=0 llite.lustre-ffff8b1fd1a39000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8b1fd1a39000.stats=clear mdc.lustre-MDT0001-mdc-ffff8b1fd1a39000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.374s user 0m0.365s sys 0m16.206s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8b1fd1a39000.statahead_batch_max=256 llite.lustre-ffff8b1fd1a39000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8b1fd1a39000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8b1fd1a39000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8b1fd1a39000.stats=clear mdc.lustre-MDT0001-mdc-ffff8b1fd1a39000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m52.973s user 0m0.261s sys 0m15.352s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8b1fd1a39000.batch_stats= snapshot_time: 1788540980.594951329 secs.nsecs start_time: 1788540916.050893784 secs.nsecs elapsed_time: 64.544057545 secs.nsecs subreqs per batch batches %% cum %% 1: 133 51 51 2: 32 12 63 4: 21 8 71 8: 8 3 74 16: 6 2 77 32: 14 5 82 64: 10 3 86 128: 0 0 86 256: 35 13 100 mdc.lustre-MDT0001-mdc-ffff8b1fd1a39000.batch_stats= snapshot_time: 1788540980.595018904 secs.nsecs start_time: 1788540916.051183464 secs.nsecs elapsed_time: 64.543835440 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8b1fd1a39000.statahead_batch_max=64 llite.lustre-ffff8b1fd1a39000.statahead_max=128