== sanity test 123ad: Verify batching statahead works correctly ========================================================== 12:08:29 (1788365309) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8c6399ce6000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c6399ce6000.batch_stats=0 llite.lustre-ffff8c6399ce6000.statahead_max=32 llite.lustre-ffff8c6399ce6000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8c6399ce6000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c6399ce6000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 0.81 seconds: 123.59 ops/second llite.lustre-ffff8c6399ce6000.statahead_max=0 101 real 0m0.771s user 0m0.004s sys 0m0.288s 'ls -l' 100 files without statahead: 1 sec llite.lustre-ffff8c6399ce6000.statahead_max=32 llite.lustre-ffff8c6399ce6000.statahead_max=32 101 real 0m0.505s user 0m0.005s sys 0m0.312s 'ls -l' 100 files with statahead: 1 sec statahead total: 114 statahead wrong: 1 agl total: 110 list_total: 28 fname_total: 84 hit_total: 3615 miss_total: 230 swrong=1 ewrong=1 total: 900 open/close in 2.34 seconds: 384.96 ops/second llite.lustre-ffff8c6399ce6000.statahead_max=0 1001 real 0m6.440s user 0m0.019s sys 0m2.483s 'ls -l' 1000 files without statahead: 7 sec llite.lustre-ffff8c6399ce6000.statahead_max=32 llite.lustre-ffff8c6399ce6000.statahead_max=32 1001 real 0m3.929s user 0m0.013s sys 0m1.889s 'ls -l' 1000 files with statahead: 4 sec statahead total: 115 statahead wrong: 1 agl total: 111 list_total: 29 fname_total: 84 hit_total: 4614 miss_total: 231 swrong=1 ewrong=1 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 7 seconds llite.lustre-ffff8c6399ce6000.statahead_stats= statahead total: 115 statahead wrong: 1 agl total: 111 list_total: 29 fname_total: 84 hit_total: 4614 miss_total: 231 mdc.lustre-MDT0000-mdc-ffff8c6399ce6000.batch_stats= snapshot_time: 1788365338.680419781 secs.nsecs start_time: 1788365311.517596558 secs.nsecs elapsed_time: 27.162823223 secs.nsecs subreqs per batch batches %% cum %% 1: 3 8 8 2: 0 0 8 4: 2 5 13 8: 2 5 18 16: 25 67 86 32: 5 13 100 mdc.lustre-MDT0001-mdc-ffff8c6399ce6000.batch_stats= snapshot_time: 1788365338.680465572 secs.nsecs start_time: 1788365311.517654300 secs.nsecs elapsed_time: 27.162811272 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 1 2 5 8: 3 7 12 16: 11 28 41 32: 23 58 100 - open/close 3579 (time 1788365349.59 total 10.00 last 357.86) - open/close 7756 (time 1788365359.59 total 20.00 last 417.62) total: 10000 open/close in 26.56 seconds: 376.44 ops/second llite.lustre-ffff8c6399ce6000.statahead_batch_max=0 llite.lustre-ffff8c6399ce6000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c6399ce6000.stats=clear mdc.lustre-MDT0001-mdc-ffff8c6399ce6000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m37.175s user 0m0.232s sys 0m35.559s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8c6399ce6000.statahead_batch_max=32 llite.lustre-ffff8c6399ce6000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c6399ce6000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8c6399ce6000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8c6399ce6000.stats=clear mdc.lustre-MDT0001-mdc-ffff8c6399ce6000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m5.077s user 0m0.210s sys 0m19.590s batched statahead RPC= hit count=9996 mdc.lustre-MDT0000-mdc-ffff8c6399ce6000.batch_stats= snapshot_time: 1788365478.721884518 secs.nsecs start_time: 1788365406.914013202 secs.nsecs elapsed_time: 71.807871316 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 2 0 0 32: 312 99 100 mdc.lustre-MDT0001-mdc-ffff8c6399ce6000.batch_stats= snapshot_time: 1788365478.721945816 secs.nsecs start_time: 1788365406.914066451 secs.nsecs elapsed_time: 71.807879365 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8c6399ce6000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c6399ce6000.batch_stats=0 llite.lustre-ffff8c6399ce6000.statahead_max=2048 llite.lustre-ffff8c6399ce6000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8c6399ce6000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c6399ce6000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 1.43 seconds: 70.11 ops/second llite.lustre-ffff8c6399ce6000.statahead_max=0 101 real 0m1.297s user 0m0.004s sys 0m0.329s 'ls -l' 100 files without statahead: 1 sec llite.lustre-ffff8c6399ce6000.statahead_max=2048 llite.lustre-ffff8c6399ce6000.statahead_max=2048 101 real 0m0.550s user 0m0.002s sys 0m0.318s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10095 miss_total: 5 swrong=0 ewrong=0 total: 900 open/close in 7.34 seconds: 122.59 ops/second llite.lustre-ffff8c6399ce6000.statahead_max=0 1001 real 0m15.580s user 0m0.027s sys 0m3.183s 'ls -l' 1000 files without statahead: 16 sec llite.lustre-ffff8c6399ce6000.statahead_max=2048 llite.lustre-ffff8c6399ce6000.statahead_max=2048 1001 real 0m4.186s user 0m0.024s sys 0m1.833s 'ls -l' 1000 files with statahead: 4 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11094 miss_total: 6 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 12 seconds llite.lustre-ffff8c6399ce6000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11094 miss_total: 6 mdc.lustre-MDT0000-mdc-ffff8c6399ce6000.batch_stats= snapshot_time: 1788365727.869932176 secs.nsecs start_time: 1788365480.469019023 secs.nsecs elapsed_time: 247.400913153 secs.nsecs subreqs per batch batches %% cum %% 1: 2 28 28 2: 0 0 28 4: 1 14 42 8: 2 28 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff8c6399ce6000.batch_stats= snapshot_time: 1788365727.870011115 secs.nsecs start_time: 1788365480.469156506 secs.nsecs elapsed_time: 247.400854609 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 11 11 8: 3 33 44 16: 1 11 55 32: 1 11 66 64: 0 0 66 128: 1 11 77 256: 2 22 100 - open/close 1076 (time 1788365739.60 total 10.00 last 107.57) - open/close 2217 (time 1788365749.61 total 20.01 last 113.99) - open/close 3312 (time 1788365759.62 total 30.02 last 109.47) - open/close 4602 (time 1788365769.62 total 40.02 last 128.97) - open/close 5755 (time 1788365779.63 total 50.03 last 115.22) - open/close 6745 (time 1788365789.63 total 60.03 last 98.95) - open/close 7767 (time 1788365799.64 total 70.04 last 102.15) - open/close 8691 (time 1788365809.64 total 80.04 last 92.38) - open/close 9739 (time 1788365819.65 total 90.05 last 104.69) total: 10000 open/close in 92.62 seconds: 107.97 ops/second llite.lustre-ffff8c6399ce6000.statahead_batch_max=0 llite.lustre-ffff8c6399ce6000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c6399ce6000.stats=clear mdc.lustre-MDT0001-mdc-ffff8c6399ce6000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m40.822s user 0m0.304s sys 0m24.126s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8c6399ce6000.statahead_batch_max=256 llite.lustre-ffff8c6399ce6000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c6399ce6000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8c6399ce6000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8c6399ce6000.stats=clear mdc.lustre-MDT0001-mdc-ffff8c6399ce6000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m36.108s user 0m0.283s sys 0m21.719s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8c6399ce6000.batch_stats= snapshot_time: 1788365914.034816320 secs.nsecs start_time: 1788365870.320944569 secs.nsecs elapsed_time: 43.713871751 secs.nsecs subreqs per batch batches %% cum %% 1: 323 62 62 2: 62 12 74 4: 43 8 83 8: 19 3 86 16: 11 2 89 32: 16 3 92 64: 6 1 93 128: 1 0 93 256: 33 6 100 mdc.lustre-MDT0001-mdc-ffff8c6399ce6000.batch_stats= snapshot_time: 1788365914.034884333 secs.nsecs start_time: 1788365870.321028079 secs.nsecs elapsed_time: 43.713856254 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8c6399ce6000.statahead_batch_max=64 llite.lustre-ffff8c6399ce6000.statahead_max=128