== sanity test 123ad: Verify batching statahead works correctly ========================================================== 00:30:03 (1788841803) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8f32c8d5c000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8f32c8d5c000.batch_stats=0 llite.lustre-ffff8f32c8d5c000.statahead_max=32 llite.lustre-ffff8f32c8d5c000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8f32c8d5c000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8f32c8d5c000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 3.10 seconds: 32.30 ops/second llite.lustre-ffff8f32c8d5c000.statahead_max=0 101 real 0m2.595s user 0m0.005s sys 0m0.307s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8f32c8d5c000.statahead_max=32 llite.lustre-ffff8f32c8d5c000.statahead_max=32 101 real 0m1.143s user 0m0.004s sys 0m0.164s 'ls -l' 100 files with statahead: 1 sec statahead total: 133 statahead wrong: 2 agl total: 128 list_total: 30 fname_total: 90 hit_total: 4089 miss_total: 1285 swrong=2 ewrong=2 - open/close 881 (time 1788841833.09 total 10.03 last 87.84) total: 900 open/close in 10.34 seconds: 87.01 ops/second llite.lustre-ffff8f32c8d5c000.statahead_max=0 1001 real 0m21.119s user 0m0.030s sys 0m2.832s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff8f32c8d5c000.statahead_max=32 llite.lustre-ffff8f32c8d5c000.statahead_max=32 1001 real 0m9.170s user 0m0.026s sys 0m1.369s 'ls -l' 1000 files with statahead: 9 sec statahead total: 134 statahead wrong: 2 agl total: 129 list_total: 31 fname_total: 90 hit_total: 5084 miss_total: 1290 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff8f32c8d5c000.statahead_stats= statahead total: 134 statahead wrong: 2 agl total: 129 list_total: 31 fname_total: 90 hit_total: 5084 miss_total: 1290 mdc.lustre-MDT0000-mdc-ffff8f32c8d5c000.batch_stats= snapshot_time: 1788841894.566490446 secs.nsecs start_time: 1788841808.637842211 secs.nsecs elapsed_time: 85.928648235 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 2 5 5 4: 1 2 8 8: 3 8 17 16: 23 65 82 32: 6 17 100 mdc.lustre-MDT0001-mdc-ffff8f32c8d5c000.batch_stats= snapshot_time: 1788841894.566597579 secs.nsecs start_time: 1788841808.637948408 secs.nsecs elapsed_time: 85.928649171 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 1 2 5 8: 4 10 15 16: 13 32 47 32: 21 52 100 - open/close 809 (time 1788841906.95 total 10.01 last 80.84) - open/close 1680 (time 1788841916.95 total 20.01 last 87.05) - open/close 2601 (time 1788841926.95 total 30.02 last 92.08) - open/close 3497 (time 1788841936.96 total 40.02 last 89.59) - open/close 4409 (time 1788841946.96 total 50.02 last 91.15) - open/close 5251 (time 1788841956.96 total 60.02 last 84.19) - open/close 6157 (time 1788841966.97 total 70.03 last 90.57) - open/close 7038 (time 1788841976.97 total 80.03 last 88.05) - open/close 7914 (time 1788841986.99 total 90.05 last 87.48) - open/close 8784 (time 1788841997.00 total 100.06 last 86.86) - open/close 9624 (time 1788842007.00 total 110.07 last 83.98) total: 10000 open/close in 114.37 seconds: 87.43 ops/second llite.lustre-ffff8f32c8d5c000.statahead_batch_max=0 llite.lustre-ffff8f32c8d5c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8f32c8d5c000.stats=clear mdc.lustre-MDT0001-mdc-ffff8f32c8d5c000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.739s user 0m0.312s sys 0m15.118s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8f32c8d5c000.statahead_batch_max=32 llite.lustre-ffff8f32c8d5c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8f32c8d5c000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8f32c8d5c000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8f32c8d5c000.stats=clear mdc.lustre-MDT0001-mdc-ffff8f32c8d5c000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m30.569s user 0m0.285s sys 0m13.969s batched statahead RPC= hit count=9980 mdc.lustre-MDT0000-mdc-ffff8f32c8d5c000.batch_stats= snapshot_time: 1788842181.220826949 secs.nsecs start_time: 1788842079.968530669 secs.nsecs elapsed_time: 101.252296280 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8f32c8d5c000.batch_stats= snapshot_time: 1788842181.221247957 secs.nsecs start_time: 1788842079.968733439 secs.nsecs elapsed_time: 101.252514518 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8f32c8d5c000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8f32c8d5c000.batch_stats=0 llite.lustre-ffff8f32c8d5c000.statahead_max=2048 llite.lustre-ffff8f32c8d5c000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8f32c8d5c000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8f32c8d5c000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.45 seconds: 40.84 ops/second llite.lustre-ffff8f32c8d5c000.statahead_max=0 101 real 0m2.028s user 0m0.005s sys 0m0.303s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8f32c8d5c000.statahead_max=2048 llite.lustre-ffff8f32c8d5c000.statahead_max=2048 101 real 0m0.680s user 0m0.006s sys 0m0.207s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10079 miss_total: 21 swrong=0 ewrong=0 - open/close 783 (time 1788842482.56 total 10.01 last 78.23) total: 900 open/close in 11.58 seconds: 77.69 ops/second llite.lustre-ffff8f32c8d5c000.statahead_max=0 1001 real 0m21.688s user 0m0.028s sys 0m2.872s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff8f32c8d5c000.statahead_max=2048 llite.lustre-ffff8f32c8d5c000.statahead_max=2048 1001 real 0m5.913s user 0m0.023s sys 0m1.470s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11078 miss_total: 22 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff8f32c8d5c000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11078 miss_total: 22 mdc.lustre-MDT0000-mdc-ffff8f32c8d5c000.batch_stats= snapshot_time: 1788842540.504716666 secs.nsecs start_time: 1788842184.027158903 secs.nsecs elapsed_time: 356.477557763 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 2 33 33 4: 1 16 50 8: 1 16 66 16: 0 0 66 32: 0 0 66 64: 0 0 66 128: 0 0 66 256: 2 33 100 mdc.lustre-MDT0001-mdc-ffff8f32c8d5c000.batch_stats= snapshot_time: 1788842540.504785837 secs.nsecs start_time: 1788842184.027332203 secs.nsecs elapsed_time: 356.477453634 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 10 10 8: 3 30 40 16: 0 0 40 32: 2 20 60 64: 2 20 80 128: 0 0 80 256: 2 20 100 - open/close 722 (time 1788842553.33 total 10.02 last 72.08) - open/close 1584 (time 1788842563.34 total 20.03 last 86.13) - open/close 2415 (time 1788842573.35 total 30.03 last 83.03) - open/close 3305 (time 1788842583.36 total 40.05 last 88.84) - open/close 4130 (time 1788842593.38 total 50.06 last 82.40) - open/close 5050 (time 1788842603.38 total 60.07 last 91.99) - open/close 5914 (time 1788842613.39 total 70.08 last 86.26) - open/close 6747 (time 1788842623.40 total 80.09 last 83.23) - open/close 7581 (time 1788842633.41 total 90.10 last 83.34) - open/close 8461 (time 1788842643.42 total 100.10 last 87.94) - open/close 9265 (time 1788842653.42 total 110.11 last 80.37) total: 10000 open/close in 118.27 seconds: 84.55 ops/second llite.lustre-ffff8f32c8d5c000.statahead_batch_max=0 llite.lustre-ffff8f32c8d5c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8f32c8d5c000.stats=clear mdc.lustre-MDT0001-mdc-ffff8f32c8d5c000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m58.308s user 0m0.306s sys 0m15.358s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8f32c8d5c000.statahead_batch_max=256 llite.lustre-ffff8f32c8d5c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8f32c8d5c000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8f32c8d5c000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8f32c8d5c000.stats=clear mdc.lustre-MDT0001-mdc-ffff8f32c8d5c000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m43.972s user 0m0.286s sys 0m15.810s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8f32c8d5c000.batch_stats= snapshot_time: 1788842779.409020854 secs.nsecs start_time: 1788842726.082544178 secs.nsecs elapsed_time: 53.326476676 secs.nsecs subreqs per batch batches %% cum %% 1: 146 55 55 2: 20 7 63 4: 17 6 69 8: 9 3 73 16: 8 3 76 32: 21 7 84 64: 5 1 85 128: 3 1 87 256: 34 12 100 mdc.lustre-MDT0001-mdc-ffff8f32c8d5c000.batch_stats= snapshot_time: 1788842779.409090307 secs.nsecs start_time: 1788842726.082717229 secs.nsecs elapsed_time: 53.326373078 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8f32c8d5c000.statahead_batch_max=64 llite.lustre-ffff8f32c8d5c000.statahead_max=128