== sanity test 123ad: Verify batching statahead works correctly ========================================================== 09:48:06 (1788875286) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9b3d4cc61000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b3d4cc61000.batch_stats=0 llite.lustre-ffff9b3d4cc61000.statahead_max=32 llite.lustre-ffff9b3d4cc61000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9b3d4cc61000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b3d4cc61000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.29 seconds: 43.60 ops/second llite.lustre-ffff9b3d4cc61000.statahead_max=0 101 real 0m1.931s user 0m0.005s sys 0m0.305s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9b3d4cc61000.statahead_max=32 llite.lustre-ffff9b3d4cc61000.statahead_max=32 101 real 0m0.890s user 0m0.005s sys 0m0.144s 'ls -l' 100 files with statahead: 1 sec statahead total: 107 statahead wrong: 1 agl total: 101 list_total: 31 fname_total: 71 hit_total: 4131 miss_total: 266 swrong=1 ewrong=1 - open/close 861 (time 1788875312.03 total 10.02 last 85.97) total: 900 open/close in 10.60 seconds: 84.87 ops/second llite.lustre-ffff9b3d4cc61000.statahead_max=0 1001 real 0m19.958s user 0m0.034s sys 0m2.900s 'ls -l' 1000 files without statahead: 20 sec llite.lustre-ffff9b3d4cc61000.statahead_max=32 llite.lustre-ffff9b3d4cc61000.statahead_max=32 1001 real 0m9.409s user 0m0.035s sys 0m1.419s 'ls -l' 1000 files with statahead: 9 sec statahead total: 108 statahead wrong: 1 agl total: 102 list_total: 32 fname_total: 71 hit_total: 5130 miss_total: 267 swrong=1 ewrong=1 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff9b3d4cc61000.statahead_stats= statahead total: 108 statahead wrong: 1 agl total: 102 list_total: 32 fname_total: 71 hit_total: 5130 miss_total: 267 mdc.lustre-MDT0000-mdc-ffff9b3d4cc61000.batch_stats= snapshot_time: 1788875375.404110424 secs.nsecs start_time: 1788875290.748087758 secs.nsecs elapsed_time: 84.656022666 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 3 7 10 8: 2 5 15 16: 20 51 66 32: 13 33 100 mdc.lustre-MDT0001-mdc-ffff9b3d4cc61000.batch_stats= snapshot_time: 1788875375.404193847 secs.nsecs start_time: 1788875290.748255536 secs.nsecs elapsed_time: 84.655938311 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 0 0 5 8: 2 5 10 16: 21 55 65 32: 13 34 100 - open/close 899 (time 1788875387.96 total 10.01 last 89.82) - open/close 1788 (time 1788875397.97 total 20.01 last 88.88) - open/close 2769 (time 1788875407.97 total 30.02 last 98.01) - open/close 3792 (time 1788875417.98 total 40.02 last 102.29) - open/close 4874 (time 1788875427.98 total 50.03 last 108.16) - open/close 5927 (time 1788875437.98 total 60.03 last 105.27) - open/close 6956 (time 1788875447.99 total 70.03 last 102.85) - open/close 8089 (time 1788875457.99 total 80.04 last 113.25) - open/close 9170 (time 1788875468.00 total 90.05 last 107.98) total: 10000 open/close in 98.41 seconds: 101.62 ops/second llite.lustre-ffff9b3d4cc61000.statahead_batch_max=0 llite.lustre-ffff9b3d4cc61000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b3d4cc61000.stats=clear mdc.lustre-MDT0001-mdc-ffff9b3d4cc61000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m51.109s user 0m0.285s sys 0m15.005s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9b3d4cc61000.statahead_batch_max=32 llite.lustre-ffff9b3d4cc61000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b3d4cc61000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9b3d4cc61000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9b3d4cc61000.stats=clear mdc.lustre-MDT0001-mdc-ffff9b3d4cc61000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m19.653s user 0m0.283s sys 0m13.407s batched statahead RPC= hit count=9978 mdc.lustre-MDT0000-mdc-ffff9b3d4cc61000.batch_stats= snapshot_time: 1788875621.662981176 secs.nsecs start_time: 1788875533.790347261 secs.nsecs elapsed_time: 87.872633915 secs.nsecs subreqs per batch batches %% cum %% 1: 1 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9b3d4cc61000.batch_stats= snapshot_time: 1788875621.663061793 secs.nsecs start_time: 1788875533.790885931 secs.nsecs elapsed_time: 87.872175862 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9b3d4cc61000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b3d4cc61000.batch_stats=0 llite.lustre-ffff9b3d4cc61000.statahead_max=2048 llite.lustre-ffff9b3d4cc61000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9b3d4cc61000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b3d4cc61000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 1.70 seconds: 58.89 ops/second llite.lustre-ffff9b3d4cc61000.statahead_max=0 101 real 0m1.912s user 0m0.003s sys 0m0.280s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9b3d4cc61000.statahead_max=2048 llite.lustre-ffff9b3d4cc61000.statahead_max=2048 101 real 0m0.582s user 0m0.004s sys 0m0.187s 'ls -l' 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10077 miss_total: 23 swrong=0 ewrong=0 total: 900 open/close in 7.84 seconds: 114.73 ops/second llite.lustre-ffff9b3d4cc61000.statahead_max=0 1001 real 0m15.393s user 0m0.033s sys 0m2.700s 'ls -l' 1000 files without statahead: 15 sec llite.lustre-ffff9b3d4cc61000.statahead_max=2048 llite.lustre-ffff9b3d4cc61000.statahead_max=2048 1001 real 0m5.272s user 0m0.030s sys 0m1.271s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 14 seconds llite.lustre-ffff9b3d4cc61000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 mdc.lustre-MDT0000-mdc-ffff9b3d4cc61000.batch_stats= snapshot_time: 1788875881.095992804 secs.nsecs start_time: 1788875623.226075102 secs.nsecs elapsed_time: 257.869917702 secs.nsecs subreqs per batch batches %% cum %% 1: 1 16 16 2: 0 0 16 4: 2 33 50 8: 1 16 66 16: 0 0 66 32: 0 0 66 64: 0 0 66 128: 0 0 66 256: 2 33 100 mdc.lustre-MDT0001-mdc-ffff9b3d4cc61000.batch_stats= snapshot_time: 1788875881.096071053 secs.nsecs start_time: 1788875623.226154202 secs.nsecs elapsed_time: 257.869916851 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 12 12 8: 3 37 50 16: 0 0 50 32: 0 0 50 64: 1 12 62 128: 1 12 75 256: 2 25 100 - open/close 992 (time 1788875893.15 total 10.01 last 99.14) - open/close 2047 (time 1788875903.16 total 20.01 last 105.42) - open/close 3094 (time 1788875913.17 total 30.02 last 104.62) - open/close 4246 (time 1788875923.18 total 40.03 last 115.12) - open/close 5346 (time 1788875933.18 total 50.03 last 110.00) - open/close 6445 (time 1788875943.18 total 60.03 last 109.88) - open/close 7714 (time 1788875953.18 total 70.04 last 126.83) - open/close 9018 (time 1788875963.18 total 80.04 last 130.39) total: 10000 open/close in 87.22 seconds: 114.65 ops/second llite.lustre-ffff9b3d4cc61000.statahead_batch_max=0 llite.lustre-ffff9b3d4cc61000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b3d4cc61000.stats=clear mdc.lustre-MDT0001-mdc-ffff9b3d4cc61000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m38.352s user 0m0.278s sys 0m17.203s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9b3d4cc61000.statahead_batch_max=256 llite.lustre-ffff9b3d4cc61000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b3d4cc61000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9b3d4cc61000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9b3d4cc61000.stats=clear mdc.lustre-MDT0001-mdc-ffff9b3d4cc61000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m32.007s user 0m0.255s sys 0m15.675s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9b3d4cc61000.batch_stats= snapshot_time: 1788876051.300612719 secs.nsecs start_time: 1788876013.195175327 secs.nsecs elapsed_time: 38.105437392 secs.nsecs subreqs per batch batches %% cum %% 1: 149 53 53 2: 32 11 64 4: 24 8 73 8: 9 3 76 16: 7 2 79 32: 17 6 85 64: 5 1 87 128: 1 0 87 256: 35 12 100 mdc.lustre-MDT0001-mdc-ffff9b3d4cc61000.batch_stats= snapshot_time: 1788876051.300701596 secs.nsecs start_time: 1788876013.195302466 secs.nsecs elapsed_time: 38.105399130 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9b3d4cc61000.statahead_batch_max=64 llite.lustre-ffff9b3d4cc61000.statahead_max=128