== sanity test 123ad: Verify batching statahead works correctly ========================================================== 20:49:43 (1788569383) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff90f4f40b1000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff90f4f40b1000.batch_stats=0 llite.lustre-ffff90f4f40b1000.statahead_max=32 llite.lustre-ffff90f4f40b1000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff90f4f40b1000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff90f4f40b1000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 1.98 seconds: 50.43 ops/second llite.lustre-ffff90f4f40b1000.statahead_max=0 101 real 0m2.512s user 0m0.003s sys 0m0.293s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff90f4f40b1000.statahead_max=32 llite.lustre-ffff90f4f40b1000.statahead_max=32 101 real 0m0.865s user 0m0.004s sys 0m0.152s 'ls -l' 100 files with statahead: 1 sec statahead total: 118 statahead wrong: 2 agl total: 112 list_total: 30 fname_total: 79 hit_total: 4052 miss_total: 1450 swrong=2 ewrong=2 total: 900 open/close in 9.99 seconds: 90.06 ops/second llite.lustre-ffff90f4f40b1000.statahead_max=0 1001 real 0m20.894s user 0m0.028s sys 0m2.903s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff90f4f40b1000.statahead_max=32 llite.lustre-ffff90f4f40b1000.statahead_max=32 1001 real 0m7.913s user 0m0.032s sys 0m1.284s 'ls -l' 1000 files with statahead: 7 sec statahead total: 119 statahead wrong: 2 agl total: 113 list_total: 31 fname_total: 79 hit_total: 5050 miss_total: 1452 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff90f4f40b1000.statahead_stats= statahead total: 119 statahead wrong: 2 agl total: 113 list_total: 31 fname_total: 79 hit_total: 5050 miss_total: 1452 mdc.lustre-MDT0000-mdc-ffff90f4f40b1000.batch_stats= snapshot_time: 1788569471.784318911 secs.nsecs start_time: 1788569388.356668679 secs.nsecs elapsed_time: 83.427650232 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 2 5 8 4: 3 8 16 8: 1 2 18 16: 26 70 89 32: 4 10 100 mdc.lustre-MDT0001-mdc-ffff90f4f40b1000.batch_stats= snapshot_time: 1788569471.784494253 secs.nsecs start_time: 1788569388.357234062 secs.nsecs elapsed_time: 83.427260191 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 1 2 7 8: 4 10 17 16: 10 25 42 32: 23 57 100 - open/close 875 (time 1788569485.01 total 10.01 last 87.44) - open/close 1887 (time 1788569495.02 total 20.02 last 101.07) - open/close 2853 (time 1788569505.02 total 30.02 last 96.60) - open/close 3712 (time 1788569515.03 total 40.02 last 85.88) - open/close 4529 (time 1788569525.03 total 50.03 last 81.65) - open/close 5235 (time 1788569535.04 total 60.04 last 70.55) - open/close 6185 (time 1788569545.04 total 70.04 last 94.96) - open/close 7157 (time 1788569555.05 total 80.04 last 97.17) - open/close 8149 (time 1788569565.05 total 90.05 last 99.17) - open/close 9010 (time 1788569575.07 total 100.06 last 85.95) - open/close 9876 (time 1788569585.07 total 110.07 last 86.56) total: 10000 open/close in 111.29 seconds: 89.86 ops/second llite.lustre-ffff90f4f40b1000.statahead_batch_max=0 llite.lustre-ffff90f4f40b1000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff90f4f40b1000.stats=clear mdc.lustre-MDT0001-mdc-ffff90f4f40b1000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m58.208s user 0m0.274s sys 0m13.803s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff90f4f40b1000.statahead_batch_max=32 llite.lustre-ffff90f4f40b1000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff90f4f40b1000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff90f4f40b1000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff90f4f40b1000.stats=clear mdc.lustre-MDT0001-mdc-ffff90f4f40b1000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m41.954s user 0m0.299s sys 0m13.089s batched statahead RPC= hit count=9984 mdc.lustre-MDT0000-mdc-ffff90f4f40b1000.batch_stats= snapshot_time: 1788569766.337159313 secs.nsecs start_time: 1788569653.338610258 secs.nsecs elapsed_time: 112.998549055 secs.nsecs subreqs per batch batches %% cum %% 1: 1 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff90f4f40b1000.batch_stats= snapshot_time: 1788569766.337284861 secs.nsecs start_time: 1788569653.338765911 secs.nsecs elapsed_time: 112.998518950 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff90f4f40b1000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff90f4f40b1000.batch_stats=0 llite.lustre-ffff90f4f40b1000.statahead_max=2048 llite.lustre-ffff90f4f40b1000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff90f4f40b1000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff90f4f40b1000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.46 seconds: 40.62 ops/second llite.lustre-ffff90f4f40b1000.statahead_max=0 101 real 0m1.748s user 0m0.009s sys 0m0.299s 'ls -l' 100 files without statahead: 1 sec llite.lustre-ffff90f4f40b1000.statahead_max=2048 llite.lustre-ffff90f4f40b1000.statahead_max=2048 101 real 0m0.844s user 0m0.006s sys 0m0.163s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10083 miss_total: 17 swrong=0 ewrong=0 total: 900 open/close in 9.84 seconds: 91.47 ops/second llite.lustre-ffff90f4f40b1000.statahead_max=0 1001 real 0m18.991s user 0m0.041s sys 0m2.831s 'ls -l' 1000 files without statahead: 19 sec llite.lustre-ffff90f4f40b1000.statahead_max=2048 llite.lustre-ffff90f4f40b1000.statahead_max=2048 1001 real 0m7.396s user 0m0.028s sys 0m1.320s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11082 miss_total: 18 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 17 seconds llite.lustre-ffff90f4f40b1000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11082 miss_total: 18 mdc.lustre-MDT0000-mdc-ffff90f4f40b1000.batch_stats= snapshot_time: 1788570090.956927163 secs.nsecs start_time: 1788569768.778554325 secs.nsecs elapsed_time: 322.178372838 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 4 57 57 16: 0 0 57 32: 0 0 57 64: 1 14 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff90f4f40b1000.batch_stats= snapshot_time: 1788570090.956993803 secs.nsecs start_time: 1788569768.778713395 secs.nsecs elapsed_time: 322.178280408 secs.nsecs subreqs per batch batches %% cum %% 1: 1 14 14 2: 0 0 14 4: 3 42 57 8: 0 0 57 16: 0 0 57 32: 0 0 57 64: 1 14 71 128: 0 0 71 256: 2 28 100 - open/close 1010 (time 1788570103.47 total 10.00 last 100.98) - open/close 1879 (time 1788570113.48 total 20.01 last 86.82) - open/close 2937 (time 1788570123.48 total 30.01 last 105.78) - open/close 3917 (time 1788570133.49 total 40.02 last 97.92) - open/close 5009 (time 1788570143.49 total 50.02 last 109.19) - open/close 5859 (time 1788570153.49 total 60.03 last 84.96) - open/close 6556 (time 1788570163.51 total 70.05 last 69.57) - open/close 7339 (time 1788570173.52 total 80.05 last 78.25) - open/close 8148 (time 1788570183.52 total 90.06 last 80.85) - open/close 8960 (time 1788570193.53 total 100.07 last 81.13) - open/close 9960 (time 1788570203.54 total 110.08 last 99.89) total: 10000 open/close in 110.41 seconds: 90.57 ops/second llite.lustre-ffff90f4f40b1000.statahead_batch_max=0 llite.lustre-ffff90f4f40b1000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff90f4f40b1000.stats=clear mdc.lustre-MDT0001-mdc-ffff90f4f40b1000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m4.832s user 0m0.314s sys 0m13.837s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff90f4f40b1000.statahead_batch_max=256 llite.lustre-ffff90f4f40b1000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff90f4f40b1000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff90f4f40b1000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff90f4f40b1000.stats=clear mdc.lustre-MDT0001-mdc-ffff90f4f40b1000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m46.857s user 0m0.298s sys 0m14.096s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff90f4f40b1000.batch_stats= snapshot_time: 1788570337.058731413 secs.nsecs start_time: 1788570277.308630337 secs.nsecs elapsed_time: 59.750101076 secs.nsecs subreqs per batch batches %% cum %% 1: 95 40 40 2: 38 16 56 4: 26 10 67 8: 10 4 71 16: 10 4 75 32: 14 5 81 64: 7 2 84 128: 2 0 85 256: 35 14 100 mdc.lustre-MDT0001-mdc-ffff90f4f40b1000.batch_stats= snapshot_time: 1788570337.059388767 secs.nsecs start_time: 1788570277.308908481 secs.nsecs elapsed_time: 59.750480286 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff90f4f40b1000.statahead_batch_max=64 llite.lustre-ffff90f4f40b1000.statahead_max=128