== sanity test 123ad: Verify batching statahead works correctly ========================================================== 09:47:02 (1788875222) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffffa09a215ec000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa09a215ec000.batch_stats=0 llite.lustre-ffffa09a215ec000.statahead_max=32 llite.lustre-ffffa09a215ec000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffffa09a215ec000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa09a215ec000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.71 seconds: 36.94 ops/second llite.lustre-ffffa09a215ec000.statahead_max=0 101 real 0m2.382s user 0m0.006s sys 0m0.295s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffffa09a215ec000.statahead_max=32 llite.lustre-ffffa09a215ec000.statahead_max=32 101 real 0m1.344s user 0m0.004s sys 0m0.149s 'ls -l' 100 files with statahead: 2 sec statahead total: 138 statahead wrong: 2 agl total: 131 list_total: 30 fname_total: 107 hit_total: 3786 miss_total: 1290 swrong=2 ewrong=2 - open/close 873 (time 1788875252.71 total 10.01 last 87.21) total: 900 open/close in 10.42 seconds: 86.38 ops/second llite.lustre-ffffa09a215ec000.statahead_max=0 1001 real 0m18.908s user 0m0.033s sys 0m2.928s 'ls -l' 1000 files without statahead: 19 sec llite.lustre-ffffa09a215ec000.statahead_max=32 llite.lustre-ffffa09a215ec000.statahead_max=32 1001 real 0m8.814s user 0m0.029s sys 0m1.408s 'ls -l' 1000 files with statahead: 9 sec statahead total: 139 statahead wrong: 2 agl total: 132 list_total: 31 fname_total: 107 hit_total: 4785 miss_total: 1291 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 16 seconds llite.lustre-ffffa09a215ec000.statahead_stats= statahead total: 139 statahead wrong: 2 agl total: 132 list_total: 31 fname_total: 107 hit_total: 4785 miss_total: 1291 mdc.lustre-MDT0000-mdc-ffffa09a215ec000.batch_stats= snapshot_time: 1788875309.321701897 secs.nsecs start_time: 1788875228.645694259 secs.nsecs elapsed_time: 80.676007638 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 2 5 7 4: 1 2 10 8: 5 12 22 16: 14 35 57 32: 17 42 100 mdc.lustre-MDT0001-mdc-ffffa09a215ec000.batch_stats= snapshot_time: 1788875309.321776647 secs.nsecs start_time: 1788875228.645828917 secs.nsecs elapsed_time: 80.675947730 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 4 10 10 8: 1 2 12 16: 24 61 74 32: 10 25 100 - open/close 935 (time 1788875321.45 total 10.01 last 93.42) - open/close 1861 (time 1788875331.45 total 20.01 last 92.58) - open/close 2485 (time 1788875341.46 total 30.02 last 62.33) - open/close 3213 (time 1788875351.46 total 40.02 last 72.79) - open/close 4032 (time 1788875361.47 total 50.03 last 81.84) - open/close 4780 (time 1788875371.48 total 60.04 last 74.73) - open/close 5685 (time 1788875381.48 total 70.05 last 90.42) - open/close 6560 (time 1788875391.50 total 80.06 last 87.39) - open/close 7556 (time 1788875401.50 total 90.07 last 99.53) - open/close 8639 (time 1788875411.53 total 100.09 last 108.07) - open/close 9752 (time 1788875421.53 total 110.09 last 111.25) total: 10000 open/close in 112.42 seconds: 88.95 ops/second llite.lustre-ffffa09a215ec000.statahead_batch_max=0 llite.lustre-ffffa09a215ec000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa09a215ec000.stats=clear mdc.lustre-MDT0001-mdc-ffffa09a215ec000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m47.787s user 0m0.328s sys 0m15.106s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffffa09a215ec000.statahead_batch_max=32 llite.lustre-ffffa09a215ec000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa09a215ec000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffffa09a215ec000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffffa09a215ec000.stats=clear mdc.lustre-MDT0001-mdc-ffffa09a215ec000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m23.707s user 0m0.308s sys 0m13.458s batched statahead RPC= hit count=9981 mdc.lustre-MDT0000-mdc-ffffa09a215ec000.batch_stats= snapshot_time: 1788875569.621171844 secs.nsecs start_time: 1788875478.189737029 secs.nsecs elapsed_time: 91.431434815 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffffa09a215ec000.batch_stats= snapshot_time: 1788875569.621318154 secs.nsecs start_time: 1788875478.189883940 secs.nsecs elapsed_time: 91.431434214 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffffa09a215ec000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa09a215ec000.batch_stats=0 llite.lustre-ffffa09a215ec000.statahead_max=2048 llite.lustre-ffffa09a215ec000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffffa09a215ec000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa09a215ec000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 1.79 seconds: 55.84 ops/second llite.lustre-ffffa09a215ec000.statahead_max=0 101 real 0m1.681s user 0m0.006s sys 0m0.283s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffffa09a215ec000.statahead_max=2048 llite.lustre-ffffa09a215ec000.statahead_max=2048 101 real 0m0.656s user 0m0.006s sys 0m0.156s 'ls -l' 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10080 miss_total: 20 swrong=0 ewrong=0 total: 900 open/close in 7.58 seconds: 118.70 ops/second llite.lustre-ffffa09a215ec000.statahead_max=0 1001 real 0m15.526s user 0m0.032s sys 0m2.715s 'ls -l' 1000 files without statahead: 15 sec llite.lustre-ffffa09a215ec000.statahead_max=2048 llite.lustre-ffffa09a215ec000.statahead_max=2048 1001 real 0m4.894s user 0m0.034s sys 0m1.418s 'ls -l' 1000 files with statahead: 5 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 12 seconds llite.lustre-ffffa09a215ec000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 mdc.lustre-MDT0000-mdc-ffffa09a215ec000.batch_stats= snapshot_time: 1788875831.761130632 secs.nsecs start_time: 1788875571.822733966 secs.nsecs elapsed_time: 259.938396666 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 12 12 4: 1 12 25 8: 3 37 62 16: 0 0 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffffa09a215ec000.batch_stats= snapshot_time: 1788875831.761214879 secs.nsecs start_time: 1788875571.826706427 secs.nsecs elapsed_time: 259.934508452 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 37 37 8: 1 12 50 16: 1 12 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 - open/close 1104 (time 1788875843.67 total 10.00 last 110.35) - open/close 2173 (time 1788875853.67 total 20.01 last 106.84) - open/close 3304 (time 1788875863.68 total 30.02 last 113.02) - open/close 4385 (time 1788875873.68 total 40.02 last 108.07) - open/close 5471 (time 1788875883.69 total 50.03 last 108.52) - open/close 6529 (time 1788875893.69 total 60.03 last 105.77) - open/close 7604 (time 1788875903.70 total 70.04 last 107.39) - open/close 8562 (time 1788875913.72 total 80.06 last 95.60) - open/close 9701 (time 1788875923.74 total 90.07 last 113.76) total: 10000 open/close in 92.50 seconds: 108.11 ops/second llite.lustre-ffffa09a215ec000.statahead_batch_max=0 llite.lustre-ffffa09a215ec000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa09a215ec000.stats=clear mdc.lustre-MDT0001-mdc-ffffa09a215ec000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m36.361s user 0m0.273s sys 0m18.555s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffffa09a215ec000.statahead_batch_max=256 llite.lustre-ffffa09a215ec000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa09a215ec000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffffa09a215ec000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffffa09a215ec000.stats=clear mdc.lustre-MDT0001-mdc-ffffa09a215ec000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m32.074s user 0m0.249s sys 0m15.254s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffffa09a215ec000.batch_stats= snapshot_time: 1788876006.004848549 secs.nsecs start_time: 1788875967.704033618 secs.nsecs elapsed_time: 38.300814931 secs.nsecs subreqs per batch batches %% cum %% 1: 49 32 32 2: 16 10 43 4: 10 6 50 8: 7 4 55 16: 6 4 59 32: 17 11 70 64: 8 5 75 128: 1 0 76 256: 35 23 100 mdc.lustre-MDT0001-mdc-ffffa09a215ec000.batch_stats= snapshot_time: 1788876006.004933256 secs.nsecs start_time: 1788875967.704157113 secs.nsecs elapsed_time: 38.300776143 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffffa09a215ec000.statahead_batch_max=64 llite.lustre-ffffa09a215ec000.statahead_max=128