== sanity test 123ad: Verify batching statahead works correctly ========================================================== 06:34:00 (1787999640) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff95a0d13ae000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff95a0d13ae000.batch_stats=0 llite.lustre-ffff95a0d13ae000.statahead_max=32 llite.lustre-ffff95a0d13ae000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff95a0d13ae000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff95a0d13ae000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.11 seconds: 47.36 ops/second llite.lustre-ffff95a0d13ae000.statahead_max=0 101 real 0m1.816s user 0m0.003s sys 0m0.303s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff95a0d13ae000.statahead_max=32 llite.lustre-ffff95a0d13ae000.statahead_max=32 101 real 0m0.629s user 0m0.002s sys 0m0.139s 'ls -l' 100 files with statahead: 0 sec statahead total: 72 statahead wrong: 1 agl total: 65 list_total: 27 fname_total: 45 hit_total: 13889 miss_total: 215 swrong=1 ewrong=1 total: 900 open/close in 6.25 seconds: 143.94 ops/second llite.lustre-ffff95a0d13ae000.statahead_max=0 1001 real 0m13.267s user 0m0.031s sys 0m2.749s 'ls -l' 1000 files without statahead: 13 sec llite.lustre-ffff95a0d13ae000.statahead_max=32 llite.lustre-ffff95a0d13ae000.statahead_max=32 1001 real 0m5.501s user 0m0.017s sys 0m1.343s 'ls -l' 1000 files with statahead: 5 sec statahead total: 73 statahead wrong: 1 agl total: 66 list_total: 28 fname_total: 45 hit_total: 14887 miss_total: 217 swrong=1 ewrong=1 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 16 seconds llite.lustre-ffff95a0d13ae000.statahead_stats= statahead total: 73 statahead wrong: 1 agl total: 66 list_total: 28 fname_total: 45 hit_total: 14887 miss_total: 217 mdc.lustre-MDT0000-mdc-ffff95a0d13ae000.batch_stats= snapshot_time: 1787999701.763782287 secs.nsecs start_time: 1787999643.450352448 secs.nsecs elapsed_time: 58.313429839 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 2 5 5 4: 2 5 10 8: 2 5 15 16: 19 48 64 32: 14 35 100 mdc.lustre-MDT0001-mdc-ffff95a0d13ae000.batch_stats= snapshot_time: 1787999701.763848306 secs.nsecs start_time: 1787999643.450454662 secs.nsecs elapsed_time: 58.313393644 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 2 5 7 8: 4 10 17 16: 21 52 70 32: 12 30 100 - open/close 1211 (time 1787999714.46 total 10.00 last 121.09) - open/close 2391 (time 1787999724.46 total 20.00 last 117.96) - open/close 3774 (time 1787999734.47 total 30.01 last 138.20) - open/close 5241 (time 1787999744.47 total 40.02 last 146.63) - open/close 6398 (time 1787999754.48 total 50.02 last 115.67) - open/close 7652 (time 1787999764.48 total 60.02 last 125.36) - open/close 9109 (time 1787999774.48 total 70.02 last 145.65) total: 10000 open/close in 76.91 seconds: 130.03 ops/second llite.lustre-ffff95a0d13ae000.statahead_batch_max=0 llite.lustre-ffff95a0d13ae000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff95a0d13ae000.stats=clear mdc.lustre-MDT0001-mdc-ffff95a0d13ae000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m40.985s user 0m0.299s sys 0m18.436s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff95a0d13ae000.statahead_batch_max=32 llite.lustre-ffff95a0d13ae000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff95a0d13ae000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff95a0d13ae000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff95a0d13ae000.stats=clear mdc.lustre-MDT0001-mdc-ffff95a0d13ae000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m57.984s user 0m0.255s sys 0m14.532s batched statahead RPC= hit count=9981 mdc.lustre-MDT0000-mdc-ffff95a0d13ae000.batch_stats= snapshot_time: 1787999891.678188638 secs.nsecs start_time: 1787999827.004221230 secs.nsecs elapsed_time: 64.673967408 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff95a0d13ae000.batch_stats= snapshot_time: 1787999891.678253100 secs.nsecs start_time: 1787999827.004366705 secs.nsecs elapsed_time: 64.673886395 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff95a0d13ae000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff95a0d13ae000.batch_stats=0 llite.lustre-ffff95a0d13ae000.statahead_max=2048 llite.lustre-ffff95a0d13ae000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff95a0d13ae000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff95a0d13ae000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 1.88 seconds: 53.12 ops/second llite.lustre-ffff95a0d13ae000.statahead_max=0 101 real 0m1.891s user 0m0.007s sys 0m0.292s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff95a0d13ae000.statahead_max=2048 llite.lustre-ffff95a0d13ae000.statahead_max=2048 101 real 0m0.603s user 0m0.002s sys 0m0.190s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10080 miss_total: 20 swrong=0 ewrong=0 total: 900 open/close in 8.63 seconds: 104.28 ops/second llite.lustre-ffff95a0d13ae000.statahead_max=0 1001 real 0m16.389s user 0m0.029s sys 0m2.945s 'ls -l' 1000 files without statahead: 17 sec llite.lustre-ffff95a0d13ae000.statahead_max=2048 llite.lustre-ffff95a0d13ae000.statahead_max=2048 1001 real 0m4.029s user 0m0.025s sys 0m1.350s 'ls -l' 1000 files with statahead: 4 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 12 seconds llite.lustre-ffff95a0d13ae000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 mdc.lustre-MDT0000-mdc-ffff95a0d13ae000.batch_stats= snapshot_time: 1788000137.485595241 secs.nsecs start_time: 1787999893.139135089 secs.nsecs elapsed_time: 244.346460152 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 9 9 8: 4 36 45 16: 3 27 72 32: 1 9 81 64: 0 0 81 128: 0 0 81 256: 2 18 100 mdc.lustre-MDT0001-mdc-ffff95a0d13ae000.batch_stats= snapshot_time: 1788000137.485667066 secs.nsecs start_time: 1787999893.139297183 secs.nsecs elapsed_time: 244.346369883 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 4 36 36 8: 1 9 45 16: 2 18 63 32: 2 18 81 64: 0 0 81 128: 0 0 81 256: 2 18 100 - open/close 1334 (time 1788000149.31 total 10.00 last 133.36) - open/close 2261 (time 1788000159.31 total 20.01 last 92.64) - open/close 3210 (time 1788000169.31 total 30.01 last 94.88) - open/close 4150 (time 1788000179.32 total 40.02 last 93.93) - open/close 5142 (time 1788000189.33 total 50.03 last 99.11) - open/close 6062 (time 1788000199.34 total 60.04 last 91.93) - open/close 7110 (time 1788000209.35 total 70.04 last 104.71) - open/close 8233 (time 1788000219.35 total 80.05 last 112.27) - open/close 9267 (time 1788000229.35 total 90.05 last 103.34) total: 10000 open/close in 96.06 seconds: 104.10 ops/second llite.lustre-ffff95a0d13ae000.statahead_batch_max=0 llite.lustre-ffff95a0d13ae000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff95a0d13ae000.stats=clear mdc.lustre-MDT0001-mdc-ffff95a0d13ae000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m32.434s user 0m0.258s sys 0m20.373s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff95a0d13ae000.statahead_batch_max=256 llite.lustre-ffff95a0d13ae000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff95a0d13ae000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff95a0d13ae000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff95a0d13ae000.stats=clear mdc.lustre-MDT0001-mdc-ffff95a0d13ae000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m28.652s user 0m0.220s sys 0m16.742s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff95a0d13ae000.batch_stats= snapshot_time: 1788000305.359439027 secs.nsecs start_time: 1788000271.799948529 secs.nsecs elapsed_time: 33.559490498 secs.nsecs subreqs per batch batches %% cum %% 1: 140 55 55 2: 20 7 63 4: 17 6 69 8: 12 4 74 16: 5 1 76 32: 13 5 81 64: 9 3 85 128: 3 1 86 256: 34 13 100 mdc.lustre-MDT0001-mdc-ffff95a0d13ae000.batch_stats= snapshot_time: 1788000305.359481910 secs.nsecs start_time: 1788000271.800109392 secs.nsecs elapsed_time: 33.559372518 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff95a0d13ae000.statahead_batch_max=64 llite.lustre-ffff95a0d13ae000.statahead_max=128