== sanity test 123ad: Verify batching statahead works correctly ========================================================== 04:40:31 (1787128831) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9ad0d00f6000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ad0d00f6000.batch_stats=0 llite.lustre-ffff9ad0d00f6000.statahead_max=32 llite.lustre-ffff9ad0d00f6000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9ad0d00f6000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ad0d00f6000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.21 seconds: 45.21 ops/second llite.lustre-ffff9ad0d00f6000.statahead_max=0 101 real 0m2.291s user 0m0.006s sys 0m0.344s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9ad0d00f6000.statahead_max=32 llite.lustre-ffff9ad0d00f6000.statahead_max=32 101 real 0m0.872s user 0m0.003s sys 0m0.183s 'ls -l' 100 files with statahead: 1 sec statahead total: 150 statahead wrong: 2 agl total: 144 list_total: 31 fname_total: 117 hit_total: 4326 miss_total: 1292 swrong=2 ewrong=2 - open/close 816 (time 1787128858.22 total 10.01 last 81.52) total: 900 open/close in 11.39 seconds: 79.01 ops/second llite.lustre-ffff9ad0d00f6000.statahead_max=0 1001 real 0m22.223s user 0m0.035s sys 0m3.263s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff9ad0d00f6000.statahead_max=32 llite.lustre-ffff9ad0d00f6000.statahead_max=32 1001 real 0m11.028s user 0m0.026s sys 0m1.692s 'ls -l' 1000 files with statahead: 11 sec statahead total: 151 statahead wrong: 2 agl total: 145 list_total: 32 fname_total: 117 hit_total: 5324 miss_total: 1294 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff9ad0d00f6000.statahead_stats= statahead total: 151 statahead wrong: 2 agl total: 145 list_total: 32 fname_total: 117 hit_total: 5324 miss_total: 1294 mdc.lustre-MDT0000-mdc-ffff9ad0d00f6000.batch_stats= snapshot_time: 1787128928.575046419 secs.nsecs start_time: 1787128836.602258343 secs.nsecs elapsed_time: 91.972788076 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 0 0 5 4: 3 7 12 8: 0 0 12 16: 18 46 58 32: 16 41 100 mdc.lustre-MDT0001-mdc-ffff9ad0d00f6000.batch_stats= snapshot_time: 1787128928.575113247 secs.nsecs start_time: 1787128836.602356739 secs.nsecs elapsed_time: 91.972756508 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 2 5 7 8: 2 5 13 16: 22 57 71 32: 11 28 100 - open/close 454 (time 1787128941.41 total 10.01 last 45.36) - open/close 1224 (time 1787128951.43 total 20.02 last 76.91) - open/close 2041 (time 1787128961.43 total 30.02 last 81.70) - open/close 2879 (time 1787128971.44 total 40.04 last 83.69) - open/close 3607 (time 1787128981.44 total 50.04 last 72.79) - open/close 4468 (time 1787128991.47 total 60.06 last 85.87) - open/close 5228 (time 1787129001.47 total 70.06 last 75.99) - open/close 6072 (time 1787129011.47 total 80.07 last 84.35) - open/close 6776 (time 1787129021.48 total 90.08 last 70.36) - open/close 7508 (time 1787129031.49 total 100.08 last 73.16) - open/close 8314 (time 1787129041.50 total 110.10 last 80.46) - open/close 9059 (time 1787129051.51 total 120.10 last 74.47) - open/close 9679 (time 1787129061.53 total 130.13 last 61.87) total: 10000 open/close in 133.70 seconds: 74.79 ops/second llite.lustre-ffff9ad0d00f6000.statahead_batch_max=0 llite.lustre-ffff9ad0d00f6000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ad0d00f6000.stats=clear mdc.lustre-MDT0001-mdc-ffff9ad0d00f6000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m17.783s user 0m0.330s sys 0m16.269s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9ad0d00f6000.statahead_batch_max=32 llite.lustre-ffff9ad0d00f6000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ad0d00f6000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9ad0d00f6000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9ad0d00f6000.stats=clear mdc.lustre-MDT0001-mdc-ffff9ad0d00f6000.stats=clear ls -l files in d123ad.sanity: 10001 real 2m4.109s user 0m0.297s sys 0m15.580s batched statahead RPC= hit count=9984 mdc.lustre-MDT0000-mdc-ffff9ad0d00f6000.batch_stats= snapshot_time: 1787129295.481366786 secs.nsecs start_time: 1787129155.208055099 secs.nsecs elapsed_time: 140.273311687 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9ad0d00f6000.batch_stats= snapshot_time: 1787129295.481559348 secs.nsecs start_time: 1787129155.208208124 secs.nsecs elapsed_time: 140.273351224 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9ad0d00f6000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ad0d00f6000.batch_stats=0 llite.lustre-ffff9ad0d00f6000.statahead_max=2048 llite.lustre-ffff9ad0d00f6000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9ad0d00f6000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ad0d00f6000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.93 seconds: 34.12 ops/second llite.lustre-ffff9ad0d00f6000.statahead_max=0 101 real 0m3.380s user 0m0.004s sys 0m0.364s 'ls -l' 100 files without statahead: 4 sec llite.lustre-ffff9ad0d00f6000.statahead_max=2048 llite.lustre-ffff9ad0d00f6000.statahead_max=2048 101 real 0m1.089s user 0m0.004s sys 0m0.179s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10083 miss_total: 17 swrong=0 ewrong=0 - open/close 677 (time 1787129614.61 total 10.02 last 67.60) total: 900 open/close in 13.81 seconds: 65.16 ops/second llite.lustre-ffff9ad0d00f6000.statahead_max=0 1001 real 0m23.974s user 0m0.043s sys 0m3.311s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff9ad0d00f6000.statahead_max=2048 llite.lustre-ffff9ad0d00f6000.statahead_max=2048 1001 real 0m8.340s user 0m0.032s sys 0m1.569s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11082 miss_total: 18 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 23 seconds llite.lustre-ffff9ad0d00f6000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11082 miss_total: 18 mdc.lustre-MDT0000-mdc-ffff9ad0d00f6000.batch_stats= snapshot_time: 1787129690.241574893 secs.nsecs start_time: 1787129297.879187044 secs.nsecs elapsed_time: 392.362387849 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 12 12 4: 2 25 37 8: 2 25 62 16: 0 0 62 32: 1 12 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff9ad0d00f6000.batch_stats= snapshot_time: 1787129690.245465623 secs.nsecs start_time: 1787129297.882253445 secs.nsecs elapsed_time: 392.363212178 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 20 20 8: 1 10 30 16: 3 30 60 32: 1 10 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 - open/close 621 (time 1787129704.11 total 10.01 last 62.07) - open/close 1317 (time 1787129714.12 total 20.02 last 69.52) - open/close 1895 (time 1787129724.13 total 30.02 last 57.78) - open/close 2483 (time 1787129734.13 total 40.02 last 58.77) - open/close 3125 (time 1787129744.14 total 50.03 last 64.15) - open/close 3866 (time 1787129754.15 total 60.04 last 74.04) - open/close 4575 (time 1787129764.16 total 70.05 last 70.82) - open/close 5315 (time 1787129774.16 total 80.06 last 73.98) - open/close 6058 (time 1787129784.16 total 90.06 last 74.30) - open/close 6909 (time 1787129794.17 total 100.07 last 85.01) - open/close 7847 (time 1787129804.18 total 110.07 last 93.78) - open/close 8628 (time 1787129814.20 total 120.09 last 77.94) - open/close 9353 (time 1787129824.20 total 130.09 last 72.50) total: 10000 open/close in 137.68 seconds: 72.63 ops/second llite.lustre-ffff9ad0d00f6000.statahead_batch_max=0 llite.lustre-ffff9ad0d00f6000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ad0d00f6000.stats=clear mdc.lustre-MDT0001-mdc-ffff9ad0d00f6000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m14.523s user 0m0.314s sys 0m16.390s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9ad0d00f6000.statahead_batch_max=256 llite.lustre-ffff9ad0d00f6000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ad0d00f6000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9ad0d00f6000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9ad0d00f6000.stats=clear mdc.lustre-MDT0001-mdc-ffff9ad0d00f6000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m7.903s user 0m0.337s sys 0m15.955s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9ad0d00f6000.batch_stats= snapshot_time: 1787130003.062275233 secs.nsecs start_time: 1787129917.796954835 secs.nsecs elapsed_time: 85.265320398 secs.nsecs subreqs per batch batches %% cum %% 1: 225 46 46 2: 103 21 67 4: 60 12 79 8: 27 5 85 16: 16 3 88 32: 17 3 92 64: 5 1 93 128: 0 0 93 256: 33 6 100 mdc.lustre-MDT0001-mdc-ffff9ad0d00f6000.batch_stats= snapshot_time: 1787130003.062345858 secs.nsecs start_time: 1787129917.797023977 secs.nsecs elapsed_time: 85.265321881 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9ad0d00f6000.statahead_batch_max=64 llite.lustre-ffff9ad0d00f6000.statahead_max=128