s390/checksum: make ip_fast_csum() faster

Convert ip_fast_csum() so it doesn't call csum_partial(), but instead
open code the checksum calculation. The problem with csum_partial() is
that it makes use of the cksm instruction, which has high startup
costs and therefore is only very fast if used on larger memory
regions.

IPv4 headers however are small in size (5-16 32-bit words). The open
coded variant calculates the checksum in ~30% of the time compared to
the old variant (z14, march=z196).

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>

diff --git a/arch/s390/include/asm/checksum.h b/arch/s390/include/asm/checksum.h
index 961c25c5124b6d02ec3ad964323c6883c4055b01..8bc6bed4715b7600e32a16c8b9db212df09947ed 100644 (file)
--- a/arch/s390/include/asm/checksum.h
+++ b/arch/s390/include/asm/checksum.h
@@ -66,7 +66,18 @@ static inline __sum16 csum_fold(__wsum sum)
  */
 static inline __sum16 ip_fast_csum(const void *iph, unsigned int ihl)
 {
-       return csum_fold(csum_partial(iph, ihl*4, 0));
+       __u64 csum = 0;
+       __u32 *ptr = (u32 *)iph;
+
+       csum += *ptr++;
+       csum += *ptr++;
+       csum += *ptr++;
+       csum += *ptr++;
+       ihl -= 4;
+       while (ihl--)
+               csum += *ptr++;
+       csum += (csum >> 32) | (csum << 32);
+       return csum_fold((__force __wsum)(csum >> 32));
 }
 
 /*