feign.FeignException: Premature end of chunk coded message body: closing chunk expected reading

[davidcanos]

Im getting problems with Tomcat (Premature end of chunk coded message body: closing chunk expected reading)

The problem only happens between my two services (the only two crowded services in my fleet) and somehow the exception raise every 15 or 20 minutes in production env only since 14th May.

Its not reproducible, the endpoint is working well.

No upgrade was made and no relevant change in the codebase on these dates.
It does not happen after any upgrade, it started to happen the 14th May, I was using 3.3 for a long time then, and upgraded a couple of days after to 3.4.6 due to this issue, but it's still happening.
I have just upgraded a while ago to 3.4.7 (this version upgrade Tomcat to 10.1.42) but it's still happening.
I have rechecked logs, before that 14th May happened once every week, being totally irrelevant.

The first issue i detected was a DecodeException due to a PREMATURE EOF (exception belowest).
After upgrading to Apache Http 5, I've noticed it's a "premature end of chunk coded message: closing chunk expected" exception. (Exception just below).

It happens with a lot of different endpoints (no pattern detected), with empty or non empty responses, large or small responses as well.

These servers are behaving properly, CPU are below 4% and memory are also fine, responses are given in <100ms

I have tried to get more debug with a custom FeignDecoder that's why the exception if wrapped in a FeignException but it doesn't spark any idea on me. I'm completely stuck.
I have tried to reproduce in another environment creating traffic with scripts to be able to share a reproducible sample but i couldn't.

The exception with Apache Http 5:

org.apache.hc.core5.http.ConnectionClosedException: Premature end of chunk coded message body: closing chunk expected
	at org.apache.hc.core5.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:264)
	at org.apache.hc.core5.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:223)
	at org.apache.hc.core5.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)
	at org.apache.hc.core5.http.io.EofSensorInputStream.read(EofSensorInputStream.java:138)
	at java.base/java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:264)
	at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:175)
	at java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:128)
	at org.apache.hc.client5.http.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:71)
	at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:350)
	at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:393)
	at java.base/sun.nio.cs.StreamDecoder.lockedRead(StreamDecoder.java:217)
	at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:171)
	at java.base/java.io.Reader.read(Reader.java:212)
	at java.base/java.io.InputStreamReader.read(InputStreamReader.java:168)
	at feign.Util.toString(Util.java:244)
	at com.xxxxx.client.FeignDecoderConfig.lambda$customDecoder$0(FeignDecoderConfig.java:30)
	at feign.InvocationContext.decode(InvocationContext.java:121)
	at feign.InvocationContext.proceed(InvocationContext.java:94)
	at feign.ResponseHandler.handleResponse(ResponseHandler.java:69)
	at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:109)
	at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:53)
	at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:104)
	at org.springframework.cloud.openfeign.FeignCachingInvocationHandlerFactory$1.proceed(FeignCachingInvocationHandlerFactory.java:66)
	at org.springframework.cache.interceptor.CacheInterceptor.lambda$invoke$0(CacheInterceptor.java:55)
	at org.springframework.cache.interceptor.CacheAspectSupport.invokeOperation(CacheAspectSupport.java:431)
	at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:416)
	at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:65)
	at org.springframework.cloud.openfeign.FeignCachingInvocationHandlerFactory.lambda$create$1(FeignCachingInvocationHandlerFactory.java:53)
	at jdk.proxy2/jdk.proxy2.$Proxy97.search(Unknown Source)
	at xxxxx(xxxxx.java:62)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.cache.interceptor.CacheInterceptor.lambda$invoke$0(CacheInterceptor.java:55)
	at org.springframework.cache.interceptor.CacheAspectSupport.invokeOperation(CacheAspectSupport.java:431)
	at org.springframework.cache.interceptor.CacheAspectSupport.evaluate(CacheAspectSupport.java:601)
	at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:448)
	at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:410)
	at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:65)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:728)
	at xxxx$$SpringCGLIB$$0.xxxxx(<generated>)
	at xxxxx.xxxx(xxxx.java:186)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:258)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:191)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:986)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:891)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:527)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:114)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:116)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:398)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:903)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1740)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1189)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:658)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Exception StackTrace with the default Http Client

feign.codec.DecodeException: Error while extracting response for type [java.util.List<com.xxxx.MyModel>] and content type [application/json]
        at feign.InvocationContext.decode(InvocationContext.java:125)
        at feign.InvocationContext.proceed(InvocationContext.java:94)
        at feign.ResponseHandler.handleResponse(ResponseHandler.java:69)
        at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:109)
        at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:53)
        at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:104)
        at org.springframework.cloud.openfeign.FeignCachingInvocationHandlerFactory$1.proceed(FeignCachingInvocationHandlerFactory.java:66)
        at org.springframework.cache.interceptor.CacheInterceptor.lambda$invoke$0(CacheInterceptor.java:55)
        at org.springframework.cache.interceptor.CacheAspectSupport.invokeOperation(CacheAspectSupport.java:431)
        at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:416)
        at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:65)
        at org.springframework.cloud.openfeign.FeignCachingInvocationHandlerFactory.lambda$create$1(FeignCachingInvocationHandlerFactory.java:53)
        at jdk.proxy2/jdk.proxy2.$Proxy95.search(Unknown Source)
        at xxxxxx(xxxxxxx.java:85)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.cache.interceptor.CacheInterceptor.lambda$invoke$0(CacheInterceptor.java:55)
        at org.springframework.cache.interceptor.CacheAspectSupport.invokeOperation(CacheAspectSupport.java:431)
        at org.springframework.cache.interceptor.CacheAspectSupport.evaluate(CacheAspectSupport.java:601)
        at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:448)
        at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:410)
        at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:65)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:728)
        at xxxxxx$$SpringCGLIB$$0.xxxxxx(<generated>)
        at xxxxxx(xxxxxx.java:114)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:258)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:191)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:986)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:891)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903)
        at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:527)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)
        at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
        at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:114)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:116)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:398)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:903)
        at [org.apache.tomcat.util.net](http://org.apache.tomcat.util.net/).NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1740)
        at [org.apache.tomcat.util.net](http://org.apache.tomcat.util.net/).SocketProcessorBase.run(SocketProcessorBase.java:52)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1189)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:658)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.springframework.web.client.RestClientException: Error while extracting response for type [java.util.List<com.MyModel>] and content type [application/json]
        at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:127)
        at org.springframework.cloud.openfeign.support.SpringDecoder.decode(SpringDecoder.java:71)
        at org.springframework.cloud.openfeign.support.ResponseEntityDecoder.decode(ResponseEntityDecoder.java:62)
        at feign.optionals.OptionalDecoder.decode(OptionalDecoder.java:38)
        at feign.InvocationContext.decode(InvocationContext.java:121)
        ... 80 more
Caused by: java.io.IOException: Premature EOF
        at java.base/sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:567)
        at java.base/sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:611)
        at java.base/sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:705)
        at java.base/[java.io](http://java.io/).FilterInputStream.read(FilterInputStream.java:119)
        at java.base/sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3692)
        at java.base/[java.io](http://java.io/).FilterInputStream.read(FilterInputStream.java:119)
        at java.base/[java.io](http://java.io/).PushbackInputStream.read(PushbackInputStream.java:197)
        at java.base/[java.io](http://java.io/).FilterInputStream.read(FilterInputStream.java:119)
        at com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.ensureLoaded(ByteSourceJsonBootstrapper.java:547)
        at com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.detectEncoding(ByteSourceJsonBootstrapper.java:137)
        at com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.constructParser(ByteSourceJsonBootstrapper.java:266)
        at com.fasterxml.jackson.core.JsonFactory._createParser(JsonFactory.java:1874)
        at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:1273)
        at com.fasterxml.jackson.databind.ObjectReader.createParser(ObjectReader.java:1123)
        at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1501)
        at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:397)
        at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.read(AbstractJackson2HttpMessageConverter.java:356)
        at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:105)
        ... 84 more

[wilkinsona]

Thanks for the report, but Spring Boot isn't involved at this level.

There's either a problem on the server side where it's sending a malformed response or there's a problem on the client side where it's failing to read a valid response. Given that the problem appears to occur with two different HTTP clients (the JDK's HttpURLConnection and the Apache HTTP client), a server-side problem would appear to be the most likely.

You say that the problem started to occur without upgrading anything. That would suggest that there may also be an infrastructure cause. For example, perhaps there's some sort of proxy in between the clients and servers that they call and it was upgraded or its configuration modified in some other way.

I can see that you're using Tomcat for the services where the HTTP client code is running. If I've understood your description, the services that you're calling are also using Tomcat. If that's the case, the Tomcat community may be able to help. Perhaps there's some logging configuration that would allow you to identify the cause without producing an overwhelming amount of output.

I'm going to close this one now as there's no evidence of a bug in Spring Boot itself. We can of course re-open the issue if your investigation shows that there is, in fact, a bug in Spring Boot.

[davidcanos]

you are totally right sorry