Why Using HTTP Client Has a Bug That You Must Avoid
Why Using HTTP Client Has a Bug That You Must Avoid
ScenarioAlmost every system has the function of asynchronously calling third-party services. The responsible system implements a message queue based on blocking queues to call third-party services.To ensure idempotency, queues are consumed sequentially. This leads to a problem, once one of the messages is blocked, the following messages cannot be consumed. When the queue is full, messages cannot be added to the queue either.It seems: In extremely occasional scenarios, the message queue is blocked for more than ten minutes. What the hell is this?The following is step-by-step troubleshooting of the problem.The first thing that comes to mind is whether there is a problem with the underlying mechanism of the message queue implementation. For example, the message queue implements continuous data access by producers and consumers through while(true) polling + sleep.Since the message is blocked, is it because the sleep is overdone? After thinking about it, even if the CPU performs fragmentation processing, it will not sleep so it will not be woken up. It is also unlikely that the thread is interrupted. Because, if it is interrupted, the entire queue will hang up, and it will return to normal after no delay.I have been confused here for a long time, and I have read the source code of the message queue implementation for a long time, but there is no breakthrough. So, I discussed it with a friend, and a sentence reminded me: it may not be a problem of sleep, but a problem of producers or consumers.So I carefully scoured the log and found that it was true: the producer lost data to the queue once, and did not lose data for a long time; the consumer still had consumption logs a few minutes after the producer lost data to the queue. Obviously, the producer is blocked by the consumer.Preliminary conclusion: The consumer consumes too long, causing the queue to be full, and the producer is blocked when adding data to the queue.Empirical guess: There are HTTP requests in consumers, and HTTP requests may hold connections for a long time without releasing them.When the analysis locates the cause of the HTTP request, it is easy to solve. First, I analyzed the log and found that there was indeed an HTTP request. The request parameters were printed before the request, but the print of the returned result was never seen.The tracking log finally saw that the log of the returned result was printed after 15 minutes, and the log content was abnormal for the other party’s service.Looking at the code again, it is found that the HTTP request is implemented based on the HTTP Client, and the person who wrote this code did not set the timeout period. In order to keep the business desensitized, I found a similar code:Like the above code, the request parameters are set, but the timeout for HttpPost is not specified. The seemingly normal code hides a huge pit, and the result is that the HTTP request has been waiting.The situation encountered by the author is not bad, the timeout time set by the other party is 15 minutes, and the result is returned. If the other party does not set the timeout time, it may wait all the time. When the business volume is relatively large, it will lead to disasters!Finding the problem is often the hardest, and when the problem is found, it is much easier to solve. Different versions of HTTP Client have different ways of setting the timeout period. This is another major drawback of HTTP Client. The API version changes too much.Configuration for version 4.3:httpClient.getParams().setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT,10000);httpClient.getParams().setParameter(CoreConnectionPNames.SO_TIMEOUT,10000);Configuration after version 4.3:RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(10000).setConnectTimeout(10000).build();httpGet.setConfig(requestConfig);Among them, setConnectTimeout is the connection timeout time in milliseconds.setSocketTimeout is the timeout for requesting data acquisition, in milliseconds. If an interface is accessed and data cannot be returned within the specified time, the call will be abandoned directly.For the use of other versions, it is recommended to refer to the relevant API instructions.