Skip to content

Instantly share code, notes, and snippets.

@bric3

bric3/HttpClientBug.java

Created September 5, 2022 10:38
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save bric3/b00cb425993d3c7e77e7c9fb96da28bc to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save bric3/b00cb425993d3c7e77e7c9fb96da28bc to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. bric3 created this gist Sep 5, 2022.
    132 changes: 132 additions & 0 deletions HttpClientBug.java
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,132 @@
    import java.io.IOException;
    import java.io.InputStream;
    import java.io.UncheckedIOException;
    import java.net.URI;
    import java.net.http.HttpClient;
    import java.net.http.HttpRequest;
    import java.net.http.HttpResponse;
    import java.time.Duration;
    import java.time.temporal.ChronoUnit;
    import java.util.function.Function;
    import java.util.function.Supplier;
    import java.util.zip.GZIPInputStream;


    /**
    * Work around <a href="https://bugs.openjdk.java.net/browse/JDK-8217627">JDK-8217627</a>,
    * <a href="https://bugs.openjdk.java.net/browse/JDK-8217264">JDK-8217264</a>
    * when using {@link java.net.http.HttpResponse.BodySubscribers#mapping(HttpResponse.BodySubscriber, Function)} and a
    * blocking call like {@link GZIPInputStream#GZIPInputStream(InputStream)}.
    *
    * <strong>Note this is fixed since JDK 13.</strong>
    */
    public class HttpClientBug {
    public static void main(String[] args) throws IOException, InterruptedException {
    System.getLogger("main").log(System.Logger.Level.INFO, System.getProperty("java.version"));

    var request = HttpRequest.newBuilder(URI.create("https://dog.ceo/api/breeds/image/random"))
    .GET()
    .header("Accept-Encoding", "gzip")
    .build();

    var httpClient = HttpClient.newBuilder()
    .version(HttpClient.Version.HTTP_1_1)
    .followRedirects(HttpClient.Redirect.NORMAL)
    .connectTimeout(Duration.of(5, ChronoUnit.SECONDS))
    .build();

    var httpResponse = httpClient.send(
    request,
    terminalBodyHandler(gzipBodyHandler())
    );

    if (httpResponse.statusCode() == 200) {
    System.out.println(httpResponse.body().get());
    } else {
    System.out.println("Unexpected status code: " + httpResponse.statusCode());
    }
    }

    /**
    * Body handler that returns a Supplier of the response body as a String.
    *
    * <p>
    * The supplier is necessary for the same reasons stated in {@link #gzipBodyHandler()},
    * as it relies on this upstream body handler. So the call to initiate the decoding of
    * the data has to be made after {@link HttpResponse#body()}.
    * </p>
    *
    * @param upstreamHandler The upstream body handler.
    * @return Body handler that decodes the input stream as a String.
    */
    private static HttpResponse.BodyHandler<Supplier<String>> terminalBodyHandler(HttpResponse.BodyHandler<Supplier<InputStream>> upstreamHandler) {
    return responseInfo -> {
    var statusCode = responseInfo.statusCode();
    var upstream = upstreamHandler.apply(responseInfo);

    if (statusCode >= 200 && statusCode < 300) {
    return HttpResponse.BodySubscribers.mapping(upstream, inputStream -> () -> {
    try (var is = inputStream.get()) {
    return new String(is.readAllBytes());
    } catch (IOException e) {
    e.printStackTrace();
    throw new UncheckedIOException(e);
    }
    });
    } else {
    return HttpResponse.BodySubscribers.mapping(upstream, inputStream -> () -> {
    try (var is = inputStream.get()) {
    return new String(is.readAllBytes());
    } catch (IOException e) {
    e.printStackTrace();
    throw new UncheckedIOException(e);
    }
    });
    }
    };
    }

    /**
    * Returns a body handler that returns a Supplier of inputstream able to handle gzip, brotli compression or nothing
    *
    * <p>
    * The use of the Supplier is necessary on JDK11u, because GZIPInputStream and BrotliInputStream
    * are reading the stream in their constructor (typically to read the compressed stream headers),
    * since this operation is blocking, it blocks the executor from performing other HttpClient tasks
    * And as such, the {@link HttpClient#send(HttpRequest, HttpResponse.BodyHandler)} call is blocked.
    * </p>
    * <p>
    * In order to prevent that it is necessary to map to a supplier that make the blocking operations
    * happen <strong>after</strong> the call to {@link HttpResponse#body()}.
    * </p>
    *
    * @see <a href="https://bugs.openjdk.java.net/browse/JDK-8217627">JDK-8217627</a>
    * @see <a href="https://bugs.openjdk.java.net/browse/JDK-8217264">JDK-8217264</a>
    * @return BodyHandler mapping to an input stream supplier
    */
    private static HttpResponse.BodyHandler<Supplier<InputStream>> gzipBodyHandler() {
    return responseInfo -> {
    var gzipped = responseInfo.headers()
    .firstValue("content-encoding")
    .orElse("")
    .equalsIgnoreCase("gzip");

    if (gzipped) {
    System.out.println("Gzipped response");
    return HttpResponse.BodySubscribers.mapping(
    HttpResponse.BodySubscribers.ofInputStream(),
    inputStream -> () -> {
    try {
    return new GZIPInputStream(inputStream);
    } catch (IOException e) {
    e.printStackTrace();
    throw new UncheckedIOException(e);
    }
    }
    );
    } else {
    return HttpResponse.BodySubscribers.mapping(HttpResponse.BodySubscribers.ofInputStream(), inputStream -> () -> inputStream);
    }
    };
    }
    }
    82 changes: 82 additions & 0 deletions without-work-around-thread-dump-stack.adoc
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,82 @@
    Without the workaround, a thread dump might have stack traces like below.
    Note https://bugs.openjdk.java.net/browse/JDK-8217264[JDK-8217264] is fixed since JDK 13.


    [source]
    ----
    "HttpClient-1-Worker-0@2251" daemon prio=5 tid=0x11 nid=NA waiting
    java.lang.Thread.State: WAITING
    at jdk.internal.misc.Unsafe.park(Unsafe.java:-1)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:417)
    at jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream.current(ResponseSubscribers.java:362)
    at jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream.read(ResponseSubscribers.java:418)
    at java.util.zip.CheckedInputStream.read(CheckedInputStream.java:60)
    at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:267)
    at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:259)
    at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:165)
    at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:80)
    at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:92)
    at HttpClientBug.lambda$gzipBodyHandler$6(HttpClientBug.java:135)
    at HttpClientBug$$Lambda$191.1885277244.apply(Unknown Source:-1)
    at java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:680)
    at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:658)
    at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2094)
    at java.util.concurrent.CompletableFuture$MinimalStage.thenApply(CompletableFuture.java:2820)
    at jdk.internal.net.http.ResponseSubscribers$MappingSubscriber.getBody(ResponseSubscribers.java:675)
    at jdk.internal.net.http.ResponseSubscribers$MappingSubscriber.getBody(ResponseSubscribers.java:675)
    at jdk.internal.net.http.Http1Response.readBody(Http1Response.java:487)
    at jdk.internal.net.http.Http1Exchange.readBodyAsync(Http1Exchange.java:375)
    at jdk.internal.net.http.Exchange.readBodyAsync(Exchange.java:175)
    at jdk.internal.net.http.MultiExchange.lambda$responseAsync0$4(MultiExchange.java:305)
    at jdk.internal.net.http.MultiExchange$$Lambda$94.1264413185.apply(Unknown Source:-1)
    at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072)
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
    at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:610)
    at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:649)
    at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
    at jdk.internal.net.http.HttpClientImpl$DelegatingExecutor.execute(HttpClientImpl.java:153)
    at java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:568)
    at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:638)
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
    at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
    at jdk.internal.net.http.Http1Response$HeadersReader.handle(Http1Response.java:693)
    at jdk.internal.net.http.Http1Response$HeadersReader.handle(Http1Response.java:619)
    at jdk.internal.net.http.Http1Response$Receiver.accept(Http1Response.java:610)
    at jdk.internal.net.http.Http1Response$HeadersReader.tryAsyncReceive(Http1Response.java:666)
    at jdk.internal.net.http.Http1AsyncReceiver.flush(Http1AsyncReceiver.java:228)
    at jdk.internal.net.http.Http1AsyncReceiver$$Lambda$117.1645547422.run(Unknown Source:-1)
    at jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
    - locked <0xb74> (a java.lang.Object)
    at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
    at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
    at jdk.internal.net.http.HttpClientImpl$DelegatingExecutor.execute(HttpClientImpl.java:153)
    at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:273)
    at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:242)
    at jdk.internal.net.http.Http1AsyncReceiver.asyncReceive(Http1AsyncReceiver.java:459)
    at jdk.internal.net.http.Http1AsyncReceiver$Http1TubeSubscriber.onNext(Http1AsyncReceiver.java:579)
    at jdk.internal.net.http.Http1AsyncReceiver$Http1TubeSubscriber.onNext(Http1AsyncReceiver.java:536)
    at jdk.internal.net.http.common.SSLTube$DelegateWrapper.onNext(SSLTube.java:202)
    at jdk.internal.net.http.common.SSLTube$SSLSubscriberWrapper.onNext(SSLTube.java:484)
    at jdk.internal.net.http.common.SSLTube$SSLSubscriberWrapper.onNext(SSLTube.java:287)
    at jdk.internal.net.http.common.SubscriberWrapper$DownstreamPusher.run1(SubscriberWrapper.java:318)
    at jdk.internal.net.http.common.SubscriberWrapper$DownstreamPusher.run(SubscriberWrapper.java:261)
    at jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
    - locked <0xb75> (a java.lang.Object)
    at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
    at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
    at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:271)
    at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:224)
    at jdk.internal.net.http.common.SubscriberWrapper.outgoing(SubscriberWrapper.java:234)
    at jdk.internal.net.http.common.SubscriberWrapper.outgoing(SubscriberWrapper.java:200)
    at jdk.internal.net.http.common.SSLFlowDelegate$Reader.processData(SSLFlowDelegate.java:403)
    at jdk.internal.net.http.common.SSLFlowDelegate$Reader$ReaderDownstreamPusher.run(SSLFlowDelegate.java:264)
    at jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
    - locked <0xb76> (a java.lang.Object)
    at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
    at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(Thread.java:829)
    ----