Storage - SocketException Socket closed

Hi fellow Google cloud java users,
We are seeing Socket closed quite often.
The StorageOptions.Builder().connectTimeout and StorageOptions.Builder().readTimeout are not set so I guess the default of 20 seconds is used.
I didn't want to change them as 20 seconds should be plenty.

Does anyone have an idea about this?

java.net.SocketException: Socket closed
    at java.net.SocketInputStream.read
    at java.net.SocketInputStream.read
    at sun.security.ssl.InputRecord.readFully
    at sun.security.ssl.InputRecord.read
    at sun.security.ssl.SSLSocketImpl.readRecord
    at sun.security.ssl.SSLSocketImpl.readDataRecord
    at sun.security.ssl.AppInputStream.read
    at java.io.BufferedInputStream.fill
    at java.io.BufferedInputStream.read1
    at java.io.BufferedInputStream.read
    at sun.net.www.http.HttpClient.parseHTTPHeader
    at sun.net.www.http.HttpClient.parseHTTP
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream
    at java.net.HttpURLConnection.getResponseCode
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode
    at com.google.api.client.http.javanet.NetHttpResponse.<init>
    at com.google.api.client.http.javanet.NetHttpRequest.execute
    at com.google.api.client.http.HttpRequest.execute
    at com.google.cloud.storage.spi.DefaultStorageRpc.write
    at com.google.cloud.storage.BlobWriteChannel$1.run
    at java.util.concurrent.Executors$RunnableAdapter.call
    at com.google.cloud.RetryHelper.doRetry
    at com.google.cloud.RetryHelper.runWithRetries
    at com.google.cloud.RetryHelper.runWithRetries
    at com.google.cloud.storage.BlobWriteChannel.flushBuffer
    at com.google.cloud.BaseWriteChannel.close
    at com.spotify.buildartifactarchiver.storage.GcsUploaderClient.storeInputStream
    at com.spotify.buildartifactarchiver.storage.ArtifactUploader.uploadData
    at com.spotify.buildartifactarchiver.ArtifactArchiver.archiveRequest
    at com.spotify.buildartifactarchiver.ArtifactArchiver.archiveBuild
    at com.spotify.buildartifactarchiver.web.ArtifactArchiverPubSubResource.messageHandler
    at com.spotify.google.cloud.pubsub.client.Puller.lambda$pullBatch$11
    at java.util.concurrent.CompletableFuture.uniWhenComplete
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire
    at java.util.concurrent.CompletableFuture.postComplete
    at java.util.concurrent.CompletableFuture.complete
    at com.spotify.google.cloud.pubsub.client.PubsubFuture.succeed
    at com.spotify.google.cloud.pubsub.client.PubsubFuture.lambda$wrap$9
    at java.util.concurrent.CompletableFuture.uniWhenComplete
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire
    at java.util.concurrent.CompletableFuture.postComplete
    at java.util.concurrent.CompletableFuture.complete
    at com.spotify.google.cloud.pubsub.client.PubsubFuture.succeed
    at com.spotify.google.cloud.pubsub.client.Pubsub.lambda$requestJavaNet$8
    at java.util.concurrent.ThreadPoolExecutor.runWorker
    at java.util.concurrent.ThreadPoolExecutor$Worker.run
    at java.lang.Thread.run
com.google.cloud.storage.StorageException: Socket closed
    at com.google.cloud.storage.spi.DefaultStorageRpc.translate
    at com.google.cloud.storage.spi.DefaultStorageRpc.write
    at com.google.cloud.storage.BlobWriteChannel$1.run
    at java.util.concurrent.Executors$RunnableAdapter.call
    at com.google.cloud.RetryHelper.doRetry
    at com.google.cloud.RetryHelper.runWithRetries
    at com.google.cloud.RetryHelper.runWithRetries
    at com.google.cloud.storage.BlobWriteChannel.flushBuffer
    at com.google.cloud.BaseWriteChannel.close
    at com.spotify.buildartifactarchiver.storage.GcsUploaderClient.storeInputStream
    at com.spotify.buildartifactarchiver.storage.ArtifactUploader.uploadData
    at com.spotify.buildartifactarchiver.ArtifactArchiver.archiveRequest
    at com.spotify.buildartifactarchiver.ArtifactArchiver.archiveBuild
    at com.spotify.buildartifactarchiver.web.ArtifactArchiverPubSubResource.messageHandler
    at com.spotify.google.cloud.pubsub.client.Puller.lambda$pullBatch$11
    at java.util.concurrent.CompletableFuture.uniWhenComplete
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire
    at java.util.concurrent.CompletableFuture.postComplete
    at java.util.concurrent.CompletableFuture.complete
    at com.spotify.google.cloud.pubsub.client.PubsubFuture.succeed
    at com.spotify.google.cloud.pubsub.client.PubsubFuture.lambda$wrap$9
    at java.util.concurrent.CompletableFuture.uniWhenComplete
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire
    at java.util.concurrent.CompletableFuture.postComplete
    at java.util.concurrent.CompletableFuture.complete
    at com.spotify.google.cloud.pubsub.client.PubsubFuture.succeed
    at com.spotify.google.cloud.pubsub.client.Pubsub.lambda$requestJavaNet$8
    at java.util.concurrent.ThreadPoolExecutor.runWorker
    at java.util.concurrent.ThreadPoolExecutor$Worker.run
    at java.lang.Thread.run

@Capstan

Hi @fonzy2013, thanks for the report.

Maybe @Capstan has some insights on this. I can confirm that this exception showed once in our integration tests recently. I have never experienced it before though. @Capstan can this be related to some recent change on your side?

@aozarov I believe we could retry this kind of exceptions just want to hear some more thoughts on this :)

I think it should be safe to add SocketException as a retryable IOException provided the call is idempotent. Basically to add it to here.

sounds great! This would make it much cleaner for us using the library. Is there anything I can do to help out with debugging the root cause of this?

@Capstan

@fonzy2013 I am working on a PR to retry SocketException and "insufficient data written" exceptions right now. I'll cc you to it!

Is there anything I can do to help out with debugging the root cause of this?

I did some digging in the java code that is throwing these exceptions. I am quite sure this has nothing to do with our library code. I suspect these exceptions are related to network link disruption or the server resetting the connection. @Capstan is the most qualified person to confirm or deny my hypothesis.

@mziccard

@mziccard that sounds great! Thanks a million for all your help.
Looking forward to it ending up in a release!

@fonzy2013 gcloud-java 0.2.8 is now out and includes this fix!

great! Thanks!

chore(main): release 2.3.2-SNAPSHOT (#1180)

chore(deps): update dependency com.google.cloud:google-cloud-logging …

deps: update dependency org.easymock:easymock to v5.2.0 (#1180)

chore(main): release 2.17.1 (#1181)

chore(deps): update dependency com.google.cloud:google-cloud-logging …

chore: add native image checks as required (#1180)

build(deps): update dependency org.sonatype.plugins:nexus-staging-mav…

Navigation Menu

Repository navigation

Storage - SocketException Socket closed #1180

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Storage - SocketException Socket closed #1180

Description

Activity

mziccard commented on Aug 22, 2016

aozarov commented on Aug 22, 2016

fondberg commented on Aug 23, 2016

mziccard commented on Aug 23, 2016

fondberg commented on Aug 25, 2016

mziccard commented on Aug 26, 2016

fondberg commented on Aug 30, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions