Hot questions for Using Amazon S3 in apache httpclient 4.x

Top Java Programmings / Amazon S3 / apache httpclient 4.x

Question:

I am trying to upload a file to Amazon S3 using a customer provided secret key. I followed this tutorial here: http://java.awsblog.com/post/TxDQ18N7AAB31J/Generating-Amazon-S3-Pre-signed-URLs-with-SSE-C-Part-5-Finale

My code:

AWSCredentials credentials = new BasicAWSCredentials("myKey", "mySecretKey");
AmazonS3 s3 = new AmazonS3Client();

try (CloseableHttpClient httpClient = HttpClientFactory.createUploadClient()) {
    GeneratePresignedUrlRequest genreq = new GeneratePresignedUrlRequest("bucketName", "test7.pdf", HttpMethod.PUT);
    SecretKey secretKey = generateSecretKey();
    SSECustomerKey sseKey = new SSECustomerKey(secretKey);
    genreq.setSSECustomerKey(sseKey);

    URL puturl = s3.generatePresignedUrl(genreq);

    HttpPut putreq = new HttpPut(URI.create(puturl.toExternalForm()));

    putreq.addHeader(Headers.SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY, sseKey.getKey());
    putreq.addHeader(Headers.SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM, SSEAlgorithm.AES256.getAlgorithm());
    putreq.addHeader(Headers.SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5, sseKey.getMd5());
    putreq.setEntity(new FileEntity(new File("filePath")));
    HttpResponse resp = httpClient.execute(putreq);
    Assert.assertTrue(resp.getStatusLine().getStatusCode() == 200);
} catch (IOException e) {
    Assert.fail();
}

the URL gets generated, and then when I try to consume I am providing the needed headers, but the problem is that the response is 403 Forbidden.

What am I missing?


Answer:

In order for the SSE-Client Specific encryption upload using pre-signed URL to work, you need to enable SigV4 (by default, SigV2 was enabled for me) There are several ways to enable this - system property, bucket policy etc. but for me this worked:

AWSCredentials credentials = new BasicAWSCredentials("accessKey", "secretKey"); s3 = new AmazonS3Client(credentials, new ClientConfiguration().withSignerOverride("AWSS3V4SignerType"));

Question:

I have simply configured Spring AWS Cloud:

<aws-context:context-credentials>
    <aws-context:simple-credentials access-key="${s3.key}" secret-key="${s3.secret}"/>
</aws-context:context-credentials>

<aws-context:context-resource-loader/>

I'm trying to upload file onto bucket

ObjectMetadata meta = new ObjectMetadata();
meta.setContentType(contentType);
meta.setContentLength(bytes.length);

try (ByteArrayInputStream bis = new ByteArrayInputStream(bytes)) {
    TransferManager transferManager = new TransferManager(this.amazonS3);
    Upload upload = transferManager.upload(BUCKET, fileName, bis, meta);
    UploadResult result = upload.waitForUploadResult(); // here is an exception!
}

Same exception appears if I use another method:

PutObjectRequest request = new PutObjectRequest(BUCKET, fileName, bis, meta);
request.setCannedAcl(CannedAccessControlList.PublicRead);
amazonS3.putObject(request);

It causes exception Stacktrace :

com.amazonaws.AmazonClientException: Unable to complete transfer: Connection pool shut down
    at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.unwrapExecutionException(AbstractTransfer.java:277)
    at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.rethrowExecutionException(AbstractTransfer.java:261)
    at com.amazonaws.services.s3.transfer.internal.UploadImpl.waitForUploadResult(UploadImpl.java:66)
...

Caused by: java.lang.IllegalStateException: Connection pool shut down
    at org.apache.http.util.Asserts.check(Asserts.java:34)
    at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:184)
    at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:217)
    at org.apache.http.impl.conn.PoolingClientConnectionManager.requestConnection(PoolingClientConnectionManager.java:186)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:72)
    at com.amazonaws.http.conn.$Proxy11.requestConnection(Unknown Source)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:416)
    at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:884)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:749)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:505)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:317)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3595)
    at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1382)
    at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInOneChunk(UploadCallable.java:131)
    at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:123)
    at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:139)
    at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:47)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    ... 1 more

It looks like PoolingClientConnectionManager has arrived here from apache httpclient 4.5.2 which I also use in my project

    <dependency>
        <groupId>org.apache.httpcomponents</groupId>
        <artifactId>httpclient</artifactId>
        <version>4.5.2</version>
    </dependency>

But it is just an assumption. The problem may be some where else. Any ideas?


Answer:

I've found workaround, but it requires completely change method of Amazon S3 configuration:

<bean id="basicAwsCredentials" class="com.amazonaws.auth.BasicAWSCredentials">
    <constructor-arg name="accessKey" value="${s3.key}"/>
    <constructor-arg name="secretKey" value="${s3.secret}"/>
</bean>

<bean id="amazonS3Client" class="com.amazonaws.services.s3.AmazonS3Client">
    <constructor-arg name="awsCredentials" ref="basicAwsCredentials"/>
</bean>

In such configuration everything works like a charm.


So, it looks like Spring's aws-context works wrongly, or I use it wrongly. But in previous configuration autowired AmazonS3 instance wasn't null, so I don't know the reason why it doesn't work properly.