Hot questions for Using Amazon S3 in pre signed url

Top Java Programmings / Amazon S3 / pre signed url

Question:


Answer:

Using some help I've found the answer, which was a combination of 2 missing pieces (one of which was referred to in the comments):

  1. Need to set this:

    System.setProperty(SDKGlobalConfiguration.ENABLE_S3_SIGV4_SYSTEM_PROPERTY, "true");
    
  2. Must set the "endPoint" (which was not required for upload or download):

    s3Client.setEndpoint(endpoint);
    

Optionally it might be useful to also add this:

s3Client.setS3ClientOptions(new S3ClientOptions().withPathStyleAccess(true));

Question:

I've the folowing code that signs a URL to an object inside an S3 Bucket

https://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURLJavaSDK.html

Now, if the Bucket has versioning enabled, how can I sign a specific version of the object link?


Answer:

Call withVersionId(String):

GeneratePresignedUrlRequest generatePresignedUrlRequest = 
                new GeneratePresignedUrlRequest(bucketName, objectKey)
                .withMethod(HttpMethod.GET)
                .withExpiration(expiration)
                .withVersionId("yourVesionId");

Sets the version ID of the object, only present if versioning has been enabled for the bucket.

Question:

Use case: we have client that can download pictures from the S3 storage using pre-signed URL and periodically refresh them. We don't need to download the picture again if it has not changed. Problem: Can we use ETag and if-none-matchheader for solve our problem if pictures URL can change(e.g. after pre-signed URL expiration)?


Answer:

We succesfully tested using Etag with S3 Pre-signed URL. You can send Etag with if-none-match header in Pre-signed URL and receive 304 not modified if content is identical. It will works regardless of URL, e.g.

  1. You get URL that expires after 10 minutes and use it for download image.
  2. Then you save Etag of this image
  3. An hour later, when the first URL expires, you get a second URL to check whether the image has changed
  4. Now you send saved Etag in header with second URL

Question:

I generate a pre sign url to access objects in S3. This url is set to expire after 1 hour. It has been working fine for months, but has all of a sudden stopped working. For every url I create, the expiry for that request is always the same time, see:

<Error>
<Code>AccessDenied</Code>
<Message>Request has expired</Message>
<X-Amz-Expires>3599</X-Amz-Expires>
<Expires>2018-05-27T22:56:29Z</Expires>
<ServerTime>2018-05-28T00:20:17Z</ServerTime>
<RequestId>xxx</RequestId>
<HostId>yyy</HostId>
</Error>

Any help would be great! Thanks!


Answer:

Signed URLs created with Signature V4 do not expire at the time you generate them plus the expiration. They expire at the time your code claims they were generated, which you can find in the X-Amz-Date field in the URL.

Compare this value to the current time in UTC.

This error suggests the clock on the server generating them is wrong, or, if the clock on that server is not set to UTC, then the time zone on the system clock may be wrong.

Question:

We are using presigned s3 urls to provide web access to images stored in s3.

The java code we are using to generate the presigned urls is similar to below

String accessKey = ...;
String secretKey = ...;
String region = ...;
com.amazonaws.HttpMethod awsHttpMethod = ...;
String bucketName = ...;
String objectKey = ...;
Date expirationDate = ...;

BasicAWSCredentials creds = new BasicAWSCredentials(accessKey, secretKey);
AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(creds)).withRegion(region).build();
GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName, objectKey);
generatePresignedUrlRequest.setMethod(awsHttpMethod);
generatePresignedUrlRequest.setExpiration(expirationDate);
URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);

The url that is generated by the code looks similar to

https://com.mycompany.personalpictures.s3.amazonaws.com/picture123.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20170623T150540Z&X-Amz-SignedHeaders=host&X-Amz-Expires=59&X-Amz-Credential=AKIAIVLB4ANK6B45G3IA%2F20170623%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=d25d407ee8efa76f339388ec93579a19be8eaead9663d6d378cf2ec6d9d9cac2

However since our bucket naming standard contains dots, a call to above URL results in a SSL: no alternative certificate subject name matches target host name 'com.mycompany.personalpictures.s3.amazonaws.com' error

I read in this post that the root cause is the dots in the bucket name and that using https://s3.amazonaws.com/com.mycompany.personalpictures/picture123.png should circumvent the problem.

How can I generate presigned urls using the url format https://s3.amazonaws.com/mybucket/myfile?


Answer:

Figured it out...

Needed to use .enablePathStyleAccess() when creating the s3 client. With that the code line now is

AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(creds)).withRegion(region).enablePathStyleAccess().build();

Question:

I am using generatePresignedUrl method of AmazonS3Client class to get a pre-signed URL. However when the method is called in the same session as that of the upload of the file. it returns url of the form :

https://mp-dev.downloads.XYZ.com.s3.ap-south-1.amazonaws.com/certificate/404/17_04_2017/XYZ/ITHLUaXYPnMmHEUbK9L21KBneJQy7oJ1jw.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20170417T090656Z&X-Amz-SignedHeaders=host&X-Amz-Expires=900&X-Amz-Credential=ASDFSFFGODQ%2F20170417%2Fap-south-1%2Fs3%2Faws4_request&X-Amz-Signature=9b2e112608c46cbeb16ff577b6e1321f889efsdfdsc850212251b231cb909d1942

But when I rerun my API and call the same method with the same params I get url of the form :

https://mp-dev.downloads.XYZ.com.s3.amazonaws.com/certificate/404/17_04_2017/XYZ/ITHLUaXYPnMmHEUbK9L21KBneJQy7oJ1jw.zip?AWSAccessKeyId=AKIAJFAFDTTISR4UHODQ&Expires=1492421420&Signature=CZuY1acATGUEEpoXN8aEQFXLX18%3D

First Url works fine but the second url results in :

The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.

I have tried to change configuration to use signature version 4 as per the following discussion :

The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256

But doesn't work. Ask me if any additional info is required.

Endpoint to generate url :

@SkipAuthentication
  @GET
  @Path("/xyz")
  public String download(@QueryParam("cer") String objectKey) {
    return testService.test(objectKey).toString();
  }

Test method of service :

public URL test(String objectKey) {
    return awsDownloadService.generateDownloadUrl(objectKey, awsServer.getDownloadsBucketName());
  }

The generateDownloadUrl method is as follows :

  public URL generateDownloadUrl(String keyName, String bucketName) {
    LOG.info("generateDownloadUrl - {} {} {} {}>",accessKeyId, secretAccessKey, keyName, bucketName);
    BasicAWSCredentials awsCreds = new BasicAWSCredentials(accessKeyId, secretAccessKey);
    AmazonS3Client s3Client = new AmazonS3Client(awsCreds);

    Date expiration = new Date();
    long msec = expiration.getTime();
    msec += 1000 * 60 * 15; // 15 minutes.
    expiration.setTime(msec);
    GeneratePresignedUrlRequest generatePresignedUrlRequest = 
                  new GeneratePresignedUrlRequest(bucketName, keyName);
    generatePresignedUrlRequest.setMethod(HttpMethod.GET); // Default.
    generatePresignedUrlRequest.setExpiration(expiration);
    URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);
    LOG.info("GeneratePreAuthenticatedUrl - url generated is <{}>",url);
    return url;
  }

When I upload the file using upload method and then call the end point to generate url it works fine but when I restart the api and call the end point to generate url, different url is returned as described earlier in the question.

Solution and new query:

I have solved the problem. Apparently ,AmazonS3Client is deprecated. Making client this way works consistently :

AmazonS3ClientBuilder builder = AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(awsCreds));
    builder.setRegion(Regions.AP_SOUTH_1.getName());
    AmazonS3 s3client = builder.build();

Still, the behavior AmazonS3Client exhibited earlier is kind of odd. Can anybody provide explanation of the earlier behavior?


Answer:

Apparently the AmazonS3Client class is deprecated in the latest SDKs. Creating client in the following manner solves the problem as it also takes region into account and hence attaches proper signature versions :

BasicAWSCredentials awsCreds = new BasicAWSCredentials(accessKeyId, secretAccessKey);
    AmazonS3ClientBuilder builder = AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(awsCreds));
    builder.setRegion(regionName);
    AmazonS3 s3client = builder.build();