Hot questions for Using Amazon S3 in spring boot

Question:

I have an auto-configured AWS, Spring Boot application, and I'm trying to setup an endpoint that will simply download a particular file from a given bucket in Amazon S3. I uploaded a JPEG file into the bucket from my computer using the AWS console - now I'm trying to download that file using my Spring Boot API.

I'm getting the following error: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied;

I have created a user and a group (user is in the group) on AWS console; the user/group has full access permissions on S3 as well as administrator access. I downloaded the access-key/secret-key pair and, for testing purposes, literally pasted the keys into my application.properties file as shown below (keys are not shown here, obviously :) ).

I'm confused as to why I'm still getting access denied. I've been searching and working on this for a while; I can't seem to find a solution to this issue that is specific to Spring Boot. Any help would be greatly appreciated.

application.properties:

cloud.aws.credentials.accessKey=myaccesskey
cloud.aws.credentials.secretKey=mysecretkey
cloud.aws.credentials.instanceProfile=false
cloud.aws.stack.auto=false

cloud.aws.region.auto=true
cloud.aws.region.static=myregion

SimpleResourceLoadingBean.java:

@RestController
public class SimpleResourceLoadingBean {

    private static Logger log = LoggerFactory.getLogger(HealthMonitorApplication.class);

    @Autowired
    private ResourceLoader resourceLoader;


    @RequestMapping("/getresource")
    public String resourceLoadingMethod() throws IOException {
        log.info("IN RESOURCE LOADER");

        Resource resource = this.resourceLoader.getResource("s3://s3.amazonaws.com/mybucket/myfile.ext");

        InputStream inputStream = resource.getInputStream();

        return inputStream.toString();
    }
}

pom.xml (Just the dependencies that are relevant to the question)

        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-aws</artifactId>
            <version>1.1.0.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-aws-autoconfigure</artifactId>
            <version>1.1.0.RELEASE</version>
        </dependency>

Answer:

Figured out the solution. Besides the application.properties configuration, I had to create a configuration class that would give me access to an AmazonS3Client object when provided the appropriate credentials. I followed this example on GitHub:

https://github.com/brant-hwang/spring-cloud-aws-example/blob/master/src/main/java/com/axisj/spring/cloud/aws/AWSConfiguration.java

AWSConfiguration.java:

import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.regions.Region;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3Client;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class AWSConfiguration {

    @Value("${cloud.aws.credentials.accessKey}")
    private String accessKey;

    @Value("${cloud.aws.credentials.secretKey}")
    private String secretKey;

    @Value("${cloud.aws.region}")
    private String region;

    @Bean
    public BasicAWSCredentials basicAWSCredentials() {
        return new BasicAWSCredentials(accessKey, secretKey);
    }

    @Bean
    public AmazonS3Client amazonS3Client(AWSCredentials awsCredentials) {
        AmazonS3Client amazonS3Client = new AmazonS3Client(awsCredentials);
        amazonS3Client.setRegion(Region.getRegion(Regions.fromName(region)));
        return amazonS3Client;
    }
}

Once this is configured, you can create AmazonS3Client objects (autowired) in your other classes, and use the client to make requests to your S3 cloud. The example uses a wrapper class as a service in order to ease the implementation of additional controller classes.

S3Wrapper.java:

import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.*;
import org.apache.commons.io.IOUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Service;
import org.springframework.util.StringUtils;
import org.springframework.web.multipart.MultipartFile;

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.net.URLEncoder;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

@Service
public class S3Wrapper {

    @Autowired
    private AmazonS3Client amazonS3Client;

    @Value("${cloud.aws.s3.bucket}")
    private String bucket;

    private PutObjectResult upload(String filePath, String uploadKey) throws FileNotFoundException {
        return upload(new FileInputStream(filePath), uploadKey);
    }

    private PutObjectResult upload(InputStream inputStream, String uploadKey) {
        PutObjectRequest putObjectRequest = new PutObjectRequest(bucket, uploadKey, inputStream, new ObjectMetadata());

        putObjectRequest.setCannedAcl(CannedAccessControlList.PublicRead);

        PutObjectResult putObjectResult = amazonS3Client.putObject(putObjectRequest);

        IOUtils.closeQuietly(inputStream);

        return putObjectResult;
    }

    public List<PutObjectResult> upload(MultipartFile[] multipartFiles) {
        List<PutObjectResult> putObjectResults = new ArrayList<>();

        Arrays.stream(multipartFiles)
                .filter(multipartFile -> !StringUtils.isEmpty(multipartFile.getOriginalFilename()))
                .forEach(multipartFile -> {
                    try {
                        putObjectResults.add(upload(multipartFile.getInputStream(), multipartFile.getOriginalFilename()));
                    } catch (IOException e) {
                        e.printStackTrace();
                    }
                });

        return putObjectResults;
    }

    public ResponseEntity<byte[]> download(String key) throws IOException {
        GetObjectRequest getObjectRequest = new GetObjectRequest(bucket, key);

        S3Object s3Object = amazonS3Client.getObject(getObjectRequest);

        S3ObjectInputStream objectInputStream = s3Object.getObjectContent();

        byte[] bytes = IOUtils.toByteArray(objectInputStream);

        String fileName = URLEncoder.encode(key, "UTF-8").replaceAll("\\+", "%20");

        HttpHeaders httpHeaders = new HttpHeaders();
        httpHeaders.setContentType(MediaType.APPLICATION_OCTET_STREAM);
        httpHeaders.setContentLength(bytes.length);
        httpHeaders.setContentDispositionFormData("attachment", fileName);

        return new ResponseEntity<>(bytes, httpHeaders, HttpStatus.OK);
    }

    public List<S3ObjectSummary> list() {
        ObjectListing objectListing = amazonS3Client.listObjects(new ListObjectsRequest().withBucketName(bucket));

        List<S3ObjectSummary> s3ObjectSummaries = objectListing.getObjectSummaries();

        return s3ObjectSummaries;
    }
}

Note: The following dependency will need to be added to pom.xml in order to use the Apache Commons IO library.

pom.xml:

<dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-io</artifactId>
    <version>1.3.2</version>
</dependency>

Question:

I am building a Spring-boot application where in all the logging is stored in a specified path in log4j.properties.

I am hosting this application on AWS Beanstalk.

Once I host the application on the AWS, the specified path for the log file will become invalid.

How can I resolve this issue where in the log file should also get stored in a different path in cloud, say Amazon S3, but not on server as the log file takes considerable amount of size.

log4j.appender.file.File=/my_log.log

How can I change the above line to store the "my_log.log" in AWS S3?


Answer:

Log file should also get stored in a different path in cloud, say Amazon S3.

I am sure that you cannot log your info directly to S3. It is a costlier process. First you have to log in EC2 instance and then you have to sync to sync your log you can use LogRotate.

To Store your log in S3 refer logrotate apache logs to amazon S3.

Question:

I am currently trying to set up an s3 bucket with my Spring Boot webapp for adding/removing images.

The guide I am following uses the following application.yml properties:

amazonProperties:
  endpointUrl: https://s3.us-east-2.amazonaws.com
  accessKey: XXXXXXXXXXXXXXXXX
  secretKey: XXXXXXXXXXXXXXXXXXXXXXXXXX
  bucketName: your-bucket-name

How can I define these properties in my application.properties file?

All help is very much appreciated, thank you!


Answer:

Try to just split them in different lines in your application.properties:

amazonProperties.endpointUrl= https://s3.us-east-2.amazonaws.com
amazonProperties.accessKey= XXXXXXXXXXXXXXXXX
amazonProperties.secretKey= XXXXXXXXXXXXXXXXXXXXXXXXXX
amazonProperties.bucketName= your-bucket-name

Question:

I'm having some drama when running the listObjects(..) method of AmazonS3. I'm certain that my credentials are set up correctly as I am able to download individual files using s3Client.getObject(..). The logs read::

com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler Caused by: org.xml.sax.SAXParseException: Premature end of file.

I understand that listObjects(..) does include in it's response some xml containing meta data. The code to reproduce the error is very simple. I can't see anything wrong here :(

ListObjectsRequest listObjectsRequest = new ListObjectsRequest() .withBucketName(ENV.getProperty("cloud.aws.s3.bucket"));

ObjectListing objectListing = amazonS3Client.listObjects(listObjectsRequest);

Here is the version of spring-cloud-aws-context I am using:: <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-aws-context</artifactId <version>1.2.1.RELEASE</version> </dependency>

Does anybody have any insight? Or know away around this issue?

Thanks in advance :)


Answer:

I encountered the exact exception *Failed to parse XML document with handler class * and the failure is not truly descriptive. But my problem was not permissions but rather trying to list the bucket subfolder directly.

I was trying to listObjects from /bucketName/subFolder/subFolder2 instead of just /bucketName and prefix.

This results in exception above (In Scala):

val path = "/bucketName/myFolder/subFolder"
val results = s3Client.listObjectsV2(path)

I needed to separate the bucket name and the prefix and then use ListObjectRequestV2

val path = "/bucketName/myFolder/subFolder"
val bucketName = "bucketName"
val prefix = "myFolder/subFolder"
val listObjectsRequest = new 
val ListObjectsV2Request().withBucketName(bucketName).withPrefix(prefix) 
val results = s3Client.listObjectsV2(path)

Question:

I've some set of objects and want to store these data as CSV file on AWS S3 bucket without creating any local file. Can anyone please suggest how could it be smoothly done without impacting performance?

For Eg.: Set<Address> addresses; //Address class contains some fields as city, state, zipcode and country.

It should create a csv file with following headers and data:

City, State, ZipCode, Country
-----------------------------

Mumbai, Maharashtra, 4200091, India

One thing I know is we can write data as InputStream and then pass it to -PutObjectRequest. But InputStream also takes filepath I don't want to waste time in creating temp files and I've multiple operations to do.

PutObjectRequest putObj = new PutObjectRequest(bucketName, KeyName, inputStream, metadata); 
s3client.putObject(putObj);

Thanks in advance for your help and time.


Answer:

You could do something like this:

  1. Create csv output stream:(Refer:)

Dependency to be added (Refer to above link for the example used):

<dependency>
    <groupId>net.sourceforge.javacsv</groupId>
    <artifactId>javacsv</artifactId>
    <version>2.0</version> </dependency>

Code :

public void createCsv() throws Exception {

    ByteArrayOutputStream stream = new ByteArrayOutputStream();
    CsvWriter writer = new CsvWriter(stream, ',', Charset
            .forName("ISO-8859-1"));

    writer.setRecordDelimiter(';');
    // WRITE Your CSV Here
    //writer.write("a;b");
    writer.endRecord();
    writer.close();

    stream.close();    }
  1. Convert output stream to input stream via byte array:

    InputStream inputStream = new ByteArrayInputStream(stream.toByteArray());

  2. Pass this stream to S3 PutObjectRequest

Question:

I need to save binary (byte[]) contents (PDF file) to S3.

I don't want to have a hard copy existing in HDD, pdf is being generated in the fly and then it needs to be sent to S3.

Java AWS SDK AmazonS3.putObject() requires File type in method signature, how can I pass my binary contents directly without saving it to hard drive?


Answer:

The SDK offers method that take InputStream and ObjectMetadata instead of File. For example there is putObject(String bucketName, String key, InputStream input, ObjectMetadata metadata).

Use one of the provided methods and supply ByteArrayInputStream if you don't want to create files.

Question:

How to iterate Amazon S3 file partially?

In the code below

 GetObjectRequest rangeObjectRequest = new GetObjectRequest(
        bucket, key);
rangeObjectRequest.setRange(0, 10);

S3Object objectPortion = amazonS3Client.getObject(rangeObjectRequest);

As per the documentation, I can only enter values from 0 to 9.

/* The first byte in an object has position 0; as an example, the first ten bytes of an object can be downloaded by specifying a range of 0 to 9.*/

http://docs.aws.amazon.com/AmazonS3/latest/dev/RetrievingObjectUsingJava.html


Answer:

You're overlooking the significance of the phrase "as an example".

If you want to read the first 10 bytes, you start at offset 0 and stop after offset 9. If you want the 101st through 200th byte, the values would be 100, 199 (the first byte is offset 0).

Any values < the total number of bytes in the object are valid.

Question:

I have integrated AWS Java SDK in my applcaition.Unfoutunately am getting "Internal Failure. Please try your request again" as the response.

This is how I have implemeneted it.

Using Maven, added this in pom.xml

<dependencies>
       <dependency>
          <groupId>software.amazon.awssdk</groupId>
          <artifactId>transcribe</artifactId>
       </dependency>
    </dependencies>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <version>2.10.12</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

And in code,

String localAudioPath = "/home/****.wav";
String key = config.awsSecretAccessKey;
String keyId = config.awsAccessKeyId;
String regionString = config.awsRegion; //"ap-south-1"
String outputBucketName = config.awsOutputBucket;
Region region = Region.of(regionString);


String inputLanguage = "en-US";
LanguageCode languageCode = LanguageCode.fromValue(inputLanguage);


AwsCredentials credentials = AwsBasicCredentials.create(keyId, key);
AwsCredentialsProvider transcribeCredentials=StaticCredentialsProvider.create(credentials);

AWSCredentialsProvider s3AwsCredentialsProvider = getS3AwsCredentialsProvider(key, keyId);

String jobName = subJob.getId()+"_"+subJob.getProgram_name().replace(" ", "");
String fileName = jobName + ".wav";

AmazonS3 s3 = 
AmazonS3ClientBuilder.standard().withRegion(regionString).withClientConfiguration(new 
ClientConfiguration()).withCredentials(s3AwsCredentialsProvider).build();
s3.putObject(outputBucketName, fileName, new File(localAudioFilePath));

String fileUri = s3.getUrl(outputBucketName, fileName).toString();

System.out.println(fileUri);
Media media = Media.builder().mediaFileUri(fileUri).build();

String mediaFormat = MediaFormat.WAV.toString();
jobName = jobName +"_"+ System.currentTimeMillis();

Settings settings = Settings.builder()
           .showSpeakerLabels(true)
           .maxSpeakerLabels(10)
           .build();

StartTranscriptionJobRequest request = StartTranscriptionJobRequest.builder()
           .languageCode(languageCode)
           .media(media)
           .mediaFormat(mediaFormat)
           .settings(settings)
           .transcriptionJobName(jobName)
           .build();

TranscribeAsyncClient client = TranscribeAsyncClient.builder()
           .region(region)
           .credentialsProvider(transcribeClientCredentialsProvider)
           .build();

CompletableFuture<StartTranscriptionJobResponse> response = 
client.startTranscriptionJob(request);

System.out.println(response.get().toString());

GetTranscriptionJobRequest jobRequest = 
GetTranscriptionJobRequest.builder().transcriptionJobName(jobName).build();

while( true ){
    CompletableFuture<GetTranscriptionJobResponse> transcriptionJobResponse = 
    client.getTranscriptionJob(jobRequest);


    GetTranscriptionJobResponse response1 = transcriptionJobResponse.get();
    if (response1 != null && response1.transcriptionJob() != null) {
          if (response1.transcriptionJob().transcriptionJobStatus() == 
                    TranscriptionJobStatus.FAILED) {

               //It comes here and gives response1.failureReason = "Internal Failure. Please try your request again".
               break;
          }
    }
}

private AWSCredentialsProvider getS3AwsCredentialsProvider(String key, String keyId) {
    return new AWSCredentialsProvider() {
                @Override
                public AWSCredentials getCredentials() {
                    return new AWSCredentials() {
                        @Override
                        public String getAWSAccessKeyId() {
                            return keyId;
                        }

                        @Override
                        public String getAWSSecretKey() {
                            return key;
                        }
                    };
                }

                @Override
                public void refresh() {

                }
    };
}

The same thing is working with Python SDK. Same region, same wav file, same language, same settings, same output bucket etc. What am doing wrong??


Answer:

Your flow looks correct. It may be an issue with the audio file you are uploading to AWS. I suggest you check it once.

Question:

My AWS S3 contains some files and I want to download those files through my application as a zip file. I'm able to download the zip file successfully but while the file I'm getting an error Unexpected end of archive and also the CSV file size is 0 inside the zip.

I did dig a lot but not able to understand the exact solutions.

Here's the code:

String zipFileName = "MyZipFile.zip";

String fileName = "test.csv";

filePath = new String(Base64.getDecoder().decode(filePath));
System.out.println("file:" + filePath);

// get file from S3
S3Object s3Obj = awsClient.downloadFile(filePath);

byte[] s3Bytes = s3Obj.getObjectContent().readAllBytes();

// create zip
ByteArrayOutputStream byteArrOutputStream = new ByteArrayOutputStream(s3Bytes.length);
ZipOutputStream zipOutStream = new ZipOutputStream(byteArrOutputStream);
ZipEntry zip = new ZipEntry(fileName);
zipOutStream.putNextEntry(zip);
zipOutStream.write(s3Bytes, 0, s3Bytes.length);
byte[] streamBytes = byteArrOutputStream.toByteArray();

// close streams
zipOutStream.closeEntry();
zipOutStream.close();
closeQuietly(byteArrOutputStream);

// prepare download
System.out.println("streamBytes:" + streamBytes + " len:" + streamBytes.length);
System.out.println("s3Bytes:" + s3Bytes + " len:" + s3Bytes.length);

HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_OCTET_STREAM);
headers.setContentLength(streamBytes.length);
headers.setContentDispositionFormData("attachment", zipFileName);

Answer:

I think you should close your zip output stream to have everything written to the underlying byte array stream before you extract the byte array... Ie reorder the lines to:

// close streams
zipOutStream.closeEntry();
zipOutStream.close();
byte[] streamBytes = byteArrOutputStream.toByteArray();
closeQuietly(byteArrOutputStream);

Question:

I have actually a problem within my spring-boot application i developed a restful api linked to an s3 bucket i've customized some exceptions but when i run my url to get an object that doesn't exsit in the console i saw this error exception in my console :

2019-07-25 09:06:45.733 ERROR 1 --- [-nio-443-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 60E24BCF6860FC66; S3 Extended Request ID: DUEnMWN7YZKug74Q15uHt4Zei3+a7SxTNYzoj99O0YW58WOwvkdM1kwYpcHrGJiTrLkRLOdUL5I=), S3 Extended Request ID: DUEnMWN7YPOAIZADg74Q15uHt4Zei3+a7SxTNYzoj99O0YW58WOwvkdM1kwYpcHrGJiTrLkRLOdUL5I=] with root cause

com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 60E24BCF6860FC66; S3 Extended Request ID: DUEnMWN7YZKug74Q15uHt4ZDFSFSQSDei3+a7SxTNYzoj99O0YW58WOwvkdM1kwYpcHrGJiTrLkRLOdUL5I=)

So my question is how to perform customization to this error in 2 points:

  1. Not having this error showing in the console (technically I don't mean change log parameters)
  2. Handle this exception with some methods

Answer:

Write custom exception handler to capture your exceptions and do graceful things.

@ControllerAdvice
@RestController
public class CustomExceptionHandler extends ResponseEntityExceptionHandler {

    @ExceptionHandler(AmazonS3Exception.class)
    public final ResponseEntity<Object> handleAmazonS3Exception(AmazonS3Exception ex, WebRequest request) {
        ApiError apiError = new ApiError(UNPROCESSABLE_ENTITY, ex.getLocalizedMessage(), ex.getErrors());
        return new ResponseEntity<>(apiError, UNPROCESSABLE_ENTITY);
    }
}

Here, UNPROCESSABLE_ENTITY is a HttpStatus (org.springframework.http.HttpStatus)

Please change the code of method handleAmazonS3Exception(...) based on your requirement.

Question:

We just created a custom AmazonS3Client with credentials on a project that was already using Amazon S3 functionality:

import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;

@Configuration
public class S3Config {

    @Bean
    public static AmazonS3Client amazonS3Client(final AWSCredentialsProvider awsCredentialsProvider) {
        return (AmazonS3Client) AmazonS3ClientBuilder.standard()
                .withCredentials(awsCredentialsProvider)
                .build();
    }
}

It has worked just fine on all other project, but for some reason, when starting up the application, we get this error:

Parameter 0 of constructor in foo.bar.MyService required a single bean, but 2 were found:
    - amazonS3Client: defined by method 'amazonS3Client' in class path resource [foo/bar/S3Config.class]
    - amazonS3: defined in null

Nowhere, absolutely nowhere on the project we have an amazonS3 Bean defined.

So, what are the contents of this Service class? Well, nothing special:

import com.amazonaws.services.s3.AmazonS3Client;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.io.ByteArrayInputStream;
import java.net.URL;

@Service
public class MyService {
    private final AmazonS3Client s3Client;

    @Autowired
    public MyService(AmazonS3Client s3Client) {
        this.s3Client = s3Client;
    }

    ...
}

It is supposed to use the AmazonS3Client we just created, and according to the first match of the error message it matched it just fine. If I delete my S3Config class, the bean duplication error is gone.

We don't want to force the project to use our AmazonS3Client implementation by adding the @Primary annotation.

So, what could we be doing wrong?


Answer:

After some hours of debugging, we realized the Service's constructor's parameter name was not exactly named as the Bean. We renamed it so that it matched the Bean's name:

@Service
public class MyService {
    private final AmazonS3Client s3Client; //Just fine

    @Autowired
    public MyService(AmazonS3Client amazonS3Client) { // Must match the bean name
        this.s3Client = amazonS3Client;
    }

    ...
}

And the Bean duplication error was gone. All we have to do is name the constructor's parameter just like the bean.

Question:

@Autowired private ResourcePatternResolver resourcePatternResolver;

String s3path = req.s3Folder+"/key1/key123/*.gz";

Resource[] allTxtFilesInFolder  = resourcePatternResolver.getResources(s3path);

I am trying to read file from s3 using spring-cloud-starter-aws. Anyhow it works well when filename is fully specified and doesn't work with wildcards.

This is what I see in log

INFO [ main] [.i.s.PathMatchingResourcePatternResolver] : Failed to resolve Amazon s3 resource [bucket='bucketname' and object='2l6hpfhfryz8422qr8nxy8x0a2-0/key1/key123'] in the file system: java.lang.UnsupportedOperationException: Amazon S3 resource can not be resolved to java.io.File objects.Use getInputStream() to retrieve the contents of the object!

why is PathMatchingResourcePatternResolver called instead of PathMatchingSimpleStorageResourcePatternResolver


Answer:

After researching I found the answer to this. I have to explicitly autowire resourcePatternResolver to be PathMatchingSimpleStorageResourcePatternResolver

private ResourcePatternResolver resourcePatternResolver;

    @Autowired
    public void setupResolver(ApplicationContext applicationContext, AmazonS3 amazonS3){
        this.resourcePatternResolver = new PathMatchingSimpleStorageResourcePatternResolver(amazonS3, applicationContext);
    }

Reference - https://cloud.spring.io/spring-cloud-static/spring-cloud-aws/2.0.0.RELEASE/multi/multi__resource_handling.html

Question:

Is it possible to get all files name from s3 bucket? In my local spring boot project, i wrote something like, and it works.

File[] files = new File("/myfiles").listFiles();
    for (File file : files) {
        if (file.isFile()) {
            filesDir.add(file.getName());
        }
    }

On AWS i try this, but it doesnt work.

File[] files = new File("https://s3.eu-central-1.amazonaws.com/bucket_name/myfiles/").listFiles();
        for (File file : files) {
            if (file.isFile()) {
                filesDir.add(file.getName());
            }
        }

What is wrong, how can i get directory path from s3?


Answer:

First of all, you need to connect to S3. To do this, following below suggest. Add in your pom the AWS API

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk</artifactId>
    <version>1.11.113</version>
</dependency>

You will create an instance of AmazonS3 service. Like this:

BasicAWSCredentials credentials = new BasicAWSCredentials("ACCESS KEY", "SECRET KEY");
AmazonS3 service AmazonS3Client.builder()
             .withClientConfiguration(clientConfiguration)
             .withEndpointConfiguration(new EndpointConfiguration("YOUR_ENDPOINT", "YOUR_REGION"))
             .withCredentials(new AWSStaticCredentialsProvider(credentials))
             .build();

After connecting, you can retrieve the information about the bucket using the service you just have created.

ListObjectsV2Request req = new ListObjectsV2Request().withBucketName("bucket").withPrefix("path_your_file_or_folder");
ListObjectsV2Result result = service.listObjectsV2(req)
for (S3ObjectSummary object: result .getObjectSummaries()){
    String key = object.getKey(); //your object it's here.
}

After getting the key to your file, you can download it. I hope this helps you.