Hot questions for Using Amazon S3 in spring mvc

Question:

I am trying to create a basic Spring Boot application that interacts with AWS S3. I am only trying to get the list of buckets:

I start out with a basic Application:

@SpringBootApplication
public class Application {

    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }
}

My problem occurs in the Controller:

@RestController
public class S3RestController {

    @RequestMapping(path = "/", method = RequestMethod.GET)
    public ResponseEntity<String> index() {

        AWSCredentials credentials;
        try {
            credentials = new ProfileCredentialsProvider().getCredentials();
        } catch (Exception e) {
            throw new AmazonClientException("Cannot load credentials.");
        }

        AmazonS3 s3 = new AmazonS3Client(credentials);
        s3.setRegion(Region.getRegion(Regions.US_EAST_1));

        s3.listBuckets();

        return ResponseEntity.ok("Hello!");
    }
}

Here is my build.gradle:

buildscript {
    ext {
        springBootVersion = '1.5.3.RELEASE'
    }
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath("org.springframework.boot:spring-boot-gradle-plugin:${springBootVersion}")
        classpath "io.spring.gradle:dependency-management-plugin:1.0.0.RC2"
    }
}

apply plugin: 'java'
apply plugin: 'eclipse'
apply plugin: 'org.springframework.boot'
apply plugin: "io.spring.dependency-management"

version = '0.0.1-SNAPSHOT'
sourceCompatibility = 1.8

repositories {
    mavenCentral()
}

dependencyManagement {
    imports {
        mavenBom 'com.amazonaws:aws-java-sdk-bom:1.10.77'
    }
}

dependencies {
    compile('com.amazonaws:aws-java-sdk-s3')
    compile("org.springframework.boot:spring-boot-starter-web")
    runtime('org.springframework.boot:spring-boot-devtools')
    testCompile('org.springframework.boot:spring-boot-starter-test')
}

The stack trace after requesting GET localhost:8080/:

java.lang.IllegalStateException: Socket not created by this factory
    at org.apache.http.util.Asserts.check(Asserts.java:34) ~[httpcore-4.4.6.jar:4.4.6]
    at org.apache.http.conn.ssl.SSLSocketFactory.isSecure(SSLSocketFactory.java:435) ~[httpclient-4.5.3.jar:4.5.3]
    at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:186) ~[httpclient-4.5.3.jar:4.5.3]
    at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:326) ~[httpclient-4.5.3.jar:4.5.3]
    at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610) ~[httpclient-4.5.3.jar:4.5.3]
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445) ~[httpclient-4.5.3.jar:4.5.3]
    at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835) ~[httpclient-4.5.3.jar:4.5.3]
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.3.jar:4.5.3]
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.3.jar:4.5.3]
    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:837) ~[aws-java-sdk-core-1.10.77.jar:na]
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:607) ~[aws-java-sdk-core-1.10.77.jar:na]
    at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:376) ~[aws-java-sdk-core-1.10.77.jar:na]
    at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:338) ~[aws-java-sdk-core-1.10.77.jar:na]
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:287) ~[aws-java-sdk-core-1.10.77.jar:na]
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3826) ~[aws-java-sdk-s3-1.10.77.jar:na]
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3778) ~[aws-java-sdk-s3-1.10.77.jar:na]
    at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:701) ~[aws-java-sdk-s3-1.10.77.jar:na]
    at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:707) ~[aws-java-sdk-s3-1.10.77.jar:na]
    at com.ordonezalex.controller.S3RestController.index(S3RestController.java:32) ~[main/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) ~[spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) ~[spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) ~[spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) ~[spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) ~[spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) ~[spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) ~[spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) ~[spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) ~[spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) ~[spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[tomcat-embed-websocket-8.5.14.jar:8.5.14]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105) ~[spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) ~[spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) ~[spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478) [tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80) [tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799) [tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861) [tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455) [tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.14.jar:8.5.14]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.14.jar:8.5.14]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]

I suspect the problem is how I am using Spring, because the same controller works in a simple Java application. Thus, I have verified my credentials are not the issue.


Answer:

This worked for me, I was using version 1.10.12 of the SDK Before I changed my dependency to version 1.11.136 of the aws-java-sdk and that resolved my issue

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk</artifactId>
  <!--  <version>1.10.12</version> -->
    <version>1.11.136</version>
</dependency>

Question:

I have a @Controller that returns an image to the front end, but currently it holds the whole thing in memory as a byte array before returning a ResponseEntity<byte[]> object.

I would like to change this so that it just streams the image out without having to hold all the requested images in memory first.

These are not static images on the file system but rather images stored in Amazon S3, we are looking into utilizing CloudFront in the future, but for now we have to act as a middle man to retrieve the files.

I found an example that I coded below however again the image is read and held as a byte array in memory as the write function needs a byte array.

Is there any way to do this without using a lot of memory?

This is going to be used in a system where the client will open a page filled with images, each having multiple levels of resolution, the client is requesting the images very rapidly so the backend has to provide the images without using all of the memory resources.

Client:

<img src="{root}/image/123/low/3" alt="Smiley face" height="42" width="42">

Backend what I have now:

@RequestMapping(value = "/image/{id}/{quality}/{pageNumber}", method = RequestMethod.GET, produces = MediaType.IMAGE_PNG_VALUE)
public void retrieveImage(@PathVariable Long id, @PathVariable String quality, @PathVariable Integer pageNumber,
        HttpServletResponse response) throws Exception{
byte[] imageBytes = getImageFromS3(id, quality, pageNumber);
return new ResponseEntity<byte[]>(imageBytes, createCachingStrategy(), HttpStatus.OK);

}

Something I found online:

@RequestMapping(value = "/image/{id}/{quality}/{pageNumber}", method = RequestMethod.GET, produces = MediaType.IMAGE_PNG_VALUE)
public void retrieveImage(@PathVariable Long id, @PathVariable String quality, @PathVariable Integer pageNumber,
        HttpServletResponse response) throws Exception{

    //Open connection
    S3Object s3Object = imageService.getS3Object(quality, pageNumber, id)
    S3ObjectInputStream s3ObjectIS = s3Object.getObjectContent();

    response.getOutputStream().write(IOUtils.toByteArray(s3ObjectIS));

    //Then close connection to amazon......
}

Answer:

You are right on one thing. You should never completely load a file to memory. Files have sizes which may exceed the total RAM capacity (5 Gb, 10 Gb) thus it is not a reliable way to handle files. Using IOUtils.toByteArray() reads an InputStream and loads it to a single byte array which is not a good thing.

What you should do is read chunk by chunk and write whenever you finish reading a chunk. More like streaming. See code change below. Here for any given time for a image there will be only 8 * 2048 bytes or 2 Kb of RAM will be used. You can increase it if you want stream faster.

@RequestMapping(value = "/image/{id}/{quality}/{pageNumber}", method = RequestMethod.GET, produces = MediaType.IMAGE_PNG_VALUE)
public void retrieveImage(@PathVariable Long id, @PathVariable String quality, @PathVariable Integer pageNumber,
        HttpServletResponse response) throws Exception{

    //Open connection
    S3Object s3Object = imageService.getS3Object(quality, pageNumber, id)
    S3ObjectInputStream s3ObjectIS = s3Object.getObjectContent();

    byte[] data = new byte[2048];
    int read = 0;
    OutputStream out = response.getOutputStream();
    while((read = s3ObjectIS.read(data)) > 0) {
        out.write(data, 0, read);
        out.flush();
    }
    out.close();

    //Then close connection to amazon......
}

Question:

Can any one please help me.The code is working correctly for image upload but I want to give byte value in file path.Because image data type is CommonsMultipartFile.

@RequestMapping(value = "/saveImage", method = RequestMethod.POST)
    public @ResponseBody ImageResultVO getImage(ImageCriteriaVO imageCriteriaVO) throws O2Exception, IOException
    {               
        CommonsMultipartFile file ;
        file =(CommonsMultipartFile) imageCriteriaVO.getImage();
        AWSCredentials credentials = new BasicAWSCredentials(System.getenv("AWS_ACCESS_KEY_ID") ,System.getenv("AWS_SECRET_ACCESS_KEY"));
          String SUFFIX = "/";
          String folderName = "images";
          String existingBucketName = "o2container";
          String keyName = folderName + SUFFIX + file.getOriginalFilename();
          System.out.println("image" +keyName);       
          String filePath = "D:/bhanu/images%20(4).JPG";
          System.out.println("filePath" +filePath);
          String amazonFileUploadLocationOriginal=existingBucketName;
          System.out.println("hello");
          AmazonS3 s3Client = new AmazonS3Client(credentials);
          System.out.println("hello1");
          FileInputStream stream = new FileInputStream(filePath);
          System.out.println("hello2");
          ObjectMetadata objectMetadata = new ObjectMetadata();
          System.out.println("hello3");
          PutObjectRequest putObjectRequest = new PutObjectRequest(amazonFileUploadLocationOriginal, keyName, stream, objectMetadata);
          System.out.println("hello4");
          PutObjectResult result = s3Client.putObject(putObjectRequest);

        System.out.println("Etag:" + result.getETag() + "-->" + result);


        return null;
}

Answer:

There is a version of putObject that accepts an InputStream instead of a file path:

putObject(String bucketName, String key, InputStream input, ObjectMetadata metadata)

Uploads the specified input stream and object metadata to Amazon S3 under the specified bucket and key name.

See: putObject documentation