Hot questions for Using Amazon S3 in tomcat

Question:

I'm very new to AWS (today is the second day of my free trial). I've created elastic beanstalk application (using Tomcat server) and deployed very simple java web-app (1 static page & 1 jsp generated by servlet). I'm not using any AWS api or so on, just upload war file to EB project. When I see my statistic there is S3 - Puts and S3 - Gets usage.

Between this screenshots I've just once opened my web-app, but S3 usage still growing(:

How to know where my application use S3, and how to prevent this?

Thank you in advance!


Answer:

Elastic Beanstalk stores application deployment artifacts on S3. Elastic Beanstalk also archives application logs to S3. It should be extremely easy for you to open the S3 console and look at the S3 buckets and objects in your account to see what is going on.

Question:

In my local I am successfully able to point my docBase like this - <Context docBase="D:/XYZ" path="/video" />.

Since my Java based application is going to be hosted on AWS, I am using Amazon ec2 to deploy my application and Amazon S3 to host the videos, thumbnails and profile photos of the users. So I changed my tomcat 7 server.xml entry on ec2 to <Context docBase="http://MyS3EndPointURL" path="/videos" />

However when I am trying to restart my tomcat, it errors out saying -

java.lang.IllegalArgumentException: Document base /usr/share/tomcat7/webapps/MyS3EndPointURL/Videos does not exist or is not a readable directory

I tried removing http:// but as I expected it didn't work as well.

I understand that I will have to give permission to Tomcat to have access to these folders, however I have these questions -

  1. How do I provide this access ?
  2. When a user uploads a video will my application have the write access to these folders ? If no, how to provide that ?

From whatever reference articles are available on AWS, none talks about this issue. Stackoverflow question searches are mostly outdated information or people are suggesting to have another static domain. I am supposed to release the product this weekend in BETA mode but I am stuck because of this issue. Any pointers, suggestions would just save me !


Answer:

If I understand your setup, your app is hosted on Tomcat, you want your users to upload content to S3, so you can refer to your assets via an S3 http endpoint.

If so, your app server configuration is incorrect. Your server will not serve any videos itself; that job is offloaded to S3. Rather than define a docBase as you have, you will want to provide a URL rewrite rule so that http://www.example.com/videos/ point s to the new S3 http endpoint EG http://assets-example-com.s3-website-us-east-1.amazonaws.com/videos/.

When a user uploads a video will my application have the write access to these folders ? If no, how to provide that.

If you have not already, you need to develop an upload interface with the appropriate permissions the the above bucket. See Java Upload File for more information

Question:

I'm using "aws-java-sdk-1.11.427.jar" in my code and tomcat is refusing to start and shows below error: But there is no compile time error..

java.util.concurrent.ExecutionException: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api]]
    at java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.util.concurrent.FutureTask.get(FutureTask.java:192)
    at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:939)
    at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:872)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1419)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1409)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api]]
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
    ... 6 more
Caused by: java.lang.NoClassDefFoundError: com/amazonaws/auth/AWSCredentials
    at java.lang.Class.getDeclaredFields0(Native Method)
    at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
    at java.lang.Class.getDeclaredFields(Class.java:1916)
    at org.apache.catalina.util.Introspection.getDeclaredFields(Introspection.java:110)
    at org.apache.catalina.startup.WebAnnotationSet.loadFieldsAnnotation(WebAnnotationSet.java:262)
    at org.apache.catalina.startup.WebAnnotationSet.loadApplicationServletAnnotations(WebAnnotationSet.java:136)
    at org.apache.catalina.startup.WebAnnotationSet.loadApplicationAnnotations(WebAnnotationSet.java:66)
    at org.apache.catalina.startup.ContextConfig.applicationAnnotationsConfig(ContextConfig.java:328)
    at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:778)
    at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:299)
    at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5105)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    ... 6 more
Caused by: java.lang.ClassNotFoundException: com.amazonaws.auth.AWSCredentials
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1291)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119)
    ... 19 more

Answer:

Just put aws-java-sdk-1.11.427.jar in tomcat lib folder

Question:

I have written a code to list files in amazon s3 bucket , but it gives me this exception :

com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to 11.36.134.067:8080 [/11.36.134.067] failed: Connection refused: connect
    com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1114)
    com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1064)
    com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
    com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
    com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
    com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
    com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
    com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
    com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4330)
    com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4277)
    com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4271)
    com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:835)
    com.focus.re.calibr.service.FileUploadService.list(FileUploadService.java:114)
    com.focus.re.calibr.controller.FileUploadController.list(FileUploadController.java:33)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:498)

Code to list files with main method:

public class App {

    String bucketName = "buketname";
    String accesskey = "accesskey";

    public static void main(String args[]) throws Exception {
        new App();

    }


    App() throws Exception {

        BasicAWSCredentials basicAWSCredentials = new BasicAWSCredentials(accesskey,
                "secretekey");

        AWSCredentialsProvider awsCredentialsProvider = new AWSStaticCredentialsProvider(basicAWSCredentials);

        AwsClientBuilder.EndpointConfiguration endpointConfiguration =
                new AwsClientBuilder.EndpointConfiguration("11.36.134.067:8080", "");

        ClientConfiguration clientCfg = new ClientConfiguration();
        SSLContextBuilder builder = new SSLContextBuilder();

        builder.loadTrustMaterial(null, new

                TrustStrategy() {
                    public boolean isTrusted(X509Certificate[] chain, String authType) throws CertificateException {
                        return true;
                    }
                });

        SSLConnectionSocketFactory sslConnectionSocketFactory =
                new SSLConnectionSocketFactory(builder.build(), new NoopHostnameVerifier());
        clientCfg.getApacheHttpClientConfig().

                withSslSocketFactory(sslConnectionSocketFactory);

        AmazonS3 amazonS3 = AmazonS3ClientBuilder.standard()
                .withCredentials(awsCredentialsProvider)
                .withClientConfiguration(clientCfg)
                .withEndpointConfiguration(endpointConfiguration)
                .build();

        listAllFiles(amazonS3);


    } 

public void listAllFiles(AmazonS3 amazonS3) {

        ListObjectsRequest listObjectsRequest = new ListObjectsRequest().withBucketName(bucketName);

        ObjectListing objectListing = amazonS3. listObjects (listObjectsRequest);

        objectListing.getObjectSummaries().forEach(
                s3ObjectSummary -> {
                    System.out.println(s3ObjectSummary.getKey() + "  " + s3ObjectSummary.getSize());

                });
    }

}

This code will work fine if i run the code by executing the main method , but if i run the code from tomcat server by calling a Spring restful webservice then the exception will occur.

Currently i am using latest Tomcat server 8.5.28 , is there any way to fix this issue ?


Answer:

EndpointConfiguration is for Amazon's endpoint, not yours. From the documentation:

Each AWS client can be configured to use a specific endpoint within a region by calling the setEndpoint method.

For example, to configure the Amazon EC2 client to use the EU (Ireland) Region, use the following code.

AmazonEC2 ec2 = new AmazonEC2(myCredentials);
ec2.setEndpoint("https://ec2.eu-west-1.amazonaws.com");

Question:

I am attempting to set a property that contains $ in it as a Environmental Property (Configure -> Software -> Environment properties) that will eventually get passed to my tomcat context.xml file (all running on AMI - Tomcat 8 with Java 8 running on 64bit Amazon Linux/3.0.1). It seems like it won't accept $ characters (source: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/environments-cfg-softwaresettings.html). It appears that they are being interpreted as bash variables. Can I escape the $ some how, I've tried \$ and (double backslash)\$ with no luck. Am I missing something or am I just out of luck.

Thanks


Answer:

So with a bit or trial and error it looks like you need 3 backslashes to escape the $ character. \\\$myData