Hot questions for Using Amazon S3 in lambda

Question:

I have gone through the question titled "Setting the AWS region programmatically 1" but it doesn't provide all the answers I need.

Q1: I'm getting a SDKClientException-Unable to find a region via the region provider chain. What am I doing wrong? or is there a typo that I missed.

public class CreateS3Bucket {

public static void main(String[] args) throws IOException {

    BasicAWSCredentials creds = new BasicAWSCredentials("aws-access-key", "aws-secret-key");
    AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(creds)).build();

    Region region = Region.getRegion(Regions.US_EAST_1);
    s3Client.setRegion(region);

    try {
        String bucketName = "testBucket" + UUID.randomUUID();
        s3Client.createBucket(bucketName);
        System.out.println("Bucket Created Successfully.");

    } catch(AmazonServiceException awse) {

        System.out.println("This means that your request made it AWS S3 but got rejected");
        System.out.println("Error Message:" +awse.getMessage());
        System.out.println("Error Message:" +awse.getErrorCode());
        System.out.println("Error Message:" +awse.getErrorType());
        System.out.println("Error Message:" +awse.getRequestId());

    } catch (AmazonClientException ace) {

        System.out.println("The Amazon Client encountered an Error with network Connectivity");
        System.out.println("Error Message:" + ace.getMessage());
    }


}

}

Q2: What code changes needs to be done if I want to build a Lambda Function out of it? I'm aware how to create a lambda function and roles that it needs. Just need to know if the code that I have written needs to changed. How should I implement the LambdaFuctionHandler class as below:

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;

 public class LambdaFunctionHandler implements RequestHandler<String, String> {

@Override
public String handleRequest(String input, Context context) {
    context.getLogger().log("Input: " + input);


    return null;
}

}

Answer:

Regarding Q1, try to build your client using the following syntax:

AmazonS3 amazonS3 = AmazonS3Client.builder()
    .withRegion("us-east-1")
    .withCredentials(new AWSStaticCredentialsProvider(creds))
    .build();

Question:

I have a lamba function to copy objects from bucket 'A' to bucket 'B', and everything was working fine, until and object with name 'New Text Document.txt' was created in bucket 'A', the json that gets built in S3 event, key as "key": "New+Text+Document.txt".

the spaces got replaced with '+'. I know it is a known issue by seraching on web. But I am not sure how to fix this and the incoming json itself has a '+' and '+' can be actually in the name of the file. like 'New+Text Document.txt'.

So I cannot blindly have logic to space '+' by ' ' in my lambda function.

Due to this issue, when code tries to find the file in bucket it fails to find it.

Please suggest.


Answer:

I came across this looking for a solution for a lambda written in python instead of java; "urllib.parse.unquote_plus" worked for me, it properly handled a file with both spaces and + signs:

from urllib.parse import unquote_plus
import boto3


bucket = 'testBucket1234'
# uploaded file with name 'foo + bar.txt' for test, s3 Put event passes following encoded object_key
object_key = 'foo %2B bar.txt'
print(object_key)
object_key = unquote_plus(object_key)
print(object_key)

client = boto3.client('s3')
client.get_object(Bucket=bucket, Key=object_key)

Question:

I have written a Lambda function. This function is uploaded in s3Bucket = "my-lambda", which is mapped to the role hello-lambda-role and the regionName = "us-west-2".

Now I wanted to access s3Bucket="some-other" where we have mapped Policy with 'hello-lambda-role' and it is in the region "eu-west-1".

Here is the API class I am using AmazonS3Client. My intention is to get some file from the bucket "some-other". But before that I need to make the connection.

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;

public class LambdaFunctionHandler implements RequestHandler<Request, Response> {

    public Response handleRequest(Request request, Context context) {
        String greetingString = String.format("Hello %s %s.",
                request.firstName, request.lastName);
        return new Response(greetingString);
    }

}

Here is the class which list the bucket.

public class Test{
    private static final Log logger = LogFactory.getLog(InvokeLambda.class);
    private static final String awsAccessKeyId = "XXXXX";
    private static final String awsSecretAccessKey = "XXXXX";
    private static final String regionName = "eu-west-1";
    private static Region region;
    private static AWSCredentials credentials;
    private static AWSLambdaClient lambdaClient;
    private static AmazonS3Client s3Client;

    private static void init() {
        credentials = new BasicAWSCredentials(awsAccessKeyId,
                awsSecretAccessKey);
        s3Client = (credentials == null) ? new AmazonS3Client()
                : new AmazonS3Client(credentials);
        region = Region.getRegion(Regions.fromName(regionName));
        s3Client.setRegion(region);
        lambdaClient = (credentials == null) ? new AWSLambdaClient()
                : new AWSLambdaClient(credentials);

        lambdaClient.setRegion(region);
        // lambdaClient.configureRegion(Regions.US_WEST_2);
    }

    /**
     * The entry point into the AWS lambda function.
     */
    public static void main(String... args) {
        init();
        getExistingBucket();
    }

    private static Bucket getExistingBucket() {
        List<Bucket> buckets = s3Client.listBuckets();
        for (Bucket bucket : buckets) {
            logger.error(bucket.getName());
        }
        return null;
    }
}

Answer:

Use same code as you would in your test, except you don't need to provide credentials when you create the AmazonS3Client. Note that the role your lambda uses needs the authority to access your S3 bucket. The region of the S3 bucket shouldn't matter; the bucket name uniquely identifies the bucket regardless of region.

Lambdas are typically triggered by an event, but you can call AWSLambdaClient.invoke() to run it manually.

For example:

public Response handleRequest(Request request, Context context) {
    AmazonS3Client s3Client = new AmazonS3Client();
    S3Object = s3Client.getObject("some-other", request.getFilename());
    ....
    return new Response(result);
}

Deploy that to AWS as "mylambda", and then invoke remotely with:

lambdaClient.invoke(new InvokeRequest().withFunctionName("mylambda").withPayload("input"));

Question:

I am trying to use Lambda function for S3 Put event notification. My Lambda function should be called once I put/add any new JSON file in my S3 bucket. The challenge I have is there are not enough documents for this to implement such Lambda function in Java. Most of doc I found are for Node.js

I want, my Lambda function should be called and then inside that Lambda function, I want to consume that added json and then send that JSON to AWS ES Service.

But what all classes I should use for this? Anyone has any idea about this? S3 abd ES are all setup and running. The auto generated code for lambda is `

@Override
public Object handleRequest(S3Event input, Context context) {
    context.getLogger().log("Input: " + input);

    // TODO: implement your handler
    return null;
}

What next??


Answer:

Handling S3 events in Lambda can be done, but you have to keep in mind, the the S3Event object only transports the reference to the object and not the object itself. To get to the actual object you have to invoke the AWS SDK yourself. Requesting a S3 Object within a lambda function would look like this:

public Object handleRequest(S3Event input, Context context) {
    AmazonS3Client s3Client = new AmazonS3Client(new DefaultAWSCredentialsProviderChain());        

    for (S3EventNotificationRecord record : input.getRecords()) {
        String s3Key = record.getS3().getObject().getKey();
        String s3Bucket = record.getS3().getBucket().getName();
        context.getLogger().log("found id: " + s3Bucket+" "+s3Key);
        // retrieve s3 object
        S3Object object = s3Client.getObject(new GetObjectRequest(s3Bucket, s3Key));
        InputStream objectData = object.getObjectContent();
        //insert object into elasticsearch
    }        
    return null;
}

Now the rather difficult part to insert this object into ElasticSearch. Sadly the AWS SDK does not provide any functions for this. The default approach would be to do a REST call against the AWS ES endpoint. There are various samples out their on how to proceed with calling an ElasticSearch instance.

Some people seem to go with the following project:

Jest - Elasticsearch Java Rest Client

Question:

I failed to get the following logic to work on AWS Lambda using Java:

1) When there is a new object created in S3 bucket, trigger a lambda function (written in java)

2) In this lambda function, list all the DynamoDB tables.

3) Create a table if there is none.

4) Write the S3 object's details as item into DynamoDB.

I only get item #1 working. when it reach item #2, I encounter a permission related error below.

Any help or suggestion?

The permission that i use is "Basic with DynamoDB", which has the following permission:

START RequestId: e9ab5aba-307b-11e5-9663-3188c327cf5e fileSize: 1024, DateTime:1970-01-01T00:00:00.000Zs3Key : HappyFace.jpgAWS credential profiles file not found in the given path: /home/sbx_user1052/.aws/credentials: java.lang.IllegalArgumentException java.lang.IllegalArgumentException: AWS credential profiles file not found in the given path: /home/sbx_user1052/.aws/credentials at com.amazonaws.auth.profile.internal.ProfilesConfigFileLoader.loadProfiles(ProfilesConfigFileLoader.java:45) at com.amazonaws.auth.profile.ProfilesConfigFile.loadProfiles(ProfilesConfigFile.java:176) at com.amazonaws.auth.profile.ProfilesConfigFile.(ProfilesConfigFile.java:112) at com.amazonaws.auth.profile.ProfilesConfigFile.(ProfilesConfigFile.java:92) at com.amazonaws.auth.profile.ProfileCredentialsProvider.getCredentials(ProfileCredentialsProvider.java:123) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1763) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.listTables(AmazonDynamoDBClient.java:1208) at com.amazonaws.services.dynamodbv2.document.internal.ListTablesCollection.firstPage(ListTablesCollection.java:46) at com.amazonaws.services.dynamodbv2.document.internal.PageIterator.next(PageIterator.java:45) at com.amazonaws.services.dynamodbv2.document.internal.IteratorSupport.nextResource(IteratorSupport.java:79) at com.amazonaws.services.dynamodbv2.document.internal.IteratorSupport.hasNext(IteratorSupport.java:47) at com.TriggerDynamoDB.handleRequest(TriggerDynamoDB.java:68) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497)

END RequestId: e9ab5aba-307b-11e5-9663-3188c327cf5e REPORT RequestId: e9ab5aba-307b-11e5-9663-3188c327cf5e Duration: 3294.97 ms Billed Duration: 3300 ms Memory Size: 512 MB Max Memory Used: 51 MB

Code is as follows:

public class TriggerDynamoDB implements RequestHandler<S3Event, String> {

public String handleRequest(S3Event s3event, Context context) {
     LambdaLogger logger = context.getLogger();
     try {           
         S3EventNotificationRecord record = s3event.getRecords().get(0);
         // Object key may have spaces or unicode non-ASCII characters.
         String srcKey = record.getS3().getObject().getKey().replace('+', ' ');
            srcKey = URLDecoder.decode(srcKey, "UTF-8");

         long fileSize = record.getS3().getObject().getSizeAsLong();
         DateTime datetime = record.getEventTime(); 

         logger.log("fileSize: " + fileSize + ", DateTime:" + datetime);
         logger.log("s3Key   : " + srcKey);

         DynamoDB dynamoDB = new DynamoDB(new AmazonDynamoDBClient(new ProfileCredentialsProvider()));
         //Table table = dynamoDB.getTable("dimensionFile");

         TableCollection<ListTablesResult> tables = dynamoDB.listTables();
         Iterator<Table> iterator = tables.iterator();

         while (iterator.hasNext()) { // this is where the error was thrown
             Table table = iterator.next();
             System.out.println(table.getTableName());
         }                    
         return "Ok";
     } catch (Exception e) {
         throw new RuntimeException(e);
     }
}
}

Answer:

The ProfileCredentialsProvider you're passing to the AmazonDynamoDBClient constructor attempts to load credentials from the ~/.aws/credentials file. This file doesn't exist on the host where your Lambda function is being run, hence the exception. This provider is mainly intended for things like integration tests that run on your dev machine and use your personal IAM user credentials.

On Lambda you want the EnvironmentVariableCredentialsProvider. This provider will load temporary credentials for the IAM Role associated with your Lambda function from environment variables that Lambda sets before invoking your code.

If you use the no-argument constructor for the AmazonDynamoDBClient, it will default to using the DefaultAWSCredentialsProviderChain to pull credentials. This provider checks environment variables, Java system properties, ~/.aws/credentials, and the EC2 Instance Metadata Service (in that order), and should pick up credentials whether you're running locally or on Lambda.

Question:

I have a shape file and i need to read the shape file from my java code. I used below code for reading shape file.

public class App {
    public static void main(String[] args) {
        File file = new File("C:\\Test\\sample.shp");
        Map<String, Object> map = new HashMap<>();//
        try {
            map.put("url", URLs.fileToUrl(file));
            DataStore dataStore = DataStoreFinder.getDataStore(map);
            String typeName = dataStore.getTypeNames()[0];
            SimpleFeatureSource source = dataStore.getFeatureSource(typeName);
            SimpleFeatureCollection collection = source.getFeatures();

            try (FeatureIterator<SimpleFeature> features = collection.features()) {
                while (features.hasNext()) {
                    SimpleFeature feature = features.next();
                    SimpleFeatureType schema = feature.getFeatureType();
                    Class<?> geomType = schema.getGeometryDescriptor().getType().getBinding();

                    String type = "";
                    if (Polygon.class.isAssignableFrom(geomType) || MultiPolygon.class.isAssignableFrom(geomType)) {

                        MultiPolygon geom = (MultiPolygon) feature.getDefaultGeometry();
                        type = "Polygon";
                        if (geom.getNumGeometries() > 1) {
                            type = "MultiPolygon";
                        }
                    } else if (LineString.class.isAssignableFrom(geomType)
                            || MultiLineString.class.isAssignableFrom(geomType)) {
                    } else {

                    }
                    System.out.println(feature.getDefaultGeometryProperty().getValue().toString());

                }
            }
        } catch (Exception e) {
            // TODO: handle exception
        }

    }
}

I got the desired output. But my requirement is write an aws lambda function to read shape file. For this 1. I created a Lambda java project of s3 event. I wrote the same code inside the handleRequest. I uploaded the java lambda project as a lanbda function and added one trigger. When I am uploading a .shp file to as s3 bucket lmbda function will automatically invoked. But I am getting an error like below

java.lang.RuntimeException: java.io.FileNotFoundException: /sample.shp (No such file or directory)

I have sample.shp file inside my s3 bucket. I go through below link. How to write an S3 object to a file?

I am getting the same error. I tried to change my code like below

  S3Object object = s3.getObject(new GetObjectRequest(bucket, key)); 
  InputStream objectData = object.getObjectContent();
  map.put("url", objectData );

instead of

File file = new File("C:\\Test\\sample.shp"); 
 map.put("url", URLs.fileToUrl(file));

:-( Now i am getting an error like below

java.lang.NullPointerException

Also I tried the below code

DataStore dataStore = DataStoreFinder.getDataStore(objectData);

instead of

DataStore dataStore = DataStoreFinder.getDataStore(map);

the error was like below

java.lang.ClassCastException: com.amazonaws.services.s3.model.S3ObjectInputStream cannot be cast to java.util.Map

Also I tried to add key directly to the map and also as DataStore object. Everything went wrong..:-(

Is there anyone who can help me? It will be very helpful if someone can do it for me...


Answer:

The DataStoreFinder.getDataStore method in geotools requires you to provide a map containing a key/value pair with key "url". The value associated with that "url" key needs to be a file URL like "file://host/path/my.shp".

You're trying to insert a Java input stream into the map. That won't work, because it's not a file URL.

The geotools library does not accept http/https URLs (see the geotools code here and here), so you need a file:// URL. That means you will need to download the file from S3 to the local Lambda filesystem and then provide a file:// URL pointing to that local file. To do that, here's Java code that should work:

// get the shape file from S3 to local filesystem
File localshp = new File("/tmp/download.shp");
s3.getObject(new GetObjectRequest(bucket, key), localshp);

// now store file:// URL in the map
map.put("url", localshp.getURI().getURL().toString());

If the geotools library had accepted real URLs (not just file:// URLs) then you could have avoided the download and simply created a time-limited, pre-signed URL for the S3 object and put that URL into the map.

Here's an example of how to do that:

// get current time and add one hour
java.util.Date expiration = new java.util.Date();
long msec = expiration.getTime();
msec += 1000 * 60 * 60;
expiration.setTime(msec);

// request pre-signed URL that will allow bearer to GET the object
GeneratePresignedUrlRequest gpur = new GeneratePresignedUrlRequest(bucket, key);
gpur.setMethod(HttpMethod.GET);
gpur.setExpiration(expiration);

// get URL that will expire in one hour
URL url = s3.generatePresignedUrl(gpur);

Question:

I am creating a java function for AWS Lambda that brings in a file from AWS S3 like this:

InputStream videoObjectStream = awsS3Video.getObjectContent();

I am also utilizing FFmpegFrameGrabber, which requires that I specify a file path whenever I create a new frameGrabber, i.e.: FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(filePath)

I am trying to convert the InputStream into a temporary file in my Lambda function, but it is not allowing me to create a file. Here is my code to convert the videoObjectStream into a file:

byte[] inputBuffer = null;

        try {
            inputBuffer = IOUtils.toByteArray(videoObjectStream);
        } catch (IOException e1) {
            // TODO Auto-generated catch block
            e1.printStackTrace();
        }

        System.out.println("The length of the byte array is " + inputBuffer.length);


        try {
            FileOutputStream videoOS = new FileOutputStream(videoDetails.get("videoFileKey"), false);
            videoOS.write(inputBuffer);
            videoOS.close();
        } catch (FileNotFoundException e1) {
            // TODO Auto-generated catch block
            e1.printStackTrace();
        } catch (IOException ex) {
            ex.printStackTrace();
        }

        File tempVideoFile = new File(videoDetails.get("videoFileKey"));

        if (tempVideoFile.exists()) {
            System.out.println("The file exists");
        } else {
            System.out.println("The file does not exist");
        }

Then, I get the following stack trace, saying that this is a read-only file system:

java.io.FileNotFoundException: currentPath1490660005410.mp4 (Read-only file system)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
    at com.amazonaws.lambda.LambdaFunctionHandler.convertVideo(LambdaFunctionHandler.java:67)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at lambdainternal.EventHandlerLoader$PojoMethodRequestHandler.handleRequest(EventHandlerLoader.java:456)
    at lambdainternal.EventHandlerLoader$PojoHandlerAsStreamHandler.handleRequest(EventHandlerLoader.java:375)
    at lambdainternal.EventHandlerLoader$2.call(EventHandlerLoader.java:1139)
    at lambdainternal.AWSLambda$2.call(AWSLambda.java:94)
    at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:290)
    at lambdainternal.AWSLambda.<clinit>(AWSLambda.java:57)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:94)

Is there any way around this? I need to manipulate the video data but cannot without making it into a file first. Any suggestions are welcome. Thank you.


Answer:

You have to create the file in /tmp. That's the only location you are allowed to write to in the Lambda environment.

Question:

when code is executed in amazon aws lambda, my @autowired spring dependencies are null. Makes sense if there is no context being loaded, but i thought SpringBeanAutowiringSupport would help. How do you inject dependencies correctly in amazon lambda?

This is my code that has null autowired fields but otherwise works fine (if i replace autowired with new:

@Component
public class ApplicationEventHandler {

@Autowired
private Foo foo;


         public ApplicationEventHandler() {
             logger.info("I'm sure the constructor is being called");
            SpringBeanAutowiringSupport.processInjectionBasedOnCurrentContext(this);
             //doesn't seem to help
         }

         public void deliveryFailedPermanentlyHandler(SNSEvent event, Context context) throws IOException {
             foo.doStuff() // causes NPE

         }

thanks in advance!


Answer:

this project on github provides a template for what i'm trying to do which works fine:

https://github.com/cagataygurturk/aws-lambda-java-boilerplate

Question:

I am an android developer and exploring Amazon AWS for my future appliaction as a backend option. I had explored various AWS services such as dynamoDB, S3, SES and cool lambda feature with JAVA. During my learing I found/prepared a code for creating thumbnail images for every image uploaded to S3 bucket. I successfully written the Lambda code in java in eclipse IDE and able to upload and use it using AWS console for the purpose it was intended.

My Concern is that when I uploaded the JAVA Lambda code to AWS it appears 49 MB in SIZE. When I seek for the reason I found that there are plenty of jars were used in project under AWS sdk for java. Is this normal or I can reduce the size of uploaded code anyhow. Please guide me How can I reduce the size efficiently. I also seen the node.js Lambda code for same and its in KBs only. Might bbe I am doing some wrong things.

Please Help...


Answer:

Simple answer - you're doing it right and package size can't be reduced to a size comparable to a node.js application.

For node.js lambda, amazon has AWS SDK libraries in place so you have to upload only your own code and third party libraries. But for java lambda, the AWS SDK must be packaged with the application.

But you can reduce package size by carefully selecting which libraries to include and excluding unnecessary dependencies.

Question:

I am trying to develop a AWS lambda function which is triggered when a file shows up in a specific s3 bucket. I am trying to follow the examples from AWS Lambda documentation, using aws-java-sdk-lambda 1.11.192, aws-java-sdk-s3 1.11.192. But, unfortunately the these examples use RequestHandler which is deprecated in the latest version of the jar.

My code is similar to this example

package example;

import java.net.URLDecoder;

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.S3Event;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.event.S3EventNotification.S3EventNotificationRecord;

public class S3GetTextBody implements RequestHandler<S3Event, String> {

public String handleRequest(S3Event s3event, Context context) {
    try {
        S3EventNotificationRecord record = s3event.getRecords().get(0);

        // Retrieve the bucket & key for the uploaded S3 object that
        // caused this Lambda function to be triggered
        String bkt = record.getS3().getBucket().getName();
        String key = record.getS3().getObject().getKey().replace('+', ' ');
        key = URLDecoder.decode(key, "UTF-8");

        // Read the source file as text
        AmazonS3 s3Client = new AmazonS3Client();
        String body = s3Client.getObjectAsString(bkt, key);
        System.out.println("Body: " + body);
        return "ok";
    } catch (Exception e) {
        System.err.println("Exception: " + e);
        return "error";
    }

  }
}

The current version of the aws sdk for lambda doesn't contain -

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.S3Event;

What are my alternatives? How can I achieve similar functionality using the newer versions of their sdk.


Answer:

You aren't required to implement the RequestHandler interface provided in their helper library. Any method will work provided the input and output parameters can be serialized properly.

See this article for more detail.

If you want to use their helper library, use the following dependency coordinates:

<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-core</artifactId>
<version>1.1.0</version>

And for the S3 event helper:

<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-events</artifactId>
<version>1.3.0</version>

It's not located within their main aws-java-sdk but instead has its own repository.

Question:

I have a lambda function configured for receiving eventes whenever a file is dropped to my s3 bucket.

My requirement is to configure a dead letter queue for lambda function, so that any failure happened, event should go to dlq.

My question is that, what response should be given from lambda function, so that it will push event to dlq?

Eg scenario: I have an event validator module within lambda, if validation fails, i wanted to move it to dlq configured for my lambda function.


Answer:

Just throw an exception and make sure your function is idempotent:

Any Lambda function invoked asynchronously is retried twice before the event is discarded

See Dead Letter Queues in AWS Lambda Developer Guide.

Question:

I am trying to scan a file using AWS Lambda, and I am getting timeout since the scan function is taking longer than expected.

I would like to increase the timeout for my client since this process is @Async and I can handle few more seconds.

This is my method:

   @Override
   @Async
   public void scanFile( String s3Bucket, String fileName, String path, String documentId, String createdBy ) throws IOException {

      FileScanInput input = new FileScanInput();
      input.setS3Bucket( s3Bucket );
      input.setS3Key( path );

      logger.debug( "Scan file: " + path + ", in S3 bucket:  " + s3Bucket );
      if ( fileScanService == null ) {
         fileScanService = buildFileScanService();
      }

      FileScanOutput fileScanOutput = fileScanService.scanFile( input );
//      TODO: if the scan process failed, ask what should be the next step.
//      for now, the file stays in S3.
      Assert.notNull( fileScanOutput );
      String status = fileScanOutput.getStatus();

      // in case the document owner was not found an infected file was file. Email approved emails
      if ( status.equalsIgnoreCase( VIRUS_SCAN_INFECTED ) ) {
         // delete file on S3
         this.deleteFile( s3Bucket, path );
         String toEmail = personInfoMapper.findStudentEmail( createdBy );
         try {
            sendVirusDetectedEmail( fileName, toEmail );
         }
         catch ( Exception e ) {
            logger.error( e.getMessage() );
         }

         //         we clean up the metadata table in case there was a virus since this is running async.
         metadataDao.cleanUpMetadata( documentId );

         logger.error( "The file is infected." );
         throw new VirusFoundException( "File is infected." );
      }
   }


   public final FileScanService buildFileScanService() {
      return LambdaInvokerFactory.builder().lambdaClient( AWSLambdaClientBuilder.defaultClient() ).build( FileScanService.class );
   }

And this is the resource configs for my Lambda function.

Update I also noticed that my Lambda function actually does its job, which means the issue is basically in this line FileScanOutput fileScanOutput = fileScanService.scanFile( input );

fileScanOutput doesn't get initialized instead I get timeout issue.

My other classes look like:

public interface FileScanService {

   @LambdaFunction( functionName = "s3-antivirus-api-scan" )
   public FileScanOutput scanFile( FileScanInput fileScanInput );
}

public class FileScanInput {

   private String s3Bucket;
   private String s3Key;

   public String getS3Bucket() {
      return s3Bucket;
   }

   public void setS3Bucket( String value ) {
      s3Bucket = value;
   }

   public String getS3Key() {
      return s3Key;
   }

   public void setS3Key( String value ) {
      s3Key = value;
   }
}



public class FileScanOutput {

   private String status;

   public FileScanOutput() {
   }

   public FileScanOutput( String status ) {
      this.status = status;
   }

   public String getStatus() {
      return status;
   }

   public void setStatus( String value ) {
      status = value;
   }
}

Answer:

When you say your client is timing out, do you mean your Lambda SDK client? If so, you may need to pass a longer socket timeout when creating your client:

AWSLambdaClientBuilder.standard()
  .withClientConfiguration(new ClientConfiguration()
    .withSocketTimeout(SOCKET_TIMEOUT_IN_MS))
  .build();

The default socket timeout is 50 seconds: https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/ClientConfiguration.java#L43

Your Lambda function itself will continue running regardless of whether the socket is closed on the client side, which is likely why you see the function completing the job.

Question:

I'm writing AWS lambda that reads protobuf obejcts from Kinesis and would like to write them to s3 as parquet file.

I saw there's a implementation of ParquetWriter for protobuf called ProtoParquetWriter, which is good. My problem is that ProtoParquetWriter expects a Path in its constructor.

What's the right way to do that without saving the content as parquet file, assuming I'm not using the file system at all?


Answer:

Assuming you have a List (can be any complex object), sample code to read/write protobuf S3 parquet

    public class SimpleS3ParquetUtilities implements S3Utilities {

    final Logger logger;
    String PATH_SCHEMA = "s3a";
    CompressionCodecName compressionCodecName;

    public SimpleS3ParquetUtilities(Logger logger) {
        this.logger = logger;
        this.compressionCodecName = CompressionCodecName.UNCOMPRESSED;
    }

    public SimpleS3ParquetUtilities(Logger logger, CompressionCodecName compressionCodecName) {
        this.logger = logger;
        this.compressionCodecName = compressionCodecName;
    }

    @Override
    public String writeTransactions(String bucket, String objectKey, List<Transaction> transactions)
            throws Exception {
        if (objectKey.charAt(0) != '/')
            objectKey = "/" + objectKey;
        Path file = new Path(PATH_SCHEMA, bucket, objectKey);
        Stopwatch sw = Stopwatch.createStarted();
        // convert the list into protobuf 
        List<TransactionProtos.Transaction> protoTransactions = Convertor.toProtoBuf(transactions);
        try (ProtoParquetWriter<TransactionProtos.Transaction> writer = new ProtoParquetWriter<TransactionProtos.Transaction>(
                file, TransactionProtos.Transaction.class, this.compressionCodecName,
                ProtoParquetWriter.DEFAULT_BLOCK_SIZE, ProtoParquetWriter.DEFAULT_PAGE_SIZE)) {

            for (TransactionProtos.Transaction transaction : protoTransactions) {
                writer.write(transaction);
            }
        }
        logger.info("Parquet write elapse:[{}{}] Time:{}ms items:{}", bucket, objectKey,
                sw.elapsed(TimeUnit.MILLISECONDS), transactions.size());
        return "";
    }

    @Override
    public List<Transaction> readTransactions(String bucket, String pathWithFileName)
            throws Exception {
        if (pathWithFileName.charAt(0) != '/')
            pathWithFileName = "/" + pathWithFileName;
        Path file = new Path(PATH_SCHEMA, bucket, pathWithFileName);
        Stopwatch sw = Stopwatch.createStarted();
        try (ParquetReader<TransactionProtos.Transaction.Builder> reader = ProtoParquetReader.<TransactionProtos.Transaction.Builder>builder(
                file).build()) {
            List<TransactionProtos.Transaction> transactions = new ArrayList<TransactionProtos.Transaction>();
            TransactionProtos.Transaction.Builder builder = reader.read();
            while (builder != null) {
                TransactionProtos.Transaction transaction = builder.build();
                transactions.add(transaction);
                builder = reader.read();
            }
            logger.info("Parquet read elapsed:[{}{}] Time:{}ms items:{}", bucket, pathWithFileName,
                    sw.elapsed(TimeUnit.MILLISECONDS), transactions.size());
            return Convertor.fromProtoBuf(transactions);
        }
    }
}

Question:

I'm trying to list keys on an S3 bucket from an AWS Lambda function written in Java. Running the code locally works fine (with hardcoded credentials).

When running the same Java code in Lambda, it hangs at listObjects

AmazonS3 s3client = new AmazonS3Client(new BasicAWSCredentials("XXXXXXXXXXXx",
           "XXXXXXXXXXZZZZZZZZZZz"));


ListObjectsRequest listObjectsRequest = new ListObjectsRequest()
        .withBucketName(bucketName)
        .withMaxKeys(10);
ObjectListing objectListing;

do {
    objectListing = s3client.listObjects(listObjectsRequest);

The hardcoded user credentials and the Lambda execution role both have full access to s3.

Why does the S3 access hangs without error? What permission configuration is wrong?

Running something comparable in Lambda with NodeJS works


Answer:

The solution was to give the Lambda more memory to work with. For most runs Lambda report arround 111 MB for the execution of the simple S3 listObjects command. So 128 MB was not enough, with 512MB it works fine.

Question:


Answer:

Use the Java S3 SDK. If you upload a file called x.db to an S3 bucket mybucket, it would look something like this:

import com.amazonaws.services.s3.*;
import com.amazonaws.services.s3.model.*;

...
AmazonS3 client = new AmazonS3Client();
S3Object xFile = client.getObject("mybucket", "x.db");
InputStream contents = xFile.getObjectContent();

In addition, you should ensure the role you've assigned to your lambda function has access to the S3 bucket. Apply a policy like this:

"Version": "2012-10-17",
"Statement": [{
  "Effect": "Allow",
  "Action": [
    "s3:*"
  ],
  "Resource": [
    "arn:aws:s3:::mybucket",
    "arn:aws:s3:::mybucket/*"
  ]
}]

Question:

I want to automatically deploy a jar to aws-lamda whenever there is any change in the version of the jar in s3 bucket.

For example: There is one S3 bucket say lambdadeploytestand a Lamda functionname autoDeployTest. I have stored an object test.jar in lambdadeploytest.

Whenever there is I upload a new version of the object test.jar, test.jar will be automatically deployed to autoDeployTest lambda function.


Answer:

You can do that with a lambda function ... At high level the steps are

  • create a lambda function that will download the test.jar and use the jar to create / update the autoDeployTest lambda function.

  • configure your new lambda function to be triggered when test.jar is modified on S3

  • configure the new lambda function with permissions to read your S3 bucket and to deploy code on Lambda.

You can follow this tutorial to create a lambda function that would be triggered by S3 events. This is the Lambda API your lambda function can call to update autodeployTest code.

Pseudo Code would be like this :

read JSON event object to get the bucket name and object name (bucket_name, test.jar)

download a copy of the object (test.jar) to the lambda container 

(do additional check on the jar file if required) 

call Lambda's UpdateFunctionCode to update autoDeployTest function code with the jar.

Question:

CONTEXT :

I am trying to read a csv file on AWS s3, compute its data and write the results on another csv in the same s3 bucket.

I tried the following code to create a class that would read my Main, with arguments stored in a text file named lambdaCmd.txt. Each time lambdaCmd.txt is modified, the lambda function is triggered and the content of lambdaCmd.txt is passed to the following class via the s3event parameter :

public class LambdaCmd implements RequestHandler<S3Event, Void>{

    static final Logger LOGGER = LogManager.getLogger(LambdaCmd.class);

    @Override
    public Void handleRequest(S3Event s3event, Context context) {

        //Getting my txt file's path
        S3EventNotification.S3EventNotificationRecord record = s3event.getRecords().get(0);
        String bkt = record.getS3().getBucket().getName();
        String key = record.getS3().getObject().getKey().replace('+', ' ');
        try {
            key = URLDecoder.decode(key, "UTF-8");
        } catch (UnsupportedEncodingException ex) {
            LOGGER.error(ex);
        }

        AmazonS3 s3Client = AmazonS3ClientBuilder.standard().build();

        try {

            //Getting my text file as a String Stream
            InputStreamReader cmdStream =
                new InputStreamReader(s3Client.getObject(bkt, key).getObjectContent());
            BufferedReader br = new BufferedReader(cmdStream);

            //Parsing the command in cmdStream
            //...
            //The command is now "String[] cmdArray" to be passed to my main

            MyMain.main(cmdArray);
            //The main function reads and write from s3 with a similar use of s3Client

            br.close();

        } catch (IOException | IllegalArgumentException | NullPointerException ex) {
            LOGGER.error(ex);
        }

        return null;
    }
}

PROBLEM :

After logging some debug messages in this code, I figured out that the lambda function stops at this line :

        AmazonS3 s3Client = AmazonS3ClientBuilder.standard().build();

Here is the CloudWatch log (I hide personal information with ##########):

START RequestId: ########## Version: $LATEST
16:12:11.596 [main] DEBUG path.to.mylambdaclass.LambdaCmd - LambdaCmd started
16:12:11.616 [main] DEBUG path.to.mylambdaclass.LambdaCmd - Just before creating s3Client
END RequestId: ##########
REPORT RequestId: ##########    Duration: 12398.45 ms   Billed Duration: 12400 ms Memory Size: 128 MB   Max Memory Used: 67 MB

I also have a log message just after creating s3Client, but it is not printed in logs.

QUESTION :

Why is my lambda ending prematurely? Do I misuse some objects?


Answer:

After adding the following try/catch wrapping all the code in handleRequest method (I was not catching Errors, this is why I did not see it) :

try {
    //My code above
} catch (Error | Exception e) {
    LOGGER.error(e);
}

I got a java.lang.OutOfMemoryError. I had set the lambda memory to 128MB since my code in local was working with less than that. But it seems that reading from s3 needs a bit more and now it works fine with 512MB.

Question:

i want to read last modified or created date in a s3 file. i have seen the coding to list all files but i wanted to find a specific file's modified date.

i used this coding to retrive the s3 file

S3Object s3ProfileObject = s3Client.getObject(new GetObjectRequest(srcBucket, fullProfilePath));

InputStream profileData = s3ProfileObject.getObjectContent();

SONObject rawProfileObject = (JSONObject)jsonParser.parse( new InputStreamReader(profileData, "UTF-8"));

is it possible to read it using s3ProfileObject ?


Answer:

You need to use S3ObjectSummary for that. Here is the documentation for it. Below is the sample code (from here):

final AmazonS3 s3 = AmazonS3ClientBuilder.defaultClient();
ListObjectsV2Result result = s3.listObjectsV2(bucket_name);
List<S3ObjectSummary> objects = result.getObjectSummaries();
for (S3ObjectSummary os: objects) {
    System.out.println("* " + os.getKey());
}

From S3ObjectSummary, you can make another request and get S3Object.

If you want to include a specific file or files with specific prefix then you can use `` object with withPrefix method along with the request, e.g.:

ListObjectsV2Request listObjectsV2Request = new ListObjectsV2Request()
    .withBucketName("test-bucket")
    .withPrefix("file-prefix");

ListObjectsV2Result result = s3client.listObjectsV2(listObjectsV2Request);

Here's the javadoc for ListObjectsV2Request.

Question:

I have this code to download a file from s3 bucket to tmp space of lambda server.

String url = "https://slack-automation.s3.amazonaws.com/slack.xlsx";

URL link = new URL(url);

/* //AmazonS3 s3Client = new AmazonS3Client(new ProfileCredentialsProvider());
   AmazonS3 s3Client = AmazonS3ClientBuilder.standard().build() ;
   S3Object object = s3Client.getObject(new GetObjectRequest(bucketName, key));
   InputStream objectData = object.getObjectContent();
   // Process the objectData stream.
   objectData.close();
 */

// Code to download
InputStream in = new BufferedInputStream(link.openStream());
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] buf = new byte[1024];
int n = 0;

while (-1 != (n = in.read(buf))) {
    out.write(buf, 0, n);
}

out.close();
in.close();
byte[] response = out.toByteArray();

FileOutputStream fos = new FileOutputStream("/tmp/" + fileName);
fos.write(response);
fos.close();
file = new File("/tmp/" + fileName);

//file = new File(filePath);
inputStream = new FileInputStream(file);
workBook = new XSSFWorkbook(inputStream);
workBookSheet = workBook.getSheet(workBook.getSheetName(0));
rowCount = workBookSheet.getPhysicalNumberOfRows();
formatter = new DataFormatter();

But the problem is to access this file, the public permission has to be given.

How can I download this file to lambda server tmp space by getting the temporary credentials which is given over here: https://docs.aws.amazon.com/AmazonS3/latest/dev/AuthUsingTempSessionTokenJava.html

I am unable to achieve, can somebody please help me with the code to download file with temporary credentials without passing my accesskey and secretkey just like in the method mentioned above?

Thanks Akshay Sing , i dont know how to comment with a code, the code which i tried after your answer is here

public void setupWorkbook(String filePath) throws IOException {

    String clientRegion = "eu-central-1";
    String roleARN = "myAwsArnRoleOverHere";
    String roleSessionName = "slackautomation";
    String bucketName = "slack-automation";
    String key = "slack.xlsx";


    String fileName = "slack.xlsx";



    try {

        AWSSecurityTokenService stsClient = AWSSecurityTokenServiceClientBuilder.standard()
                                                .withCredentials(new ProfileCredentialsProvider())
                                                .withRegion(clientRegion)
                                                .build();

        AssumeRoleRequest roleRequest = new AssumeRoleRequest()
                                                .withRoleArn(roleARN)
                                                .withRoleSessionName(roleSessionName);
        stsClient.assumeRole(roleRequest);

        // Start a session.
        GetSessionTokenRequest getSessionTokenRequest = new GetSessionTokenRequest();
        // The duration can be set to more than 3600 seconds only if temporary
        // credentials are requested by an IAM user rather than an account owner.
        getSessionTokenRequest.setDurationSeconds(900);
        GetSessionTokenResult sessionTokenResult = stsClient.getSessionToken(getSessionTokenRequest);
        Credentials sessionCredentials = sessionTokenResult.getCredentials();

        // Package the temporary security credentials as a BasicSessionCredentials object 
        // for an Amazon S3 client object to use.
        BasicSessionCredentials basicSessionCredentials = new BasicSessionCredentials(
                sessionCredentials.getAccessKeyId(), sessionCredentials.getSecretAccessKey(),
                sessionCredentials.getSessionToken());

        // Provide temporary security credentials so that the Amazon S3 client 
        // can send authenticated requests to Amazon S3. You create the client 
        // using the basicSessionCredentials object.
        AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
                                .withCredentials(new AWSStaticCredentialsProvider(basicSessionCredentials))
                                .withRegion(clientRegion)
                                .build();

        // Verify that assuming the role worked and the permissions are set correctly
        // by getting a set of object keys from the bucket.
        ObjectListing objects = s3Client.listObjects(bucketName);

        System.out.println("No. of Objects: " + objects.getObjectSummaries().size());
        S3Object fileObject = s3Client.getObject(new GetObjectRequest(bucketName, key));         
        InputStream in = new BufferedInputStream(fileObject.getObjectContent());

        ByteArrayOutputStream out = new ByteArrayOutputStream();

        byte[] buf = new byte[1024];

        int n = 0;

        while (-1 != (n = in.read(buf))) {

            out.write(buf, 0, n);

        }

        out.close();
        in.close();
        byte[] response = out.toByteArray();
        FileOutputStream fos = new FileOutputStream("/tmp/" + fileName);

        fos.write(response);

        fos.close();
        file = new File("/tmp/" + fileName);

        //file = new File(filePath);

        inputStream = new FileInputStream(file);

        workBook = new XSSFWorkbook(inputStream);

        workBookSheet = workBook.getSheet(workBook.getSheetName(0));

        rowCount = workBookSheet.getPhysicalNumberOfRows();

        formatter = new DataFormatter();


    }
    catch(AmazonServiceException e) {
        // The call was transmitted successfully, but Amazon S3 couldn't process 
        // it, so it returned an error response.
        e.printStackTrace();
    }
    catch(SdkClientException e) {
        // Amazon S3 couldn't be contacted for a response, or the client
        // couldn't parse the response from Amazon S3.
        e.printStackTrace();
    }

But it is not working it is throwing an error over "stsClient.assumeRole(roleRequest);"

error with just the line number not much clue for it.


Answer:

The Solution for my above question, i am writing the entire code over here

public void setupWorkbook(String filePath) throws IOException {



   // String bucketName = "slack-automation";
    String key = "slack.xlsx";


    String fileName = "slack.xlsx";


    String bucketName = System.getenv("bucket_name");

    try {



        AmazonS3 s3Client = new AmazonS3Client();

        ObjectListing objects = s3Client.listObjects(bucketName);

        System.out.println("No. of Objects: " + objects.getObjectSummaries().size());
        S3Object fileObject = s3Client.getObject(new GetObjectRequest(bucketName, key));         
        InputStream in = new BufferedInputStream(fileObject.getObjectContent());

        ByteArrayOutputStream out = new ByteArrayOutputStream();

        byte[] buf = new byte[1024];

        int n = 0;

        while (-1 != (n = in.read(buf))) {

            out.write(buf, 0, n);

        }

        out.close();
        in.close();
        byte[] response = out.toByteArray();
        FileOutputStream fos = new FileOutputStream("/tmp/" + fileName);

        fos.write(response);

        fos.close();
        file = new File("/tmp/" + fileName);

        //file = new File(filePath);

        inputStream = new FileInputStream(file);

        workBook = new XSSFWorkbook(inputStream);

        workBookSheet = workBook.getSheet(workBook.getSheetName(0));

        rowCount = workBookSheet.getPhysicalNumberOfRows();

        formatter = new DataFormatter();


    }
    catch(AmazonServiceException e) {
        // The call was transmitted successfully, but Amazon S3 couldn't process 
        // it, so it returned an error response.
        e.printStackTrace();
        System.out.println("FIRST EXCEPTION"+e.getMessage());
    }
    catch(SdkClientException e) {
        // Amazon S3 couldn't be contacted for a response, or the client
        // couldn't parse the response from Amazon S3.
        e.printStackTrace();
        System.out.println("SECOND EXCEPTION"+e.getMessage());
    }

It was just this part of the code was required to add

   AmazonS3 s3Client = new AmazonS3Client();

    ObjectListing objects = s3Client.listObjects(bucketName);

    System.out.println("No. of Objects: " + objects.getObjectSummaries().size());
    S3Object fileObject = s3Client.getObject(new GetObjectRequest(bucketName, key));         
    InputStream in = new BufferedInputStream(fileObject.getObjectContent());

The line "AmazonS3 s3Client = new AmazonS3Client()" would assume the IAM Role and will get the temporary credentials without passing my accesskey and secretkey since the code is running from AWS Lambda. That's it, Thank you all for your help

Question:

I have 2 accounts

Account A and Account B

In account A I have deployed (Amazon S3) my Lambda function.

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;

public class LambdaFunctionHandler implements RequestHandler<Request, Response> {

    public Response handleRequest(Request request, Context context) {
        String greetingString = String.format("Hello %s %s.",
                request.firstName, request.lastName);
        //Here I need to get the Account B's bucket info
        return new Response(greetingString);
    }

}

In account A I am creating the IAM Role 'my-lambda' and same is mapped to the user X

In Account B I have created the policy to grant user permission with role 'my-lambda' How to get the bucket info of the Account B by using user X's IAM Role???

Note: I am able to get the bucket info of Account B if I give the credentials directly

AWSCredentials longTermCredentials_ = new PropertiesCredentials(LambdaFunctionHandler .class.getResourceAsStream("/resources/"+"AwsCredentials.properties"));
AWSSecurityTokenServiceClient stsClient = new AWSSecurityTokenServiceClient(longTermCredentials_);
GetSessionTokenRequest getSessionTokenRequest = new GetSessionTokenRequest();
GetSessionTokenResult sessionTokenResult = stsClient.getSessionToken(getSessionTokenRequest);
Credentials sessionCredentials = sessionTokenResult.getCredentials();
BasicSessionCredentials basicSessionCredentials = new BasicSessionCredentials(sessionCredentials.getAccessKeyId(),sessionCredentials.getSecretAccessKey(),sessionCredentials.getSessionToken());
AmazonS3Client s3Client = new AmazonS3Client(basicSessionCredentials);
ListObjectsRequest listObjectsRequest = new ListObjectsRequest().withBucketName("bucketName");
ObjectListing objectListing;
 do {
            objectListing = s3.listObjects(listObjectsRequest);
            for (S3ObjectSummary objectSummary : objectListing
                    .getObjectSummaries()) {
                String key = objectSummary.getKey();

            }
            listObjectsRequest.setMarker(objectListing.getNextMarker());
        } while (objectListing.isTruncated());

Answer:

You can use the STSAssumeRoleSessionCredentialsProvider class to help assume the role based on your long-term credentials and getting temporary credentials to the S3 Client.

AWSCredentials longTermCredentials_ =  ...
STSAssumeRoleSessionCredentialsProvider roleCredsProvider = 
    new STSAssumeRoleSessionCredentialsProvider(
        longTermCredentials_, 
        "my_lambda", 
        "BucketListSession");
AmazonS3Client s3Client = new AmazonS3Client(roleCredsProvider);

Question:

I am creating an aws lambda function which downloads an s3 file and processes it according an event it receives everytime. However i don't want to download the s3 file from s3 everytime. Can anyone suggest me how to download s3 file only once and process the incoming events without having to download the s3 file everytime?

Currently its downloading everytime even if i put the code to dowload from s3 in constructor of the lambdafunctionhandler class

If you make any code references or examples, please use java. Thanks in advance


Answer:

If you run several lamdbas in parallel thee context is not reused. So you need to download the file in all lambdas. For storing files use /tmp/. It has a limit of 512MB.

However, if you run a lamdba after another one, the context probably will be reused and therefore the file will exist. Keep in mind cold boot.

Extracted from AWS Lamdba DOC.

After a Lambda function is executed, AWS Lambda maintains the execution context for some time in anticipation of another Lambda function invocation. In effect, the service freezes the execution context after a Lambda function completes, and thaws the context for reuse, if AWS Lambda chooses to reuse the context when the Lambda function is invoked again. This execution context reuse approach has the following implications:

  • Objects declared outside of the function's handler method remain initialized, providing additional optimization when the function is invoked again. For example, if your Lambda function establishes a database connection, instead of reestablishing the connection, the original connection is used in subsequent invocations. We suggest adding logic in your code to check if a connection exists before creating one. Each execution context provides 512 MB of additional disk space in the /tmp directory. The directory content remains when the execution context is frozen, providing transient cache that can be used for multiple invocations. You can add extra code to check if the cache has the data that you stored. For information on deployment limits, see AWS Lambda Limits. Background processes or callbacks initiated by your Lambda function that did not complete when the function ended resume if AWS Lambda chooses to reuse the execution context. You should make sure any background processes or callbacks in your code are complete before the code exits.

Example code for downloading an obejct from S3:

AmazonS3 s3client = AmazonS3ClientBuilder
                  .standard()
                  .withRegion(Regions.EU_WEST_1)
                  .build();

        //S3 download file

        GetObjectRequest getObjectRequest = new GetObjectRequest(System.getenv("bucket"), "key");
        s3client.getObject(getObjectRequest, new File("/tmp/example.png")); 

EDIT 1: Lambdas and Serverless in general is not recommended for apps that need to maintain the state between different invocations.

Question:

I have a an existing index of lucene index files and the java code to perform search functions on it.

What I would like to do is perform the same thing on a server so users of an app could simply pass a query that will be taken as an input parameter by the java program and run it against the existing index to return the document in which it occurs.

All the implementation has been tested on my local pc,but what I need to do is implement it in an Android app.

So far I have read around and concluded that porting the code in AWS lambda and using S3 to store the files and calling the s3 objects from lambda.

Is this the right approach?Any resources that point to the this approach or alternative suggestions are also appreciated.

Thanks in advance.


Answer:

Every time your Android app sends a request to AWS Lambda (via AWS API Gateway I assume) the Lambda function will have to download the entire index file from S3 to the Lambda /tmp directory (where Lambda has a 512MB limit) and then perform a search against that index file. This seems extremely inefficient, and depending on how large your index file is, it might perform terribly or it might not even fit into the space you have available on Lambda.

I would suggest looking into the AWS Elasticsearch Service. This is a fully managed search engine service, based on Lucene, that you should be able to query directly from your Android application.

Question:

I am new to AWS and I am currently trying to understand Lambda functions and to trigger it when I upload file to S3 bucket. I wrote a handler class for this:

public class Hello implements RequestHandler<Employee, String> {
    public String handleRequest(Employee input, Context context) {
        context.getLogger().log("helloWorld");
        return "Hello World " ;
    }
}

This was just a basic and I could see the "helloworld" printed in logs in CloudWatch when I upload a file to S3 bucket.

But Now what I want to log the metadata of the file (fileName, createdTime etc.).

I went thru a sample template example in AWS Lambda page, where I can see using Nodejs, we have the event as the argument and we can extract the name and other fields using this field.

const aws = require('aws-sdk');
const s3 = new aws.S3({ apiVersion: '2006-03-01' });
exports.handler = (event, context, callback) => {
    const bucket = event.Records[0].s3.bucket.name;
    ...

}

But as a Java developer, I tried to use S3EventNotification as the argument:

public class Hello implements RequestHandler<S3EventNotification, String> {
    public String handleRequest(S3EventNotification input, Context context) {
        context.getLogger().log(input.getRecords().get(0).getEventSource());
        return "Hello World " ;
    }
}

But I am getting below error:

An error occurred during JSON parsing: java.lang.RuntimeException
java.lang.RuntimeException: An error occurred during JSON parsing
Caused by: lambdainternal.util.ReflectUtil$ReflectException: java.lang.NoSuchMethodException: com.amazonaws.services.s3.event.S3EventNotification$S3ObjectEntity.<init>(java.lang.String, java.lang.Long, java.lang.String, java.lang.String)
Caused by: java.lang.NoSuchMethodException: com.amazonaws.services.s3.event.S3EventNotification$S3ObjectEntity.<init>(java.lang.String, java.lang.Long, java.lang.String, java.lang.String)

How can I achieve the same thing in Java? Thanks.


Answer:

Try some variant of the following:

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.S3Event;

public class Hello implements RequestHandler<S3Event, Void> {
    @Override
    public Void handleRequest(S3Event s3event, Context context) {
        try {
            S3EventNotificationRecord record = s3event.getRecords().get(0);

            String bkt = record.getS3().getBucket().getName();
            String key = record.getS3().getObject().getKey().replace('+', ' ');
            key = URLDecoder.decode(key, "UTF-8");
       } catch (Exception e) {
           // do something
       }
       return null;
    }
}

And here are the corresponding dependencies that I used in pom.xml:

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk-s3</artifactId>
    <version>1.11.228</version>
</dependency>
<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-lambda-java-core</artifactId>
    <version>1.1.0</version>
</dependency>
<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-lambda-java-events</artifactId>
    <version>2.0.1</version>
</dependency>

And here is the build specification from my pom.xml (which will cause dependent classes to be pulled into my built JAR):

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>2.3</version>
            <configuration>
                <createDependencyReducedPom>false</createDependencyReducedPom>
            </configuration>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

None of this is very simple, unfortunately, but that's Java and Maven for you. AWS Lambda programming in Node.js or Python is much simpler (and more fun) than in Java, so if there's no strong requirement to write it in Java, you're better off not writing in Java.

Also note that if the Lambda is going to be invoked asynchronously then the output type should be Void rather than String (see docs).

Question:

I created a basic AWS Lambda java function to convert an xml message to json. The function is triggered off of an S3 event (message is converted and dropped in different S3 bucket). It appears to be successful, I can see the steps in cloudwatch and the converted message is in the S3 destination bucket. However, I see the timeout warning in the cloud watch logs (set timeout to 15 seconds). I must be missing something, definitely a newby when it comes to Lambda. Do I need to provide the context with a done or success? Any suggestions or tips would be greatly appreciated.

Code Snippet:

    public void msgConvertToJson(S3Event s3event, Context context) {

    final String key = s3event.getRecords().get(0).getS3().getObject().getKey();
    final String bucketName = s3event.getRecords().get(0).getS3().getBucket().getName();
    log.info("key: " + key + " bucketName:" + bucketName);
    String action = "";


    try {

        //Attempt to retrieve the message from S3 on notification
        log.info("attempting to get message");
        action = "get";
        final String message = s3Client.getObjectAsString(bucketName, key);

        //Attempt to parse and convert the message to JSON
        log.info("attempting to parse message");
        String msgJson = new MessageParser(message).getMessageJson();

        //Attempt to write the converted message to a new S3 bucket
        final String parsedBucket = "parsed-" + bucketName;
        final String newKey = key.replace(".xml",".json");
        log.info("newKey: " + newKey + " parsedBucketName:" + parsedBucket);
        log.info("attempting to put message");
        action = "put";

        s3Client.putObject(parsedBucket, newKey, msgJson );

    } catch (AmazonServiceException ase) {
        log.error("Caught an AmazonServiceException trying to " + action +  " file " + key + ", which " +
                "means your request made it " +
                "to Amazon S3, but was rejected with an error response" +
                " for some reason.");
        log.error("Error Message:    " + ase.getMessage());
        log.error("HTTP Status Code: " + ase.getStatusCode());
        log.error("AWS Error Code:   " + ase.getErrorCode());
        log.error("Error Type:       " + ase.getErrorType());
        log.error("Request ID:       " + ase.getRequestId());
    } catch (AmazonClientException ace) {
        log.error("Caught an AmazonClientException while trying to " + action +  " file " + key + ", which " +
                "means the client encountered " +
                "an internal error while trying to " +
                "communicate with S3, " +
                "such as not being able to access the network.");
        log.error("Error Message: " + ace.getMessage());
    }
}

Answer:

It actually turned out that it was just the timeout was too short. The cold start for the JVM, must be taking longer than I would have thought. I was just assuming I had another issue in my code. Bumped memory to 192 and timeout to 45 seconds. Hope this helps someone else.

Really unfortunate I was marked down by someone that was pointing me at the wrong information (NodeJS) instead of Java.

Question:

I have a very simple spring route that im attempting to run on aws lambda. The route simply returns the text/string "redirect:/upload" instead of redirecting. I have the html file in the /resources/templates folder.

@RequestMapping(path = "/test", method = RequestMethod.POST)
    public String UploadPage2() {
        return "redirect:/upload";
    }

Answer:

I think the problem is from the return type of method: String.

You can do:

public RedirectView UploadPage2() {
    return new RedirectView("/upload");
}

Second question

To return an view on path /test with GET request, you need another method with same path but different method

@RequestMapping(path = "/test", method = RequestMethod.GET) 
public ModelAndView testGet(){
    return new ModelAndView("uploadview");
}

Question:

I've a requirement to generate file's (.docx , .pdf) using docx4j and iText. And end generated documents should be store at S3 bucket using AWS lambda function.

I've tried couple of options but none of the solution works for me. I can able to upload a local file to S3 bucket and write a normal text file to AWS S3 bucket using my java code.

However my require is, firstly I've to create a file path's for (.docx, .pdf) of the desire location i.e. S3 bucket then give a call to my code to generate the files(.docx, .pdf) and save it to S3 bucket.

NOTE: I wish to call my save method (lambda handler) which will directly save files in S3 bucket. Or either way if s3 doesn't allow to save files directly then whats the alternative aws provide to save files locally/temp so later i can push those temp files into s3 bucket??

If any ever work on this or have an idea how to do that please share their inputs.


Answer:

There are two ways you can do this. First method would be storing the file in /tmp/ first then use the putobject API to upload to s3 specifying the bucket and keyname.

Second method is directly uploading the object to s3 using the same putobject but specifying your object.

The docs for this can be found here

A detailed example of this can be found here

Question:


Answer:

There are many ways to do this (depending on what you're actually trying to do!).

Scheduling

If you run code on an Amazon EC2 instance, you can use cron to trigger a script.

If you run code as an AWS Lambda function, you can add a schedule to the function.

Processing

You did not supply much information about the content of the files and how they should be processed (eg whether each file is individually processed, or if multiple files are processed together).

Some processing options are:

  • If the files can be processed individually, you can use S3 Select, which can run simple SQL-like commands over the file, without needing to download the file
  • For more complex queries, including processing multiple files together, use Amazon Athena
  • If the files need to be processed with custom code, either run a script on an Amazon EC2 instance, or run an AWS Lambda function (Note: Lambda has a maximum execution time of 15 minutes)

S3 Select and Amazon Athena can be called from a Lambda function.

Inventory

If you have a large number of files (tens of thousands or more), it might be efficient to use Amazon S3 Inventory to provide a weekly list of input files. You code can then use that list to perform operations.

Real-time processing

Rather than processing data as a weekly batch, you could also consider performing real-time processing when the data is placed in Amazon S3. You can configure an Amazon S3 Event to trigger an AWS Lambda function when the file is created.

Sending to users

Rather than sending a file to users, it is generally better to provide a link back to a file. Your email administrators will thank you!

Regardless, your AWS Lambda function can use Amazon Simple Email Service (SES) to send emails. However, you'll need some way of identifying who should receive such notifications.

One option is to send a notification to Amazon Simple Notification Service (SNS) and users can subscribe to the message. It could point them to the report (but also consider security of the report).

Question:

I am accessing a lambda function from my java code and I would like to know if I am using Https or Https for that request. How can I do that.

This is how my method looks like.

public interface FileScanService {

   @LambdaFunction( functionName = "s3-antivirus-api-scan" )
   public FileScanOutput scanFile( FileScanInput fileScanInput );
}

PS: this method is fully functional.


Answer:

UPDATE: See the comment by Michael below. It turns out there is a way how to do it directly via HTTPS.

A Lambda function is not invoked directly via HTTP, so the answer to your question is neither. Instead, it is invoked by one or more of the event sources.

It can however sit behind an API Gateway which would enable it being (indirectly) invoked by an HTTP request. In that case, the request is transformed into an invocation event and that is what your Lambda function operates with.

Question:

I am using a lambda function written in java that listen to s3 bucket create object The requesthandler will call another class "testDownload" to download the object

something is happening when AmazonS3 s3 = AmazonS3ClientBuilder.standard().build(); is initialized, i am using it in a try catch block and checking if an exception is thrown and no error is displaying in cloud watch. Can you please help me understand whats going on

public class S3EventProcessor implements RequestHandler<S3Event, String> {
private final org.slf4j.Logger logger = LoggerFactory.getLogger(this.getClass());
private final String ASSET_TYPE = (String) "jpg";
public String handleRequest(S3Event s3event, Context context) {
....
....
t.download(Bucket,Key);


import com.amazonaws.AmazonServiceException;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.S3Object;
import com.amazonaws.services.s3.model.S3ObjectInputStream;
import org.slf4j.LoggerFactory;

import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;


public class testDownload {
private final org.slf4j.Logger logger =  LoggerFactory.getLogger(this.getClass());

public void download(String bucketName,String key) throws IOException {


    try {
        logger.info("Downloading an object \n");
        logger.info("Building s3 client");
        AmazonS3 s3 = AmazonS3ClientBuilder.standard().build();
        logger.info("success");
        S3Object o = s3.getObject(bucketName, key);
        S3ObjectInputStream s3is = o.getObjectContent();
        FileOutputStream fos = new FileOutputStream(new File(key));
        byte[] read_buf = new byte[1024];
        int read_len = 0;
        while ((read_len = s3is.read(read_buf)) > 0) {
            fos.write(read_buf, 0, read_len);
        }
        s3is.close();
        fos.close();
    } catch (AmazonServiceException e) {
        logger.error(e.getErrorMessage());
        System.exit(1);
    } catch (FileNotFoundException e) {
        logger.error(e.getMessage());
        System.exit(1);
    } catch (IOException e) {
        logger.error(e.getMessage());
        System.exit(1);
    } catch (Exception e){
        logger.error(e.getStackTrace().toString());
        System.exit(1);
    }
}

}

Cloud Logs

19:25:34,229 INFO testDownload:24 - Downloading an object 

19:25:34,229 INFO testDownload:25 - Building s3 client
END RequestId: someid
REPORT RequestId: someid    Duration:   3003.13 ms  Billed Duration: 3000 ms Memory Size: 128 MB    Max Memory     Used: 52 MB  
2017-10-21T19:25:36.150Z someid Task timed out after 3.00 seconds

Answer:

The default timeout for a Lambda is 3 seconds. Your Lambda runs for 3 seconds and then is killed. Try to configure the Lambda to run longer. Additionally, what are you trying to do with the output file - save it on the Lambda machine?

Question:

Im trying to create an AWS Lambda Function in C# to crop an image which is stored in the S3 bucket. Once cropped, the new image need to be pushed to another bucket. And this function has to be invoked manually using the AWS Lambda Client. As im a beginner and there is not much tutorials available on the web, please help me on this.

This is how my code looks like

 using System;
 using System.Threading.Tasks;
 using System.IO;

 using Amazon.Lambda.Core;
 using Amazon.Lambda.Serialization;
 using Amazon.S3;
 using Amazon;
 using Amazon.S3.Model;
 using Pomelo.Data.MySql;
 using System.Drawing;
 using Amazon.S3.Transfer;
 //using ImageNet;
 //using ImageResizer;

 // Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
 [assembly: LambdaSerializerAttribute(typeof(Amazon.Lambda.Serialization.Json.JsonSerializer))]

 namespace Cropper
 {
     public class Crop
     {
         IAmazonS3 s3Client { get; set; }
         private Image orgImg;
         private MySqlConnection dbConn = new MySqlConnection();
         private MySqlCommand cmd = new MySqlCommand(); 

         public Crop()
         {
             s3Client = new AmazonS3Client();
         }

         /*private static string s3AccessKey = Environment.GetEnvironmentVariable("S3AccessKey");
         private static string s3SecretKey = Environment.GetEnvironmentVariable("S3SecretKey");
         private static string s3Region = Environment.GetEnvironmentVariable("S3Region");
         private AmazonS3Client s3Client = new AmazonS3Client(s3AccessKey,s3SecretKey, RegionEndpoint.GetBySystemName(s3Region));
         */

         // public async Task<string> CropImage(string sourceBucket, string key, string destBucket, string permission)
         //public async void CropImage(string sourceBucket, string key, string destBucket, string permission)
         public async Task<string> CropImage(string input, ILambdaContext context)
         {
             s3Client = new AmazonS3Client();

             string[] arr = input.Split(',');
             string sourceBucket = arr[0].Trim();
             string key = arr[1].Trim();
             string destBucket = arr[2].Trim();
             //string size = arr[3].Trim();
             //string crop = arr[3].Trim();
             string permission = arr[3].Trim();

             string path = Path.Combine("/tmp", key);

             try
             {
                 Console.WriteLine("Checkpoint - 0");

                 TransferUtility fileTransferUtility = new TransferUtility(s3Client);
                 TransferUtilityDownloadRequest downloadRequest = new TransferUtilityDownloadRequest();

                 Console.WriteLine("path - " + path);

                 if (File.Exists(path))
                 {
                     File.Delete(path);
                 }
                 downloadRequest.FilePath = path;
                 downloadRequest.BucketName = sourceBucket;
                 downloadRequest.Key = key;
                 fileTransferUtility.Download(downloadRequest);

                 Console.WriteLine("Checkpoint - file transfer");

                 orgImg = Image.FromFile(path);
                 orgImg.RotateFlip(RotateFlipType.Rotate90FlipNone);
                 //FluentImage.FromFile(path)
                 //   .Resize.Width(200)
                 //   .Save("/tmp/test_file.png", OutputFormat.Png);



                 Console.WriteLine("Checkpoint - Img creation");

                 TransferUtilityUploadRequest uploadRequest = new TransferUtilityUploadRequest();
                 //uploadRequest.FilePath = "/tmp/test_file.png";
                 uploadRequest.FilePath = path;
                 uploadRequest.BucketName = destBucket;
                 uploadRequest.Key = key;
                 if (permission.ToUpper() == "PUBLIC")
                 {
                     uploadRequest.CannedACL = S3CannedACL.PublicRead;
                 }
                 else if (permission.ToUpper() == "PRIVATE")
                 {
                     uploadRequest.CannedACL = S3CannedACL.Private;
                 }
                 else if (permission.ToUpper() == "NONE")
                 {
                     uploadRequest.CannedACL = S3CannedACL.NoACL;
                 }
                 fileTransferUtility.Upload(uploadRequest);

                 Console.WriteLine("Checkpoint - Done");

                 return context.AwsRequestId.ToString();
             }
             catch (Exception ex)
             {
                 Console.WriteLine("ex message - " + ex.Message);
                 Console.WriteLine("ex stack - " + ex.StackTrace);
                 return ex.Message;
                 //context.Logger.LogLine(ex.Message);
             }
             finally
             {
                 if (File.Exists(path))
                 {
                     File.Delete(path);
                 }
                 if (orgImg != null)
                 {
                     orgImg.Dispose();
                 }
             }
         }

         //private byte[] ToArrayBytes(Stream input)
         //{
         //    byte[] buffer = new byte[16 * 1024];
         //    using (MemoryStream ms = new MemoryStream())
         //    {
         //        int read;
         //        while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
         //        {
         //            ms.Write(buffer, 0, read);
         //        }
         //        return ms.ToArray();
         //    }
         //}

         //private byte[] ImageToByte(Image img)
         //{
         //    ImageConverter converter = new ImageConverter();
         //    return (byte[])converter.ConvertTo(img, typeof(byte[]));
         //}
     }
 }

This is the project.json

 {
   "version": "1.0.0-*",
   "buildOptions": {
   },

   "dependencies": {
     "Microsoft.NETCore.App": {
       "type": "platform",
       "version": "1.0.0"
     },
     "Amazon.Lambda.Core": "1.0.0*",
     "Amazon.Lambda.Serialization.Json": "1.0.1",
     "Amazon.Lambda.Tools": {
       "type": "build",
       "version": "1.0.4-preview1"
     },
     "AWSSDK.S3": "3.3.4.1",
     "Pomelo.Data.MySql": "1.0.0",
     "Microsoft.NETCore.Portable.Compatibility": "1.0.1",
     //"System.Drawing-dotnet-core": "1.0.2",
     //"Shaman.System.Drawing": "1.0.1.1",
     //"ZKWeb.System.Drawing": "2.0.1"
     //"System.Drawing.Primitives": "4.3.0"
     //"ImageProcessor-strong-name": "2.3.0",
     //"Microsoft.NETCore.Runtime.CoreCLR": "1.1.0",
     //"Sitecore.ImageProcessor": "1.1.0",
     //"DynamicImage": "3.1.1",
     //"ImageLibrary": "3.0.0",
     //"FluentImage": "1.1.0",
     "CoreCompat.System.Drawing": "1.0.0-beta006",
     "ImageResizer.Mvc": "4.0.5"
     //"NETStandard.Library": "1.6.1"
     //"omu.drawing": "1.0.0"
   },
     "tools": {
       "Amazon.Lambda.Tools": "1.0.4-preview1"
     },
     "frameworks": {
       "netcoreapp1.0": {
         "imports": [
           "dnxcore50",
           "dotnet5.6",
           "net46"
         ]
       }
     }
 }

And, this is the stack trace

 2017-11-03 23:11:43: START RequestId: e3975f78-d1db-11e6-9923-974cd0941dbb Version: $LATEST
 2017-11-03 23:11:44: Checkpoint - 0
 2017-11-03 23:11:44: path - /tmp/r3_small.jpg
 2017-11-03 23:11:47: Checkpoint - file transfer
 2017-11-03 23:11:47: ex message - The type initializer for 'System.Drawing.GDIPlus' threw an exception.
 2017-11-03 23:11:48: ex stack -    at System.Drawing.GDIPlus.GdipLoadImageFromFile(String filename, IntPtr& image)
 2017-11-03 23:11:48:    at System.Drawing.Image.FromFile(String filename, Boolean useEmbeddedColorManagement)
 2017-11-03 23:11:48:    at System.Drawing.Image.FromFile(String filename)
 2017-11-03 23:11:48:    at Cropper.Crop.<CropImage>d__8.MoveNext()
 2017-11-03 23:11:48: END RequestId: e3975f78-d1db-11e6-9923-974cd0941dbb
 2017-11-03 23:11:48: REPORT RequestId: e3975f78-d1db-11e6-9923-974cd0941dbb    Duration: 4970.70 ms    Billed Duration: 5000 ms    Memory Size: 256 MB Max Memory Used: 63 MB  

Code used to crop and rotate the image

 public void cropLocal( string url, int x, int y, int w, int h, int a)
 {
    try
    {
        if (x != 0 && y != 0 && w != 0 && h != 0)
        {
            if (File.Exists(url))
            {
                orgImg = Image.FromFile(url);
                switch (a)
                {
                    case 90:
                    case -270:
                        orgImg.RotateFlip(RotateFlipType.Rotate90FlipNone);
                        break;
                    case 180:
                    case -180:
                        orgImg.RotateFlip(RotateFlipType.Rotate180FlipNone);
                        break;
                    case 270:
                    case -90:
                        orgImg.RotateFlip(RotateFlipType.Rotate270FlipNone);
                        break;
                }

                if (x < 0) { x = 0; }
                if (y < 0) { y = 0; }
                if (w > orgImg.Width) { w = orgImg.Width; }
                if (h > orgImg.Height) { h = orgImg.Height; }

                bitmap = new Bitmap(orgImg);
                orgImg.Dispose();
                orgImg = null;

                rect = new Rectangle(x, y, w, h);
                cropped = bitmap.Clone(rect, bitmap.PixelFormat);
                bitmap.Dispose();
                bitmap = null;

                //delete original file
                File.Delete(url);
                //Save new file
                cropped.Save(url);
                cropped.Dispose();
                cropped = null;
            }
        }
    }
    catch(Exception ex)
    {
        string excep = ex.Message;
    }
    finally
    {
        if (orgImg != null)
        {
            orgImg.Dispose();
            orgImg = null;
        }
        if (bitmap != null)
        {
            bitmap.Dispose();
            bitmap = null;
        }
        if (cropped != null)
        {
            cropped.Dispose();
            cropped = null;
        }
    }

 }

Answer:

Found that its hard to do it in C# at this point of time and i have to learn java instead. See below the Java code to download image from s3, resize (without losing aspect ratio), crop, rotate and upload to back to s3 (to a different bucket). It works perfectly.

 package imageprocessor;

 import java.awt.Graphics2D;
 import java.awt.Rectangle;
 import java.awt.geom.AffineTransform;
 import java.awt.image.BufferedImage;
 import java.io.File;
 import java.util.UUID;

 import javax.imageio.ImageIO;
 import javax.swing.UIManager;

 import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
 import com.amazonaws.services.lambda.runtime.Context;
 import com.amazonaws.services.lambda.runtime.RequestHandler;
 import com.amazonaws.services.s3.AmazonS3Client;
 import com.amazonaws.services.s3.model.CannedAccessControlList;
 import com.amazonaws.services.s3.model.GetObjectRequest;
 import com.amazonaws.services.s3.model.PutObjectRequest;


 public class LambdaImageProcessor implements RequestHandler<String, String> {

    private static final String filePath = "/tmp/";
    private static File localFile, outputfileCropped, outputfileRotated, outputfileSized;
     private static final float xsize = 500;   //max width for resize
     private static final float ysize = 500;   //max height for resize

     @Override
     public String handleRequest(String input, Context context) {

        AmazonS3Client s3Client = new AmazonS3Client(
                 new DefaultAWSCredentialsProviderChain());

         context.getLogger().log("Input: " + input + "\n");

         try
         {
            String[] arr = input.split(",");
            String sourceBucket = arr[0];
            String key = arr[1];
             String destBucket = arr[2];
             String permission = arr[3];

            //download
             context.getLogger().log("AWS S3 downloading image\n");
            GetObjectRequest downloadReq = new GetObjectRequest(sourceBucket, key);

            String uniqueID = UUID.randomUUID().toString().replaceAll("-", "");
            localFile = new File(filePath + uniqueID + "_" + key);
            outputfileCropped = new File(filePath + uniqueID + "_" + "c_" + key);
            outputfileRotated = new File(filePath + uniqueID + "_" + "r_" + key);
            outputfileSized = new File(filePath + uniqueID + "_" + "s_" + key);

            if(localFile.exists())
            {
                localFile.delete();
            }
            if(outputfileCropped.exists())
            {
                outputfileCropped.delete();
            }
            if(outputfileRotated.exists())
            {
                outputfileRotated.delete();
            }

            context.getLogger().log("File name - " + localFile.toString() + "\n");
            s3Client.getObject(downloadReq, localFile);      
            context.getLogger().log("AWS S3 download - Done\n");

            //upload
            if(localFile.exists())
            {
                //context.getLogger().log("AWS S3 cropping image\n");

                BufferedImage bufferedImage = ImageIO.read(localFile);  

                BufferedImage resizeBuffered = resizeImage(bufferedImage, xsize, ysize);
                //ImageIO.write(resizeBuffered, "jpg", outputfileSized);    
                context.getLogger().log("AWS S3 Resize - Done\n");

                Rectangle rect = new Rectangle(100, 100, 300, 150);
                BufferedImage cropBuffered = cropImage(resizeBuffered, rect);
                context.getLogger().log("AWS S3 Crop - Done\n");

                ImageIO.write(rotateImage(cropBuffered, 90), "jpg", outputfileRotated);
                context.getLogger().log("AWS S3 Rotate - Done\n");

                context.getLogger().log("AWS S3 uploading image\n");
                PutObjectRequest uploadReq = new PutObjectRequest(destBucket, key, outputfileRotated);        

                if(permission.toUpperCase().equals("PUBLIC"))
                {
                    uploadReq.setCannedAcl(CannedAccessControlList.PublicRead);
                }
                else if(permission.toUpperCase().equals("PRIVATE"))
                {
                    uploadReq.setCannedAcl(CannedAccessControlList.Private);
                }
                s3Client.putObject(uploadReq);
                context.getLogger().log("AWS S3 upload - Done\n");
            }
            else
            {
                context.getLogger().log("Downloaded file not found\n");
            }
            return context.getAwsRequestId();
         }
         catch (Exception ex)
         {
            context.getLogger().log("Exception - " + ex.getMessage().toString() + " " + ex.getStackTrace().toString() + "\n");
            return "Exception - " + ex.getMessage().toString() + " " + ex.getStackTrace().toString();
         }
         finally 
         {          
            if(localFile.exists())
            {
                localFile.delete();
                context.getLogger().log("Temp Local File Deleted\n");
            }  
            if(outputfileCropped.exists())
            {
                outputfileCropped.delete();
                context.getLogger().log("Temp Cropped File Deleted\n");
            }
            if(outputfileRotated.exists())
            {
                outputfileRotated.delete();
                context.getLogger().log("Temp Rotated File Deleted\n");
            }
            if(outputfileSized.exists())
            {
                outputfileSized.delete();
                context.getLogger().log("Temp ReSized File Deleted\n");
            }
        }
     }

     private static BufferedImage cropImage(BufferedImage src, Rectangle rect) {
       BufferedImage dest = src.getSubimage(rect.x, rect.y, rect.width, rect.height);
       return dest; 
    }

     private static BufferedImage rotateImage(BufferedImage bufferedImage, int angle) { 

        double theta = Math.toRadians(angle);
        double cos = Math.abs(Math.cos(theta));
         double sin = Math.abs(Math.sin(theta));
         double width  = bufferedImage.getWidth();
         double height = bufferedImage.getHeight();
         int w = (int)(width * cos + height * sin);
         int h = (int)(width * sin + height * cos);

         BufferedImage out = new BufferedImage(w, h, bufferedImage.getType());
         Graphics2D g2 = out.createGraphics();
         g2.setPaint(UIManager.getColor("Panel.background"));
         g2.fillRect(0,0,w,h);
         double x = w/2;
         double y = h/2;
         AffineTransform at = AffineTransform.getRotateInstance(theta, x, y);
         x = (w - width)/2;
         y = (h - height)/2;
         at.translate(x, y);
         g2.drawRenderedImage(bufferedImage, at);
         g2.dispose();
         return out; 
    }

     private static BufferedImage resizeImage(BufferedImage bufferedImage, float sizeX, float sizeY)
    {
        float ratioX = sizeX / bufferedImage.getWidth();
        float ratioY = sizeY / bufferedImage.getHeight();
        float ratio = Math.min(ratioX, ratioY);

        float newX = bufferedImage.getWidth() * ratio;
        float newY = bufferedImage.getHeight() * ratio;

        BufferedImage resizedImage = new BufferedImage((int)newX, (int)newY, bufferedImage.getType());
         Graphics2D g = resizedImage.createGraphics();
         g.drawImage(bufferedImage, 0, 0, (int)newX, (int)newY, null);
         g.dispose();

         return resizedImage;

    }

 }

Question:

I am currently working on a project that use AWS lambda and AWS S3. I would like to create a Lambda that will write into a S3 bucket. Here is my lambda written in Java :

public class WriteIntoBucketHandler implements RequestHandler<SNSEvent, PutObjectResult> {

    @Override
    public PutObjectResult handleRequest(SNSEvent snsEvent, Context context) {
        System.out.println("Connect to S3...");
        AmazonS3 client = AmazonS3ClientBuilder.defaultClient();
        System.out.println("Connected to S3 !");
        System.out.println("Send data to S3...");
        PutObjectResult result =  client.putObject(
                "my-bucket",
                "tata",
                snsEvent.getRecords().get(0).getSNS().getMessage()
        );
        System.out.println("Data sent to S3 !");
        return result;
    }
}

This lambda log correctly Connect to S3... but fall into timeout just after that and consequently does not display Connected to S3 ! (timeout is set to 10 secondes)

I configured my lamda to have AmazonS3FullAccess and have no VPC.

I have already read a lot of posts into internet and stack overflow. I found nothing that can solve it. Anyone have an idea ?

EDIT :

I set timeout to one minute and I got this log before to reach timeout :

START RequestId: c300eec4-42bb-4985-9360-7e1e0d9fbdfb Version: $LATEST
Connect to S3...
Metaspace: java.lang.OutOfMemoryError java.lang.OutOfMemoryError: Metaspace at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at ja
Exception in thread "main" java.lang.Error: java.lang.OutOfMemoryError: Metaspace
at lambdainternal.AWSLambda.<clinit>(AWSLambda.java:66)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:114)
Caused by: java.lang.OutOfMemoryError: Metaspace
END RequestId: c300eec4-42bb-4985-9360-7e1e0d9fbdfb
REPORT RequestId: c300eec4-42bb-4985-9360-7e1e0d9fbdfb  Duration: 13202.94 ms   Billed Duration: 13300 ms Memory Size: 128 MB   Max Memory Used: 103 MB
Metaspace java.lang.OutOfMemoryError

Answer:

I found a solution. I set the lambda to fall into timeout after 1 minute. After that I got a new exception :

START RequestId: c300eec4-42bb-4985-9360-7e1e0d9fbdfb Version: $LATEST
Connect to S3...
Metaspace: java.lang.OutOfMemoryError java.lang.OutOfMemoryError: Metaspace at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at ja
Exception in thread "main" java.lang.Error: java.lang.OutOfMemoryError: Metaspace
at lambdainternal.AWSLambda.<clinit>(AWSLambda.java:66)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:114)
Caused by: java.lang.OutOfMemoryError: Metaspace
END RequestId: c300eec4-42bb-4985-9360-7e1e0d9fbdfb
REPORT RequestId: c300eec4-42bb-4985-9360-7e1e0d9fbdfb  Duration: 13202.94 ms   Billed Duration: 13300 ms Memory Size: 128 MB   Max Memory Used: 103 MB
Metaspace java.lang.OutOfMemoryError

As it said in this exception 128MB is not enought to instance the S3 client. S3 client need at least 192MB so I had to increase my lambda size memory.

Important note : Only the first instance take more that 10 seconds to be executed. All lambda executions will be far more faster after that. After 1 hour, however, the lambda will need to authenticate again and will take more than 10 seconds.

Question:

I am trying to upload and retrieve data from S3 via a lambda, however it currently returns data as a string. How do I return it as a JSON object.

For example this is the data I'm storing, through manual upload to S3:

{ "data": [ { "encryptedString": "D31BKNI8yVwknKXadWIP9LSL06Oss9Xhc5qyZSBHTBDj8TQihTHjoinKJEoKRT03Pt8U/b81ZLxuSOJxw3MU+ZX/CgSolBUPWeH/gD6zA/yKR+aQ0vb/t3g1SysTaOiKK2i5cGuqy3FLbgenn2U43sYKpb97B9h/WKCjGISBsMw=" } ] }

Lambda is triggered with this Java code:

public static void cryptoFromLambda(S3Event s3Event) {
    System.out.println("Called from Lambda. Records are:");
    s3Event.getRecords().forEach(e -> {
        try {
            String bucket = e.getS3().getBucket().getName();
            String key = e.getS3().getObject().getKey().replace('+', ' ');
            key = URLDecoder.decode(key, "UTF-8");

            AmazonS3Client s3Client = new AmazonS3Client();
            String body = s3Client.getObjectAsString(bucket, key);
            System.out.println(body);
        } catch (UnsupportedEncodingException ex) {
            System.out.println(ex.toString());
        }
    });
    System.out.println("end lambda");
}

Each line in CloudWatch is printed...

 15:26:46 {

 15:26:46 "data": [{

 15:26:46 "encryptedString": "D31BKNI8yVwknKXadWIP9LSL06Oss9Xhc5qyZSBHTBDj8TQihTHjoinKJEoKRT03Pt8U/b81ZLxuSOJxw3MU+ZX/CgSolBUPWeH/gD6zA/yKR+aQ0vb/t3g1SysTaOiKK2i5cGuqy3FLbgenn2U43sYKpb97B9h/WKCjGISBsMw="

 15:26:46 }]

 15:26:46 }

... whereas I need each line as an element of JSON, of which I can then parse.

I think my issue is more in how I tell S3 that it should take what I input as JSON rather than just a text file, if it's possible to do that.


Answer:

Found the answer is to getObject and then get the content as a stream. One can then use Jackson's JsonParser to parse the stream

S3Object s3Object = s3Client.getObject(bucket, key);
S3ObjectInputStream s3Stream = s3Object.getObjectContent();

ObjectMapper objectMapper = new ObjectMapper();
JsonFactory jsonFactory = objectMapper.getFactory();
JsonParser jsonParser = jsonFactory.createParser(s3Stream);

JsonParser.nextToken()...etc

Question:

Info about my use case -

  • Lambda is in AWS account A
  • S3 in AWS Account B

File upload event in S3 (Present in AWS account B) triggers lambda function (Present in AWS account A), lambda function take S3 event as input and must download file from S3. Lambda is configured with IAM role which has a policy attached. Policy has access to S3 (Present in AWS account B).

In order for lambda to interact with S3, lambda server must have aws token set (which it does itself using IAM role) and then assume role to set token profile for interacting with S3 without getting access denied. And to do so I have written AWSTokenManager (class code shared below) in my lambda function which when executed throws this error -

Unable to load credentials from service endpoint: com.amazonaws.SdkClientException
    com.amazonaws.SdkClientException: Unable to load credentials from service endpoint
    at com.amazonaws.auth.EC2CredentialsFetcher.handleError(EC2CredentialsFetcher.java:183)
    at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:162)
    at com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82)
    at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:164)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1166)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:762)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:724)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
    at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)
    at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)
    at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)
    at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    Caused by: java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
    at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
    at sun.net.www.http.HttpClient.New(HttpClient.java:339)
    at sun.net.www.http.HttpClient.New(HttpClient.java:357)
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1199)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
    at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
    at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:54)
    at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:116)
    at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:87)
    at com.amazonaws.auth.InstanceProfileCredentialsProvider$InstanceMetadataCredentialsEndpointProvider.getCredentialsEndpoint(InstanceProfileCredentialsProvider.java:189)
    at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:122)

And here is AWSTokenManager class code - package com.packagename;

import com.amazonaws.auth.BasicSessionCredentials;
import com.amazonaws.auth.InstanceProfileCredentialsProvider;
import com.amazonaws.services.securitytoken.AWSSecurityTokenService;
import com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClientBuilder;
import com.amazonaws.services.securitytoken.model.AssumeRoleRequest;
import com.amazonaws.services.securitytoken.model.Credentials;

public class AWSTokenManager {

public void awsTokenManager() {
    System.out.println("Inside awsTokenManager method");
    basicSessionCredentials();
}


BasicSessionCredentials basicSessionCredentials() {
    System.out.println("Call reached inside basicSessionCredentials method");

    AWSSecurityTokenService stsClient = AWSSecurityTokenServiceClientBuilder.standard()
            .withCredentials(InstanceProfileCredentialsProvider.getInstance()).build();

    AssumeRoleRequest roleRequest = new AssumeRoleRequest()
            .withRoleArn("arn:aws:iam::111:role/iamrolenamehere").withRoleSessionName("role_profilename");

    return gets3Credentials(stsClient, roleRequest);
}

public static BasicSessionCredentials gets3Credentials(AWSSecurityTokenService stsClient, AssumeRoleRequest roleRequest) {
    Credentials s3Credentials = stsClient.assumeRole(roleRequest).getCredentials();

    BasicSessionCredentials basicSessionCredentials = new BasicSessionCredentials(
            s3Credentials.getAccessKeyId(), s3Credentials.getSecretAccessKey(),
            s3Credentials.getSessionToken());

    return basicSessionCredentials;
 }
}

Also note that no_proxy and NO_PROXY environment variable in lambda configuration has 169.254.169.254. Reason I did this is to avoid proxy issue while connecting to AWS service which according to my understanding runs on ip given above.


Answer:

I was able to get pass above error by updating awstokenManager method to -

public void awsTokenManager() {
        System.out.println("Inside awsTokenManager method");
        STSAssumeRoleSessionCredentialsProvider stsAssumeRoleSessionCredentialsProvider = new STSAssumeRoleSessionCredentialsProvider.Builder(iAM_ROLE_ARN_TO_ASSUME, "us-east-1b")
                .withStsClient(AWSSecurityTokenServiceClientBuilder.standard().build())
                .withRoleSessionDurationSeconds(900)
                .build();
        System.out.println("SessionCredentials awsaccesskeyid is - " + stsAssumeRoleSessionCredentialsProvider.getCredentials().getAWSAccessKeyId());
        System.out.println("SessionCredentials awsaccesskeyid is - " + stsAssumeRoleSessionCredentialsProvider.getCredentials().getAWSSecretKey());
        System.out.println("SessionCredentials awsaccesskeyid is - " + stsAssumeRoleSessionCredentialsProvider.getCredentials().getSessionToken());
        }

Question:

I have been experiencing UserKeyMustBeSpecified errors lately when deleting multiple objects from s3, using keys without version.

The operation is performed in a Java lambda function, which uses the following code:

public class S3Dao {

    private final AmazonS3 s3;
    private Logger logger;

    public S3Dao() {
        BasicAWSCredentials creds = new BasicAWSCredentials(accessKey, secretKey);
        ClientConfiguration config = new ClientConfiguration();
        config.setConnectionTimeout(220_000);
        config.setClientExecutionTimeout(220_000);
        this.s3 = AmazonS3ClientBuilder.standard()
                .withClientConfiguration(config)
                .withCredentials(new AWSStaticCredentialsProvider(creds))
                .build();
    }

    public void deleteKeys(Collection<String> s3keysToDelete) {
        logger.log("Deleting S3 " + s3keysToDelete.size() + " keys");
        if (s3keysToDelete.isEmpty()) {
            return;
        }
        DeleteObjectsRequest deleteRequest = new DeleteObjectsRequest(bucketName)
                .withKeys(s3keysToDelete.toArray(new String[] {}));
        DeleteObjectsResult deleteObjectsResult = s3.deleteObjects(deleteRequest);
        logger.log("Deleted " + deleteObjectsResult.getDeletedObjects().size() + " s3 objects");
    }
}

I double checked the access keys, but they're fine.

Does anyone know what this "user key" is, and how I can specify it?

Thanks!


Answer:

One of the keys I was trying to delete was null. Problem solved :)

Question:

I have an AWS Step Function that calls Lambda Functions for each state and I'm passing a LinkedList of PartETag objects from one state to another for a MultipartUpload.

Not sure if it'll help, but here's the code snippet:

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;

import UploadJob;
import UploadJobParameters;

public class JobFinishHandler implements RequestHandler<ExportJobParameters, ExportJobParameters> {

    @Override
    public ExportJobParameters handleRequest(ExportJobParameters parameters, Context context) {

        UploadJob uploadJob = new UploadJob(parameters, context);
        context.getLogger().log("JobFinishHandler");
        context.getLogger().log("parameters: " + parameters.toString());
        context.getLogger().log("parameters.getPartETags().size(): " + parameters.getPartETags().size());
        context.getLogger().log(parameters.getPartETags().toString());
        context.getLogger().log("parameters.getPartETags().hashCode(): " + parameters.getPartETags().hashCode());
        uploadJob.continueS3UploadStream();
        uploadJob.jobFinish();
        uploadJob.jobAbort();
        return parameters;
    }

}

This is the input for the state:

{ "uploadId":"1234567890", "partETags":[ { "partNumber":1, "ETag":"67c335990ba21ff95db1d09a3c2c4a11" } ] }

I get this error:

An error occurred during JSON parsing: java.lang.RuntimeException
java.lang.RuntimeException: An error occurred during JSON parsing
Caused by: java.io.UncheckedIOException: com.fasterxml.jackson.databind.JsonMappingException: No suitable constructor found for type [simple type, class com.amazonaws.services.s3.model.PartETag]: can not instantiate from JSON object (missing default constructor or creator, or perhaps need to add/enable type information?)
at [Source: lambdainternal.util.NativeMemoryAsInputStream@369f73a2; line: 1, column: 458] (through reference chain: gov.osdls.exportservices.export.common.ExportJobParameters["partETags"]->java.util.ArrayList[0])
Caused by: com.fasterxml.jackson.databind.JsonMappingException: No suitable constructor found for type [simple type, class com.amazonaws.services.s3.model.PartETag]: can not instantiate from JSON object (missing default constructor or creator, or perhaps need to add/enable type information?)
at [Source: lambdainternal.util.NativeMemoryAsInputStream@369f73a2; line: 1, column: 458] (through reference chain: gov.osdls.exportservices.export.common.ExportJobParameters["partETags"]->java.util.ArrayList[0])
at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:148)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1106)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:296)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:133)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:245)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:217)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:25)
at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:520)
at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:95)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:258)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:125)
at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:1511)
at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1102)

I checked the PartETag class to see if there's a default constructor and there isn't so I'm going to override this class and make a default constructor. But I feel like since PartETag is an amazon object, shouldn't they have thought of this already? Am I doing this wrong? Is there a better way to do this?


Answer:

Just in case anybody is looking for the answer to this question, I just created a custom class that extends PartETag and converted the list before completing the MultipartUpload:

import com.amazonaws.services.s3.model.PartETag;
import com.amazonaws.util.json.Jackson;

public class PartETagExt extends PartETag {

    public PartETagExt() {
        super(0, "a");
    }

    public PartETagExt(PartETag partETag) {
        super(partETag.getPartNumber(), partETag.getETag());
    }

    @Override
    public String toString() {
        return Jackson.toJsonString(this);
    }
}

then converted the list to regular PartETag:

List<PartETag> partETags = new LinkedList<PartETag>();
partETags.addAll(partETagExts);
CompleteMultipartUploadRequest compRequest = 
                new CompleteMultipartUploadRequest(bucket, key, uploadId, partETags);

Question:

When assuming IAM role using STS client in Java based lambda, did anyone face and solved this error - Connect to sts.amazonaws.com:443 [sts.amazonaws.com/54.239.29.25] failed: connect timed out: com.amazonaws.SdkClientException

I wonder - When I have aws-java-sdk-stsclient dependency in my pom and hence in my shaded jar, why would there be in internet call? Since lambda is in VPC with SG’s configured, internet access is blocked.

Note - I have to assume role in order to access files on S3 from another AWS account. Updating IAM trust policy to have list access is not an option.

Here is dependency I have in my pom -

<!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-sts -->
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-sts</artifactId>
            <version>1.11.163</version>
        </dependency>

And Here is my java class where I am assuming role using sts client -

public class AWSTokenManager {
        public void awsTokenManager() {
            System.out.println("Inside awsTokenManager method");
            STSAssumeRoleSessionCredentialsProvider stsAssumeRoleSessionCredentialsProvider = new STSAssumeRoleSessionCredentialsProvider.Builder("role-arn-here", "us-east-1b")
                    .withStsClient(AWSSecurityTokenServiceClientBuilder.standard().build())
                    .withRoleSessionDurationSeconds(900)
                    .build();
            System.out.println("SessionCredentials awsaccesskeyid is - " + stsAssumeRoleSessionCredentialsProvider.getCredentials().getAWSAccessKeyId());
            System.out.println("SessionCredentials awsaccesskeyid is - " + stsAssumeRoleSessionCredentialsProvider.getCredentials().getAWSSecretKey());
            System.out.println("SessionCredentials awsaccesskeyid is - " + stsAssumeRoleSessionCredentialsProvider.getCredentials().getSessionToken());
            }

Answer:

why would there be in internet call?

Why would there not be? You're accessing the STS service, using the STS client.

Unless you created and configured a VPC Endpoint for STS (which you did not mention), the STS endpoint needs to be accessed via the Internet in order to send the request to call AssumeRole.

Question:


Answer:

Regularly scanning an Amazon S3 bucket is not efficient — buckets can be quite large, requiring many API calls, and work is performed even if no new files have been added.

The better way to implement this is:

  • Configure an Amazon S3 Event that will trigger an AWS Lambda function when a new file is created
  • The AWS Lambda function will receive the Bucket and Key of the new object via the event field, so that it can perform actions on the new object

This is a simple, clean solution compared to running code at regular intervals to scan for new objects.