Hot questions for Using Amazon S3 in json

Question:

Is it possible to store JSON data into Amazon S3? Lets say that I want to store this JSON data:

{
    "glossary": {
        "title": "example glossary",
        "GlossDiv": {
            "title": "S",
            "GlossList": {
                "GlossEntry": {
                    "ID": "SGML",
                    "SortAs": "SGML",
                    "GlossTerm": "Standard Generalized Markup Language",
                    "Acronym": "SGML",
                    "Abbrev": "ISO 8879:1986",
                    "GlossDef": {
                        "para": "A meta-markup language, used to create markup languages such as DocBook.",
                        "GlossSeeAlso": ["GML", "XML"]
                    },
                    "GlossSee": "markup"
                }
            }
        }
    }
}

I checked here says it is possible but then it is using jQuery but I could not find the corrresponding thing in Java. Even if it is possible, in what form will the JSON be stored? Will it be dumped into a file?


Answer:

Yes, you're pretty much dealing with bytes here so whatever format these bytes represent has no impact at all.

In your java app, convert whatever object you have into bytes then stream that out directly (or write to a file first, then upload). Sample code:

 ObjectMapper objectMapper = new ObjectMapper(); 
 byte[] bytesToWrite = objectMapper.writeValueAsBytes(yourObject)

 ObjectMetadata omd = new ObjectMetadata();
 omd.setContentLength(bytesToWrite.length);
 transferManager.upload(bucketName, filename, new ByteArrayInputStream(bytesToWrite), omd);

The java client can be found here: https://aws.amazon.com/sdk-for-java/

Question:

I have a very simple call to just create a new AmazonS3Client (using Java AWSSDK 1.10.12) like this:

AmazonS3 s3Client = new AmazonS3Client();

And during this call I'm getting the exception message from the AWSDK: Internalconfig.java - "awssdk_config_override.json file not found"

Actual message: 6656 [http-bio-8080-exec-8] DEBUG com.amazonaws.internal.config.InternalConfig -Configuration override awssdk_config_override.json not found.

Totally baffled what's going on here. I can currently only reproduce this on one project, but can't reproduce it when creating new projects from scratch.

Any idea what this json file is even used for? Or why it can't find it or feels the need to find it?


Answer:

The SDK comes bundled with a file called awssdk_config_default.json with various bits of trivia about how to find and authenticate to the different AWS services. When loading up, it also looks for a file on the classpath called "awssdk_config_override.json" that you can optionally provide to override these settings - for example if you want to use a different authentication protocol than what the SDK chooses by default.

This file is not at all required, and almost all users of the SDK will be perfectly happy with the default configuration. The SDK logs a message at the DEBUG level if it can't find the file to help debug what's going wrong if you're expecting it to pick up an override config file but it's not for some reason. This message (and all of the other messages the SDK logs at the DEBUG level) are safe to ignore unless you're trying to figure out why something isn't working.

I would definitely recommend turning off DEBUG level logging before deploying to production - the DEBUG logs can get quite verbose.

Question:

I cant figure out how to read a JSON file from S3 into memory as String.

The examples I find calls getObjectContent() however this is not available for the GetObjectResponse I get from the S3AsyncClient.

The code I experiment is the sample code from AWS.

// Creates a default async client with credentials and AWS Region loaded from the
// environment
S3AsyncClient client = S3AsyncClient.create();

// Start the call to Amazon S3, not blocking to wait for the result
CompletableFuture<GetObjectResponse> responseFuture =
        client.getObject(GetObjectRequest.builder()
                                         .bucket("my-bucket")
                                         .key("my-object-key")
                                         .build(),
                         AsyncResponseTransformer.toFile(Paths.get("my-file.out")));

// When future is complete (either successfully or in error), handle the response
CompletableFuture<GetObjectResponse> operationCompleteFuture =
        responseFuture.whenComplete((getObjectResponse, exception) -> {
            if (getObjectResponse != null) {
                // At this point, the file my-file.out has been created with the data
                // from S3; let's just print the object version
                System.out.println(getObjectResponse.versionId());
            } else {
                // Handle the error
                exception.printStackTrace();
            }
        });

// We could do other work while waiting for the AWS call to complete in
// the background, but we'll just wait for "whenComplete" to finish instead
operationCompleteFuture.join();

How should this code be modified so that I can get the actual JSON content from the GetObjectResponse?


Answer:

After response is transformed to bytes it can be transformed to string:

S3AsyncClient client = S3AsyncClient.create();

GetObjectRequest getObjectRequest = GetObjectRequest.builder().bucket("my-bucket").key("my-object-key").build();

client.getObject(getObjectRequest, AsyncResponseTransformer.toBytes())
      .thenApply(ResponseBytes::asUtf8String)
      .whenComplete((stringContent, exception) -> {
          if (stringContent != null)
              System.out.println(stringContent);
          else
              exception.printStackTrace();
      });

Question:

I am developing an app that is able to read from JSON file located on AWS S3 servers in a bucket with read and write permissions. App also allows to edit some of the values in that JSON file.

Problem

Problem is that everyone with the JSON file URL can then alter the included data. I only want the file to be modified from this app.

JSON file

{
  "females": [
    {
      "id": 1,
      "name": "First Name",
      "actions": [
        {
          "action_1": 123,
          "action_2": 456,
          "action_3": 789
        }
      ]
    }, ...    
}

Users are able to modify values located withing actions array.

Any recommendations on how to limit write access of the JSON file only to this app?


Answer:

Cloudfront will not help here.

Solution is

  • authenticate your application's users (Amazon Cognito will help you) OR use Amazon Cognito with Anonymous User.

  • Give your Amazon Cognito role the permission to read and write on your S3 bucket

  • remove access policies from S3 (to make the bucket private)

That way, only application users will be able to read and write the files.

Question:

I'm trying to convert an Object to JSON then convert it to File to be able to send it to AWS S3.

Is there a way to convert the String in the most efficient way? Thank you!

Here is my code:

String messageBody = new Gson().toJson(thisIsADtoObject);

And for the S3

PutObjectRequest request = new PutObjectRequest(s3BucketName, key, file);
        amazonS3.putObject(request);

Answer:

As far as I know, to create a file object to send to AWS, you will have to create the actual file on disk, e.g. with a PrintStream:

File file = new File("path/to/your/file.name");
try (PrintStream out = new PrintStream(new FileOutputStream(file))) {
    out.print(messageBody);
} 

Instead of using the constructor taking a file, you might want to use the one which takes an InputStream:

PutObjectRequest request = new PutObjectRequest(s3BucketName, key, inputStream, metadata);
        amazonS3.putObject(request);

To convert a String to an InputStream, use

new ByteArrayInputStream(messageBody.getBytes(StandardCharsets.UTF_8));

Link to SDK JavaDoc