Hot questions for Using Amazon S3 in encryption

Question:

I am trying for encryption and decryption using amazon aws. I got exception like

 Exception in thread "main" com.amazonaws.AmazonClientException: Unable to build cipher: Illegal key size
    Make sure you have the JCE unlimited strength policy files installed and configured for your JVM
        at  com.amazonaws.services.s3.internal.crypto.ContentCryptoScheme.createCipherLite(ContentCryptoScheme.java:190)
        at com.amazonaws.services.s3.internal.crypto.ContentCryptoMaterial.wrap(ContentCryptoMaterial.java:823)
        at com.amazonaws.services.s3.internal.crypto.S3CryptoModuleBase.buildContentCryptoMaterial(S3CryptoModuleBase.java:535)
        at com.amazonaws.services.s3.internal.crypto.S3CryptoModuleBase.newContentCryptoMaterial(S3CryptoModuleBase.java:483)
        at com.amazonaws.services.s3.internal.crypto.S3CryptoModuleBase.createContentCryptoMaterial(S3CryptoModuleBase.java:449)
        at com.amazonaws.services.s3.internal.crypto.S3CryptoModuleBase.putObjectUsingMetadata(S3CryptoModuleBase.java:165)
        at com.amazonaws.services.s3.internal.crypto.S3CryptoModuleBase.putObjectSecurely(S3CryptoModuleBase.java:159)
        at com.amazonaws.services.s3.internal.crypto.CryptoModuleDispatcher.putObjectSecurely(CryptoModuleDispatcher.java:107)
        at com.amazonaws.services.s3.AmazonS3EncryptionClient.putObject(AmazonS3EncryptionClient.java:485)
        at testKMSkeyUploadObject.main(testKMSkeyUploadObject.java:91)
    Caused by: java.security.InvalidKeyException: Illegal key size
        at javax.crypto.Cipher.checkCryptoPerm(Cipher.java:1039)
        at javax.crypto.Cipher.implInit(Cipher.java:805)
        at javax.crypto.Cipher.chooseProvider(Cipher.java:864)
        at javax.crypto.Cipher.init(Cipher.java:1396)
        at javax.crypto.Cipher.init(Cipher.java:1327)
        at com.amazonaws.services.s3.internal.crypto.ContentCryptoScheme.createCipherLite(ContentCryptoScheme.java:187)
        ... 9 more

please help me.

when I was trying to put object for doing encryption using AmazonS3EncryptionClient I am getting exception. How to resolve this error.

AmazonS3EncryptionClient s3 = new AmazonS3EncryptionClient(credentials,materialProvider);

PutObjectRequest putRequest = new PutObjectRequest(
                            bucket, kms_cmk_id, new ByteArrayInputStream(plaintext), metadata);

ObjectMetadata objectMetadata = new ObjectMetadata();
                    objectMetadata.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION);   

putRequest.setMetadata(objectMetadata);
System.out.println(putRequest.getKey());
s3.putObject(putRequest); //getting exception here

Answer:

It seems the problem is with key size and IMO Amazon has hard coded it somewhere in their code. The solution may be to go for unlimited strength file which you can download from:

Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 6

Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 7 Download

Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 8 Download

Install the file in ${java.home}/jre/lib/security/.

Question:

For copying file in S3, I am using vfs-s3-2.2.1.jar I found S3FileObject class under com.intridea.io.vfs.provider.s3 package. In which I am using public void copyFrom(final FileObject file, final FileSelector selector) method for copy file. In this method I found following code :

try {
    if (srcFile.getType().hasChildren()) {
        destFile.createFolder();
        // do server side copy if both source and dest are in S3 and using same credentials
    } else if (srcFile instanceof S3FileObject) {
        S3FileObject s3SrcFile = (S3FileObject)srcFile;
        String srcBucketName = s3SrcFile.getBucket().getName();
        String srcFileName = s3SrcFile.getS3Key();
        String destBucketName = destFile.getBucket().getName();
        String destFileName = destFile.getS3Key();
        CopyObjectRequest copy = new CopyObjectRequest(
                srcBucketName, srcFileName, destBucketName, destFileName);
        if (srcFile.getType() == FileType.FILE && getServerSideEncryption()) {
            ObjectMetadata meta = s3SrcFile.getObjectMetadata();
            meta.setSSEAlgorithm(AES_256_SERVER_SIDE_ENCRYPTION);
            copy.setNewObjectMetadata(meta);
        }
        getService().copyObject(copy);
    } else if (srcFile.getType().hasContent() && srcFile.getURL().getProtocol().equals("file")) {
        // do direct upload from file to avoid overhead of making a copy of the file
        try {
            File localFile = new File(srcFile.getURL().toURI());
            destFile.upload(localFile);
        } catch (URISyntaxException e) {
            // couldn't convert URL to URI, but should still be able to do the slower way
            super.copyFrom(file, selector);
        }
    } else {
        super.copyFrom(file, selector);
    }
} catch (IOException e) {
    throw new FileSystemException("vfs.provider/copy-file.error", new Object[]{srcFile, destFile}, e);
} catch (AmazonClientException e) {
    throw new FileSystemException("vfs.provider/copy-file.error", new Object[]{srcFile, destFile}, e);
} finally {
    destFile.close();
}

In official reference it uses these method

withSourceSSECustomerKey(sseKey)
withDestinationSSECustomerKey(newSseKey);

In copyFrom method of vfs-s3-2.2.1.jar S3FileObject I can't find any method to set SSECustomerKey How can I achieve the same. Thanks for looking here.


Answer:

I did not test but I look at the lib/code quickly - in https://github.com/abashev/vfs-s3/blob/branch-2.3.x/src/main/java/com/intridea/io/vfs/provider/s3/S3FileSystemConfigBuilder.java there is a method to set the server-side encryption

/**
 * use server-side encryption.
 *
 * @param opts The FileSystemOptions.
 * @param serverSideEncryption true if server-side encryption should be used.
 */
public void setServerSideEncryption(FileSystemOptions opts, boolean serverSideEncryption)
{
    setParam(opts, SERVER_SIDE_ENCRYPTION, serverSideEncryption);
}

so before you're calling the copyFrom you can do

    S3FileSystemConfigBuilder.getInstance().setServerSideEncryption(
        S3FileSystemConfigBuilder.getInstance().getFileSystem().getFileSystemOptions(), 
        true);

Question:

I've been unsuccessful in locating a call that would allow me to create a KMS encrypted bucket in S3 (using the Java AWS SDK).

Does such a method exist? And if so, where can I find examples/documentation?


Answer:

I found the answer. The version of the AWS Java SDK I was using wasn't recent enough to have the method.

Here is how to do it:

    Bucket bucket =  amazonS3Client.createBucket( bucketName );
    ServerSideEncryptionRule serverSideEncryptionRule = new ServerSideEncryptionRule();

    ServerSideEncryptionByDefault serverSideEncryptionByDefault = new ServerSideEncryptionByDefault();
    serverSideEncryptionByDefault.setKMSMasterKeyID( "xxxxxxxxx-xxx-xxxxx-xxxx-xxxxx-xxxx-xxxxxxx" );
    serverSideEncryptionByDefault.setSSEAlgorithm( SSEAlgorithm.KMS.getAlgorithm() );

    serverSideEncryptionRule.setApplyServerSideEncryptionByDefault( serverSideEncryptionByDefault );

    SetBucketEncryptionRequest setBucketEncryptionRequest = new SetBucketEncryptionRequest();
    setBucketEncryptionRequest.setBucketName( bucket.getName() );

    ServerSideEncryptionConfiguration serverSideEncryptionConfiguration = new ServerSideEncryptionConfiguration();

    ArrayList< ServerSideEncryptionRule > serverSideEncryptionRules = new ArrayList<>();
    serverSideEncryptionRules.add( serverSideEncryptionRule );
    serverSideEncryptionConfiguration.setRules( serverSideEncryptionRules );

    setBucketEncryptionRequest.setServerSideEncryptionConfiguration( serverSideEncryptionConfiguration );

    amazonS3Client.setBucketEncryption( setBucketEncryptionRequest );

Question:

I am using the S3 Java API and I want to enable the default encryption (AES-256) option for buckets that I create programmatically through the Java API. I can't seem to find the correct API call or documentation that describes how to enable this option through code. Is there a way to do this through the Java API and not have to explicitly enable it on the S3 Web Console?

I don't have any issues uploading objects that have the default server side encryption set using the ObjectMetadata class, but can't seem to find the option to enable it on the bucket it self. Thanks in advance


Answer:

Looks like my S3 API was out of date. After updating to the latest version (1.11.257) I now have access to com.amazonaws.services.s3.model.SetBucketEncryptionRequest

Question:

I am using vfs s3 plugin to perform file related operation in S3. Official guide for Server-Side Encryption with Customer-Provided Encryption Keys (SSE-C) is here http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html but I am using vfs s3 plugin.

I have checked code of copyFrom method in S3FileObject in com.intridea.io.vfs.provider.s3 package but I didn't found any code to implement Server-Side Encryption with Customer-Provided Encryption Keys (SSE-C) There is option for set Server Side Encryption but how can achieve it with Customer-Provided Encryption Keys

Is there only way to modify code of s3 plugin? or there is any other way to do the same


Answer:

Right now it is not possible to use customer-provided encryption keys with vfs-s3 library but I can take your PR or need some time to implement it - you are first one who have asked for it.

Question:

I want to encrypt stream and then send it using Amazon S3. I'm using legacy code and have two important parameters: non-encrypted InputStream and its length. This is important as AmazonS3Client wants to know the length of the stream before it uploads it.

Encrypting a stream is not very difficult task:

        InputStream in = new FileInputStream("path-to-file");
        KeyGenerator keygen = KeyGenerator.getInstance("AES");
        Key key = keygen.generateKey();

        Cipher cipher = Cipher.getInstance("AES/CBC/NoPadding");
        cipher.init(Cipher.ENCRYPT_MODE, key);

        OutputStream encryptedStream = new ByteArrayOutputStream();
        CipherOutputStream out = new CipherOutputStream(encryptedStream, cipher);
        out.write(in.read());
        out.flush();

        byte[] bytes = ((ByteArrayOutputStream) encryptedStream).toByteArray();
        InputStream encryptedInput = new ByteArrayInputStream(bytes);

However, this approach could be used for small files only as their are written to a byte array held in memory. The thing is that I'd like to encrypt big files (> 1TB). How do I do this? More specifically, I get InputStream and my job is to return encrypted InputStream that will be used by Amazon S3.


Answer:

do not use ByteArrayOutputStream

    InputStream in = new FileInputStream("path-to-file");
    KeyGenerator keygen = KeyGenerator.getInstance("AES");
    Key key = keygen.generateKey();

    Cipher cipher = Cipher.getInstance("AES/CBC/NoPadding");
    cipher.init(Cipher.ENCRYPT_MODE, key);

    try(CipherOutputStream out = new CipherOutputStream(
            new FileOutputStream(new File("path to enc file")), cipher))
    {
        byte[] buffer = new byte[8192];
        int count;
        while ((count = in.read(buffer)) > 0)
        {
            out.write(buffer, 0, count);
        }
    }

    return new FileInputStream(new File("path to enc file"));

One more answer with piped streams.

    InputStream in = new FileInputStream("path-to-file");
    KeyGenerator keygen = KeyGenerator.getInstance("AES");
    Key key = keygen.generateKey();

    Cipher cipher = Cipher.getInstance("AES/CBC/NoPadding");
    cipher.init(Cipher.ENCRYPT_MODE, key);

    PipedOutputStream pout = new PipedOutputStream();
    //write data in a separate thread.
    new Thread(() -> {
        try(CipherOutputStream out = new CipherOutputStream(
                pout, cipher))
        {
            byte[] buffer = new byte[8192];
            int count;
            while ((count = in.read(buffer)) > 0)
            {
                out.write(buffer, 0, count);
            }
        } catch (IOException  e)
        {
            e.printStackTrace();
        }
    }).start();

    return new PipedInputStream(pout);

Question:

I have created a java service to return the meta-data of the S3 objects but in my json response it's not returning the Server-side encryption and Object URL

Here is my code

 @GetMapping(value = "/ListS3BucketsObjects")
    public List<S3ObjectSummary> ListS3BucketsObjects(@RequestParam String bucketName) {
        try {
            BasicAWSCredentials creds = new BasicAWSCredentials("xxxx",
                    "xxxxxxx");
            final AmazonS3 s3client = AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(creds))
                    .withRegion(Regions.US_EAST_1).build();
            ObjectListing objectListing = s3client.listObjects(bucketName);
            return objectListing.getObjectSummaries();
        } catch (Exception Ex) {
            throw Ex;
        }
    }

I'm getting result like this

[
    {
        "bucketName": "xxxx",
        "key": "xx",
        "size": 489583,
        "lastModified": "2020-01-01T06:59:52.000+0000",
        "storageClass": "STANDARD",
        "owner": {
            "displayName": "xxxxx",
            "id": "xxx"
        },
        "etag": "xxxxx"
    }]

how to get the Object URL and Server-side encryption?


Answer:

Check this answer. Based S3ObjectSummary you can retrieve metadata:

ObjectMetadata objectMetadata = s3client.getObjectMetadata(bucketName, objectSummary.getKey());