Hot questions for Using Amazon S3 in maven

Question:

I have a private repository in S3 that I use to release a library that then I use in another project.

To release, I have this setup:

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.7.0</version>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
        </plugin>
    </plugins>

    <extensions>
        <extension>
            <groupId>com.allogy.maven.wagon</groupId>
            <artifactId>maven-s3-wagon</artifactId>
            <version>1.1.0</version>
        </extension>
    </extensions>
</build>

<distributionManagement>
    <repository>
        <id>projectx-aws-release</id>
        <name>ProjectX AWS Release Repository</name>
        <url>s3://projectx-support/maven2</url>
    </repository>
    <snapshotRepository>
        <id>projectx-aws-snapshots</id>
        <name>Projectx AWS Snapshot Repository</name>
        <url>s3://projectx-support/maven2</url>
    </snapshotRepository>
</distributionManagement>

and then, on the application that's using that library:

<build>
    ...

    <extensions>
        <extension>
            <groupId>com.allogy.maven.wagon</groupId>
            <artifactId>maven-s3-wagon</artifactId>
            <version>1.1.0</version>
        </extension>
    </extensions>
</build>

<repositories>
    <repository>
        <id>projectX-aws</id>
        <url>s3://projectX-support/maven2</url>
        <releases><enabled>true</enabled></releases>
        <snapshots><enabled>true</enabled></snapshots>
    </repository>
</repositories>

with .circleci/settings.xml having the credentials for the S3 repo:

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
    <servers>
        <server>
            <id>projectx-aws</id>
            <username>....</username>
            <password>....</password>
        </server>
    </servers>
</settings>

When running mvn --batch-mode --settings .circleci/settings.xml dependency:go-offline something odd happens. It quickly downloads some dependencies:

mvn --batch-mode --settings .circleci/settings.xml dependency:go-offline
[INFO] Scanning for projects...
Downloading: http://repo.maven.apache.org/maven2/com/allogy/maven/wagon/maven-s3-wagon/1.1.0/maven-s3-wagon-1.1.0.pom
Downloaded: http://repo.maven.apache.org/maven2/com/allogy/maven/wagon/maven-s3-wagon/1.1.0/maven-s3-wagon-1.1.0.pom (7 KB at 47.0 KB/sec)
Downloading: http://repo.maven.apache.org/maven2/com/allogy/allogy-maven-pom-oss/2.0/allogy-maven-pom-oss-2.0.pom
Downloaded: http://repo.maven.apache.org/maven2/com/allogy/allogy-maven-pom-oss/2.0/allogy-maven-pom-oss-2.0.pom (10 KB at 808.4 KB/sec)
Downloading: http://repo.maven.apache.org/maven2/com/google/guava/guava/15.0/guava-15.0.pom
Downloaded: http://repo.maven.apache.org/maven2/com/google/guava/guava/15.0/guava-15.0.pom (7 KB at 508.8 KB/sec)
.
.
.
Downloading: http://repo.maven.apache.org/maven2/org/apache/apache/16/apache-16.pom
Downloaded: http://repo.maven.apache.org/maven2/org/apache/apache/16/apache-16.pom (16 KB at 1879.5 KB/sec)
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/2.10/maven-dependency-plugin-2.10.jar
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/2.10/maven-dependency-plugin-2.10.jar (157 KB at 12014.5 KB/sec)

then it shows this:

[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building projectx 1.0.0-beta.2
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] >>> maven-dependency-plugin:2.10:go-offline (default-cli) @ projectx >>>
[INFO] 
[INFO] --- maven-dependency-plugin:2.10:resolve-plugins (resolve-plugins) @ projectx ---

and downloads some more deps:

Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact/2.2.1/maven-artifact-2.2.1.pom
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact/2.2.1/maven-artifact-2.2.1.pom (2 KB at 171.5 KB/sec)
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/maven/2.2.1/maven-2.2.1.pom
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/maven/2.2.1/maven-2.2.1.pom (22 KB at 2431.5 KB/sec)
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/11/maven-parent-11.pom
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/11/maven-parent-11.pom (32 KB at 3516.6 KB/sec)
.
.
.
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.6/maven-resources-plugin-2.6.jar
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.6/maven-resources-plugin-2.6.jar (29 KB at 4118.2 KB/sec)
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-jar-plugin/2.2/maven-jar-plugin-2.2.jar
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-jar-plugin/2.2/maven-jar-plugin-2.2.jar (27 KB at 3252.7 KB/sec)
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-install-plugin/2.3/maven-install-plugin-2.3.jar
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-install-plugin/2.3/maven-install-plugin-2.3.jar (23 KB at 2775.0 KB/sec)

but then it attempts to download jars that should be and are in Maven Central from the S3 repository:

[INFO] Plugin Resolved: copy-rename-maven-plugin-1.0.1.jar
Downloading: s3://projectx-support/maven2/com/coderplus/maven/plugins/copy-rename-maven-plugin/1.0.1/copy-rename-maven-plugin-1.0.1.pom
[INFO] File permissions: Private
[INFO] Logged in - projectx-support
[INFO] Downloading: s3://projectx-support/maven2/com/coderplus/maven/plugins/copy-rename-maven-plugin/1.0.1/copy-rename-maven-plugin-1.0.1.pom
[INFO] Logged off - projectx-support
[INFO] Transfers: 1 Time: 27ms Amount: 0 bytes Throughput: 0.000 bytes/s
Downloading: http://repo.maven.apache.org/maven2/com/coderplus/maven/plugins/copy-rename-maven-plugin/1.0.1/copy-rename-maven-plugin-1.0.1.pom
Downloaded: http://repo.maven.apache.org/maven2/com/coderplus/maven/plugins/copy-rename-maven-plugin/1.0.1/copy-rename-maven-plugin-1.0.1.pom (5 KB at 418.0 KB/sec)
Downloading: s3://projectx-support/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
[INFO] File permissions: Private
[INFO] Logged in - projectx-support
[INFO] Downloading: s3://projectx-support/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
[INFO] Logged off - projectx-support
[INFO] Transfers: 1 Time: 11ms Amount: 0 bytes Throughput: 0.000 bytes/s
Downloading: http://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
Downloaded: http://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom (7 KB at 712.8 KB/sec)
Downloading: s3://projectx-support/maven2/org/apache/maven/maven-model/2.0.9/maven-model-2.0.9.jar
[INFO] File permissions: Private
[INFO] Logged in - projectx-support
[INFO] Downloading: s3://projectx-support/maven2/org/apache/maven/maven-model/2.0.9/maven-model-2.0.9.jar
[INFO] Logged off - projectx-support
[INFO] Transfers: 1 Time: 12ms Amount: 0 bytes Throughput: 0.000 bytes/s
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/maven-model/2.0.9/maven-model-2.0.9.jar
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/maven-model/2.0.9/maven-model-2.0.9.jar (86 KB at 9475.0 KB/sec)
Downloading: s3://projectx-support/maven2/org/apache/maven/maven-project/2.0.9/maven-project-2.0.9.jar
[INFO] File permissions: Private
[INFO] Logged in - projectx-support
[INFO] Downloading: s3://projectx-support/maven2/org/apache/maven/maven-project/2.0.9/maven-project-2.0.9.jar
[INFO] Logged off - projectx-support
[INFO] Transfers: 1 Time: 12ms Amount: 0 bytes Throughput: 0.000 bytes/s
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/maven-project/2.0.9/maven-project-2.0.9.jar
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/maven-project/2.0.9/maven-project-2.0.9.jar (119 KB at 11892.6 KB/sec)
Downloading: s3://projectx-support/maven2/org/apache/maven/maven-core/2.0.9/maven-core-2.0.9.jar

After a while of that, AWS starts to throw Connection refused errors (some protection system maybe?) and the dependency that is only present in the S3 repo is not downloaded:

Downloading: s3://projectx-support/maven2/tech/projectx/projectxcommon/1.0.0-SNAPSHOT/maven-metadata.xml
[WARN] s3://projectx-support/maven2 - Connection refused
[INFO] Logged off - projectx-support
[WARNING] Could not transfer metadata tech.projectx:projectxcommon:1.0.0-SNAPSHOT/maven-metadata.xml from/to projectx-aws (s3://projectx-support/maven2): Could not connect to repository
[WARNING] Failure to transfer tech.projectx:projectxcommon:1.0.0-SNAPSHOT/maven-metadata.xml from s3://projectx-support/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of projectx-aws has elapsed or updates are forced. Original error: Could not transfer metadata tech.projectx:projectxcommon:1.0.0-SNAPSHOT/maven-metadata.xml from/to projectx-aws (s3://projectx-support/maven2): Could not connect to repository
Downloading: s3://projectx-support/maven2/tech/projectx/projectxcommon/1.0.0-SNAPSHOT/projectxcommon-1.0.0-SNAPSHOT.pom
[WARN] s3://projectx-support/maven2 - Connection refused
[INFO] Logged off - projectx-support

which causes the compilation to fail:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 17.929s
[INFO] Finished at: Wed Apr 18 16:03:19 UTC 2018
[INFO] Final Memory: 32M/1067M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project projectx: Could not resolve dependencies for project tech.projectx:projectx:jar:1.0.0-beta.2: Failed to collect dependencies for [tech.projectx:projectxcommon:jar:1.0.0-SNAPSHOT (compile), org.projectlombok:lombok:jar:1.16.20 (compile), org.springframework:spring-web:jar:5.0.5.RELEASE (compile), org.springframework.retry:spring-retry:jar:1.2.2.RELEASE (compile), com.fasterxml.jackson.core:jackson-databind:jar:2.9.5 (compile), org.kordamp.ikonli:ikonli-javafx:jar:2.1.1 (compile), org.kordamp.ikonli:ikonli-fontawesome5-pack:jar:2.1.1 (compile), net.harawata:appdirs:jar:1.0.1 (compile), io.sentry:sentry:jar:1.7.3 (compile), org.javassist:javassist:jar:3.22.0-CR2 (compile), junit:junit:jar:4.12 (test)]: Failed to read artifact descriptor for tech.projectx:projectxcommon:jar:1.0.0-SNAPSHOT: Could not transfer artifact tech.projectx:projectxcommon:pom:1.0.0-SNAPSHOT from/to projectx-aws (s3://projectx-support/maven2): Could not connect to repository: Couldn't initialize a SAX driver to create an XMLReader: SAX2 driver class org.apache.xerces.parsers.SAXParser not found -> [Help 1]

Any ideas what's going on? This seems to be non-deterministic. It comes and goes with changes that shouldn't have any effect on it, although it seems to be quite consistent in that re-building doesn't seem to change the outcome. I'm sure the credentials work because they are used in two projects and one is working and they are the same that were used in this project when it was working.

This has been happening since I switched from Gradle to Maven, but since that's a completely different system, it's not a very useful comparison I believe.

When I run mvn --batch-mode --settings .circleci/settings.xml dependency:go-offline install -e -X on my own computer, after deleting ~/.m2/repository, the output looks like this: https://gist.github.com/pupeno/da4f53a6e9673ff611909021306191a4#file-maven-local-machine-log and when I do the same in CircleCI, it looks like this: https://gist.github.com/pupeno/da4f53a6e9673ff611909021306191a4#file-maven-circleci-log

When listing the repositories present, following the answer at How can I ask Maven for a list of the default repositories?, I see this:

<repositories>
    <repository>
        <id>projectx-aws</id>
        <url>s3://projectx-support/maven2</url>
        <layout>default</layout>
        <locations class="linked-hash-map">
            <entry>
                <string></string>
                <inputLocation>
                    <lineNumber>461</lineNumber>
                    <columnNumber>21</columnNumber>
                    <source>
                        <modelId>tech.projectx:projectx:1.0.0-beta.2</modelId>
                        <location>C:\Users\pupeno\Documents\projectx\code\projectx\pom.xml</location>
                    </source>
                </inputLocation>
            </entry>
            <entry>
                <string>id</string>
                <inputLocation>
                    <lineNumber>462</lineNumber>
                    <columnNumber>17</columnNumber>
                    <source reference="../../../entry/inputLocation/source"/>
                </inputLocation>
            </entry>
            <entry>
                <string>url</string>
                <inputLocation>
                    <lineNumber>463</lineNumber>
                    <columnNumber>18</columnNumber>
                    <source reference="../../../entry/inputLocation/source"/>
                </inputLocation>
            </entry>
        </locations>
        <releases>
            <enabled>true</enabled>
            <locations class="linked-hash-map">
                <entry>
                    <string></string>
                    <inputLocation>
                        <lineNumber>464</lineNumber>
                        <columnNumber>23</columnNumber>
                        <source reference="../../../../../locations/entry/inputLocation/source"/>
                    </inputLocation>
                </entry>
                <entry>
                    <string>enabled</string>
                    <inputLocation>
                        <lineNumber>465</lineNumber>
                        <columnNumber>26</columnNumber>
                        <source reference="../../../../../locations/entry/inputLocation/source"/>
                    </inputLocation>
                </entry>
            </locations>
        </releases>
        <snapshots>
            <enabled>true</enabled>
            <locations class="linked-hash-map">
                <entry>
                    <string></string>
                    <inputLocation>
                        <lineNumber>467</lineNumber>
                        <columnNumber>24</columnNumber>
                        <source reference="../../../../../locations/entry/inputLocation/source"/>
                    </inputLocation>
                </entry>
                <entry>
                    <string>enabled</string>
                    <inputLocation>
                        <lineNumber>468</lineNumber>
                        <columnNumber>26</columnNumber>
                        <source reference="../../../../../locations/entry/inputLocation/source"/>
                    </inputLocation>
                </entry>
            </locations>
        </snapshots>
    </repository>
    <repository>
        <id>central</id>
        <name>Central Repository</name>
        <url>https://repo.maven.apache.org/maven2</url>
        <layout>default</layout>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
    </repository>
</repositories>

s3://projectx-support/maven2/tech/projectx/projectxcommon/1.0.0-SNAPSHOT/maven-metadata.xml is present and contains the following:

<?xml version="1.0" encoding="UTF-8"?>
<metadata modelVersion="1.1.0">
  <groupId>tech.projectx</groupId>
  <artifactId>projectxcommon</artifactId>
  <version>1.0.0-SNAPSHOT</version>
  <versioning>
    <snapshot>
      <timestamp>20180418.113836</timestamp>
      <buildNumber>5</buildNumber>
    </snapshot>
    <lastUpdated>20180418113836</lastUpdated>
    <snapshotVersions>
      <snapshotVersion>
        <extension>jar</extension>
        <value>1.0.0-20180418.113836-5</value>
        <updated>20180418113836</updated>
      </snapshotVersion>
      <snapshotVersion>
        <extension>pom</extension>
        <value>1.0.0-20180418.113836-5</value>
        <updated>20180418113836</updated>
      </snapshotVersion>
    </snapshotVersions>
  </versioning>
</metadata>

The dependency tree of my app looks like this:

> mvn dependency:tree
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------< tech.projectx:projectx >------------------------
[INFO] Building projectx 1.0.0-beta.2
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:2.10:tree (default-cli) @ projectx ---
[INFO] tech.projectx:projectx:jar:1.0.0-beta.2
[INFO] +- tech.projectx:projectxcommon:jar:1.0.0-SNAPSHOT:compile
[INFO] |  +- org.bouncycastle:bcprov-jdk15on:jar:1.59:compile
[INFO] |  +- javax.validation:validation-api:jar:2.0.1.Final:compile
[INFO] |  +- org.hibernate.validator:hibernate-validator:jar:6.0.9.Final:compile
[INFO] |  |  +- org.jboss.logging:jboss-logging:jar:3.3.2.Final:compile
[INFO] |  |  \- com.fasterxml:classmate:jar:1.3.4:compile
[INFO] |  +- org.glassfish:javax.el:jar:3.0.1-b10:compile
[INFO] |  +- org.springframework:spring-context:jar:5.0.5.RELEASE:compile
[INFO] |  |  +- org.springframework:spring-aop:jar:5.0.5.RELEASE:compile
[INFO] |  |  \- org.springframework:spring-expression:jar:5.0.5.RELEASE:compile
[INFO] |  +- com.pubnub:pubnub-gson:jar:4.19.0:compile
[INFO] |  |  +- com.squareup.retrofit2:retrofit:jar:2.3.0:compile
[INFO] |  |  +- com.squareup.okhttp3:okhttp:jar:3.9.1:compile
[INFO] |  |  |  \- com.squareup.okio:okio:jar:1.13.0:compile
[INFO] |  |  +- com.squareup.okhttp3:logging-interceptor:jar:3.9.1:compile
[INFO] |  |  +- com.google.code.gson:gson:jar:2.8.2:compile
[INFO] |  |  \- com.squareup.retrofit2:converter-gson:jar:2.3.0:compile
[INFO] |  +- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.9.5:compile
[INFO] |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.9.5:compile
[INFO] |  \- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.9.5:compile
[INFO] +- org.projectlombok:lombok:jar:1.16.20:compile
[INFO] +- org.springframework:spring-web:jar:5.0.5.RELEASE:compile
[INFO] |  +- org.springframework:spring-beans:jar:5.0.5.RELEASE:compile
[INFO] |  \- org.springframework:spring-core:jar:5.0.5.RELEASE:compile
[INFO] |     \- org.springframework:spring-jcl:jar:5.0.5.RELEASE:compile
[INFO] +- org.springframework.retry:spring-retry:jar:1.2.2.RELEASE:compile
[INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.9.5:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.0:compile
[INFO] |  \- com.fasterxml.jackson.core:jackson-core:jar:2.9.5:compile
[INFO] +- org.kordamp.ikonli:ikonli-javafx:jar:2.1.1:compile
[INFO] |  \- org.kordamp.ikonli:ikonli-core:jar:2.1.1:compile
[INFO] +- org.kordamp.ikonli:ikonli-fontawesome5-pack:jar:2.1.1:compile
[INFO] +- net.harawata:appdirs:jar:1.0.1:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:1.7.25:compile
[INFO] |  \- net.java.dev.jna:jna-platform:jar:4.5.0:compile
[INFO] |     \- net.java.dev.jna:jna:jar:4.5.0:compile
[INFO] +- io.sentry:sentry:jar:1.7.3:compile
[INFO] +- org.javassist:javassist:jar:3.22.0-CR2:compile
[INFO] \- junit:junit:jar:4.12:test
[INFO]    \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.188 s
[INFO] Finished at: 2018-04-23T20:38:39+01:00
[INFO] ------------------------------------------------------------------------

My .cicleci\config.yml file looks like this:

# Java Gradle CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-java/ for more details
#
version: 2
jobs:
  build:
    docker:
      # specify the version you desire here
      - image: circleci/jdk8:0.1.1

      # Specify service dependencies here if necessary
      # CircleCI maintains a library of pre-built images
      # documented at https://circleci.com/docs/2.0/circleci-images/
      # - image: circleci/postgres:9.4

    working_directory: ~/repo

    environment:
      # Customize the JVM maximum heap limit
      JVM_OPTS: -Xmx3200m
      TERM: dumb

    steps:
       - checkout

       - run: apt-get update

       - run: apt-get install --assume-yes maven

       - restore_cache:
           key: projectx-{{ checksum "pom.xml" }}

       - run: mvn --batch-mode --settings .circleci/settings.xml dependency:go-offline install -e -X

       - save_cache:
           paths:
             - ~/.m2
           key: projectx-{{ checksum "pom.xml" }}

       - run: mvn --batch-mode --settings .circleci/settings.xml test

I have logging enabled for this bucket and one entry looks like this (with the expected 404):

https://gist.github.com/pupeno/da4f53a6e9673ff611909021306191a4#file-2018-04-23-21-10-39-7467973073c586b9

These are the last 3 accesses to maven-metadata.xml:


Answer:

Probably main problem was with too many requests to Amazon S3 repository.

By analysing logs we can see over 120 request of downloading files from Amazon S3 in short time 1-3 seconds:

  • Maven is downloading artefacts that don't exist in Amazon S3, then download it from Maven Central, those requests to Amazon S3 are useless
  • Every download needs a login and logout request, that's strange!
  • Last 40 request are connection refused
  • Last 40 request don't have [INFO] Logged in - projectx-support statement for them, so maybe some limits of log in in short time?

To solve the problem, there are atleast 2 solutions:

  1. Force maven to download artefact FIRSTLY from Maven Central, then rest from Amazon S3, to decrease requests to Amazon S3. This is a good solutions, but remember if you will have more artefacts to download only from Amazon S3 you will get same problem!

    By default Maven Central comes LAST in repository, to set highest priority and download first from Maven Center, we have to add it manually in FIRST position in our repositories, because the order of the repositories will also DECIDE the order of the repository access. (Source):

    <repositories>
        <repository>
            <id>central</id>
            <name>Central Repository</name>
            <url>https://repo.maven.apache.org/maven2</url>
            <layout>default</layout>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>projectX-aws</id>
            <url>s3://projectX-support/maven2</url>
            <releases><enabled>true</enabled></releases>
            <snapshots><enabled>true</enabled></snapshots>
        </repository>
    </repositories>
    
  2. Find a way to change limits of requests by IP on Amazon S3 repository. This solution will always works but your Amazon S3 will get many useless requests.

So the best way to solve the problem is to mix this two solutions. :)

Question:

When I try to create bucket at aws s3, get this error: Exception in thread "main" java.lang.NoSuchFieldError: JAVA_VENDOR at software.amazon.awssdk.core.internal.util.UserAgentUtils.userAgent(UserAgentUtils.java:87) at software.amazon.awssdk.core.internal.util.UserAgentUtils.initializeUserAgent(UserAgentUtils.java:73).

I have openjdk 11; results of java -version command

openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment 18.9 (build 11.0.1+13)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.1+13, mixed mode)

Result of mvn -version command: ...Apache Maven 3.5.4...

This is my pom file:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>c</groupId>
    <artifactId>e</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>

        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3</artifactId>
            <version>2.5.19</version>
        </dependency>

        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>regions</artifactId>
            <version>2.5.19</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-nop -->
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-nop</artifactId>
            <version>1.7.25</version>
        </dependency>
    </dependencies>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <version>2.0.0</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

Simple create bucket example from aws:

import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.CreateBucketConfiguration;
import software.amazon.awssdk.services.s3.model.CreateBucketRequest;

public class Main {


    public static void main(String[] args) {
        System.setProperty("JAVA_VENDOR","Oracle Corporation");

        Region region = Region.US_WEST_2;
        S3Client s3 = S3Client.builder().region(region).build();
        String bucket = "bucket" + System.currentTimeMillis();
        CreateBucketRequest createBucketRequest = CreateBucketRequest
                .builder()
                .bucket(bucket)
                .createBucketConfiguration(CreateBucketConfiguration.builder()
                        .locationConstraint(region.id())
                        .build())
                .build();
        s3.createBucket(createBucketRequest);
    }
}

I made some efforts to win it: 1) try this : System.setProperty("JAVA_VENDOR","Oracle Corporation");, because the real system property has nave "java.vendor" 2) I found problem place in at software.amazon.awssdk.core.internal.util.UserAgentUtils.userAgent: it calls JAVA_VENDOR from software.amazon.awssdk.utils.JavaSystemSettings

.replace("{java.vendor}", sanitizeInput(JavaSystemSetting.JAVA_VENDOR.getStringValue().orElse(null)))

But there is no field with such name in the JavaSystemSettings class. How can i handle it?

TR; DR UPDATE! It work for me with next maven pom file:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>c</groupId>
<artifactId>e</artifactId>
<version>1.0-SNAPSHOT</version>

<dependencies>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-nop</artifactId>
        <version>1.7.25</version>
    </dependency>
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>sdk-core</artifactId>
        <version>2.5.19</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/software.amazon.awssdk/s3 -->
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>s3</artifactId>
        <version>2.5.19</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/software.amazon.awssdk/regions -->
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>regions</artifactId>
        <version>2.5.19</version>
    </dependency>
</dependencies>
<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>bom</artifactId>
            <version>2.5.19</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
        </plugin>
    </plugins>
</build>


Answer:

A NoSuchFieldError is caused by a version mismatch when loading a class.

(Setting properties won't fix this. The exception refers to a field of a class, not a property in a Properties object.)

But there is no field with such name in the JavaSystemSettings class.

Exactly! You have a version mismatch between JARs containing the UserAgentUtils class and the JavaSystemSettings class.

Question:

I've got a file and get the error mentioned in the title very early:

import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.Bucket;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.PutObjectRequest;
import com.amazonaws.services.s3.model.PutObjectResult;
import java.io.File;

public static void main(String[] args) throws Exception {
    createAndPopulateSimpleBucket();
}

public static void createAndPopulateSimpleBucket() throws Exception {
    AmazonS3 s3client = new AmazonS3ClientBuilder().standard().build();
}

I get the error in the title when I call new AmazonS3ClientBuilder().

I'm new to Maven, and I think I have my pom set up correctly. Here it is:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>groupId</groupId>
    <artifactId>AWSJavaHelloWorld</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>com.amazonaws</groupId>
                <artifactId>aws-java-sdk-bom</artifactId>
                <version>1.11.715</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>

        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-s3</artifactId>
        </dependency>

    </dependencies>
</project>

I'm guessing something is wrong with my pom, but I can't figure out what. I also get the same error when I include everything from the AWS SDK (instead of only really including the S3 dependency).

So, can anyone spot what's wrong?


Answer:

The documentation of AmazonS3ClientBuilder states that this class does not expose a public no-arg constructor. This means that you cannot invoke new AmazonS3ClientBuilder() from your class.

Fortunately, this class provides two static factory methods; one of these methods, AmazonS3ClientBuilder#defaultClient, creates an AmazonS3 instance (the client), and the other method, AmazonS3ClientBuilder#standard, creates an instance of the builder.

Knowing this, you can replace your code with one of the following snippets:

public static void createAndPopulateSimpleBucket() throws Exception {
    AmazonS3 s3client = AmazonS3ClientBuilder.defaultClient();
}

Or:

public static void createAndPopulateSimpleBucket() throws Exception {
    AmazonS3 s3client = AmazonS3ClientBuilder.standard().build();
}

Question:

I am working on my first time using maven to build a java project. This project requires a jar project that isn't hosted on the maven repository, so I am going to use S3 as a remote repository.

An issue that I appear to be running into is that the the S3 credentials must be stored in ~/.m2/settings.xml. I am not the only person working on this project, so ideally, I would like to keep those credentials in a per project basis, as opposed to a per user basis, as to ease the automation of checking out and building of the project. Is this possible?


Answer:

Automation can be done within with a appropriate plugin Config File Provider which supports different settings.xml/toolchains.xm etc.

Apart from that create a git repository from your ${HOME}/.m2 including the settings.xml which stores the information you need so anyone else can clone this repository...

Question:

While executing the SWF in maven project getting the following error, can u please help me how to resolve the this error

ERROR execute, Failure processing activity task with taskId=6, workflowGenerationId=MainWorkflowPCompress, activity={Name: Activity.DownloadFromS3,Version: 22.0}, activityInstanceId=1 com.amazonaws.services.simpleworkflow.flow.ActivityFailureException: com.amazonaws.AmazonWebServiceRequest.copyPrivateRequestParameters()Ljava/util/Map; : ["java.lang.NoSuchMethodError",{"cause":null,"stackTrace":[{"methodName":"invoke","fileName":"AmazonS3Client.java","lineNumber":3651,"className":"com.amazonaws.services.s3.AmazonS3Client","nativeMethod":false},{"methodName":"getObject","fileName":"AmazonS3Client.java","lineNumber":1160,"className":"com.amazonaws.services.s3.AmazonS3Client","nativeMethod":false},{"methodName":"getS3ObjectStream","fileName":"AmazonS3Client.java","lineNumber":1248,"className":"com.amazonaws.services.s3.AmazonS3Client$3","nativeMethod":false},{"methodName":"retryableDownloadS3ObjectToFile","fileName":"ServiceUtils.java","lineNumber":382,"className":"com.amazonaws.services.s3.internal.ServiceUtils","nativeMethod":false},{"methodName":"getObject","fileName":"AmazonS3Client.java","lineNumber":1244,"className":"com.amazonaws.services.s3.AmazonS3Client","nativeMethod":false},{"methodName":"DownloadFromS3","fileName":"ActivityImpl.java","lineNumber":135,"className":"com.quantiphi.Pagination.Compress.ActivityImpl","nativeMethod":false},{"methodName":"invoke0","fileName":null,"lineNumber":-2,"className":"sun.reflect.NativeMethodAccessorImpl","nativeMethod":true},{"methodName":"invoke","fileName":null,"lineNumber":-1,"className":"sun.reflect.NativeMethodAccessorImpl","nativeMethod":false},{"methodName":"invoke","fileName":null,"lineNumber":-1,"className":"sun.reflect.DelegatingMethodAccessorImpl","nativeMethod":false},{"methodName":"invoke","fileName":null,"lineNumber":-1,"className":"java.lang.reflect.Method","nativeMethod":false},{"methodName":"execute","fileName":"POJOActivityImplementation.java","lineNumber":64,"className":"com.amazonaws.services.simpleworkflow.flow.pojo.POJOActivityImplementation","nativeMethod":false},{"methodName":"execute","fileName":"ActivityImplementationBase.java","lineNumber":46,"className":"com.amazonaws.services.simpleworkflow.flow.generic.ActivityImplementationBase","nativeMethod":false},{"methodName":"execute","fileName":"SynchronousActivityTaskPoller.java","lineNumber":196,"className":"com.amazonaws.services.simpleworkflow.flow.worker.SynchronousActivityTaskPoller","nativeMethod":false},{"methodName":"run","fileName":"ActivityTaskPoller.java","lineNumber":92,"className":"com.amazonaws.services.simpleworkflow.flow.worker.ActivityTaskPoller$2","nativeMethod":false},{"methodName":"runWorker","fileName":null,"lineNumber":-1,"className":"java.util.concurrent.ThreadPoolExecutor","nativeMethod":false},{"methodName":"run","fileName":null,"lineNumber":-1,"className":"java.util.concurrent.ThreadPoolExecutor$Worker","nativeMethod":false},{"methodName":"run","fileName":null,"lineNumber":-1,"className":"java.lang.Thread","nativeMethod":false}],"localizedMessage":"com.amazonaws.AmazonWebServiceRequest.copyPrivateRequestParameters()Ljava/util/Map;","message":"com.amazonaws.AmazonWebServiceRequest.copyPrivateRequestParameters()Ljava/util/Map;","suppressed":["[Ljava.lang.Throwable;",[]]}] at com.amazonaws.services.simpleworkflow.flow.pojo.POJOActivityImplementation.throwActivityFailureException(POJOActivityImplementation.java:110) at com.amazonaws.services.simpleworkflow.flow.pojo.POJOActivityImplementation.execute(POJOActivityImplementation.java:67) at com.amazonaws.services.simpleworkflow.flow.generic.ActivityImplementationBase.execute(ActivityImplementationBase.java:46) at com.amazonaws.services.simpleworkflow.flow.worker.SynchronousActivityTaskPoller.execute(SynchronousActivityTaskPoller.java:196) at com.amazonaws.services.simpleworkflow.flow.worker.ActivityTaskPoller$2.run(ActivityTaskPoller.java:92) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source)


Answer:

You are getting java.lang.NoSuchMethodError from method "invoke" of "AmazonS3Client.java" at "lineNumber":3651

So the exception is not related to the Simple Workflow but rather to S3 calling into some incompatible code. Make sure that all dependent libraries are of the correct version.

Question:

When trying to build a project on my computer using mvn test, the Amazon S3 client is unable to find both the credentials and the region, which are stored in ~/.aws/credentials and ~/.aws/config respectively.

When running the program using mvn exec, the S3 client correctly authenticates and there are no issues with the same function.

Does anyone know why this is happening? Would be willing to look into an alternative solution, as I'd like to move the app to a docker container and don't want to have to send credentials to that container anyway.

Full error message:

com.airdev.serena.AWSTest  Time elapsed: 10.628 sec  <<< FAILURE!
com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.
at com.airdev.serena.AWSTest.before(AWSTest.java:32)

Contents of ~/.aws/config:

[default]
region = us-west-1

There shouldn't be any permissions issue - the ~/.aws/config file is readable and I ran the test suite with sudo anyway.

EDIT:

I've tried the following ways of building the S3 Client, and a couple other combinations that I don't have the code for.

s3Client = AmazonS3ClientBuilder.defaultClient(); <- works for maven exec command and when running from Eclipse IDE

s3Client = AmazonS3ClientBuilder.standard()
                                     .withRegion("us-west-1") // The first region to try your request against
                                     .withForceGlobalBucketAccessEnabled(true) // If a bucket is in a different region, try again in the correct region
                                     .build();

Neither of these seem to be picking up the default credential chain, even when I set environment variables or delete the .aws folder and re-set the credentials through the aws configure command.


Answer:

Configure default credential provider using aws cli, as you are able to pick with maven exec, this may not be a reason. Use command

aws configure

To use default credential provider

AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
                       .withRegion(Regions.EU_WEST_1)
                       .build();

You can also provide keys in code explicitly. As you are willing to use docker, credentials need to be inside the docker environment. To use in code

BasicAWSCredentials awsCreds = new BasicAWSCredentials("access_key_id","secret_key_id");
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
                        .withCredentials(new AWSStaticCredentialsProvider(awsCreds))
                        .build();

Question:

I am currently trying to use the AWS SDK (specifically the s3 SDK) in my project but keep getting the exception

java.lang.NoClassDefFoundError:com/amazonaws/services/s3/AmazonS3ClientBuilder

I have imported the SDK into my project using maven as shown in the SDK documentation here. The code that I am running that causes this to occur is

AmazonS3 amazonS3 = AmazonS3ClientBuilder.standard().withRegion(Regions.EU_WEST_1).build();

I suspected the issue may have been occurring due to a conflict in Jackson versions between the version required by the AWS SDK and the version I was importing myself for use else where in the project although changing this does not seem to have resolved the issue. I will include my POM.xml file below.

Thanks in advance.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

<modelVersion>4.0.0</modelVersion>

<groupId>com.projectname.restservice</groupId>
<artifactId>ProjectName</artifactId>
<packaging>war</packaging>
<version>1.0-SNAPSHOT</version>
<name>ProjectName</name>

<build>
    <finalName>ProjectName</finalName>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>2.5.1</version>
            <inherited>true</inherited>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
        </plugin>
    </plugins>
</build>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.glassfish.jersey</groupId>
            <artifactId>jersey-bom</artifactId>
            <version>2.16</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-bom</artifactId>
            <version>1.11.22</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
    <dependency>
        <groupId>org.glassfish.jersey.containers</groupId>
        <artifactId>jersey-container-servlet-core</artifactId>
    </dependency>

    <!--Removed due to possible conflict with AWS SDK?-->
    <!--<dependency>
        <groupId>org.glassfish.jersey.media</groupId>
        <artifactId>jersey-media-json-jackson</artifactId>
    </dependency>-->

    <dependency>
        <groupId>org.glassfish.jersey.media</groupId>
        <artifactId>jersey-media-multipart</artifactId>
    </dependency>

    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
        <version>5.1.35</version>
    </dependency>

    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-s3</artifactId>
    </dependency>

</dependencies>

Output of: mvn dependency:tree -Dverbose -Dincludes=com.amazonaws

com.trailfinder.restservice:TrailFinderRestService:war:1.0-SNAPSHOT
[INFO] \- com.amazonaws:aws-java-sdk-s3:jar:1.11.22:compile
[INFO]    +- com.amazonaws:aws-java-sdk-kms:jar:1.11.22:compile
[INFO]    |  \- (com.amazonaws:aws-java-sdk-core:jar:1.11.22:compile - omitted for duplicate)
[INFO]    \- com.amazonaws:aws-java-sdk-core:jar:1.11.22:compile

Answer:

I think, I've found a solution / workaround to this issue. Thanks to suggestions from @DaveMaple I tried building the app through the command line and deploying the .war manually which seems to have worked.

According to this question IntelliJ IDEA uses its own build process and not Mavens. This leads me to believe that the issue was with IntelliJ's build process and nothing to do with AWS SDK or Maven. To work around this issue I configured IntelliJ to not use its own build process but instead to use Maven and then to deploy the resulting .war. Explanation on how to do this can be found here.