Hot questions for Using Neo4j in netbeans

Question:

I am trying to load 16807 nodes with 17.210.368 relationships in a graph in Neo4j. As to do so, i load a file that contains a vincinity table and i get a list with the nodes that must be connected with a relationship.

Find my code below:

    String inputFile = "Desktop\kron7cd_unix.t01";
    FileInputStream in = new FileInputStream(inputFile);
    FileChannel ch = in.getChannel();
    ByteBuffer buf = ByteBuffer.allocate(1024);

    ArrayList<Integer> list = new ArrayList<Integer>();
    int NumOfOnes = 0;
    int column=-1;
    int row=0;
    int rd;
    while ((rd = ch.read( buf )) != -1){
        buf.flip();
        while (buf.hasRemaining()){
            byte byteVal = buf.get();
            if((byteVal == 48) || (byteVal == 49)){// when finds 1 or 0
                column++;
            }
            if (byteVal == 92){//when finds '/'
                    row++;
                    column=-1;
            }
            if(byteVal == 49){//when finds 1
                NumOfOnes++;
                list.add(column);
                list.add(row);
            }
        }
    buf.clear();
    }
    ch.close();


    GraphDatabaseFactory dbFactory = new GraphDatabaseFactory();
    GraphDatabaseService graphDb = dbFactory.newEmbeddedDatabase("C:\Neo4j\default.graphdb");

    Transaction tx = graphDb.beginTx();
    try {

        Label myLabel = DynamicLabel.label("Data");
        ArrayList<Node> nodelist = new ArrayList<Node>();

        for (int k = 0; k < row; k++) {
            nodelist.add(graphDb.createNode());
        }

        for (int k = 0; k < row; k++) {
            nodelist.get(k).setProperty("ID", k);
            nodelist.get(k).setProperty("Group","Random");
            nodelist.get(k).addLabel(myLabel);
        }

        Relationship rel;
        final RelationshipType type2 = DynamicRelationshipType.withName("Rel");

        for (int j = 0; j < list.size()-1 ; j += 2) { //list.size()=34420736

            rel = nodelist.get(list.get(j)).createRelationshipTo(nodelist.get(list.get(j+1)), type2);
            rel.setProperty("Team", "Common");

            if (j > 0 && j % 10000 == 0) {// as to commit transaction every now and then and dont throw heap space
                tx.success();
                tx.close();
                tx = graphDb.beginTx();
            }                

        }

        tx.success();
    }

    finally {
        tx.close();
    }
    graphDb.shutdown();

When i run this code it throws me the following error. I use Neo4j 2.3.3 and Netbeans 8.1 with Java 8. I want to understand if the problem is the heap space or when it tries to commit the transaction. I have also add the command-line option -Xmx1g in my project as to increrase the heap space.

Any ideas?

Find error messages below:

Exception in thread "main" org.neo4j.graphdb.TransactionFailureException: Transaction was marked as successful, but unable to commit transaction so rolled back.
at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:121)
at com.mycompany.traverse_test.traverse_main.main(traverse_main.java:232)

Caused by: org.neo4j.kernel.api.exceptions.TransactionFailureException: Could not apply the transaction to the store after written to log
at org.neo4j.kernel.impl.api.TransactionRepresentationCommitProcess.applyToStore(TransactionRepresentationCommitProcess.java:105)
at org.neo4j.kernel.impl.api.TransactionRepresentationCommitProcess.commit(TransactionRepresentationCommitProcess.java:58)
at org.neo4j.kernel.impl.api.KernelTransactionImplementation.commit(KernelTransactionImplementation.java:565)
at org.neo4j.kernel.impl.api.KernelTransactionImplementation.close(KernelTransactionImplementation.java:458)
at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:97)
... 1 more

Caused by: java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:386)
at org.neo4j.unsafe.impl.internal.dragons.MemoryManager$Slab.<init>(MemoryManager.java:111)
at org.neo4j.unsafe.impl.internal.dragons.MemoryManager.allocateAligned(MemoryManager.java:82)
at org.neo4j.io.pagecache.impl.muninn.MuninnPage.initBuffer(MuninnPage.java:417)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pageFault(MuninnPageCursor.java:230)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pin(MuninnPageCursor.java:157)
at org.neo4j.io.pagecache.impl.muninn.MuninnWritePageCursor.next(MuninnWritePageCursor.java:58)
at org.neo4j.kernel.impl.store.PropertyStore.updateRecord(PropertyStore.java:144)
at org.neo4j.kernel.impl.transaction.command.NeoStoreTransactionApplier.visitPropertyCommand(NeoStoreTransactionApplier.java:99)
at org.neo4j.kernel.impl.api.CommandApplierFacade.visitPropertyCommand(CommandApplierFacade.java:120)
at org.neo4j.kernel.impl.transaction.command.Command$PropertyCommand.handle(Command.java:288)
at org.neo4j.kernel.impl.api.CommandApplierFacade.visit(CommandApplierFacade.java:82)
at org.neo4j.kernel.impl.api.CommandApplierFacade.visit(CommandApplierFacade.java:45)
at org.neo4j.kernel.impl.transaction.log.PhysicalTransactionRepresentation.accept(PhysicalTransactionRepresentation.java:69)
at org.neo4j.kernel.impl.api.TransactionRepresentationStoreApplier.apply(TransactionRepresentationStoreApplier.java:111)
at org.neo4j.kernel.impl.api.TransactionRepresentationCommitProcess.applyToStore(TransactionRepresentationCommitProcess.java:100)
... 5 more

Answer:

You should commit the transactions every 1k - 10k elements to avoid holding all the data on the heap. See, for example, https://github.com/graphaware/neo4j-framework/tree/master/tx-executor#batch-transactional-operations

Question:

Error starting org.neo4j.kernel.EmbeddedGraphDatabase, /home/manish/Softwares/neo4j-enterprise-2.1.6/data/graph.db

CausedBy: Component 'org.neo4j.kernel.impl.transaction.XaDataSourceManager@2a792260' was successfully initialized, but failed to start. Please see attached cause exception.

CausedBy: Component 'org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource@23d7c3d2' was successfully initialized, but failed to start. Please see attached cause exception.

CausedBy : No schema index provider org.neo4j.kernel.api.index.SchemaIndexProvider found. Kernel extensions available on classpath: Class path entries: /home/manish/NetBeansProjects/ClassLibOperations/store/ClassLibNeo4jOperations.jar /home/manish/NetBeansProjects/ClassLib2/store/2.jar /home/manish/NetBeansProjects/ClassLibB/store/3.jar /home/manish/NetBeansProjects/ExecuteJarFile/build/classes

I am getting the above exception while executing the line :

database = new GraphDatabaseFactory().newEmbeddedDatabase( DB_PATH );

The scenario is something like this : I have a class library for operation with neo4j graph db named ClassLibNeo4jOperations. This class library has all the references required for working with neo4j. Now I am packaging this ClassLibrary as ClassLibNeo4jOperations.jar and adding it as reference in a Console Application ExecuteJarFile. Now when i am calling a method of this jar file this line database = new GraphDatabaseFactory().newEmbeddedDatabase( DB_PATH ); is executed which throws the above mentioned exception.

The JAR (ClassLibNeo4jOperations.jar) is a fat jar ,created using the method mentioned at How to build fat jar


Answer:

Make sure you have neo4j-lucene-index.jar on your classpath.

Question:

My configuration is : * OS : Windows 8 * Neo4j community 2.1.6 (latest stable) * IDE : NetBeans 8.0.2 (latest) * java -version

java version "1.8.0_25"
Java(TM) SE Runtime Environment (build 1.8.0_25-b18)
Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)
  1. I created a maven Java project and put the Hello world example provided by Neo4j guys for the appropriate version.
  2. I added all the main dependencies (neo4j-kernel 2.1.6) and maven transitively added the remaining dependencies.
  3. Build process is 100% OK

I still get a the following exception at runtime :

Exception in thread "main" java.lang.RuntimeException: Error starting org.neo4j.kernel.EmbeddedGraphDatabase, C:\Users\Mehdy\Documents\NetBeansProjects\neo4j-example\target\neo4j-hello-db

I looked in the stack trace and I found $EmbeddedGraphDatabase constructor is deprecated.

Here's the whole console output :

------------------------------------------------------------------------
Building neo4j-example 1.0-SNAPSHOT
------------------------------------------------------------------------

--- exec-maven-plugin:1.2.1:exec (default-cli) @ neo4j-example ---
Exception in thread "main" java.lang.RuntimeException: Error starting org.neo4j.kernel.EmbeddedGraphDatabase, C:\Users\Mehdy\Documents\NetBeansProjects\neo4j-example\target\neo4j-hello-db
    at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:366)
    at org.neo4j.kernel.EmbeddedGraphDatabase.<init>(EmbeddedGraphDatabase.java:59)
    at org.neo4j.graphdb.factory.GraphDatabaseFactory$1.newDatabase(GraphDatabaseFactory.java:91)
    at org.neo4j.graphdb.factory.GraphDatabaseBuilder.newGraphDatabase(GraphDatabaseBuilder.java:181)
    at org.neo4j.graphdb.factory.GraphDatabaseFactory.newEmbeddedDatabase(GraphDatabaseFactory.java:71)
    at com.lirmm.neo4j.example.Hello.createDb(Hello.java:51)
    at com.lirmm.neo4j.example.Hello.main(Hello.java:41)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.transaction.XaDataSourceManager@53f65459' was successfully initialized, but failed to start. Please see attached cause exception.
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:513)
    at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:115)
    at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:343)
    ... 6 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource@4d95d2a2' was successfully initialized, but failed to start. Please see attached cause exception.
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:513)
    at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:115)
    at org.neo4j.kernel.impl.transaction.XaDataSourceManager.start(XaDataSourceManager.java:164)
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:507)
    ... 8 more
Caused by: java.lang.IllegalArgumentException: No schema index provider org.neo4j.kernel.api.index.SchemaIndexProvider found. Kernel extensions available on classpath: 

Class path entries:
  C:\Users\Mehdy\Documents\NetBeansProjects\neo4j-example\target\classes
  C:\Users\Mehdy\.m2\repository\org\neo4j\neo4j-kernel\2.1.6\neo4j-kernel-2.1.6.jar
  C:\Users\Mehdy\.m2\repository\org\neo4j\neo4j-primitive-collections\2.1.6\neo4j-primitive-collections-2.1.6.jar
  C:\Users\Mehdy\.m2\repository\org\apache\geronimo\specs\geronimo-jta_1.1_spec\1.1.1\geronimo-jta_1.1_spec-1.1.1.jar
    at org.neo4j.kernel.api.index.SchemaIndexProvider$2.select(SchemaIndexProvider.java:140)
    at org.neo4j.kernel.extension.KernelExtensions.resolveDependency(KernelExtensions.java:206)
    at org.neo4j.kernel.InternalAbstractGraphDatabase$DependencyResolverImpl.resolveDependency(InternalAbstractGraphDatabase.java:1524)
    at org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource.start(NeoStoreXaDataSource.java:337)
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:507)
    ... 11 more
------------------------------------------------------------------------
BUILD FAILURE
------------------------------------------------------------------------
Total time: 10.883s
Finished at: Wed Dec 10 12:55:47 CET 2014
Final Memory: 5M/72M
------------------------------------------------------------------------
Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (default-cli) on project neo4j-example: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (default-cli) on project neo4j-example: Command execution failed.
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:217)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
    at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
Caused by: org.apache.maven.plugin.MojoExecutionException: Command execution failed.
    at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:362)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
    ... 19 more
Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
    at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:377)
    at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:160)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:610)
    at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:352)
    ... 21 more

Re-run Maven using the -X switch to enable full debug logging.

For more information about the errors and possible solutions, please read the following articles:
[Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

I don't think it's a dependency problem but the use of a deprecated method. How can I get rid of this code snippet and use something compliant neo4j 2.1.6 ?

I tried different solutions found in SO, but still the same result. What should I do to make this work ?


Answer:

You're missing a dependency. Neo4j requires a schema index provider which is located in neo4j-lucene-index. Since it's an exchangeable component (you can write your own index provider) it's bundled in a separate artifact.

Amend your dependencies in pom.xml with:

<dependency>
    <groupId>org.neo4j</groupId>
    <artifactId>neo4j-lucene-index</artifactId>
    <version>2.1.6</version>
</dependency>

If you only want to have one single dependency for all neo4j stuff in your pom.xml use:

<dependency>
    <groupId>org.neo4j</groupId>
    <artifactId>neo4j-community</artifactId>
    <version>2.1.6</version>
</dependency>