Hot questions for Using Neo4j in neo4j embedded

Top Java Programmings / Neo4j / neo4j embedded

Question:

I looked at how I can easily get a Node by ID through cypher, yet the methods that are available for the Node interface do not appear to support a way to return a particular Node object based on ID.

How can I return a particular node by ID with the Neo4j embedded Java library? Same question with relationships...


Answer:

You use GraphDatabaseService.getNodeById and GraphDatabaseService.getRelationshipById.

Question:

The question is regarding the

findNode(label, propertyName, propertyValue)

function. Once I have the node, what operations are cheap and what operations are expensive ? For example, does the node object get retrieved together with all its properties or does each

node.getProperty

call incur additional performance cost ? Cheers !


Answer:

findNode(label, propertyName, propertyValue) uses an index to retrieve the matching nodes. The properties won't be loaded with the resulting nodes.

node.getProperty will cause the entire property chain to be loaded (i.e. all the node's properties). So the first call to this method on any node/relationship is relatively "expensive", the subsequent ones will be cheap.

Question:

Context

I have a RESTful API for a versus fighting game, using JAX-RS, tomcat8 and Neo4j embedded.

Today I figured that a lot of queries will be done in a limited time, I'm using embedded for faster queries but I still want to go as fast as possible.

Problem

In fact, the problem is a bit different but not that much.

Actually, I'm using a Singleton with a getDabatase() method returning the current GraphDatabaseServiceinstance to begin a transaction, once it's done, the transaction is closed... and that's all.

I don't know if the best solution for optimal perfs is a Singleton pattern or a pool one (like creating XX instances of database connection, and reuse them when the database operation is finished).

I can't test it myself actually, because I don't have enough connections to even know which one is the fastest (and the best overall).

Also, I wonder if I create a pool of GraphDatabaseService instances, will they all be able to access the same datas without getting blocked by the lock?


Answer:

Crate only one on GraphDatabaseService instance and use it everywhere. There are no need to create instance pool for them. GraphDatabaseService is completely thread-safe, so you can not worry about concurrency (note: transaction are thread-bound, so you can't run multiple transactions in same thread).

All operations in Neo4j should be executed in Transaction. On commit transaction is written in transaction log, and then persisted into database. General rules are:

  • Always close transaction as early as possible (use try-with-resource)
  • Close all resources as early as possible (ResourceIterator returned by findNodes() and execute())

Here you can find information about locking strategy.


To be sure that you have best performance, you should:

  • Check database settings (memory mapping)
  • Check OS settings (file system)
  • Check JVM settings (GC, heap size)
  • Data model

Here you can find some articles about Neo4j configuration & optimizations. All of them have useful information.

Question:

I'm experimenting with Neo4J via Embedded Java API. My Build path seems ok (no Exceptions during runtime). When I create some nodes and relations, I can query it directly after it with success. But after shutting down and re-run my programm, i'm only getting the data I created in the new runtime and none of them before. But if I look at my directory, I see, that the size has grown with each runtime, I perform a creating of data.

Here's my code:

public static void main(String[] args)
{
    GraphDatabaseService gdb = new GraphDatabaseFactory().newEmbeddedDatabase( "/mytestdb/" );

    create( gdb );
    query( gdb );

    gdb.shutdown();
}

private static void query( GraphDatabaseService gdb )
{
    StringLogger sl = StringLogger.wrap( new Writer()
    {

        @Override
        public void write( char[] arg0, int arg1, int arg2 ) throws IOException
        {
            for( int i=arg1; i<=arg2; i++ ) System.out.print( arg0[i] );

        }

        @Override
        public void flush() throws IOException
        {}

        @Override
        public void close() throws IOException
        {}
    } );
    ExecutionEngine ee = new ExecutionEngine( gdb, sl );
    ExecutionResult result = ee.execute( "MATCH (p:Privilleg) RETURN p" );
    System.out.println( result.dumpToString() );
}

private static void create( GraphDatabaseService gdb )
{
    Transaction tx = gdb.beginTx(); 
    Node project = gdb.createNode( MyLabels.Project );
    Node user = gdb.createNode( MyLabels.User );
    Node priv1 = gdb.createNode( MyLabels.Privilleg );
    Node priv2 = gdb.createNode( MyLabels.Privilleg );

    user.setProperty( "name", "Heinz" );
    user.setProperty( "email", "heinz@gmx.net" );
    priv1.setProperty( "name", "Allowed to read all" );
    priv1.setProperty( "targets", Short.MAX_VALUE );
    priv1.setProperty( "read", true );
    priv1.setProperty( "write", false );
    priv2.setProperty( "name", "Allowed to write all" );
    priv2.setProperty( "targets", Short.MAX_VALUE );
    priv2.setProperty( "read", false );
    priv2.setProperty( "write", true );
    project.setProperty( "name", "My first project" );
    project.setProperty( "sname", "STARTUP" );
    user.createRelationshipTo( priv1, MyRelationships.UserPrivilleg );
    user.createRelationshipTo( priv2, MyRelationships.UserPrivilleg );
    priv1.createRelationshipTo( project, MyRelationships.ProjectPrivilleg );
    priv2.createRelationshipTo( project, MyRelationships.ProjectPrivilleg );
    tx.success();
}

Answer:

Your code doesn't close the transaction. Typically you use a try-with-resources block:

try (Transaction tx=gdb.beginTx()) {
    // do stuff in the graph
    tx.success();
}

Since Transaction is AutoClosable its close() method will be called implicitly upon leaving the code block. If (for whatever) reason you decide not to use try-with-resources, be sure to explicitly call close().

On a different notice: your code uses ExecutionEngine. Since Neo4j 2.2 you directly call gdb.execute(myCypherString) instead.

Question:

I'm loading a large number of nodes and relationships into an embedded Neo4j database. After about 10,000 inserts, it dies. If I stay under that point, then everything works great. Queries return as they should, as do inserts. It looks like somehow a database file is getting deleted in the middle of the inserts, which is causing everything to fall apart. My database builds itself from scratch, so if I completely delete my graphdb folder and restart it, it runs exactly the same every time. So how do you handle large embedded Neo4j databases?

Here are the pertinent errors.

From the Java output side

The transactions start not committing:

WorkerThread exception::org.neo4j.graphdb.TransactionFailureException::Unable to commit transaction
org.neo4j.graphdb.TransactionFailureException: Unable to commit transaction
        at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:140)
        ...
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.neo4j.graphdb.TransactionFailureException: commit threw exception
        at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:500)
        at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:385)
        at org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:123)
        at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:124)
        ... 4 more
Caused by: javax.transaction.xa.XAException

Then it informs me that there's a missing file in the database:

saction.TransactionImpl.doCommit(TransactionImpl.java:560)

at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:448)
        ... 7 more
Caused by: org.neo4j.kernel.impl.nioneo.store.UnderlyingStorageException: java.io.FileNotFoundException: /home/user/graphdb/schema/label/lucene/_1z6.frq (Protocol error)
        at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.updateLabelScanStore(NeoStoreTransaction.java:814)
        at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.applyCommit(NeoStoreTransaction.java:699)
        at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.doCommit(NeoStoreTransaction.java:631)
        at org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:327)
        at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commitWriteTx(XaResourceManager.java:632)
        at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:533)
        at org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64)
        at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:548)
        ... 8 more
Caused by: java.io.FileNotFoundException: /home/user/graphdb/schema/label/lucene/_1z6.frq (Protocol error)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
        at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:441)
        at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306)
        at org.apache.lucene.index.FormatPostingsDocsWriter.<init>(FormatPostingsDocsWriter.java:47)
        at org.apache.lucene.index.FormatPostingsTermsWriter.<init>(FormatPostingsTermsWriter.java:33)
        at org.apache.lucene.index.FormatPostingsFieldsWriter.<init>(FormatPostingsFieldsWriter.java:51)
        at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:85)
        at org.apache.lucene.index.TermsHash.flush(TermsHash.java:113)
        at org.apache.lucene.index.DocInverter.flush(DocInverter.java:70)
        at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:60)
        at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:581)
        at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3587)
        at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3552)
        at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:450)
        at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
        at org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413)
        at org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432)
        at org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375)
        at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508)
        at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109)
        at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57)
        at org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137)
        at org.neo4j.kernel.api.impl.index.LuceneLabelScanStore.refreshSearcher(LuceneLabelScanStore.java:159)
        at org.neo4j.kernel.api.impl.index.LuceneLabelScanWriter.close(LuceneLabelScanWriter.java:82)
        at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.updateLabelScanStore(NeoStoreTransaction.java:811)
        ... 15 more

Then I can no longer get a new transaction:

WorkerThread exception::org.neo4j.graphdb.TransactionFailureException::Unable to get transaction.
org.neo4j.graphdb.TransactionFailureException: Unable to get transaction.
        at org.neo4j.kernel.InternalAbstractGraphDatabase.transactionRunning(InternalAbstractGraphDatabase.java:1064)
        at org.neo4j.kernel.InternalAbstractGraphDatabase.beginTx(InternalAbstractGraphDatabase.java:1037)
        at org.neo4j.kernel.TransactionBuilderImpl.begin(TransactionBuilderImpl.java:43)
        at org.neo4j.kernel.InternalAbstractGraphDatabase.beginTx(InternalAbstractGraphDatabase.java:1024)
        ...
        at java.lang.Thread.run(Thread.java:745)
Caused by: javax.transaction.SystemException: Kernel has encountered some problem, please perform neccesary action (tx recovery/restart)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.neo4j.kernel.impl.transaction.KernelHealth.assertHealthy(KernelHealth.java:61)
        at org.neo4j.kernel.impl.transaction.TxManager.assertTmOk(TxManager.java:339)
        at org.neo4j.kernel.impl.transaction.TxManager.getTransaction(TxManager.java:725)
        at org.neo4j.kernel.InternalAbstractGraphDatabase.transactionRunning(InternalAbstractGraphDatabase.java:1060)
        ... 7 more
Caused by: javax.transaction.xa.XAException
        at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:560)
        at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:448)
        at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:385)
        at org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:123)
        at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:124)
        ... 4 more
Caused by: org.neo4j.kernel.impl.nioneo.store.UnderlyingStorageException: java.io.FileNotFoundException: /home/user/graphdb/schema/label/lucene/_1z6.frq (Protocol error)
        at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.updateLabelScanStore(NeoStoreTransaction.java:814)
        at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.applyCommit(NeoStoreTransaction.java:699)
        at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.doCommit(NeoStoreTransaction.java:631)
        at org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:327)
        at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commitWriteTx(XaResourceManager.java:632)
        at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:533)
        at org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64)
        at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:548)
        ... 8 more
Caused by: java.io.FileNotFoundException: /home/user/graphdb/schema/label/lucene/_1z6.frq (Protocol error)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
        at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:441)
        at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306)
        at org.apache.lucene.index.FormatPostingsDocsWriter.<init>(FormatPostingsDocsWriter.java:47)
        at org.apache.lucene.index.FormatPostingsTermsWriter.<init>(FormatPostingsTermsWriter.java:33)
        at org.apache.lucene.index.FormatPostingsFieldsWriter.<init>(FormatPostingsFieldsWriter.java:51)
        at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:85)
        at org.apache.lucene.index.TermsHash.flush(TermsHash.java:113)
        at org.apache.lucene.index.DocInverter.flush(DocInverter.java:70)
        at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:60)
        at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:581)
        at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3587)
        at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3552)
        at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:450)
        at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
        at org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413)
        at org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432)
        at org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375)
        at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508)
        at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109)
        at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57)
        at org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137)
        at org.neo4j.kernel.api.impl.index.LuceneLabelScanStore.refreshSearcher(LuceneLabelScanStore.java:159)
        at org.neo4j.kernel.api.impl.index.LuceneLabelScanWriter.close(LuceneLabelScanWriter.java:82)
        at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.updateLabelScanStore(NeoStoreTransaction.java:811)
        ... 15 more

From the messages.log side

It starts off getting memory mapping errors. This actually happens first when the database first comes online. But more trickle in before it totally dies:

2015-01-27 21:51:29.112+0000 ERROR [org.neo4j]: [/home/user/graphdb/neostore.nodestore.db] Unable to memory map Unable to map pos=0 recordSize=15 totalSize=1048575
org.neo4j.kernel.impl.nioneo.store.MappedMemException: Unable to map pos=0 recordSize=15 totalSize=1048575
        at org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow.<init>(MappedPersistenceWindow.java:59)
        at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.allocateNewWindow(PersistenceWindowPool.java:656)
        at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.expandBricks(PersistenceWindowPool.java:617)
        at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:144)
        at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:546)
        at org.neo4j.kernel.impl.nioneo.store.NodeStore.forceGetRecord(NodeStore.java:149)
        at org.neo4j.kernel.impl.nioneo.xa.NeoStoreIndexStoreView$NodeStoreScan.run(NeoStoreIndexStoreView.java:327)
        at org.neo4j.kernel.impl.api.index.IndexPopulationJob.indexAllNodes(IndexPopulationJob.java:212)
        at org.neo4j.kernel.impl.api.index.IndexPopulationJob.run(IndexPopulationJob.java:107)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Invalid argument
        at sun.nio.ch.FileChannelImpl.map0(Native Method)
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:875)
        at org.neo4j.kernel.impl.nioneo.store.StoreFileChannel.map(StoreFileChannel.java:57)
        at org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow.<init>(MappedPersistenceWindow.java:53)
        ... 13 more

Then, and I've figured out that this is when everything falls apart, I get this error in messages.log:

2015-01-27 21:59:41.516+0000 ERROR [org.neo4j]: setting TM not OK. Kernel has encountered some problem, please perform neccesary action (tx recovery/restart) null
javax.transaction.xa.XAException
        at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:560)
        at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:448)
        at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:385)
        at org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:123)
        at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:124)
        ...
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.neo4j.kernel.impl.nioneo.store.UnderlyingStorageException: java.io.FileNotFoundException: /home/user/graphdb/schema/label/lucene/_1z6.frq (Protocol error)
        at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.updateLabelScanStore(NeoStoreTransaction.java:814)
        at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.applyCommit(NeoStoreTransaction.java:699)
        at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.doCommit(NeoStoreTransaction.java:631)
        at org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:327)
        at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commitWriteTx(XaResourceManager.java:632)
        at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:533)
        at org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64)
        at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:548)
        ... 8 more
Caused by: java.io.FileNotFoundException: /home/user/graphdb/schema/label/lucene/_1z6.frq (Protocol error)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
        at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:441)
        at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306)
        at org.apache.lucene.index.FormatPostingsDocsWriter.<init>(FormatPostingsDocsWriter.java:47)
        at org.apache.lucene.index.FormatPostingsTermsWriter.<init>(FormatPostingsTermsWriter.java:33)
        at org.apache.lucene.index.FormatPostingsFieldsWriter.<init>(FormatPostingsFieldsWriter.java:51)
        at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:85)
        at org.apache.lucene.index.TermsHash.flush(TermsHash.java:113)
        at org.apache.lucene.index.DocInverter.flush(DocInverter.java:70)
        at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:60)
        at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:581)
        at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3587)
        at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3552)
        at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:450)
        at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
        at org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413)
        at org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432)
        at org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375)
        at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508)
        at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109)
        at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57)
        at org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137)
        at org.neo4j.kernel.api.impl.index.LuceneLabelScanStore.refreshSearcher(LuceneLabelScanStore.java:159)
        at org.neo4j.kernel.api.impl.index.LuceneLabelScanWriter.close(LuceneLabelScanWriter.java:82)
        at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.updateLabelScanStore(NeoStoreTransaction.java:811)
        ... 15 more
2015-01-27 21:59:41.519+0000 ERROR [org.neo4j]: TM error tx commit commit threw exception

Any ideas on what's causing that .frq file to disappear?


Answer:

We resolved it in a side-conversation, maximum open files was too low (4000) which is also reported at startup.

That causes Lucene to break internally.

After increasing the limit the OP could import the data successfully.

Question:

I have a neo4j embedded DB in my application and I want to load a .csv file to fill the database. I've managed to create the .csv file in the /import folder but when I try to load it, I get a Couldn't load the external resource at: file:/csv_file.csv. I've read it may be something about permissions but I cannot change them since it creates the whole folder structure every time I run my application (I try to change them after creating the file with Runtime.getRuntime().exec("chmod 777 /path_to_file/csv_file.csv") but it never works)

This is my code:

public void addCSVtoDB() {
    try ( Transaction ignored = graphDb.beginTx();
            Result result = graphDb.execute( "LOAD CSV WITH HEADERS FROM \"file:///csv_file.csv\" AS csvLine\n" )
        {

    } 

I'm using MacOSX 10.11 so the / are supposed to be allright. Any idea?


Answer:

Ok so I finllay solved it: everywhere it says that the .csv files had to be in the /import folder but that doesn't applies to embedded databases. The path_to_file is absolute so I put the .csv file in my /users/MY_USER/ folder and made the cypher query as follows:

public void addCSVtoDB() {
    try ( Transaction ignored = graphDb.beginTx();
            Result result = graphDb.execute( "LOAD CSV WITH HEADERS FROM \"file:////Users/My_User/csv_file.csv\" AS csvLine\n" )
        {

    }

Question:

I am using Neo4J embedded database with OGM and creating the database service in a directory via the OGM SessionFactory:

Configuration configuration = new Configuration.Builder()
                .uris("C:\neoEmbeddedDb")
                .build();

        factory = new SessionFactory(configuration, packages);

This works well, but now i want to browse the created database with the Neo4J Browser Tool. As i read, i have to expose my database via Bolt to be able to access it.

In the Neo4J Embedded documentation, they use the GraphDatabaseService and simply specify an additional bolt driver to expose the database:

GraphDatabaseService graphDb = new GraphDatabaseFactory()
                .newEmbeddedDatabaseBuilder( DB_PATH )
                .setConfig( bolt.type, "BOLT" )
                .setConfig( bolt.enabled, "true" )
                .setConfig( bolt.address, "localhost:7687" )
                .newGraphDatabase();

But unfortunately, i don't have this option when using the OGM SessionFactory. I tried to call the Configuration Builder with multiple URIs:

Configuration configuration = new Configuration.Builder()
                .uris(new String[]{this.databasePath.toUri().toString(), "localhost:7687"})
                .build();

But it seems to ignore the first URI (my file location) and instead creates the database in a temporary location.

The debug output logs a corresponding message to the console:

Creating temporary file store: file:/C:/Temp/neo4jTmpEmbedded.db2736315981519762299/database/

Can anyone explain how i can expose my embedded database via bolt or access it otherwise with the Neo4J Browser?

Many thanks!


Solution

With meistermeier's help i was able to create a real EmbeddedDatabase and connect my OGM to it. I added the Bolt connection options as i found them in the documentation. Now, the database is created and properly exposed via Bolt. I can connect with my Neo4J Desktop Windows Browser.

The final code is

BoltConnector boltConnector = new BoltConnector(_BOLT_CONNECTION_STRING);

GraphDatabaseService graphDb = new GraphDatabaseFactory()
        .newEmbeddedDatabaseBuilder(databasePath.toFile())
        .setConfig(boltConnector.type, "BOLT" )
        .setConfig(boltConnector.enabled, "true" )
        .setConfig(boltConnector.listen_address, "localhost:7687" )
        .setConfig(GraphDatabaseSettings.auth_enabled, "false")
        .newGraphDatabase();

registerShutdownHook(graphDb);

// connect OGM session factory to embedded database
EmbeddedDriver driver = new EmbeddedDriver(graphDb);
final String[] packages = new String[] {
        "Entity domain package",
};

factory = new SessionFactory(driver, packages);

Answer:

First of all: Neo4j-OGM is not in charge of any more complex database start besides creating a connection to it. Providing an instance of Neo4j with a custom configuration will make your application responsible for setting it up correctly.

Secondly: What you are trying to achieve is not possible. At least not with another server instance that serves the Neo4j Browser that can connect to any bolt protocol server once started. You can start an embedded instance and, if you really want this, open the bolt port. But there will no built-in Neo4j browser been deployed.

But after all let me explain why you see the temporary DB folder message: Within your setup snippets I can see two instances getting created: One by creating an explicit instance of the GraphDatabaseService and the other one through the configuration. The first one is obvious, the second one not so much. I just assume that you are not using the very latest version of Neo4j-OGM (3.1.8 at the moment of writing this answer).

The method/parameter uris is just for additional uris (prior 3.1.8) regarding causal cluster and routing. You would have to use uri with a single uri (either bolt://.... or file:///...) in your case. The behaviour of Neo4j-OGM in the case of an undefined uri is to fall back to an temporary embedded instance.

Another solution for this problem is to not provide a Configuration but an EmbeddedDriver instance for the SessionFactory. This can wrap the already configured GraphDatabaseService like new SessionFactory(new EmbeddedDriver(graphDatabaseService), ...). Then you would have the exposed bolt port but unfortunately not a running browser.

Question:

I would like to add a weight to a relationship using Embedded Neo4j Java API.

For example : A knows B very well so their relationship should be weighted 5. On the other hand, A knows C very little so their relationship should be weighted 1.

How can I do that?

PS : I have already tried the example here : http://neo4j.com/docs/stable/tutorials-java-embedded-graph-algo.html but it doesn't recognize th functions createNode( "name", "A", "x", 0d, "y", 0d ) and createRelationship( nodeA, nodeC, "length", 2d ).

This is the code :

package com.neo4j.test.test1;

import org.neo4j.graphalgo.CommonEvaluators;
import org.neo4j.graphalgo.EstimateEvaluator;
import org.neo4j.graphalgo.GraphAlgoFactory;
import org.neo4j.graphalgo.PathFinder;
import org.neo4j.graphalgo.WeightedPath;
import org.neo4j.graphdb.GraphDatabaseService;
import org.neo4j.graphdb.Node;
import org.neo4j.graphdb.PathExpanders;
import org.neo4j.graphdb.Relationship;
import org.neo4j.graphdb.Transaction;
import org.neo4j.graphdb.factory.GraphDatabaseFactory;

import com.neo4j.test.labels.NodeLabels;
import com.neo4j.test.labels.TypeRelation;

public class Test1 {

    public static void main(String[] args) {

        GraphDatabaseFactory dbFactory = new GraphDatabaseFactory();
        GraphDatabaseService db = dbFactory.newEmbeddedDatabase("C:\\Zakaria\\NeoTests\\Test2");

        try (Transaction tx = db.beginTx()) {

            Node nodeA = createNode( "name", "A", "x", 0d, "y", 0d );
            Node nodeB = createNode( "name", "B", "x", 7d, "y", 0d );
            Node nodeC = createNode( "name", "C", "x", 2d, "y", 1d );
            Relationship relAB = createRelationship( nodeA, nodeC, "length", 2d );
            Relationship relBC = createRelationship( nodeC, nodeB, "length", 3d );
            Relationship relAC = createRelationship( nodeA, nodeB, "length", 10d );

            EstimateEvaluator<Double> estimateEvaluator = new EstimateEvaluator<Double>()
            {
                @Override
                public Double getCost( final Node node, final Node goal )
                {
                    double dx = (Double) node.getProperty( "x" ) - (Double) goal.getProperty( "x" );
                    double dy = (Double) node.getProperty( "y" ) - (Double) goal.getProperty( "y" );
                    double result = Math.sqrt( Math.pow( dx, 2 ) + Math.pow( dy, 2 ) );
                    return result;
                }
            };
            PathFinder<WeightedPath> astar = GraphAlgoFactory.aStar(
                    PathExpanders.allTypesAndDirections(),
                    CommonEvaluators.doubleCostEvaluator( "length" ), estimateEvaluator );
            WeightedPath path = astar.findSinglePath( nodeA, nodeB );

            tx.success();

        }

        System.out.println("Done!");

    }

}

It should give this result :

It says that these following functions are not defined :

Node nodeA = createNode( "name", "A", "x", 0d, "y", 0d );
Node nodeB = createNode( "name", "B", "x", 7d, "y", 0d );
Node nodeC = createNode( "name", "C", "x", 2d, "y", 1d );
Relationship relAB = createRelationship( nodeA, nodeC, "length", 2d );
Relationship relBC = createRelationship( nodeC, nodeB, "length", 3d );
Relationship relAC = createRelationship( nodeA, nodeB, "length", 10d );

Thanks!


Answer:

As said Ryan, Java compiler doesn't detect the methods createNode() and createRelationship() that you use.

This is a way to create the Nodes, and works for me:

try (Transaction Tx = gdbs.beginTx(){
    Node nodo = gdbs.createNode();
    nodo.addLabel(p);   // if you have Labels
    nodo.setProperty("property1", someValue);
    Tx.success();
    Tx.close();
} catch (Exception e){//do something}

For relationships, only show you how to add properties:

relationship = firstNode.createRelationshipTo(secondNode, RelTypes.KNOWS );
relationship.setProperty( "message", "brave Neo4j " );

Depending of your Neo4j version, you should see how create Nodes and Relationships. Last, I have you create nodes and relationships, and without commit them, you create the PathFinder. I recommend you persist the nodes before querying the graph as good practice.

Question:

I'm beginning in Neo4j java embedded graph. I have made a first test but I cant visualize my graph in neo4j-community.

Here is my code for creating the graph :

package connection;

import java.io.File;

import org.neo4j.graphdb.GraphDatabaseService;
import org.neo4j.graphdb.Label;
import org.neo4j.graphdb.Node;
import org.neo4j.graphdb.RelationshipType;
import org.neo4j.graphdb.Transaction;
import org.neo4j.graphdb.factory.GraphDatabaseFactory;

public class embededdedGraph {

    public static void main(String... args) throws Exception {
        GraphDatabaseFactory graphDbFactory = new GraphDatabaseFactory();
        File graphDir = new File("/home/nicolas/neo4j-community-3.5.14/data/databases/cars.db");
        GraphDatabaseService graphDb = graphDbFactory.newEmbeddedDatabase(graphDir);
        Transaction tx = graphDb.beginTx();

        createNode(graphDb);

        tx.close();

    }

    public static void createNode(GraphDatabaseService graphDb) {
        Node car = graphDb.createNode();
        car.addLabel(Label.label("Car"));

        car.setProperty("make", "tesla");
        car.setProperty("model", "model3");

        Node owner = graphDb.createNode(Label.label("Person"));
        owner.setProperty("firstName", "Oliver");
        owner.setProperty("lastName", "John");
        owner.createRelationshipTo(car, RelationshipType.withName("owner"));
    }

}

Next I changed the value of "#dbms.active_database" to "dbms.active_database=cars.db in the /neo4j-community-3.5.14/conf/neo4.conf file.

When I restart the neo4j-community, the database name is "cars.db" but it indicated that there are no labels and relationships in it.

What is the problem I cannot figure?

Nicolas


Answer:

It looks like you need to call tx.success() or tx.fail() before tx.close().

https://neo4j.com/docs/java-reference/3.5/javadocs/org/neo4j/graphdb/Transaction.html

I am also new at this API, I hope it helps

Question:

Below is my code. When i call graphDatabaseService.beginTx() it is giving nullPointerException. Is there any other ways to create Transaction in Neo4j

 protected Record getNext() throws Exception {  
Transaction tx = graphDatabaseService.beginTx();
    try {   
    if (hits == null) { 
        hits = index.query(query);
       tx.success(); 
    }
    T next = hits.next();
    if (next == null) {
        return finished();
    } else {
        return nodeToRecord.call(next);
    }
    }   
    finally{    
        closeIndexResults(hits);
         tx.close();
    }
}

Answer:

Your graphDatabaseService variable has a null value. Make sure you have properly initialized it before getNext() is called. For example:

GraphDatabaseService graphDatabaseService = new GraphDatabaseFactory().newEmbeddedDatabase(dbPath);

Question:

I have upgraded my neo4j embedded DB from 2.3.9 to 3.2.3 in SINGLE mode, it has upgraded successfully. After upgrade, I have enabled "HA" mode. While running neo4j with 3 clusters, I am facing below issue.

Individually servers are running fine in HA mode. (i.e. ha.initial_hosts = "ip_address_1:5101"), but if I add three servers under initial_hosts (as shown in the config), all three servers are stopping immediately.

Am I missing any configuration? Please suggest.

Config:

neo4j {
            # Enable these two options while upgrading neo4j database.
            # dbms.allow_format_migration=true

            # or weak or strong
    cache_type = "weak"
            # Reduce the default page cache memory allocation
            dbms.memory.pagecache.size="6G"

            # Port to listen to for incoming backup requests.
            dbms.backup.address = ${local.private-ip}":6367"

            # Unique server id for this Neo4j instance
            # can not be negative id and must be unique
            ha.server_id="1"

            # List of other known instances in this cluster
            ha.initial_hosts = "ip_1:5101,ip_2:5101,ip_3:5101"

            # ha.initial_hosts = "ip_1:5101"
            # ha.cluster_server = ${local.private-ip}":5101"

            # IP and port for this instance to bind to for communicating cluster information
            # with the other neo4j instances in the cluster.
            ha.host.coordination = ${local.private-ip}":5101"

            # IP and port for this instance to bind to for communicating data with the
            # other neo4j instances in the cluster.
            ha.host.data = ${local.private-ip}":6365"

            # HA - High Availability
            # SINGLE - Single mode, default.
            dbms.mode="HA"

            # HTTP Connector
            dbms.connector.http.enabled="true"
            dbms.connector.http.listen_address=":7474"

            # Bolt connector
            dbms.connector.bolt.enabled="true"
            dbms.connector.bolt.tls_level="OPTIONAL"
            dbms.connector.bolt.listen_address=":7689"
}

From the neo4j debug.log:

2017-10-09 12:35:47.153+0000 ERROR [o.n.k.h.c.m.HighAvailabilityModeSwitcher] Error while trying to switch to slave Cannot find the master among [] with master serverId=1 and uri=ha://ip_address_1:6365?serverId=1
    java.lang.IllegalStateException: Cannot find the master among [] with master serverId=1 and uri=ha://ip_address_1:6365?serverId=1
            at org.neo4j.kernel.ha.cluster.SwitchToSlave.checkMyStoreIdAndMastersStoreId(SwitchToSlave.java:263)
            at org.neo4j.kernel.ha.cluster.SwitchToSlaveBranchThenCopy.checkDataConsistency(SwitchToSlaveBranchThenCopy.java:142)
            at org.neo4j.kernel.ha.cluster.SwitchToSlave.executeConsistencyChecks(SwitchToSlave.java:478)
            at org.neo4j.kernel.ha.cluster.SwitchToSlave.switchToSlave(SwitchToSlave.java:221)
            at org.neo4j.kernel.ha.cluster.modeswitch.HighAvailabilityModeSwitcher$1.run(HighAvailabilityModeSwitcher.java:355)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            at java.lang.Thread.run(Thread.java:745)
            at org.neo4j.helpers.NamedThreadFactory$2.run(NamedThreadFactory.java:109)
    2017-10-09 12:35:47.154+0000 INFO [o.n.k.h.c.m.HighAvailabilityModeSwitcher] Attempting to switch to slave in 300s

Answer:

Default Value for join_timeout is 30 seconds.

Timeout for joining a cluster. Defaults to ha.broadcast_timeout. Note that if the timeout expires during cluster formation, the operator may have to restart the instance or instances.

ha.join_timeout=10m

https://neo4j.com/docs/operations-manual/current/reference/configuration-settings/#config_ha.join_timeout