Hot questions for Using Neo4j in neo4j java api

Top Java Programmings / Neo4j / neo4j java api

Question:

The Neo4j Java API automatically updates outdated databases to the current version when I call

new GraphDatabaseFactory().newEmbeddedDatabase(File storeDir)

I would like to check what version the database is before doing that. Is there a way to do that with the Java API? Or alternatively: Where is the database version stored so I can read it out manually?


Answer:

Release version

I dug into the Neo4j API source and found an answer. Neo4j reads out the previous version from the debug.log file in the logs directory. Whenever the database is started the version is printed into the log file as Kernel version: (this is where you'll find the version). For instance it could look like this:

2017-11-21 06:21:43.460+0000 INFO [o.n.k.i.DiagnosticsManager] Kernel version: 3.3.0,5b700972242a5ec3e0140261120f2845fb3520ad

You could read out the debug.log the Neo4j way:

import java.io.File;

import org.neo4j.io.fs.DefaultFileSystemAbstraction;
import org.neo4j.io.fs.FileSystemAbstraction;
import org.neo4j.kernel.impl.transaction.log.LogTailScanner;
import org.neo4j.kernel.impl.transaction.log.PhysicalLogFile;
import org.neo4j.kernel.impl.transaction.log.PhysicalLogFiles;
import org.neo4j.kernel.impl.transaction.log.ReadableClosablePositionAwareChannel;
import org.neo4j.kernel.impl.transaction.log.entry.LogEntryReader;
import org.neo4j.kernel.impl.transaction.log.entry.LogEntryVersion;
import org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogEntryReader;

public class Neo4jVersionChecker {

    //Note that this method needs the store directory NOT the debug.log file
    public static String getNeo4jVersion(File storeDir) {   
        FileSystemAbstraction fileSystem = new DefaultFileSystemAbstraction();
        final PhysicalLogFiles logFiles = new PhysicalLogFiles( storeDir, PhysicalLogFile.DEFAULT_NAME, fileSystem );
        final LogEntryReader<ReadableClosablePositionAwareChannel> logEntryReader = new VersionAwareLogEntryReader<>();
        LogTailScanner tailScanner = new LogTailScanner( logFiles, fileSystem, logEntryReader );

        LogEntryVersion version = tailScanner.getTailInformation().latestLogEntryVersion;

        if(version!=null) {
                return version.toString();
        } else {
                return null;
        }
    }

}

The above method returns V3_0_10 for a debug.log whose latest Kernel version entry is the one above.

Unfortunately the Neo4j way is not very precise. As you can see the Kernel version from the debug.log starts with 3.3.0 but the Neo method says it is V3_0_10. I'm assuming this has something to do with the way Neo handles versions internally.

But since we now know how Neo4j gets the version, we can do the same thing in a more exact way:

import java.io.File;
import java.io.IOException;
import java.nio.charset.StandardCharsets;

import org.apache.commons.io.input.ReversedLinesFileReader;

public class VersionChecker {

    public static String getVersion(File storeDir) {
        File debugLog = new File(storeDir, "logs" + File.separator + "debug.log");
        if(debugLog.exists()) {
            try {
                //The ReversedLinesFileReader reads the last line of a file first and so on
                ReversedLinesFileReader reader = new ReversedLinesFileReader(debugLog, StandardCharsets.UTF_8);
                //Read last line
                String line = reader.readLine();
                while(line!=null) {
                    //Line can't be null at this point

                    if(line.contains("Kernel version: ")) {
                        //This line contains the version
                        line = line.substring(line.indexOf("Kernel version: ")).substring(16);  //get rid of everything except the version
                        line = line.split(",")[0];  //get rid of the second part of the Kernel version that we don't want
                        return line;
                    }

                    //Next line
                    line = reader.readLine();
                }
                reader.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
        return null;
    }

}

The method above will return 3.3.0.

Store version

Of course both of these methods only work if there is a debug.log file. Not all previous Neo4j versions have them. As long as the store directory contains the neostore file you can read out the store version which is not as nice as reading out the release version but at least it is something. So here's how that works:

There is a Neo4j class called StoreVersionCheck which contains a very handy method called getVersion(File neostoreFile). Unfortunately we need an instance of something called a PageCache to initialize an instance of StoreVersionCheck. We can make a PageCache, so that is what we'll do.

import java.io.File;
import java.io.IOException;
import java.util.Optional;
import java.util.function.Consumer;

import org.neo4j.graphdb.factory.GraphDatabaseSettings;
import org.neo4j.io.fs.DefaultFileSystemAbstraction;
import org.neo4j.io.fs.FileSystemAbstraction;
import org.neo4j.io.pagecache.PageCache;
import org.neo4j.kernel.configuration.Config;
import org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory;
import org.neo4j.kernel.impl.pagecache.ConfiguringPageCacheFactory;
import org.neo4j.kernel.impl.storemigration.StoreVersionCheck;
import org.neo4j.kernel.impl.util.Neo4jJobScheduler;
import org.neo4j.kernel.monitoring.Monitors;
import org.neo4j.kernel.monitoring.tracing.Tracers;
import org.neo4j.logging.Log;
import org.neo4j.logging.Logger;
import org.neo4j.scheduler.JobScheduler;

public class StoreVersionChecker {

    public static String getStoreVersion(File storeDir) {
        File storeFile = new File(storeDir, "neostore");
        if(!storeFile.exists()) {
            return null;
        }
        StoreVersionCheck check = new StoreVersionCheck(buildPageCache());
        try {
            Optional<String> version = check.getVersion(storeFile);
            if(version.isPresent()) {
                return version.get();
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
        return null;
    }

    private static PageCache buildPageCache() {
        FileSystemAbstraction fileSystem = new DefaultFileSystemAbstraction();
        Config config = Config.defaults();
        Log pageCacheLog = new DummyLog();
        String desiredImplementationName = config.get( GraphDatabaseFacadeFactory.Configuration.tracer );
        Monitors monitors = new Monitors();
        JobScheduler jobScheduler = new Neo4jJobScheduler();
        Tracers tracers = new Tracers( desiredImplementationName, new DummyLog(), monitors, jobScheduler );
        ConfiguringPageCacheFactory pageCacheFactory = new ConfiguringPageCacheFactory(fileSystem, config, tracers.pageCacheTracer, tracers.pageCursorTracerSupplier, pageCacheLog );
        PageCache pageCache = pageCacheFactory.getOrCreatePageCache();

        if ( config.get( GraphDatabaseSettings.dump_configuration ) )
        {
            pageCacheFactory.dumpConfiguration();
        }
        return pageCache;
    }

    //We need this so we can give the Tracers a Log
    private static class DummyLog implements Log {

        @Override
        public boolean isDebugEnabled() {return false;}

        @Override
        public Logger debugLogger() {return null;}

        @Override
        public void debug(String message) {}

        @Override
        public void debug(String message, Throwable throwable) {}

        @Override
        public void debug(String format, Object... arguments) {}

        @Override
        public Logger infoLogger() {return null;}

        @Override
        public void info(String message) {}

        @Override
        public void info(String message, Throwable throwable) {}

        @Override
        public void info(String format, Object... arguments) {}

        @Override
        public Logger warnLogger() {return null;}

        @Override
        public void warn(String message) {}

        @Override
        public void warn(String message, Throwable throwable) {}

        @Override
        public void warn(String format, Object... arguments) {}

        @Override
        public Logger errorLogger() {return null;}

        @Override
        public void error(String message) {}

        @Override
        public void error(String message, Throwable throwable) {}

        @Override
        public void error(String format, Object... arguments) {}

        @Override
        public void bulk(Consumer<Log> consumer) {}

    }

}

Question:

Is it possible to determine if a Node is in a transaction? It is possible to get a GraphDatabaseService by the method Node.getGraphDatabase.

I would like to do something like this:

public class Neo4JHelper {
    public void setProperty(Node node, String key, Object value) {
        if(isInTransaction(node) {
            node.setProperty(key, value);
        } else {
            throw new MyOwnException("You are trying to set a node outside a transaction... you suck");
        }
    }

    private boolean isInTransaction(Node node) {
        //Something
    }
}

The reason I want to do this is because I would like to give my users a custom error when trying to use my class Neo4JHelperoutside a transaction.

Another solution would be if it is possible to somehow tell the compiler that you need a transaction to use the method/class and otherwise give a compile error.


Answer:

I have several different takes on this, I'm not sure which one is most helpful.

Point #1 is that you don't need to do this checking. If you attempt to create any dummy node, and you're not inside of a transaction, the DB will throw an exception indicating that you're not in a transaction. So if you wanted to detect this situation, just try to create a trivial test node. No exception? You're in a transaction.

Point #2 is maybe you're asking if a particular node is in a transaction. I'm not aware of any way to do that, maybe a developer can add something on that. Within transactions, it's possible to acquire a read/write lock on an individual node. So if you had the Transaction object (which your method doesn't) then a surrogate method might be to determine whether there's a lock on a given node.

Point #3 while I'm not 100% sure what you're doing, your code suggest there's a different way of going about this problem so you don't even have to answer this question. If you want your users to get a custom error, go ahead and try to modify the property -- if neo4j throws an exception that you're not in a transaction, again, there's your answer. Catch that exception, then throw your custom error.

Question:

I’m using Neo4j 2.3.0 in Java. I have 16 GB RAM, running the code on MAC OSX Laptop, using "-Xmx12g -Xms12g" as VM arguments.

I’ve encountered a "GC overhead limit exceeded" problem in Neo4j Java API.

In order to do experiments with lots of queries, I have a program which opens a transaction over different query.db's and get the answers of that from my own framework which is wrapped in an object (It runs a query and print its running time in a file).

So, for running the query, I don’t use Cypher.

For each query I open two transactions over a query.db and a data.db, initialize my framework and run it. The memory usage slightly increases and the "GC overhead" finally happens.

try (Transaction txG = knowledgeGraph.beginTx()) {
     try (Transaction txQ = queryGraph.beginTx()) {
          MyObj myFramework = new MyObj();
          printTheResultsIntoTheFile(framework.run());
          myFramework =null;
          txQ.success();
          txQ.close();

These are some of my endeavors to get rid of this error:

  1. After I’ve used a monitoring program to dump the heap, I’ve found that there is some problem with this "org.neo4j.io.pagecache.impl.muninn.MuninnPageCache" So, I’ve tried to set the page cache size and limit it to a small value:

    dataGraph = new GraphDatabaseFactory().newEmbeddedDatabaseBuilder(MODELGRAPH_DB_PATH) .setConfig(GraphDatabaseSettings.pagecache_memory, "500M").newGraphDatabase();

However, still the "memory leakage" problem exists.

  1. After tx.success(), I called the tx.close() to make sure that it doesn’t use the memory.

  2. After using my framework(object) to find the answers of a query, I explicitly set it to null. topkFramework=null;

  3. I called System.gc(); and System.runFinalization();

  4. I changed all of my static variables like MyCacheServer or MyNeighborIndexer to non-static ones and in each query, I made them clear, and explicitly set them to null.

    queryNodeIdSet.clear(); queryNodeIdSet = null; queryNodeIdSet = new HashSet<Long>();


Answer:

After lots of digging into Neo4j, I've found that it's related to create a lot of query graphs one after one. Although I called db.shutdown() after my work with each query, it seems that cache won't be empty.

smallGraph = new GraphDatabaseFactory().newEmbeddedDatabaseBuilder(graphPath)
            .setConfig(GraphDatabaseSettings.pagecache_memory, "240k").newGraphDatabase();

I've added this config and set it to minimum possible amount. Right now the memory leakage is not too much to break my process. After running around 1000 queries it's still running. Earlier it consumed all of my memory (12 GB) after running 200 queries.

This was my stacktrace:

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

    at org.neo4j.io.pagecache.impl.muninn.MuninnPageCache.<init>(MuninnPageCache.java:246)

    at org.neo4j.kernel.impl.pagecache.ConfiguringPageCacheFactory.createPageCache(ConfiguringPageCacheFactory.java:96)

    at org.neo4j.kernel.impl.pagecache.ConfiguringPageCacheFactory.getOrCreatePageCache(ConfiguringPageCacheFactory.java:87)

    at org.neo4j.kernel.impl.factory.PlatformModule.createPageCache(PlatformModule.java:277)

    at org.neo4j.kernel.impl.factory.PlatformModule.<init>(PlatformModule.java:154)

    at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.createPlatform(GraphDatabaseFacadeFactory.java:181)

    at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:124)

    at org.neo4j.kernel.impl.factory.CommunityFacadeFactory.newFacade(CommunityFacadeFactory.java:43)

    at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:108)

    at org.neo4j.graphdb.factory.GraphDatabaseFactory.newDatabase(GraphDatabaseFactory.java:129)

    at org.neo4j.graphdb.factory.GraphDatabaseFactory$1.newDatabase(GraphDatabaseFactory.java:117)

    at org.neo4j.graphdb.factory.GraphDatabaseBuilder.newGraphDatabase(GraphDatabaseBuilder.java:185)

    at org.neo4j.graphdb.factory.GraphDatabaseFactory.newEmbeddedDatabase(GraphDatabaseFactory.java:79)

    at org.neo4j.graphdb.factory.GraphDatabaseFactory.newEmbeddedDatabase(GraphDatabaseFactory.java:74)

Question:

Is there a way to get a node by specific attributes using the Java API? Specifically:

  • By Name
  • By Label
  • By properties - (various properties)

The only functions I found were:

findNode(label)
findNode(label, key, value)

Which only supports one property. Or do I have to use Cypher to get this?


Answer:

If your question is about how to find nodes that have specific multiple property values, the Java API has no method that does that task. In general, it is easier to use Cypher for that.

Question:

I wanted to use neo4j DB in my Android project. I'm trying to connect with Neo4j database with Android, while connecting I'm getting an error. I have added a dependency in my Android app level gradle .

compile 'org.neo4j.driver:neo4j-java-driver:1.3.0'

MainActivity.java

Driver driver = GraphDatabase.driver( "bolt://localhost:7687", AuthTokens.basic("neo4j", "neo4j") );
                Session session = driver.session();
                session.run( "CREATE (a:Person {name: {name}, title: {title}})",
                        parameters( "name", "Arthur", "title", "King" ) );

                StatementResult result = session.run( "MATCH (a:Person) WHERE a.name = {name} " +
                                "RETURN a.name AS name, a.title AS title",
                        parameters( "name", "Arthur" ) );
                while ( result.hasNext() )
                {
                    Record record = result.next();
                    System.out.println( record.get( "title" ).asString() + " " + record.get( "name" ).asString() );
                }
                session.close();
                driver.close();

Logs

06-22 16:24:05.627 4990-4990/com.example.prateek.neo4jtest E/AndroidRuntime: FATAL EXCEPTION: main
Process: com.example.prateek.neo4jtest, PID: 4990
java.lang.NoClassDefFoundError: java.net.StandardSocketOptions
at org.neo4j.driver.internal.net.ChannelFactory.create(ChannelFactory.java:39)
at org.neo4j.driver.internal.net.SocketClient.start(SocketClient.java:124)
at org.neo4j.driver.internal.net.SocketConnection.startSocketClient(SocketConnection.java:92)
at org.neo4j.driver.internal.net.SocketConnection.<init>(SocketConnection.java:67)
at org.neo4j.driver.internal.net.SocketConnector.createConnection(SocketConnector.java:77)
at org.neo4j.driver.internal.net.SocketConnector.connect(SocketConnector.java:50)
at org.neo4j.driver.internal.net.pooling.SocketConnectionPool$ConnectionSupplier.get(SocketConnectionPool.java:204)
at org.neo4j.driver.internal.net.pooling.SocketConnectionPool$ConnectionSupplier.get(SocketConnectionPool.java:186)
at org.neo4j.driver.internal.net.pooling.BlockingPooledConnectionQueue.acquire(BlockingPooledConnectionQueue.java:96)
at org.neo4j.driver.internal.net.pooling.SocketConnectionPool.acquireConnection(SocketConnectionPool.java:137)
at org.neo4j.driver.internal.net.pooling.SocketConnectionPool.acquire(SocketConnectionPool.java:76)
at org.neo4j.driver.internal.DirectConnectionProvider.acquireConnection(DirectConnectionProvider.java:45)
at org.neo4j.driver.internal.NetworkSession.acquireConnection(NetworkSession.java:340)
at org.neo4j.driver.internal.NetworkSession.run(NetworkSession.java:104)
at org.neo4j.driver.internal.NetworkSession.run(NetworkSession.java:94)
at com.example.prateek.neo4jtest.MainActivity$1.onClick(MainActivity.java:29)
at android.view.View.performClick(View.java:4438)
at android.view.View$PerformClick.run(View.java:18431)
at android.os.Handler.handleCallback(Handler.java:733)
at android.os.Handler.dispatchMessage(Handler.java:95)
at android.os.Looper.loop(Looper.java:149)
at android.app.ActivityThread.main(ActivityThread.java:5045)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:515)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:794)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:610)
at dalvik.system.NativeStart.main(Native Method)

Any help will be appreciated. Thanks!


Answer:

Just because there's a Java Driver, doesn't mean it works on Android.

(Although, it seems there's an outdated Android driver on Github)

Neo4j exposes a REST API you can use given HTTP requests, which is probably the safest approach to use, although it'll require learning how that works on Android and dealing with JSON parsing.

More specifically to the error, StandardSocketOptions is not part of Android until API 24, so you could try making your compileSdk and minimumSdk to be 24, but no guarantees it'll still work.


By the way, you can't connect to localhost from the Android device unless you are running Neo4j Server on that Android device itself

Question:

Calling

new GraphDatabaseFactory().newEmbeddedDatabase(File storeDir);

in the Neo4j Java API not only opens a database at a given location but also creates one if it doesn't already exist.

I would like to check if a Neo4j database exists in a given directory without creating one if it doesn't. How can I accomplish that?


Answer:

I now do this by checking if there is a neostore file in the store directory and by trying to get it's version.

See how you can get the store version here: Determine Neo4j database version

Question:

I have list of String and i want to import all the elements to the graph database. By saying import i mean, i want to set the String as the Node's property. The size of the list is gonna be massive. So is there any way to automate Node naming ? Because by the traditional way, you have to create Nodes by calling graphDb.createNode() 100 times, if the size of the list is 100.


Answer:

You can pass your list of strings as a parameter to a Cypher query. Here is a sample snippet:

List<String> names = ...;
try ( Transaction tx = graphDb.beginTx() )
{
    String queryString = "UNWIND {names} AS name CREATE (n:User {name: name})";
    Map<String, Object> parameters = new HashMap<>();
    parameters.put( "names", names );
    graphDb.execute( queryString, parameters );
    tx.success();
}

Note: If the list of strings is "too long", the above approach will not work, as the server could run out of memory trying to do all that processing in a single transaction. In that case, you may want to use an APOC procedure like apoc.periodic.iterate to create the nodes in smaller batches.

Question:

  1. My java version:java version "1.8.0_25" Java(TM) SE Runtime Environment (build 1.8.0_25-b18) Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)
  2. My Neo4j version: neo4j-community-2.3.0-RC1
  3. I imported all jar files in 'neo4j-community-2.3.0-RC1\lib' direcory to classpath with Eclipse
  4. I can find the messages.log file in "C:/TPNeo4jDB", the log is: 2015-10-15 06:19:03.151+0000 INFO [o.n.k.i.f.CommunityFacadeFactory] No locking implementation specified, defaulting to 'community'
  5. I use windows 8 now

When my program runs GraphDatabaseService db = dbFactory.newEmbeddedDatabase("C:/TPNeo4jDB");

I get error:

Exception in thread "main" java.lang.NullPointerException
at org.neo4j.kernel.impl.factory.CommunityEditionModule.determineEdition(CommunityEditionModule.java:142)
at org.neo4j.kernel.impl.factory.CommunityEditionModule.publishEditionInfo(CommunityEditionModule.java:132)
at org.neo4j.kernel.impl.factory.CommunityEditionModule.(CommunityEditionModule.java:122)
at org.neo4j.kernel.impl.factory.CommunityFacadeFactory.createEdition(CommunityFacadeFactory.java:50)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:125)
at org.neo4j.kernel.impl.factory.CommunityFacadeFactory.newFacade(CommunityFacadeFactory.java:43)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:108)
at org.neo4j.graphdb.factory.GraphDatabaseFactory.newDatabase(GraphDatabaseFactory.java:129)
at org.neo4j.graphdb.factory.GraphDatabaseFactory$1.newDatabase(GraphDatabaseFactory.java:117)
at org.neo4j.graphdb.factory.GraphDatabaseBuilder.newGraphDatabase(GraphDatabaseBuilder.java:185)
at org.neo4j.graphdb.factory.GraphDatabaseFactory.newEmbeddedDatabase(GraphDatabaseFactory.java:79)
at org.neo4j.graphdb.factory.GraphDatabaseFactory.newEmbeddedDatabase(GraphDatabaseFactory.java:74)
at com.gongjiaolaile.view.Test.main(Test.java:9)

Answer:

I guess that you are missing some libraries. First of all there are several places in Neo4j directory with libraries:

  • lib/
  • system/lib/

Second one - you should use build management tool like Maven.

In Neo4j documentation there is Using Neo4j embedded in Java applications section, that gives you overview how to use Neo4j. And here is described how to add Neo4j as dependency.