Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Solr is a standalone enterprise search server with a REST-like API. Solr is highly reliable, scalable and fault tolerant, providing distributed indexing, replication and load-balanced querying, automated failover and recovery, centralized configuration and more.
JanusGraph supports Apache Solr as an index backend. Here are some of the Solr features supported by JanusGraph:
- Full-Text: Supports all
Textpredicates to search for text properties that matches a given word, prefix or regular expression.
- Geo: Supports all
Geopredicates to search for geo properties that are intersecting, within, disjoint to or contained in a given query geometry. Supports points, lines and polygons for indexing. Supports circles, boxes and polygons for querying point properties and all shapes for querying non-point properties.
- Numeric Range: Supports all numeric comparisons in
- TTL: Supports automatically expiring indexed elements.
- Temporal: Millisecond granularity temporal indexing.
- Custom Analyzer: Choose to use a custom analyzer
Please see Version Compatibility for details on what versions of Solr will work with JanusGraph.
Solr Configuration Overview
JanusGraph supports Solr running in either a SolrCloud or Solr
Standalone (HTTP) configuration for use with a mixed index
(see Mixed Index).
The desired connection mode is configured via the
mode which must be set to either
former being the default value. For example, to explicitly specify that
Solr is running in a SolrCloud configuration the following property is
specified as a JanusGraph configuration property:
These are some key Solr terms:
- Core: A single index on a single machine
- Configuration: solrconfig.xml, schema.xml, and other files required to define a core.
- Collection: A single logical index that can span multiple cores on different machines.
- Configset: A shared configuration that can be reused by multiple cores.
Connecting to SolrCloud
When connecting to a SolrCloud cluster by setting the
mode equal to
cloud, the Zookeeper URL (and optionally port) must be specified so
that JanusGraph can discover and interact with the Solr cluster.
1 2 3
A number of additional configuration options pertaining to the creation of new collections (which is only supported in SolrCloud operation mode) can be configured to control sharding behavior among other things. Refer to the Configuration Reference for a complete listing of those options.
SolrCloud leverages Zookeeper to coordinate collection and configset information between the Solr servers. The use of Zookeeper with SolrCloud provides the opportunity to significantly reduce the amount of manual configuration required to use Solr as a back end index for JanusGraph.
A configset is required to create a collection. The configset is stored in Zookeeper to enable access to it across the Solr servers.
Each collection can provide its own configset when it is created, so that each collection may have a different configuration. With this approach, each collection must be created manually.
A shared configset can be uploaded separately to Zookeeper if it will be reused by multiple collections. With this approach, JanusGraph can create collections automatically by using the shared configset. Another benefit is that reusing a configset significantly reduces the amount of data stored in Zookeeper.
Using an Individual Configset
In this example, a collection named
verticesByAge is created manually
using the default JanusGraph configuration for Solr that is found in the
distribution. When the collection is created, the configuration is
uploaded into Zookeeper, using the same collection name
for the configset name. Refer to the Solr Reference Guide
for available parameters.
Define a mixed index using
JanusGraphManagement and the same
1 2 3 4
Using a Shared Configset
When using a shared configset, it is most convenient to upload the
configuration first as a one time operation. In this example, a
janusgraph-configset is uploaded in to Zookeeper using
the default JanusGraph configuration for Solr that is found in the
distribution. Refer to the Solr Reference Guide
for available parameters.
1 2 3 4 5 6 7
When configuring the SolrCloud indexing backend for JanusGraph, make
sure to provide the name of the shared configset using the
1 2 3 4
Define a mixed index using
JanusGraphManagement and the collection
1 2 3 4
Connecting to Solr Standalone (HTTP)
When connecting to Solr Standalone via HTTP by setting the
http, a single or list of URLs for the Solr instances must be
1 2 3
Additional configuration options for controlling the maximum number of connections, connection timeout and transmission compression are available for the HTTP mode. Refer to the Configuration Reference for a complete listing of those options.
Solr Standalone is used for a single instance, and it keeps configuration information on the file system. A core must be created manually for each mixed index.
To create a core, a
core_name and a
configuration directory is
required. Refer to the Solr Reference Guide
for available parameters. In this example, a core named
is created using the default JanusGraph configuration for Solr that is
found in the distribution.
Define a mixed index using
JanusGraphManagement and the same core
1 2 3 4
When connecting to a Solr environment that is protected by Kerberos we must specify that Kerberos is being used and reference a JAAS configuration file to properly configure the Solr Clients. This configuration is required when Kerberos is in use regardless of the mode in which Solr is operating (SolrCloud or Solr Standalone).
The JAAS configuration file is supplied by ensuring that you set the java system property
java.security.auth.login.config with the absolute path to the file. This property should be set using JVM options. For example to run
gremlin.sh you would need to set the
JAVA_OPTIONS environment variable prior to running the script:
For details on the content required in the JAAS configuration file refer to the https://lucene.apache.org/solr/guide/7_0/kerberos-authentication-plugin.html#define-a-jaas-configuration-file[Solr Reference Guide].
Solr Schema Design
Dynamic Field Definition
By default, JanusGraph uses Solr’s Dynamic Fields feature to define the field types for all indexed keys. This requires no extra configuration when adding property keys to a mixed index backed by Solr and provides better performance than schemaless mode.
JanusGraph assumes the following dynamic field tags are defined in the
backing Solr collection’s schema.xml file. Please note that there is
additional xml definition of the following fields required in a solr
schema.xml file in order to use them. Reference the example schema.xml
file provided in the
./conf/solr/schema.xml directory in a JanusGraph
installation for more information.
1 2 3 4 5 6 7 8 9 10
In JanusGraph’s default configuration, property key names do not have to end with the type-appropriate suffix to take advantage of Solr’s dynamic field feature. JanusGraph generates the Solr field name from the property key name by encoding the property key definition’s numeric identifier and the type-appropriate suffix. This means that JanusGraph uses synthetic field names with type-appropriate suffixes behind the scenes, regardless of the property key names defined and used by application code using JanusGraph. This field name mapping can be overridden through non-default configuration. That’s described in the next section.
Manual Field Definition
If the user would rather manually define the field types for each of the
indexed fields in a collection, the configuration option
needs to be disabled. It is important that the field for each indexed
property key is defined in the backing Solr schema before the property
key is added to the index.
In this scenario, it is advisable to enable explicit property key name to field mapping in order to fix the field names for their explicit definition. This can be achieved in one of two ways:
- Configuring the name of the field by providing a
mapped-nameparameter when adding the property key to the index. See Individual Field Mapping for more information.
- By enabling the
map-nameconfiguration option for the Solr index which will use the property key name as the field name in Solr. See Global Field Mapping for more information.
JanusGraph can also interact with a SolrCloud cluster that is configured
for schemaless mode.
In this scenario, the configuration option
dyn-fields should be
disabled since Solr will infer the field type from the values and not
the field name.
Note, however, that schemaless mode is recommended only for prototyping and initial application development and NOT recommended for production use.
Collection Does Not Exist
The collection (and all of the required configuration files) must be initialized before a defined index can use the collection. See Connecting to SolrCloud for more information.
When using SolrCloud, the Zookeeper zkCli.sh command line tool can be used to inspect the configurations loaded into Zookeeper. Also verify that the default JanusGraph configuration files are copied to the correct location under solr and that the directory where the files are copied is correct.
Cannot Find the Specified Configset
When using SolrCloud, a configset is required to create a mixed index for JanusGraph. See Configset Configuration for more information.
- If using an individual configset, the collection must be created manually first.
- If using a shared configset, the configset must be uploaded into Zookeeper first.
You can verify that the configset and its configuration files are in
/configs. Refer to the Solr Reference Guide
for other Zookeeper operations.
1 2 3 4 5
HTTP Error 404
This error may be encountered when using Solr Standalone (HTTP) mode. An example of the error:
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Make sure to create the core manually before attempting to store data into the index. See Core Configuration for more information.
Invalid core or collection name
The core or collection name is an identifier. It must consist entirely of periods, underscores, hyphens, and/or alphanumerics, and also it may not start with a hyphen.
Irrespective of the operation mode, a Solr instance or a cluster of Solr instances must be running and accessible from the JanusGraph instance(s) in order for JanusGraph to use Solr as an indexing backend. Check that the Solr cluster is running correctly and that it is visible and accessible over the network (or locally) from the JanusGraph instances.
JTS ClassNotFoundException with Geo Data
Solr relies on Spatial4j for geo processing. Spatial4j declares an
optional dependency on JTS ("JTS Topology Suite"). JTS is required for
some geo field definition and query functionality. If the JTS jar is not
on the Solr daemon’s classpath and a field in schema.xml uses a geo
type, then Solr may throw a ClassNotFoundException on one of the missing
JTS classes. The exception can appear when starting Solr using a
schema.xml file designed to work with JanusGraph, but can also appear
CREATE in the Solr CoreAdmin API. The exception appears in
slightly different formats on the client and server sides, although the
root cause is identical.
Here’s a representative example from a Solr server log:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Here’s what normally appears in the output of the client that issued the
CREATE command to the CoreAdmin API:
1 2 3 4 5
This is resolved by adding the JTS jar to the classpath of JanusGraph and/or the Solr server. JTS is not included in JanusGraph distributions by default due to its LGPL license. Users must download the JTS jar file separately and copy it into the JanusGraph and/or Solr server lib directory. If using Solr’s built in web server, the JTS jar may be copied to the example/solr-webapp/webapp/WEB-INF/lib directory to include it in the classpath. Solr can be restarted, and the exception should be gone. Solr must be started once with the correct schema.xml file in place first, for the example/solr-webapp/webapp/WEB-INF/lib directory to exist.
To determine the ideal JTS version for Solr server, first check the
version of Spatial4j in use by the Solr cluster, then determine the
version of JTS against which that Spatial4j version was compiled.
Spatial4j declares its target JTS version in the pom for the
Copy the JTS jar to the server/solr-webapp/webapp/WEB-INF/lib directory
in your solr installation.
Advanced Solr Configuration
This section covers installation and configuration of JanusGraph with DataStax Enterprise (DSE) Search. There are multiple ways to install DSE, but this section focuses on DSE’s binary tarball install option on Linux. Most of the steps in this section can be generalized to the other install options for DSE.
Install DataStax Enterprise as directed by the page Installing DataStax Enterprise using the binary tarball.
DSE_HOME and append to
PATH in your shell environment. Here’s
an example using Bash syntax:
Install JTS for Solr. The appropriate version varies with the Spatial4j version. As of DSE 4.5.2, the appropriate version is 1.13.
Start DSE Cassandra and Solr in a single background daemon:
1 2 3 4 5
The previous command will write some startup information to the console
and to the logfile path
log4j.appender.R.File configured in
Once DSE with Cassandra and Solr has started normally, check the cluster
nodetool status. A single-instance ring should show one
node with flags *U*p and *N*ormal:
1 2 3 4 5 6 7
Next, switch to Gremlin Console and open a JanusGraph database against the DSE instance. This will create JanusGraph’s keyspace and column families.
1 2 3 4 5 6 7 8 9 10 11
Keep this Gremlin Console open. We’ll take a break now to install a Solr core. Then we’ll come back to this console to load some sample data.
Next, upload configuration files for JanusGraph’s Solr collection, then create the core in DSE:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
Now the JanusGraph database and backing Solr core are ready for use. We can test it out with the Graph of the Gods dataset. Picking up the Gremlin Console session started above:
1 2 3
Now we can run any of the queries described in Getting started.
Queries involving text and geo predicates will be served by Solr. For
more verbose reporting from JanusGraph and the Solr client, run
gremlin.sh -l DEBUG and issue some index-backed queries.