Semantics Web for the Microsoft .NET Framework

Querying via a ClientModel

2010-10-11T07:29:00.001-07:00

The .NET API available in the Semantics Platform allows an application to query one or more sources of data in a federated fashion using SPARQL. The data sources can be local in-memory graphs or remote graphs hosted on another server. The ClientModel class provides the primary interface for executing SPARQL queries.

In this posting I will go through a simple example of how to use a ClientModel to query the contents of two RDF files.

I will start by constructing two in-memory GraphDataSource objects. Each data source will be filled with the contents of a local RDF file.

GraphDataSource dataGraph = new GraphDataSource();

using (StreamReader r = new StreamReader("c:\\data.nt"))
    dataGraph.Read<NTriplesReader>();

GraphDataSource ontologyGraph = new GraphDataSource();

using (StreamReader r = new StreamReader("c:\\ontology.nt"))
    ontologyGraph.Read<NTriplesReader>();

Next I construct the ClientModel object and add both of the graph object to the DataSources collection. Each data source added to a ClientModel must have a unique URI to identify it.

ClientModel model = new ClientModel();

model.DataSources["http://example.org/data"] = dataGraph;
model.DataSources["http://example.org/ontology"] = ontologyGraph;

As a convenience, I will define some namespaces on the ClientModel to make my SPARQL queries easier to write.

model.ParserOptions.Namespaces["rdf"] = NS.Rdf;
model.ParserOptions.Namespaces["rdfs"] = NS.Rdfs;
model.ParserOptions.Namespaces["x"] = "http://example.org/";

To execute the SPARQL query I simply call the Query method with a valid SPARQL query string. Note because I defined the namespaces on the ClientModel object I do not need to include them in the query using PREFIX statements.

Table results = model.Query(@"
    select ?label 
    from x:data 
    from x:ontology
    where {
        ?s rdf:type ?type; rdfs:label ?label.
        ?type rdfs:subClassOf x:Thing.
    }");

Finally I write the results of the query to the console window by iterating over the rows in the result table.

foreach (TableRow row in results.GetRows())
    Console.WriteLine(row[0].Value);

Loading large data sets via the Jobs Framework

2010-09-30T08:23:00.001-07:00

The Semantics.Datacenter Jobs Framework provides a powerful mechanism for processing and loading very large data sets into an RDF graph. For very large data sets it can be impractical to load all of the data into memory at once for processing. The solution the Jobs Framework provides is based on the following approach:

Breaking the input data files into a larger set of smaller files that can processed independently, as tasks, by the worker nodes in the Semantics.Datacenter cluster.
Each task running on the worker nodes then combine a subset of these data files into a new set of files that provide the required date or serve as input into another processing step (repeat step #1).

In the case of a typical data load job, it would look like:

Run multiple tasks to split the input data files into a smaller set of data files. These files may or may not be RDF data files.
Run multiple tasks to process the data files from step #1. Each task would operate on a subset of the files produced in step #1. A task would produce RDF statements and write each statement to a file that is partitioned typically by subject. These files provide an efficient binary representation of the RDF statements and are referred to as shard files.
Run multiple tasks that each produce an in-memory graph partition file that is suitable for handling queries. Each task would gather the shard files that contain the RDF data for its respective partition an produce a partition file.
(Optional) Run multiple tasks to compile each partition file into a read-only partition file. These files are much smaller than a writable partition file and therefore will consume less memory once loaded into a Semantics.Datacenter graph. This step can be skipped if a writable graph is desired.
Run multiple tasks, one for each partition, to load the partition files into a Semantics.Datacenter hosted graph.

For more information on Semantics.Datacenter Partitioned In-Memory Graphs read this posting.
For more information on setting up a Semantics.Datacenter cluster read this posting.

Workflow Job

Semantics.Datacenter allows you to create jobs based on the Windows Workflow Foundation. Semantics.Datacenter includes a set of pre-build Workflow Activities that can be used to load your data. You can also create your own Workflow Activities using C# if you require some custom processing that is part of the data load or create your own Job type that is not based on Windows Workflows. In this article I will focus on the Workflow Job using our standard Activities.

Job Workspace

A data load job requires a set of directories that can be used by the tasks for processing data. Before creating your Workflow you should figure out this structure. The table below describes the directory structure I will use for my job.

Directory	Contents
c:\job\rdf	The input RDF files.
c:\job\rdf\split	The split set of RDF files.
c:\job\parts\shards	The shard files resulting from parsing the RDF files.
c:\job\parts	The partition files generated from the shard files.
c:\job\parts\ro	The read-only partition files compiled from the writable partition files.

Creating the Workflow Job

The Workflow Job can be created using the Model Manager. Connect to the Semantics.Datacenter cluster coordinator server and select the “Create…” option from the Jobs context menu in the tree control. This will display the Workflow Job designer where Activities can be added and configured to a Workflow Job. To add an Activity simply select one from the list and drag it into the workflow. Select the Activity in the workflow to display its property page where the activity is configured.

The following sections will describe the Activities I added to my Workflow Job in the order that they will be executed.

SplitFilesActivity

The SplitFilesActivity will split all the RDF files in a directory into a set of smaller files. The table below lists the settings for this activity.

Property	Value	Description
Name	SplitFiles	The name of the activity instance.
InputDirectory	c:\job\rdf	The directory containing the input RDF files
OutputDirectory	c:\job\rdf\split	The output directory containing the split RDF files.
NumLines	50000	The maximum number of statements allowed in an output file.
ParseType	NTriples	The format of the input RDF files.
OverwriteDirectory	True	Deletes the contents of the output directory before processing the input.

ParseFileActivity

The ParseFilesActivity will parse all the RDF files in a directory into a set of shard files that contain a binary representation of the RDF statements. The table below lists the settings for this activity.

Property	Value	Description
Name	ParseFiles	The name of the activity instance.
InputDirectory	SplitFiles.OutputDirectory	The directory containing the input RDF files
OutputDirectory	c:\job\parts\shards	The output directory containing the shard files.
NumLines	50000	The maximum number of statements allowed in an output file.
ParseType	NTriples	The format of the input RDF files.
OverwriteDirectory	True	Deletes the contents of the output directory before processing the input.
BatchSize	10	The maximum number of input files to be processed by a single task.
PartitionCount	16	The number of partition files that can be created from the output shard files.
IgnoreParseErrors	False	If set to True any parse errors will be ignored and processing will continue.

LoadPartitionsActivity

The LoadPartitionsActivity will load all of the shard files into a set of graph partition files. The table below lists the settings for this activity.

Property	Value	Description
Name	CreatePartitions	The name of the activity instance.
InputDirectory	ParseFiles.OutputDirectory	The directory containing the input shard files
OutputDirectory	c:\job\parts	The output directory containing the partition files.
Arity	3	The arity of the graph (3=triples, 4=quads)
IncludeCaseInsensitiveIndex	False	If True, a case-insensitive index is created
IncludeFullTextIndex	False	If True, a full-text index is created
PartitionCount	ParseFiles.PartitionCount	The number of partition files that will be created from the shard files.

CompilePartitionsActivity

The CompilePartitionsActivity will compile a set of writable partition files into a set of read-only partition files.The table below lists the settings for this activity.

Property	Value	Description
Name	CreateReadOnly	The name of the activity instance.
InputDirectory	CreatePartitions.OutputDirectory	The directory containing the writable partition files
OutputDirectory	c:\job\parts\ro	The output directory containing the read-only partition files.

LoadGraphPartitionsActivity

The LoadGraphPartitionsActivity will load a set of partition files into a Semantics.Datacenter hosted graph. The graph must exist otherwise the CreateGraphActivity should be used prior to this activity to create it. The table below lists the settings for this activity.

Property	Value	Description
Name	LoadGraph	The name of the activity instance.
InputDirectory	CreateReadOnly.OutputDirectory	The directory containing the partition files
ConnectionString	net.tcp://host:7055/DataService	The connection string to the Semantics.Datacenter server that is hosting the graph to be loaded.
GraphUri	http://example.org/data	The URI of the graph.

Save and Run

After creating the Workflow Job you should save it to a file so it can be used again, especially if your job fails for some reason. This is done by clicking on the “Save…” button and choosing a file name. Then click on “OK” button on the Workflow Job designer and to submit the job to the job queue where its tasks will be processed by one or more worker nodes in the Semantics.Datacenter cluster.

Semantics Platform v2.0 Maintenance Release Available.

2010-09-29T11:21:00.001-07:00

(September 29, 2010) A maintenance release of the Semantics Platform v2.0.1.4 is now available for download from the Intellidimension web site at:

http://www.intellidimension.com/downloads/

This release includes the following updates:

Fixed issue with range constraints in read-only in-memory graphs.
Fixes to SPARQL query compiler.
Query engine optimizations.
Resolved issue with compilation of free-text expression that include Unicode characters.
Enhancements to build in workflow tasks in the Semantics.Datacenter Job Framework.
Improvements to job workflow editor in Model Manager.
Other minor bug fixes

Partitioned In-Memory Graphs

2010-09-27T05:09:00.001-07:00

The Semantics Platform supports partitioned in-memory graphs for storing and querying RDF data. These graphs use an extremely compact indexing format that minimizes the system memory required. As the name implies, the data in the graph can be partitioned to allow parallel read and write access to the data. This parallel access can greatly improve query execution times. Partitioned in-memory graphs manage all their data within the memory space of a single computer process. This is in contrast to a distributed graph which can manage RDF data in multiple processes on multiple machines (I will discuss these is another post).

A writable partitioned in-memory graph can compiled into a read-only form that requires even less memory to hold the same amount of data.

In the case of Semantics.Datacenter, partitioned in-memory graphs are backed by files, with support for journaling, to provide persistent storage of RDF data. Each partition in graph is storage in a separate file. This enables parallel read/write access to the data in the graph and vastly improves data load times.

Creating a Partitioned In-Memory Graph

Using the Model Manager you can create a partitioned in-memory graph in local client model or on a running instance of a Semantics.Datacenter server. The image below shows the screen for creating a graph on Semantics.Datacenter. The graph can be configured to store either triples or quads and also supports full-text indexing.

Loading a Partitioned In-Memory Graph

In a previous posting I discussed how to load data into a partitioned in-memory graph hosted in a local client model using the Model Manger (see Working with large RDF files in the Model Manager). However in the case of a graph hosted by Semantics.Datacenter this is accomplished much more efficiently by using the Semantics.Datacenter Jobs Framework. The Jobs Framework allows you partition the data load process into a set of tasks that can be run in parallel which often requires much less memory. I will discuss this process in another post.

An introduction to Semantics.Datacenter clusters

2010-09-23T06:32:00.001-07:00

Semantics.Datacenter is based on Intellidimension’s proprietary distributed graph store called IMDB. Semantics.Datacenter is deployed in a clustered configuration. A Semantics.Datacenter cluster is a set of network addressable services (“servers”) based on WCF that can be running one or more computers. A cluster must have one coordinator server and one or more worker servers. A cluster also requires a shared directory that is accessible by all servers in the cluster for both reading and writing files. If the cluster is deployed on multiple computers the shared directory must be on a networked storage device. The coordinator and worker nodes can be deployed to a single computer using a local file system directory.

The coordinator server manages all the activities for the cluster and is generally not used for application data storage or processing. The coordinator server manages the following:

The Job Queue
Server failover
Replication Coordination

The worker servers in the cluster are responsible for providing the bulk of the services for an application such as:

Hosting of Graphs
Execution of Tasks (a unit of work in a Job)

Deploying a cluster involves deciding on the hardware resources that you wish to use. This includes the physical computers as well as any network storage that will be required. Each server in the cluster will also require its own network address and the cluster coordinator network address must be reachable by all the worker servers in the cluster.

Step 1: Install Semantics.Datacenter
The first step is to install the product on each computer that will be used in the cluster.

Step 2: Install a Coordinator Server
Launch the IMDB Setup utility on the computer that will be hosting the coordinator server. From the main screen select the option to “Add a new server” followed by the option to install “A coordinator node in a new cluster”. Follow the steps on the rest of the screens that will prompt you for a network address and the location of the shared directory for the cluster.

Step 3: Install the Worker Server(s)
For each worker server on each computer repeat Step 2 except on the second screen select the option to install “A worker node in an existing cluster”. In order to complete the installation you will be required to provide the network address of coordination server.

Next Steps
Now that the cluster is up and running you will need to configure it for your application. This configuration will vary significantly from application to application and often requires a large amount of testing and tuning to get right depending on the application. I will address several of these topics in future postings, including:

Graph Partitioning
Distributed Graphs
Data Replication
SQL Server Replication
Data Load Jobs

Working with large RDF files in the Model Manager

2010-09-15T05:02:00.001-07:00

Often it is necessary to load and query a large RDF file to understand the data you are working with. The Model Manager is a good tool for this but has some limitations that I will describe in this posting.

In-Memory Graphs are intended to be used for short term storage of RDF data for performing some application processing. They are not intended to be used as a persistent store of RDF data. In the case of the Model Manger, any data imported into an In-Memory graph is stored in the project file. A Model Manager project file is primarily for storing the configuration of a model. It is not very good at all for storing large amounts of RDF data. Meaning it is not a database.

We have two database products that are very good at storing large amounts of RDF data. These products are called Semantics.Server and Semantics.Datacenter.

The links below provide information about both of these products.

http://www.intellidimension.com/products/semantics-server/

http://www.intellidimension.com/products/semantics-datacenter/

We have two instructional videos that show how to setup, load and query RDF data using each (see links below).

http://www.intellidimension.com/developers/videos/v2.aspx

http://www.intellidimension.com/developers/videos/v3.aspx

If you do not want to use one of our database server products. There is a more efficient way to work with large RDF data files using In-Memory graphs in the Model Manager. You can "Import" a RDF file into an In-Memory graph but then save it as an image file. This is done by creating an image file or read-only image file from the menu on the In-Memory graph. These files have a very compact storage format and can be loaded much more quickly than parsing an RDF file via the import menu. If you don't not intend on modifying any of the data I suggest you use read-only image file.

Now here is the IMPORTANT step. Drop the graph before saving your project so its contents are not saved to the project file. We are working on a better solution for this that will be available in a future release.

When you open your project file next time in the Model Manager. Just create a new In-Memory graph and load the image file using the menu on that graph. This will load much faster than an RDF file because it does not need to parse the file and it can be read directly into memory. Of coarse you will still need enough memory on you local machine to hold the entire contents of the image file but this will be less than the memory that is needed to parse the RDF file or load the data from the Model Manager project file.

Registering Custom SPARQL Functions

2010-09-14T10:09:00.001-07:00

The Semantics Platform allows custom SPARQL functions to be registered with a client model. Once a function has been registered it can be used in a SPARQL query using the URI it was registered with. A custom SPARQL function can be implemented as a static method on a .NET class that has the proper delegate type. The Semantics Platform enables all the custom SPARQL functions in an assembly to be loaded provided they are declared with the proper attributes.

The example below shows a custom SPARQL function implemented in C# that supports the delegate type (ClientFunctionCall). The function calculates the area of a circle based on a single argument that represents the radius. The function checks that the proper number of arguments are provided. It calculates the area and returns it as the result argument. If the function is called with a result argument set to a value then it must be compared to the calculated area to see if the expression evaluates to true. The function should return false if the function fails to generate a value or the comparison fails.

[ClientFunctionAttribute(
    "http://example.org/fn/areaOfCircle", 
    "[bf]b", 
    Deterministic=true)]
public static bool AreaOfCircleFunction(
    FunctionCallParams funcParams)
{
    TableRow funcArgValues = funcParams.ArgRow;

    // Make sure we have exactly 2 arguments in 
    // our arguments collection.
    //
    // index 0: The result value
    // index 1: The radius
    if (funcArgValues.Count == 2 && 
        funcArgValues[1] != null && 
        funcArgValues[1] is RdfLiteral && 
        ((RdfLiteral)funcArgValues[1]).IsNumeric)
    {
        // A = PI * R^2
        double area = Math.PI * Math.Pow(
            (double)(RdfLiteral)funcArgValues[1], 2);

        RdfLiteral resultValue = area;

        // If a result value was provided (index 0) 
        // then test for equality.
        if (RdfValue.IsNull(funcArgValues[0]) || 
            funcArgValues[0] == resultValue)
        {
            funcArgValues[0] = resultValue;
            return true;
        }
    }

    return false; //evaluated as false
}

Notice, the method is declared with an attribute called ClientFunctionAttribute. This attribute requires a URI to identify the SPARQL function and a binding pattern. A binding pattern is a form of a regular express where ‘b’ represents a bound argument and ‘f’ represents an unbound argument (aka free). In the example above the binding pattern ‘[bf]b’ indicates the result argument can be bound or free and the first argument must be bound. The method attribute is also sets the property Deterministic to true to indicate it is a deterministic function.

One way to register the custom SPARQL function is using the Model Manager. Open a model, new or existing, and right click on the Functions folder to display the menu and select ‘Load Assembly…’.

This will display the dialog box shown above. Simply browse and select the assembly file that contains the custom SPARQL function(s). The dialog will display all the methods found in the assembly that are declared with the proper attribute and delegate type. You may select specific SPARQL functions to load or just load them all.

Once loaded they may be used in a SPARQL query using the Model Manager query window as shown below. The function is called using the URI it was registered with.

The custom functions assembly can also be loaded via the Semantics Platform .NET API using the helper class ClientFunctionLoader. This class will create the required class for registering a SPARQL function with an instance of a ClientModel. The C# code below provides and example of its use.

ClientModel model = new ClientModel();

ClientFunctionLoader loader = new ClientFunctionLoader(
    "c:\\SPARQL\\CustomFunctions.dll");

foreach (RdfUri uri in loader.FunctionUris)
{
    model.RegisterFunction(uri, loader.CreateFunction(uri));
}

This is the easiest way to register a custom SPARQL function.

Semantics Platform v2.0 Released: Semantic Web technology for Windows, RDF Libraries for .NET/C# and SPARQL on SQL Server.

2010-06-07T04:47:00.001-07:00

Intellidimension has released Semantics Platform v2.0 that provides the most comprehensive set of Semantic Web technology for Microsoft Windows, the .NET Framework (C#) and SQL Server.

http://www.intellidimension.com/products/semantics-platform/

Semantics Platform is a family of products that includes:

Semantics.Datacenter a distributed RDF data store that allows both graphs and processing jobs to be partitioned over a cluster of servers. It supports both distributed SPARQL query and inference rule execution. Semantics.Datacenter also contains a library of built-in jobs that can be extended using C#. Jobs are configured using a simple workflow editor and when executed they can leverage the parallel processing capabilities of the cluster.

http://www.intellidimension.com/products/semantics-datacenter/

Semantics.Server continues to provide a robust option for storing RDF data within SQL Server along with allowing SQL Server to execute SPARQL queries and inference rules.

http://www.intellidimension.com/products/semantics-server/

Semantics.Framework Express Edition is all the core RDF libraries used in the Semantics Platform that are free for use. C# programmers can use these .NET libraries to load and store RDF data along with perform complex SPARQL and inference rule processing. The Express Edition also includes visual tools for working with RDF files and testing SPARQL queries and inference rules.

http://www.intellidimension.com/products/semantics-framework/

Semantics.Developer provides developers with full access to all the technology in the Semantics Platform in a cost-effective manner. A free 60-day evaluation is available for this product.

http://www.intellidimension.com/products/semantics-developer/

Semantics Platform 2.0 (Beta) Released

2010-05-03T13:45:00.001-07:00

Intellidimension has just released the Semantics Platform 2.0 Beta for free download and evaluation. The Semantics Platform 2.0 includes the following improvements:

A new highly-scalable in-memory graph database (IMDB). IMDB provides a scale-out strategy with parallel query execution and failover recovery. IMDB is included in a new product call Semantics.Datacenter.
Significant improvements to query plan, compilation and execution for SPARQL queries and inference rules.
Numerous enhancements to Model Manager including new multiple file project format.

Visit the following URL for more information about the Semantics Platform 2.0:

http://www.intellidimension.com/products/semantics-platform/

Check out our new online video library for product overviews and tutorials:

http://www.intellidimension.com/developers/videos/

Querying for intersections between sets.

2010-02-05T08:24:00.001-08:00

A SPARQL query is often needed to find the intersection between sets in a data model. This article provides a simple example based on landmarks and regions. In this data model a landmark belongs to a region and regions can belong to other regions creating a hierarchical collection of sets. The goal is to find the common regions for all landmarks. The trick is we want the best match for the region. For example Boston Common and Fenway Park are both located in Boston; therefore, we do not want to return results showing regions that contain Boston such as Massachusetts, United States of America or North America.

The SPARQL query below uses a rule to generate some data for the examples. It uses a couple of other rules to recursively generation all the containment relationships for the regions. The query itself uses negation to eliminate results that have a region that contains another region for the same two landmarks.

prefix x: <http://example.org/>

rulebase (
# generate some facts for our example
construct {
# places
x:Boston x:isPartOf x:Massachusetts.
x:Massachusetts x:isPartOf x:UnitedStatesOfAmerica.
x:UnitedStatesOfAmerica x:isPartOf x:NorthAmerica.
x:Boston x:isPartOf x:Massachusetts.
x:Paris x:isPartOf x:France.
x:France x:isPartOf x:Europe.
x:Toronto x:isPartOf x:Ontario.
x:Ontario x:isPartOf x:Canada.
x:Canada x:isPartOf x:NorthAmerica.

#landmarks
x:BostonCommon x:hasLocation x:Boston.
x:FenwayPark x:hasLocation x:Boston.
x:EiffelTower x:hasLocation x:Paris.
x:CNTower x:hasLocation x:Toronto.
x:MississippiRiver x:hasLocation x:UnitedStatesOfAmerica.
x:GreatLakes x:HasLocation x:NorthAmerica.
}

# walk the region classification
construct {?a x:isPartOf ?b} where {?a x:isPartOf ?c. ?c x:isPartOf ?b}
construct {?a x:isSameAsOrPartOf ?a} where {{?a x:isPartOf ?x.}union{?x x:isPartOf ?a}}
construct {?a x:isSameAsOrPartOf ?b} where {?a x:isPartOf ?b}
)

# find the smallest common location between landmarks
select ?lm1 ?lm2 ?region where {
?lm1 x:hasLocation ?loc1.
?lm2 x:hasLocation ?loc2.
filter(?lm1 != ?lm2)
?loc1 x:isSameAsOrPartOf ?region.
?loc2 x:isSameAsOrPartOf ?region.
#make sure there is not a closer match
not(?loc1 ?loc2 ?region)
{
    ?loc1 x:isSameAsOrPartOf ?region2.
    ?loc2 x:isSameAsOrPartOf ?region2.
    ?region2 x:isPartOf ?region.
}
}

When the query above is executed it produces the following results.

        1: <http://example.org/BostonCommon>    <http://example.org/CNTower>    <http://example.org/NorthAmerica>.

       2: <http://example.org/BostonCommon>    <http://example.org/FenwayPark>    <http://example.org/Boston>.

       3: <http://example.org/BostonCommon>    <http://example.org/MississippiRiver>    <http://example.org/UnitedStatesOfAmerica>.

       4: <http://example.org/CNTower>    <http://example.org/BostonCommon>    <http://example.org/NorthAmerica>.

       5: <http://example.org/CNTower>    <http://example.org/FenwayPark>    <http://example.org/NorthAmerica>.

       6: <http://example.org/CNTower>    <http://example.org/MississippiRiver>    <http://example.org/NorthAmerica>.

       7: <http://example.org/FenwayPark>    <http://example.org/BostonCommon>    <http://example.org/Boston>.

       8: <http://example.org/FenwayPark>    <http://example.org/CNTower>    <http://example.org/NorthAmerica>.

       9: <http://example.org/FenwayPark>    <http://example.org/MississippiRiver>    <http://example.org/UnitedStatesOfAmerica>.

      10: <http://example.org/MississippiRiver>    <http://example.org/BostonCommon>    <http://example.org/UnitedStatesOfAmerica>.

      11: <http://example.org/MississippiRiver>    <http://example.org/CNTower>    <http://example.org/NorthAmerica>.

      12: <http://example.org/MississippiRiver>    <http://example.org/FenwayPark>    <http://example.org/UnitedStatesOfAmerica>.

Using rules to implement a cascading delete

2010-02-03T05:21:00.001-08:00

In an RDF model some entities are wholly owned by other entities. These are often referenced to as contained objects. When deleting an entity the desired logic is:

Delete all statements about the entity
Delete all statements about any contained objects referenced by the entity
Delete any statements that reference the contained object.

One way to implement this logic is to add some additional information about the predicates that are used to reference a contained object. For example for a property “foo” we can add the following statement to the ontology.

x:foo x:refType x:containedSubject.

This statement denotes that any entity that is the object value of a statement with the predicate x:foo is a contained object. Using this approach a generic set of recursive delete rules can be implemented for removing contained objects and the statements that reference them. It traditional database terminology this is referred to a a cascading delete.

The SPARQL rulebase and DELETE query shown below provides an example of how to implement this logic.

prefix x: <http://example.org/>

rulebase (
# remove references to the contained object.
select ?s ?p ?o ?r where {?s ?p ?o. 
  ?p x:refType x:containedObject. filter(?o = ?r)} as x:cascadeDelete

# remove all statements about the contained object.
select ?s ?p ?o ?r where {?s ?p ?o. 
  filter(?s=?r).} as x:cascadeDelete

# remove all referenced contained objects
select ?s ?p ?o ?r where {?r ?x ?y. 
  ?x x:refType x:containedObject. 
  x:cascadeDelete(?s, ?p, ?o, ?y).} as x:cascadeDelete
)

# delete a node where @target = the target URI
delete from <xxx> {?s ?p ?o} 
from <xxx>
where {x:cascadeDelete(?s, ?p, ?o, @target).}

Maintenance Release v1.2.0.56 Available

2010-01-05T04:50:00.001-08:00

January, 5 2010 - A new maintenance release of Semantics.SDK and Semantics.Server is now available for download at:

http://www.intellidimension.com/downloads

Release Notes:

Improvements to Model Manager SPARQL editor
Fixes to resource view in Model Manager
Fix to “index out of range” exception that sometimes occurred when using multiple data sources in a query.
Improvements to optimization algorithms in the query and rule compiler.
Fix to defect with SQL Server (SH) assemblies that resulted in an exception.

Using the SQL Data Source

2009-12-16T06:20:00.001-08:00

The SQL Data Source is a component available in the Semantics.SDK. It allows an application to connect to a traditional relational database and expose its data as RDF data. It is most commonly used by an semantic application to import relational data in to its RDF database. This article will provide a short example of how to use it.

Sample Data

Let’s first examine a very simple relational database. It contains one table “Employee” with its schema described below.

Column	Description
ID	The unique identifier of the employee. (PK, int)
Name	The employee name. (NOT NULL, varchar(255))

The table contains the following two records.

ID	Name
1	Fred Flintstone
2	Barney Rubble

Model Manager

I will first show how you can query the relational data as RDF using SPARQL via the Model Manager. First open the Model Manager and use the context menu on the Data Sources folder under the default model to add a SQL Data Source. The following dialog box will be displayed.

The table below describes the fields in the dialog box.

Data Source Uri	The URI for the graph representing the relational data in your local model. This will be the URI you will use in the FROM clause of your SPARQL queries.
Base Uri	The base URI is used by the SQL Data Source when generating URIs to describe the relational data as RDF data.
SQL Connection String	The connection string for the relational database.
Query Mode	The SQL Data Source can be configured to return RDF data that describes the data in the relational tables, that describes the relational schema, or both.

I configured the SQL Data Source to return both Table & Schema Data so we can see both. Using the Model Manager query window I executed the following SPARQL query.

select ?s ?p ?o 
from <http://example.org/graph/Employee>
where {?s ?p ?o} order by ?s ?p

I save the results as to turtle file (via a context menu on the query results) that contains the RDF data shown below:

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>.
@prefix ns1: <http://www.w3.org/2001/XMLSchema#>.
@prefix ns2: <http://www.w3.org/2000/01/rdf-schema#>.
@prefix ns3: <http://example.org/data/>.
@prefix ns4: <http://www.w3.org/2002/07/owl#>.

ns3:Employee_Name a ns4:DatatypeProperty;
    ns2:range [
        ns1:maxLength "255"^^ns1:int;
        ns2:subClassOf ns1:string];
    ns2:domain ns3:Employee.
ns3:Employee a ns4:Class.
ns3:Employee_ID a ns4:DatatypeProperty;
    ns2:range ns1:int;
    ns2:domain ns3:Employee.
<http://example.org/data/Employee,1> ns3:Employee_Name "Fred Flintstone"^^ns1:string;
    a ns3:Employee;
    ns3:Employee_ID "1"^^ns1:int.
<http://example.org/data/Employee,2> ns3:Employee_Name "Barney Rubble"^^ns1:string;
    a ns3:Employee;
    ns3:Employee_ID "2"^^ns1:int.

If you examine the RDF data closely you will see the table schema is described as an ontology using RDFS and OWL and the table data is described in terms of that ontology.

Using .NET

The Semantics.SDK allows you do perform the same task I performed above in your own .NET application. This is because the Model Manage itself is based on the Semantics.SDK .NET API. The code segment shown below creates SQL Data Source connected to the relation database, executes a SPARQL query on it, and writes the results to a turtle file.

ClientModel cm = new ClientModel();

SqlDatasource ds = new SqlDatasource(
    "trusted_connection=yes;;server=localhost\mssql2008;database=test", 
    "http://www.example.org/data/");

ds.QueryScope = QUERY_SCOPE.DATA | QUERY_SCOPE.ONTOLOGY;

cm.DataSources["http://www.example.org/graph/Employee"] = ds;

Table res = cm.Query(@"
        select ?s ?p ?o from <http://www.example.org/graph/Employee> 
        where {?s ?p ?o} order by ?s ?p");

GraphDataSource g = new GraphDataSource(res);

using (StreamWriter s = new StreamWriter("c:\\sample.ttl"))
{
    g.Format<TurtleFormatter>(s);
}

Ontology Mapping

Note that the ontology used by the SQL Data Source to describe the relational data isn’t always how you want to model the data in your application. However, now that you have the data described as RDF it in one ontology it is easy to map that data into another ontology using inference rules. That topic I will save for another posting.

Full Text Expressions

2009-12-09T07:23:00.001-08:00

Many applications need to efficiently search a RDF graph for any statements that contain a set of terms in a literal value. Semantics.SDK and Semantics.Server both provide support for full text expressions to meet this need. In the case of Semantics.Server these full text expressions are evaluated using full text capabilities of Microsoft SQL Server.

Full text expressions are based on the following constructs:

pattern	Any simple word match that contains no spaces or punctuation. The matched text must contain the exact word as specified in the pattern. ex: bluebird
prefix pattern	Any valid pattern followed by an asterisk (‘’). The matched text must contain a word that starts with the specified pattern. ex: blue
simple expression	A simple expression contains one or more patterns. The matched text must match all the patterns specified. ex: song bluebird
set expression	A set expression contains one or more sets. The matched text must match all the sets in the expression. ex: {blue red}{large medium}!{dog}
set	A set contains one or more patterns delimited by curly braces (‘{}’). The text must match at least one of the patterns in the set. ex: {blue* red*}
negated set	A negated set is a set that is prefixed with an exclamation mark (‘!’). The text must not match any of the patterns in the set. ex: !{dog cat}

Examples:

The following examples are based on the familiar input string:

“The quick fox jumps over the lazy brown dog.”

Expression	Match?
dog laz*	yes
{dog laz*}{quick}	yes
{dog laz*}!{quick}	no
umps	no

SPARQL:

The contains function allows you to use a full text expression in a SPARQL query. The example below shows the use of a full text expression in a SPARQL query.

select 1 where {filter(contains("The quick fox jumps over the lazy brown dog.", "dog laz*"))}

Getting counts via SPARQL

2009-11-11T06:26:00.001-08:00

Many applications need to acquire a count of either statements or subjects to provide some statistics about the information in an RDF graph. There are multiple ways to accomplish this:

Programmatically via the Semantics.SDK API.
Using the SPARQL aggregation function: count.

In this article we will focus on the SPARQL count function. The example below shows a SPARQL query that uses the count function to return the number of unique subjects that have a property x:gender equal to x:Male. The inference rules used in the example are solely for generating some sample data for this example.

prefix x: <http://example.org/>
prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

#infer some sample data
rulebase (
  construct {
    x:id-1 rdf:type x:Person.
    x:id-1 rdf:type x:Employee.
    x:id-1 rdf:type x:SalesPerson.
    x:id-1 x:gender x:Male.
    x:id-2 rdf:type x:Customer.
    x:id-2 x:gender x:Male.}
)
select count(?s) where {?s x:gender x:Male.}

When the query above is executed it returns a value of:

"2"^^<http://www.w3.org/2001/XMLSchema#int>

The count function can be used with multiple arguments to provide a count of distinct combinations of variables. For example, if we wanted to get a count of the the number of distinct combinations of subject and object values for a particular predicate value (rdf:type) the query could be modified to:

select count(?s, ?o) where {
  ?s ?p ?o. filter(?p=rdf:type)
} group by ?p

Notice we include a GROUP BY clause to force an aggregation of ?s and ?o. This query produces a result of:

"4"^^<http://www.w3.org/2001/XMLSchema#int>

If you would like to get a count of all the statements in a graph(s) you can execute the following query:

select count(?s, ?p, ?o) where  {?s ?p ?o. filter(?x=1)} group by ?x

We need to force a grouping on a dummy variable ?x to avoid having the query default to grouping on all variables (?s, ?p and ?o). This behavior will be modified in a future release to make the default grouping be on no variables so a dummy variable is not required. The query above produces the result:

"6"^^<http://www.w3.org/2001/XMLSchema#int>

I will try and get an article on using the Semantics.SDK API to get counts sometime soon.

OWL Union and Intersection Rules

2009-11-06T08:56:00.001-08:00

This article provides a simple example of how to use inference rules to map owl:unionOf and owl:intersectionOf class descriptions into rdfs:subClassOf relationships.

The example below shows a SPARQL query that has rules to:

(1) Interpret the RDF collection notation to assign membership.

(2) Create RDFS subclass relationships based on the OWL union and intersection descriptions.

(3) Generates some sample data to demonstrate the first two rules.

prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>
prefix owl: <http://www.w3.org/2002/07/owl#>
prefix x: <http://www.example.org/>

rulebase (
# (1) some rdf collection rules
  construct {?l rdfs:member ?i} where {?l rdf:first ?i}
  construct {?l rdfs:member ?i} where {?l rdf:rest ?r. ?r rdfs:member ?i}

# (2) owl to rdfs subclass rules
  construct {?s rdfs:subClassOf ?c} where {?s owl:intersectionOf ?l. ?l rdfs:member ?c}
  construct {?c rdfs:subClassOf ?u} where {?u owl:unionOf ?l. ?l rdfs:member ?c}

# (3) some sample data
  construct {x:HumanBeing  owl:unionOf  (x:Man x:Woman)}
  construct {x:TallMan  owl:intersectionOf  (x:Man x:TallThing)}
)

select ?sub ?super where {?sub rdfs:subClassOf ?super}

The SELECT query then produces a list of classes with their corresponding base class. When the query is executed it produces the following results:

<http://www.example.org/Man>    <http://www.example.org/HumanBeing>
<http://www.example.org/TallMan>    <http://www.example.org/Man>
<http://www.example.org/TallMan>    <http://www.example.org/TallThing>
<http://www.example.org/Woman>    <http://www.example.org/HumanBeing>

Note, the RDFS subclass rules are not recursive and just show the immediate base class.

Semantics v1.2 Released

2009-10-28T15:20:00.000-07:00

Intellidimension has just released version 1.2 of its Semantics.Server and Semantics.SDK. Both products are available for download from its web site.

http://www.intellidimension.com/downloads

Release Notes:

General:

Added native list data type and associated functions (list, nil, first, rest, listlength, listmember).

Semantics.Server:

Improvements to full-text search.
Additional auxiliary SQL statistics including expandable histogram (error based vs. fixed size)

Semantics.SDK:

Improvements to query compilation to deal with unsafe rules and aggregates.
Added rule rewriter support
Enhanced full-text search expressions.
Fixes to turtle parsing.
XSPARQL support
New data source (InMemoryGraph) with dramatically reduced memory footprint with full text index.
Improved query caching in ClientModel
Events added for improved manageability of models (ex Caching Events)
Reduced size of over the wire serialization of graphs and tables
Math functions added
Improvements to RDF serialization of objects.
Added hash index based data source (HashDataSource).
Modified quad serialization to be compatible with other implementations.
Added support for function cardinality callbacks in function registration
Enforce proper namespace conventions (blank namespace not allowed)

Model Manager:

Support for multiple models within a project file.
Support for multiple query windows.
Addition of Model Manager extension API
Improvements to Semantics.Server assembly installer.
Added SQL Data Source extension.
Added Semantics.Server extension.
Added Local Client Model extension.
New In-Memory Graph extension.
Support for adding custom functions via user interface.
Support for query parameters.
General improvements to user interface.
Addition of model namespace configuration.
Load and save of query results
New baseball sample project

Model Manager Improvements (v1.2)

2009-10-14T12:11:00.001-07:00

The Model Manager has a number of significant improvements in version 1.2 of the Semantics.SDK. The Model Manager is a Windows desktop application that exposes many of the features of the Semantics.SDK and Semantics.Server.

Projects and Models

The Model Manager operates on a single project at a time that consists of one or more models. A model is fundamentally a set of RDF graphs that can be queried. The Model Manager supports different types of models each with additional capabilities beyond supporting graphs and queries. The Model Manager allows you to create, edit and remove models from your project.

A model can be created from either the main application file menu or from a context menu on the models collection item in the tree control. A model can be saved to file as part of a project or in its own file where it can be opened in another project.

Local Model

A Local Model is a model that runs in the same process as the Model Manager. It supports federated queries across local in-memory RDF graphs, as well as, external sources of data such a relational store (SQL Data Source) or a Semantics.Server graph (Server Graph). In addition of data sources to a Local Model also supports:

Custom Functions
Rulebases
Graph Aliases
Namespaces

Server Model

A Server Model is a model that runs in a separate process from that of the Model Manager. Server Models support one or more RDF graphs and can be queried. Server Model graphs can also be added as data sources to a Local Model to perform federated queries with graphs not part of that Server Model. A Local Model can also be created from a Server Model to include all the graphs from that Server Model in the Local Model. The advantage being now you can make use of all the features supported by the Local Model (ex: custom functions or namespaces). Each type of Server Model supports a different set of features that may or may not be the same of those available in a Local Model.

Semantics.Server Model

A Semantics.Server Model is added to a project by selecting it from the new model menu. This will open the dialog shown below that allows you to define the SQL Server connection settings for the database.

If the database does not exist or Semantics.Server has not been installed on that database then you must first run the Semantics.Server installer by clicking on the “Setup” button.

SQL Data Source

The SQL Data Source exposes existing relational databases as RDF and allows them to be queried using SPARQL. You can add a SQL Data Source to your Local Model by selecting it on the add menu available as a context menu on the data source collection for the model. The following configuration dialog will open that allows you to defined the URI for the data source, the SQL connection string for the relational DB, the base URI for the relation data, and the query mode (data, ontology or both).

Queries

The Model Manager now supports a multiple query files open at the same time! To execute a query you must select the model you wish to query using the main menu and either open an existing query file or type in a query and press “F5”. The query results are shown in the grid below. The query results can be saved to a file using SPARQL Results XML format by using the context menu available from the grid.

Baseball Sample Model

2009-10-12T12:25:00.001-07:00

The v1.2 release of the Semantics.SDK includes a sample model that can be loaded into the Model Manager. The model contains a simple ontology for describing a baseball team lineup and a sample lineup that uses the ontology. To try out the sample, start the Model Manager application, click the menu item “Help -> Open Sample Project…” and select the “baseball.project” file.

The model has two graphs that contain the ontology (http://www.intellidimension.com/schema/baseball/20090721#Ontology) and instance data (http://www.intellidimension.com/schema/baseball/20090721#Data). The RDF data for these graphs are available in the files “baseball-ontology.ttl” and “team-data.ttl” respectively.

The sample includes a couple of SPARQL query files that can be used with this model. To open a SPARQL query click on the “File->Open->Query” menu and open one of the queries in the directory “<install dir>\samples\models\baseball”.

The query file “lineup-card.sparql” contains a SPARQL SELECT query that creates a result set in a format similar to lineup card. The other query file “validate-lineup.sparql” runs a set of inference rules over the lineup data to ensure that it’s valid. This query adds the same batter twice to the lineup to create a violation and report on it.

Security via SPARQL Expression Parsing

2009-09-30T11:33:00.001-07:00

Some common questions that often arise in the context of wanting to secure a SPARQL service are:

How do I prevent the execution of SPARQL INSERT and DELETE commands?
How do I protect certain graphs in model from being accessed?

An easy way to implement application level security in your services is to parse the incoming SPARQL query into an expression tree and interrogate it and either reject the query or modify it before allowing it to be executed. This is accomplished in the Semantics.SDK by using the SparqlExpression class.

The code snip below shows how to prevent INSERT and DELETE queries from being executed by first parsing a SPARQL query string and then examining its command type.

using Intellidimension.Sparql;

SparqlExpression expr = SparqlExpression.Parse(
"delete from <data> {?s ?p ?o} from <data> where {?s ?p ?o}");

if (expr.Command is CommandDelete || expr.Command is CommandInsert)
throw new Exception("Access denied!");

Access to a particular graph can be prevented by checking the graphs that are specified in the FROM or FROM NAMED clause of a SPARQL query. This can be accomplished using an expression by testing both the DefaultGraphs and NamedGraphs collections on the command.

SparqlExpression expr = SparqlExpression.Parse(
    "select ?s ?p ?o from <admin> where {?s ?p ?o}");

if (expr.Command.DefaultGraphs.Contains("admin") || 
    expr.Command.NamedGraphs.Contains("admin"))
    throw new Exception("Access denied!");

Expression parsing has uses outside of security such a dynamically building SPARQL queries or modifying existing ones. For example, its often used by services to set a default graph when none is specified in a query. This is done by modifying the expression object and then calling its ToString method to produce a SPARQL query string.

I find myself using the SparqlExpression class in almost every application I write.

SPARQL Switch/Case

2009-08-28T11:36:00.001-07:00

The Intellidimension SPARQL processor supports a Switch/Case syntax that allows for the implementation of If…Else If…Else If…Else… logic in your queries. The syntax of a switch statement consists of a variable list followed by a body that contains one or more graph patterns preceded by the case keyword.

switch ([variable list]) {
case {[graph pattern 1]}
..
case {[graph pattern N]}
}

The variable list defines one or more variables to be grouped by when evaluating the body of the statement. The graph patterns for the cases are evaluated once for each unique binding of the variables in the list. The case graph patterns are evaluated in order and once a pattern is matched for a set of variable bindings then no more patterns are evaluated (for that set of variable bindings).

The example below uses some inference rules to create some sample data and then runs a SPARQL SELECT query that utilizes a switch statement to get a label for a resource. The logic is to first use the property <name>, then <label> and if neither exists then to turn the subject URI into a literal.

# infer some data
rulebase (
  construct {<a> <label> "a"}
  construct {<b> <label> "b"}  
  construct {<b> <name> "bee"}  
  construct {<c> <size> "large"}
)

select ?s ?txt where {
  ?s ?p ?o.
  switch(?s) {
    case {?s <name> ?txt} #name first
    case {?s <label> ?txt} #then label
    case {filter(?txt=str(?s))} #default
  }
}

When this query is executed it produces the following results.

<a>    "a"
<b>    "bee"
<c>    "c"

The switch statement is perfect for generating a single value for a property based on some predetermined ranking of predicates.

Customizing the SPARQL DESCRIBE command

2009-07-25T11:39:00.001-07:00

The default behavior of the SPARQL DESCRIBE command is to return all statements where the specified URI is the subject value (ex: describe <some uri>). Many applications require a much richer description of a resource that may include information about referenced URIs also. When using Intellidimension Semantics the behavior of the DESCRIBE command can be modified by providing some special inference rules. The DESCRIBE command gets its behavior from a special inference rules call <description>. This rule (“relation”) takes four arguments as shown below.

The first three arguments are the statements describing the URI in the fourth argument.

select ?s ?p ?o ?r where {?s ?p ?o. filter(?s=?r)} as <description>

So consider the first three arguments outputs and the fourth as an input. This relation provides the default implementation of the DESCRIBE command by returning all statements about a specified URI.

To customize the DESCRIBE command, you must provide one or more rules that implement the <description> relation. For example, if you wanted to create a custom description for a factory entity that included information about chemicals it uses you could implement the following rules and include them in your DESCRIBE query.

rulebase (
    select ?s ?p ?o ?r where {
    ?r x:usesChemical ?s. 
    ?s ?p ?o.} as <description>
)
describe <somefactoryuri> from <data>

By adding this rule you will now get all statements about the factory plus all statements about any chemicals it uses. So you can now create different sets of rules for different entity types each with its own custom description.

Another way to accomplish this goal is to add some additional information to your ontology so that a single set of rules can be used for all entity types. For example if we added the statement {x:FactoryClass x:includedReference x:usesChemical} to the ontology a generic set of description rules could be written as follows.

rulebase (
    select ?s ?p ?o ?r where 
    {?s ?p ?o. filter(?s=?r)} as <description>
    
    # special rule to look for all marked predicates for the
    # class of the entity (?r) and recursively describe all
    # included references.
    select ?s ?p ?o ?r where {
    ?r rdf:type ?class.
    ?class x:includedReference ?ref.
    ?r ?ref ?included.
    <description>(?s, ?p, ?o, ?included).
    }as <description>
  )
describe <somefactoryuri>

Taking this approach we only need a single set of rules for our descriptions.

An Introduction to Inference Rules

2009-07-16T11:27:00.001-07:00

Intellidimension’s SPARQL extensions for inference rules are a powerful tool when developing a semantics based application. In this article I will show you a few simple examples to get you started. Intellidiemsion has added a SPARQL extension called a rulebase for defining inference rules to be used in any SPARQL query. When an SPARQL query is executed all the rules in all the rulebases are evaluated. A rulebase can be loaded into a model and referenced by uri or declared right inline with your SPARQL query. In this article let’s take a looks at an inline rulebase.

prefix rdf:<http://www.w3.org/1999/02/22-rdf-syntax-ns#>
prefix rdfs:<http://www.w3.org/2000/01/rdf-schema#>
prefix x:<http://example.org/>

rulebase(
# create some facts
construct {x:D rdfs:subClassOf x:B}
construct {x:C rdfs:subClassOf x:B}
construct {x:B rdfs:subClassOf x:A}

#transitive subclass rule
construct {?class rdfs:subClassOf ?super} where {
    ?class rdfs:subClassOf ?x. 
    ?x rdfs:subClassOf ?super }

# root class relation
select ?class ?root where {
    ?class rdfs:subClassOf ?root.
    not(?root) {?root rdfs:subClassOf ?super}
} as :rootClass

# aggregation relation
select ?class :count(?sub) as ?count where {
    ?sub rdfs:subClassOf ?class.
} group by ?class as :numberOfSubClasses

)

select ?super where {x:C rdfs:subClassOf ?super}
#select ?root where {:rootClass(x:C, ?root)}
#select ?count where {:numberOfSubClasses(x:B, ?count)}

The example about creates a rulebase with some facts about subclass relationships and some inference rules to derive some additional information about those class relationships. The rule:

construct {?class rdfs:subClassOf ?super} where {
?class rdfs:subClassOf ?x.
?x rdfs:subClassOf ?super }

Recursively generates all the subclass relationships so when the following query is run:


select ?super where {x:C rdfs:subClassOf ?super}

it results in:

<http://example.org/A> 
<http://example.org/B>

The rulebase also contains a relation rule for finding the root class (the top base class) of any class. Note that this rule actually makes use of our previous rule for recursively traversing the class hierarchy.


select ?class ?root where { 
 ?class rdfs:subClassOf ?root. 
 not(?root) {?root rdfs:subClassOf ?super} 
} as :rootClass

When we execute this rule in a query by replacing the SELECT query in the example with the following:


select ?root where {:rootClass(x:C, ?root)}

We get:


<http://example.org/A>

The last rule in the rulebase provides both an example of the use of a SPARQL function as well as an aggregation. This rule calculates how many subclasses a class has.

select ?class :count(?sub) as ?count where { 
 ?sub rdfs:subClassOf ?class. 
} group by ?class as :numberOfSubClasses

When we execute this rule in a query (replace the SELECT in the example):


select ?count where {:numberOfSubClasses(x:B, ?count)}

We get:


"2"^^<http://www.w3.org/2001/XMLSchema#int>

Hopefully this is enough to get you started with inference rules. I will post some more complicated use of rules in the future.

Client Model Pooling

2009-07-06T15:01:00.001-07:00

One simple way to improve performance of your data access services is to create a client model pool. Creating a new instance of a ClientModel for each service request can get expensive especially if the ClientModel makes use of a SemanticServerGraph. By creating a pool of client models that can be reused you benefit from cached statistics and compiled queries.

Client model pooling is implemented is by creating a single instance of a ClientModelPool. The example below creates a pool that holds a maximum of 10 client models that expire after 5 minutes.

ClientModel pool = new ClientModelPool(10, new TimeSpan(0, 5, 0));

// assign it to some global
app.ModelPool = pool;

You will want to keep this pool around in as a static member or in some other global container such as the ASP.NET Application object. Next you must create a method for creating a new instance of a client model that is configured for your application. This can be done in one of two ways:

Derive an application specific class from ClientModelPool and override its Create method.
Implement your own method for creating a client model on some centralized configuration class that already exists in your application.

Option 2 seems to be the most popular choice. So a simple method for getting a client model in your application class might look like.

public ClientModel GetClientModel()
{
    //get a pooled model 
    ClientModel model = ModelPool.Get();

    if (model == null)
    {
        // create a new one and make it pooled
        model = new ClientModel();
        model.SetPool(ModelPool);

        // TODO: configure for your application
    }

    return model;
}

You should call this method in your application in the context of a using statement. When the client model is disposed then it is returned to the pool.

using (ClientModel cm = app.GetClientModel())
{
    //do something
}

You should quickly discover this can have a big effect on performance.

Maintenance Release 1.1.0.29 Available

2009-06-16T06:42:00.001-07:00

June 16, 2009 – A new maintenance release of Semantics.SDK and Semantics.Server is now available for download at http://www.intellidimension.com/downloads.

* Fix to problem with XSD date in gmt +xxx timezones.
* Fix to stats when using certain quad based graph patterns in queries.
* Improvements to Model Manager model explorer.
* Fix to SemanticServer API for getting SQL extension functions.
* Fix to SemanticServer API for setting graph update mode.
* Addition of URI fo SemanticServerModel.
* Fix delete key handling in SPARQL editor Model Manager.
* New events added to ClientModelPool.
* Fixed expiration bug in ClientModelPool
* Renumber in rules fixed

Maintenance Release 1.1.0.28 Available

2009-06-10T07:42:00.001-07:00

June 10, 2009 – A new maintenance release of Semantics.SDK and Semantics.Server is now available for download at http://www.intellidimension.com/downloads.

Fix for ServerModel compilation of queries to SQL. In some cases this would produce SQL syntax errors that would be seen when using the sw_sparql extension.

Semantics 1.1 Released

2009-06-01T14:36:00.001-07:00

Intellidimension has just released its 1.1 version of Semantics.SDK and Semantics.Server. Both are available for download on our web site

Installation

You should download (click here) and install both Semantics.SDK and Semantics.Server. Be sure to install Semantics SDK first.

The Model Manager

The Model Manager is a new tool included in the 1.1 release of the Semantics.SDK. This tool exposes most of the fundamental features of our .NET API through a simple GUI. You should become familiar with the Model Manager functionality before diving into the API.

The Model Manager can be started by running the executable ModelManager.exe located in the [install dir]\bin directory of the SDK.

The Model Manger allows a user to create, save and edit a client model. A client model is a local RDF model that serves as the primary interface in the API for executing SPARQL queries. The table below describes some of the most basic components of a client model.

Data Sources	A data source is a container of triples (or quads) that can be local or remote. A model can have one or more data sources associated with it. The model manager provides a file load feature for quickly adding data to a data source. Each data source identified by a URI.
Rulebases	A rulebase is a collection of inference rules that are identified by a URI. One or more rulebases can be included in a query to execute the rules as part of the query.
Functions	A client model comes with a set of predefined SPARQL functions. However you may define custom functions of your own.

All of these components can be added, edited and removed via the tree control on the left hand side of the Model Manager.

Once you have setup your client model you can use the “Query” tab to execute and debug SPARQL queries running against your model.

Semantics.Server Setup

Semantics.Server provides an implementation of a server model. A server model is hosted on an instance of SQL Server and stores its data in a SQL Server database. The Model Manager provides a simple interface to:

Connect to a SQL Server database.
Install the Semantics.Server assemblies in the SQL Server database.
Create data sources (graphs) in the server model that are backed by SQL Server tables.

To get started you need to define a connection to Semantics.Server model. Right mouse click on the “Connections” item in the tree control and select the “Add Semantics.Server Connection…” menu item.

The dialog box that is displayed allows you to install all the Semantics.Server assemblies on an instance of SQL Server. Once you define a Semantic.Server connection you can define some graphs that you will be able to include as data sources in your client model. This is how you move data between your client application and your SQL Server database.

Disclaimer

We acknowledge that the Model Manager is still a bit rough around the edges but we wanted to get it in the hands of our developer community. We welcome any feedback you might have and we are actively improving its usability (user-friendliness) as well as its feature set.

Now start writing that semantic application!

Semantics 1.1 Coming Soon

2009-05-22T10:33:00.001-07:00

Intellidimension will soon be releasing a new version of it’s Semantics.SDK and Semantics.Server (v 1.1). Here are some of the key features:

Much improved query compilation. This includes better graph index statistics and caching of compiled query plans.
Support for distributed graphs. A graph can be partitioned over multiple servers for applications that require high performance querying on large data models. Distributed graph partitions can be backed by a SQL Server databases or just use the high performance in-memory r/w cache. The framework ensures maximum parallelization during queries and loads.
A set of new visual tools for setting up models and testing and debugging queries. Many improvements to these tools are planned over the next several months.

Both products will be available for a free 60-day evaluation.

Stay tuned!

Defining an Entity Description Type

2009-05-05T08:52:00.001-07:00

One of the first design issues encountered when using the Entity Framework is the design of your description types. The role of a description type is to define the data that makes up an entity. So you need to start with a clear understanding of your data model. Let’s start with a simple class of object like a Person and work through an example.

A Person will have a couple of simple literal properties such as:

Full Name
First Name
Last Name

The simplest way to define a description type with the Entity Framework is with a rulebase. So let’s define a rulebase for the Person description.


select ?s ?p ?o ?r where {?s ?p ?o. filter(?s=?r).} as <description>

We start by adding a description rule that includes all direct properties of the entity. This is the most basic form of a description type. Things get more complicated when we introduce references to other objects. This introduces some rather common data modeling concepts such as is an object contained by another object or just referenced. So let’s add an example of each type of property.

Address (contains an Address object)
Knows (references another Person object)

So I will add two additional rules to the description rulebase to include these new properties.

select ?s ?p ?o ?r where {
    ?r x:Address ?s. ?s ?p ?o.
}as <description>

 
select ?s ?p ?o ?r where {
    ?r x:Knows ?s. 
    ?s ?p ?o. 
    filter(?p=x:FullName)
} as <description>

The first rule includes all the direct properties of the contained Address object. The second rule includes just the full name of the referenced Person object to provide some user-friendly identity for the reference.

This description type is useful for retrieving an entity from the store. However get and put operations are often asymmetric. Meaning you often retrieve more data than you want to store after editing. In this example we would not want to store any a statements about the referenced Person object. To handle this we use a different description type when storing the entity. One that does not include any of the statements for the referenced object.

So we would remove the following rule from the storage description type.

select ?s ?p ?o ?r where {
    ?r x:Knows ?s.     
    ?s ?p ?o.     
    filter(?p=x:FullName)
} as <description>

Note these description type rules are for illustrative purposes. It it sometimes much easier to classify containment vs. reference through annotations on the property in the ontology. In this case, you can have one set of rules for all Classes in your data model.

For example we could get all contained objects using a generic rule.

select ?s ?p ?o where {
    ?r ?x ?y. 
    ?x x:referenceType x:Containment.
    <description>(?s, ?p, ?o, ?y).
} 
as <description>

The rule above recursively includes all properties that are marked with a reference type of Containment.

As you can see description types are just a tool that can be configured to meet the needs of your data model.

Setting up an Entity Model

2009-05-04T12:33:00.001-07:00

This article is about the RDF Entity Framework. It shows you how to setup a basic entity model using Semantics.Server. Setting up Semantics.Server The first step to setting up an entity model is configuing a Semantics.Server database with the graphs that you will need to hold your entity model. We will use the Semantics.Server API in the Semantics.SDK to do this. Create a provider graph that will hold the information about your model setup.

SemanticServerModel entityStore = new SemanticServerModel(ConnectionString);

entityStore.Graphs.CreateMultiGraph("http://entitystore/graph-provider");

Create a graph to hold the facts about the entity instances.

entityStore.Graphs.CreateGraph("http://entitystore/graph-fact");

Create a graph to hold facts about the ontology.

entityStore.Graphs.CreateGraph("http://entitystore/graph-ontology");

Create a rulebase for your CBD description type.

entityStore.Rulebases.CreateRulebase("http://entitystore/rulebase-cbd-description", EntityProviderRulebase.ConciseBoundedDescription);

Setting up the Entity Model We will use the ModelSetup helper class to defined a entity model for the Semantics.Server database we just setup. First we define a URI for our model.

ModelSetup modelSetup = new ModelSetup("http://entitystore/model/model-1");

Map the fact graph into the entity model. This connects the logic graph name with the physical name.

GraphSetup factGraph = new GraphSetup(EntityGraphUri.Fact, true);
factGraph.AddTarget("http://entitystore/graph-fact");
modelSetup.AddGraph(factGraph);

Map the ontology graph into the entity model.

GraphSetup ontologyGraph = new GraphSetup(EntityGraphUri.Ontology, true);
ontologyGraph.AddTarget("http://entitystore/graph-ontology");
modelSetup.AddGraph(ontologyGraph);

Map the description graph onto the fact graph since we are not using any other entity data graphs in this model.

GraphSetup descriptionGraph = new GraphSetup(EntityGraphUri.Description);
descriptionGraph.AddTarget("http://entitystore/graph-fact");
modelSetup.AddDefaultGraph(descriptionGraph);

Create a description type based on our CBD rules.

DescriptionTypeSetup cbdType = new DescriptionTypeSetup("http://entitystore/model/model-1/type-default");
cbdType.AddRulebase("http://entitystore/rulebase-cbd-description");
modelSetup.AddDefaultDescriptionType(cbdType);

Create the entity model.

SemanticServerProvider provider = new SemanticServerProvider(ConnectionString);
provider.Settings.ProviderGraphUri = "http://entitystore/graph-provider";
EntityModel model = modelSetup.CreateEntityModel(provider);

That's it.

Entity Framework Overview

2009-05-04T11:41:00.000-07:00

The Entity framework is built on top of the Semantics.SDK and providesentity-based transactions on a RDF store such as Semantics.Server. This article will cover some of the basic concepts of the Entity framework. Entity An entity consists of two things: (1) A URI that indentifies the entity and (2) and set of RDF statements that describe the entity. Description Type A description type defines the statements associated with an entity. In the simple case just the statements with the entity URI as the subject value are considered part of the entity. In real applications an entity description is often much richer than this. A common description type is called a CBD (concise bounded description). A CBD is defined as all statements with the entity URI as the subject value plus any anonymous nodes recursively referenced by any of those statements. Description types can be defined using rules and/or custom .NET code. The example below shows the CBD definition using rules.

RULEBASE 
(
    SELECT ?s ?p ?o ?r WHERE {?s ?p ?o. filter(?r=?s)} AS <description>
    SELECT ?s ?p ?o ?r WHERE {
        ?r ?x ?y.
        {<description>(?s, ?p, ?o, ?y)}
        filter(isblank(?y))
    } AS <description>
)

Retrieving an Entity When an entity is retrieved using the RdfEntity framework via the EntityModel class. An entity model requires a URI for the model and a EntityServiceProvider in its constructor. We will discuss the EntityServiceProvider class later but it can be thought of as an entity store driver.

EntityModel model = new EntityModel(provider, “model:MyModel”);
model.UriResolver.AddNamespace("dc", "http://purl.org/dc/elements/1.1/");
model.UriResolver.AddNamespace("foaf", "http://xmlns.com/foaf/0.1/");
model.UriResolver.AddNamespace("model", "http://entitystore/model/model-1/");
model.UriResolver.AddNamespace("store", "http://entitystore/");

Entity derrish = model.GetEntity("model:DerrishRepchick");
derrish.DescriptionType = "model:CBD";

An entity is constructed when EntityModel.GetEntity is called. The entity is bound to the specifed URI and the entity model and therefore the underlying service provider. All calls on the instance of the entity will be routed through the entity model that created it. Notice how we set the DescriptionType property on the entity. This tells the framework which statements to retrieve for this entity. The statements are not actually retrieved until a graph is accessed for that entity.

string creator = derrish.NamedNode[EntityGraphUri.Metadata]["dc:creator"].FirstOrDefault();

Entity Graphs An entity model is logically and/or physically partitioned into multiple graphs. The Entity framework has some well-known graphs that are defined as follows. You may choose which, if any, of these graphs to support in your entity model. However, some are required for certain features of the Entity framework. Ontology Graph Contains statements about the onotology. This is a required graph. Fact Graph Contains explicit facts about entities. This is a required graph. Description Graph Contains full entity description (facts + inferences). This is a required read-only graph. Inference Graph Contains only inferences about entities. This is an optional read-only graph. Metadata Graph Contains metadata about the entities. This is an optional read-only graph. Provider Graph Contains metadata about the underlying service provider. This is a required (optional for some providers) read-only graph. Storing an Entity A modified entity can be stored by calling the SaveToStore method. You will want to make sure to set the correct description type on the entity before storing. It is often the case that an entity will have asymmetric description types. Meaning often more statements are retrieved then stored. For example you may retrieve facts and inferences but only want to store facts. A similar issue exists when removing an entity from the store. Service Providers The Entity framework contains a built in provider for Semantics.Server called SemanticServerProvider. This service provider allows an entity model to be maintained in a Semantics.Server database. The SemanticServerProvider has a convenient helper class for setting up an entity model called ModelSetup.

Querying a graph using SPARQL

2009-05-04T08:49:00.000-07:00

The Intellidimension Semantics.SDK supports the SPARQL syntax for querying RDF data. This article will focus on using SPARQL to query a single in-memory graph and accessing the results. The DataSource.Query Method As mentioned in earlier articles, the abstract base class DataSource provides the interface to all sources of RDF data. All DataSource objects support a Query method with several overloads. The simplest overload takes a single string parameter that is the SPARQL query string to be executed on the DataSource object. The Query method returns a Table object that contains any results for the query. SELECT Command One of the most commonly used SPARQL commands is the SELECT command. When executing a SELECT command against a single DataSource object the name of the graph does not need to be included in the FROM clause of the command since it is implied, as shown below.

GraphDataSource g = new GraphDataSource();

Table results = g.Query(@"
  PREFIX dc: <http://purl.org/dc/elements/1.1/>
  SELECT ?title WHERE
  {<http://example.org/book1> dc:title ?title}");

Per the SPARQL specification, the graph pattern is matched against the statements in the DataSource object. The variables in the select list are bound as specified in the graph pattern and returned as results. Query Results SPARQL commands such as the SELECT command return query results in a Table object. The Table object has a row for each distinct set of column values. The column values are defined in the select list of the SELECT command. Below is an example of how to iterate over the results of a SELECT command.

for (int i = 0; i < results.RowCount; i++)
RdfLiteral title = results[i][0] as RdfLiteral;

The Semantics.SDK provides support for the XML serialization of query results using the SPARQL result syntax, as shown below.

SparqlXmlFormatter fmt = new SparqlXmlFormatter(
results, typeof(CommandSelect));
fmt.Write(stream);

Parameterized Queries SPARQL queries can be parameterized using the Semantics.SDK via the QueryParameters object. The QueryParameter object is used to bind a RdfValue object to a named parameter that is specified in the query using the @ character followed by any valid variable name. A query parameter can be used anywhere that a query variable can be specified. Below shows an example of the use of a query parameter in a SPARQL SELECT command.

QueryParameters qp = new QueryParameters();
qp.Add("id", new RdfUri("http://example.org/"));

g.Query(@"SELECT ?s ?p ?o WHERE
{?s ?p ?o. filter(?s=@id)}", qp);

Conclusion This article provided an introduction to the objects and methods used to execute a SPARQL query and access the results using the Semantics.SDK. However this article did not provide much insight into the Semantics.SDK support for all the SPARQL commands and extensions. The Semantics.SDK supports a variety of SPARQL commands such as: SELECT, DESCRIBE, CONSTRUCT, ASK, INSERT, and DELETE. In addition, it provides support for inference rules and a variety of extension functions. These will all be discussed in future articles.

Reading and writing a RDF Graph

2009-05-04T08:39:00.000-07:00

The Intellidimension Semantics.SDK contains a suite of RDF readers and formatters for .NET. These allow applications to read and write serialized RDF data from and to streams. RdfReader and RdfFormatter The Semantics.SDK has two abstract base classes that provide the interfaces for reading and formatting RDF data, respectively. These base classes provide an interface that allows each to operate on an instance of a stream object that must be created in the application code prior to using either of these classes. This allows the RdfReader and RdfFormatter classes to operate on virtually any source that supports the TextReader or TextWriter interface as defined in System.IO. The RdfReader class is design to read data from a stream into an instance of a DataSource. While the RdfFormatter class will write data from an instance of a DataSource to a stream. RDF Syntaxes The Semantics.SDK supports most of the standard RDF syntaxes. The table below lists the RDF syntax along with the RdfReader and RdfFormatter classes that implement the serialization for that syntax. RDF/XML: RdfXmlReader, RdfXmlFormatter N Triples: NTriplesReader, NTriplesFormatter Turtle: TurtleReader, TurtleFormatter RDFa: RdfaReader Reading RDF In the Semantics.SDK, all RDF graphs derive from the base class DataSource. This base class defines several overloaded Read methods for loading RDF data into an instance of a DataSource. The Read method is used to load RDF data into the DataSource from a string, stream or location specified as a URI. Each of these overloaded methods provides the option of specifying the default base URI for any relative URIs used in the RDF data. The code below shows an example of how to read a RDF/XML file into an in-memory graph. Note, that one of the overloaded Read methods is a generic method in which the class of the RdfReader is specified.

GraphDataSource g = new GraphDataSource();
StreamReader s = new StreamReader(@"c:\sample.rdf");
g.Read(s);

Writing RDF The DataSource class also defines several overloaded Format methods for writing RDF data from an instance of a DataSource to a string or stream. Like the Read method, the Format method also has an overloaded generic form in which the class of the RdfFormatter is specified. The code below shows an example of how to store the contents of an in-memory graph to a file using the N Triples syntax.

GraphDataSource g = new GraphDataSource();
StreamWriter s =
    new StreamWriter(@"c:\sample.rdf");
g.Format(s);

The code below shows an example of how to store the contents of an in-memory graph to a string using the turtle syntax.

GraphDataSource g = new GraphDataSource();
String s = g.Format();

Conclusion This article provided a brief introduction into how to serialize RDF data to and from a stream. The Semantics.SDK also allows RDF data to be stored and retrieved from a Microsoft SQL Server® database using Intellidimension Semantics.Server. This will be discussed in detail in future articles.

Getting Started with Graphs

2009-05-04T07:18:00.000-07:00

The Intellidimension Semantics.SDK provides a simple object-oriented API for working with RDF data using Microsoft .NET. The most basic structure for managing RDF data is a graph, which is a collection of RDF statements. This article will focus on the basic interface the Semantics.SDK provides for manipulating statements in a graph. In-memory Graphs The Semantics.SDK provides an abstract base class DataSource that provides an interface for working with a RDF graph. The most commonly used implementation of a DataSource is the GraphDataSource which implements an in-memory RDF graph. Adding Statements One of several ways statements can be added to any DataSource via its Add method.

GraphDataSource g = new GraphDataSource();

g.Add(new RdfUri("http://www.intellidimension.com/sdj.pdf"),
new RdfUri("http://www.w3.org/2000/01/rdf-schema#label"),
new RdfLiteral("Getting Started With Graphs"));

The parameters to the Add method specify the subject, predicate and object values, respectively, of the statement to be added to the graph. There are several overloads to the Add method. Removing Statements Similarly, a statement can be removed from a DataSource by calling it’s Remove method and specify the subject, predicate and object values of the statement to be removed. Any one of these parameters can be set to null. In that case the parameters for the Remove method act as a statement mask where each null value is treated as a wildcard value for matching statements in the DataSource. In this manner multiple or all the statement can be removed from the DataSource with a single call. Getting Statements A DataSource object is a collection of RDF statements each represented by an instance of the class Statement. All DataSource objects provide access to the collection of statements via their GetStatements method. The code below shows how to iterate over all the statements in a DataSource.

foreach (Statement stmt in g.GetStatements())
{
    RdfValue s = stmt.Subject;
    RdfValue p = stmt.Predicate;
    RdfValue o = stmt.Object;
}

For each Statement the Subject, Predicate and Object properties are retrieved as an instance of the RdfValue class. The RdfValue Class The RdfValue class is an abstract base class that is used to represent subject, predicate and object values of a statement. A RDF resource is represented by the RdfUri class and a RDF literal is represented by the RdfLiteral class.

double d = (double)(RdfLiteral)stmt.Object;
DateTime dt = (DateTime)(RdfLiteral)stmt.Object;

The RdfValue class is widely used throughout the Semantics.SDK and it supports some conveniences such as conversion operators to simplify the integration with other data types in .NET. Conclusion This article provided a brief introduction into how to manipulate statements in an RDF graph. The Semantics.SDK provides other capabilities such as issuing queries to graphs that will be discussed in future.

What is Semantics.Server?

2009-05-04T06:34:00.000-07:00

Semantics.Server is a set of .NET assemblies that are installed on Microsoft SQL Server enabling it to store, query and perform inferencing on RDF data. All the RDF data stored using Semantics.Server is backed by relational tables thereby leveraging the capabilities of Microsoft SQL Server. Even inferencing can be backed by relational tables. This eliminates memory limits that are often reached when executing inference rules over large models. An evaluation copy of Semantics.Server can be downloaded from: http://www.intellidimension.com/

What is the Semantics.SDK?

2009-05-04T06:20:00.000-07:00

The Semantics.SDK is a commercially available product for adding semantics based computing to your .NET application. It consists of visual tools for working if RDF, RDFS/OWL ontologies, SPARQL queries and inference rules. These tools are based on a complete .NET based API that is also available to the developer. Semantics.SDK features:

In-memory RDF graphs
SPARQL query engine
Inference engine
Extensible data service interface for integration
API for Semantics.Server

An evaluation copy of Semantics.SDK can be downloaded from:

http://www.intellidimension.com/