VOYAGES OF THE SEMANTIC ENTERPRISE

Publishing HTML created with SPARQL Web Pages

2011-12-02T07:09:00.000-08:00

Last week we saw how to use SPARQL Web Pages (SWP) to render customized HTML of individual class instances and how to create a web page of all that class's instances with a title at the top. The fine-grained control that SWP gives us over the generated HTML let us take advantage of the jQuery Mobile libraries so that the sample TopBraid application generated web pages appropriate for a smartphone interface, with buttons that expand and collapse at your touch to display details about each class instance.

Testing this application meant choosing from two alternatives:

The first was to run it on TopBraid Composer's built-in TopBraid Live Personal Server, which let us look at the page from any web browser running on the same machine.
Uploading the application's project to a TopBraid Live Enterprise Server, where multiple devices, including phones, could access it.

Either way, because TopBraid Live generates these web pages dynamically, if the underlying data is changed, refreshed versions of the web page would reflect this, making TopBraid a great platform for interactive semantic web applications for any device.

You don't have to have a TopBraid Live Enterprise server to deliver pages generated by SWP, though. A simple SPARQLMotion script can save your formatted HTML in disk files that you can copy to a web server that may or may not have TopBraid Live installed. Using this technique, you can use the TopBraid platform to create semantic content publishing applications as well as interactive applications.

The following SPARQLMotion script, which is stored in the application file described last week, does this for the mobile Kennedys web application.

The first module is an sml:ImportRDFFromWorkspace module that reads the file that this script is stored in. That file has the Kennedys data and the SWP formatting markup so that this data can be fed to the next step in the process.

The second module, named mk:GenerateHTML, is an sml:CreateUISPINDocument SPARQLMotion module (from the Text Processing section of the SPARQLMotion palette) whose key setting is its sml:view property, which has the following:

<ui:resourceView
   ui:resource="&lt;http://topbraidlive.org/mobileKennedys&gt;"/>

It's a snippet of XML specifying that the module should create a resource view for the specified resource, which is identified here with a complete URI. (The URI's delimiting angle brackets are escaped because they're in an XML attribute.) The real work to make this happen was all described in the last blog entry, which showed how the SWP code to generate a complete web page was attached to the resource. The mk:GenerateHTML module in this script also specifies that this generated markup will be stored in a variable named doc.

The final mk:SaveFile module in the script is an sml:ExportToTextFile module that saves the contents of the doc variable (set in the module's sml:text property as the SPARQL expression ?doc) to a file called output.html. I also set sml:replace to true so that repeated execution of the script wouldn't append the output onto the result of previous runs.

After you run this script you'll have a web page called output.html that looks like the display shown in the phone browsers in last week's blog entry, and you can copy this file to any web server you want.

This script is very simple. As you bring other SPARQLMotion capabilities into it such as inferencing and reading from all the data formats that TopBraid understands, you can make it much more sophisticated. You can also configure the script to save a collection of multiple files, letting you publish large collections of data in pieces that are digestible for typical browsers. (Phone browsers in particular can get sluggish; my Android LG Ally is not a recent model, and the expanding and collapsing of information about each person on the display of this app is not as quick on the Ally as I'd like it to be.)

So, use your imagination to add new features to this SPARQLMotion script, and you can create dynamic or static web pages for phones or any other kinds of browsers, with all the power of TopBraid behind your application development.

Creating a TopBraid mobile web app with SPARQL Web Pages

2011-11-22T07:18:00.000-08:00

I've written here before about how SPARQL Web Pages (SWP) let you convert your RDF to HTML or XML by embedding SPARQL queries into the appropriate markup. In that very simple example, I showed how to create a web page for an address book entry and then display it both in TopBraid Composer and in a regular web browser.

Today I'm going to show how I did something similar to display a single Person instance from the Kennedys sample data included with TopBraid Composer and then defined a page that showed all the people in that data model. You can download and try the project here. The fun part was displaying it so that it looks like a proper mobile web page on a phone's web browser, as shown here on an Android phone and on an iPhone turned sideways to test the re-orienting capability of the display.

Touching someone's name on the phone expands the display to show the remaining property names and values about that person underneath his or her name. In the picture, I've just touched Andrew Cuomo's name on the Android phone and Edward Kennedy Jr's name on the iPhone, displaying details about each of them below their names. Touching the names again hides their data.

In the picture, the two phone browsers are displaying the output of a TopBraid Live server running this application. As we'll see in the sequel to this blog entry, you can use the same SPARQL Web Page configuration to save HTML disk files with all of this formatting so that the phone browsers could view the static web pages stored on a server that didn't have TopBraid Live installed.

To enable proper mobile display, I used the jQuery Mobile library. jQuery is a set of Javascript and CSS libraries designed to let you add sophisticated user interfaces to your web pages without worrying about cross-browser compatibility, and jQuery Mobile is a branch of this project specialized for mobile phones. You don't need to know any JavaScript or CSS to use these libraries; if you're happy with one of their display configuration, using these libraries is usually just a matter of including the right file links in your HTML's head element and then setting certain attributes in your HTML elements to reference the libraries.

I began this application by creating an RDF/SPARQLMotion file in TopBraid Composer with a base URI of http://topbraidlive.org/mobileKennedys. I needed SPARQLMotion for the script that creates the static disk file version of the Kennedys display that we'll learn about next week. Next, I imported the kennedys.rdf model from the /TopBraid/Examples folder in the Navigator view. I also imported the SWP html.rdf and tui.rdf models from the Navigator's /TopBraid/UISPIN folder. (This all works the same when the files to import are Turtle ttl files instead of RDF/XML files.)

After importing the necessary files, the next step was to set up the display of data about a Person instance. After importing the files described above, clicking on kennedys:Person under owl:Thing on the Class view shows that the presence of the SWP libraries has added a ui:instanceView property to the kennedys:Person class form. I could have put the HTML to display a person here, like I did with the address book display in the blog entry mentioned above, but for greater flexibility, I created a separate PersonView class to store this markup and pointed at this class from the Person class's ui:instanceView value.

I created this mk:PersonView class (I had assigned the prefix "mk:" to the URI http://topbraidlive.org/mobileKennedys#) as a child of the ui:Element class, which is a child of the ui:Node class added by the SWP libraries. The ui:prototype property on this class's form is the place for the formatting code and markup, but I did a few setup steps before setting it:

Because the app needs to pass a parameter to the code in ui:prototype specifying which person to display, I had to define that parameter. To do this, I created an sp:person child of the sp:arg property in the Properties view to represent the person argument value passed to the prototype. Next, I dragged the new property from the Properties view to the spin:constraint property name on the mk:PersonView form to indicate that this would store the argument passed to the code and markup used to display a single person. This displays the "Create from SPIN template" wizard with all the values filled out the way I needed them, so I just clicked the OK button.
JQuery implements some of its magic with HTML extension attributes named data-collapsed and data-role. TopBraid Composer helps you assemble proper HTML by flagging any non-HTML markup, and it won't like these because they're not declared as HTML 4 properties. So, I declared them myself by making two clones of the html:class property (a subproperty of html:attributes) and renamed them html:data-collapsed and html:data-role. This way, TopBraid Composer wouldn't prevent me from saving HTML markup that used these properties as attributes.
When listing each person's property names and values (for example, Andrew Cuomo's year of birth and first name in the picture above), I certainly didn't want to list the full URI of each property name. Ideally, each property would have an rdfs:label value that I could display instead; if not, I thought it best to just show the local name of the property's URI. To make this easier, I created a new function called mk:bestName as a subclass of spin:Functions (itself a subclass of spin:Modules). I defined a spin:constraint of sp:arg1 for this function and then defined this spin:body for it:
```
SELECT ?label
WHERE {
    BIND (spif:name(?arg1) AS ?name) .
    BIND (IF(fn:contains(?name, ":"), afn:localname(?arg1), ?name) AS ?label) .
}
```
mk:bestName is a good general-purpose function. It calls the SPIN spif:name function, which gets a resource's skos:prefLabel value if available or an rdfs:label value as a second choice. If neither is available, mk:bestName takes the local name of the URI or prefixed name that got returned.
Because members of the kennedys:Person class might have a kennedys:name value that I'd prefer the application to use if available, I declared a similar but more specialized function for the Kennedys data called mk:bestKennedyName. This is also as a subclass of spin:Functions, and has a spin:constraint of sp:arg1 and the following as a spin:body:
```
SELECT ?label
WHERE {
    OPTIONAL {
        ?arg1 kennedys:name ?kname .
    } .
    BIND (spif:name(?arg1) AS ?name) .
    BIND (COALESCE(?kname, IF(fn:contains(?name, ":"), afn:localname(?arg1), ?name)) AS ?label) .
}}
```
This function body takes advantage of SPARQL 1.1's new COALESCE() function, which returns the value of the first parameter passed to it that can be evaluated without an error.

With the functions, the HTML extensions, and the argument to pass to it all set up for the formatting markup in the mk:PersonView class, I was ready to add that markup and SPARQL code to the ui:prototype property of my new class. It's mostly HTML div elements with attributes set according to the models I saw in the source of the jQuery Mobile demos. The "collapsible" part means that initially only the kennedys:name value will display, as an h3 element, and that clicking on that name (or, on a phone, touching it) will toggle the display of the remaining property names and values about that person.

<div data-collapsed="true" data-role="collapsible">
   <h3>{= spl:object(?person, kennedys:name) }</h3>
   <div class="ui-grid-a">
       <ui:forEach ui:resultSet="{#
               SELECT ?propertyName ?bestValueLabel
               WHERE {
                   ?person ?property ?value .
                   BIND (mk:bestName(?property) AS ?propertyName) .
                   BIND (IF(isIRI(?value), mk:bestKennedyName(?value), ?value)
                      AS ?bestValueLabel) .
               }
               ORDER BY (?property) }">
           <div class="ui-block-a">
               <div class="ui-bar ui-bar-c">{= ?propertyName }</div>
           </div>
           <div class="ui-block-b">
               <div class="ui-bar ui-bar-c">{= ?bestValueLabel }</div>
           </div>
       </ui:forEach>
   </div>
</div>

When you use SWP to define an HTML div element with the data and markup to display something, the SWP engine will create html, head, and body wrapper elements to ensure that a browser viewing the HTML gets a complete web page. The SWP ui:headIncludes property, which you'll see on the mk:PersonView class form with ui:prototype and the other properties there, lets you specify custom markup to add to the HTML head element when the SWP engine sends the web page to the requesting browser. I added the following to this property; it has the meta, link, and script elements necessary to make the resulting HTML a proper jQuery Mobile page:

<ui:group>
   <meta content="width=device-width, minimum-scale=1.0, maximum-scale=1.0"
         name="viewport"/>
   <link href="http://code.jquery.com/mobile/1.0/jquery.mobile-1.0.min.css"
         rel="stylesheet"/>
   <script src="http://code.jquery.com/jquery-1.6.4.min.js"/>
   <script src="http://code.jquery.com/mobile/1.0/jquery.mobile-1.0.min.js"/>
</ui:group>

Then, going back to the kennedys:Person element, I added this ui:instanceView value for it to point at the mk:PersonView class I had created:

<mk:PersonView sp:person="{= ?this }"/>

The ?this variable passes the Person instance currently being processed to be used as the ?person value in the SPARQL query in the mk:PersonView ui:prototype value.

This is all enough to display a single person, but I wanted to display all the Person instances in a sorted list. I attached this view's definition to the ontology resource itself by clicking on the little house icon at the top of TopBraid Composer and then adding this ui:view value to it (note that ui:view wasn't already part of the form, so I dragged it on there from TopBraid Composer's Properties view):

<div>
   <div data-role="header">
       <h1>Kennedys List</h1>
   </div>
   <div data-role="collapsible-set">
       <ui:forEach ui:resultSet="{#
               SELECT ?p
               WHERE {
                   ?p a kennedys:Person .
                   ?p kennedys:lastName ?lname .
               }
               ORDER BY (?lname) }">
           <ui:resourceView ui:resource="{= ?p }"/>
       </ui:forEach>
   </div>
</div>

As with the code to display each individual Person instance, this markup is mostly div elements with attribute settings based on the source of the jQuery Mobile demos I saw. The ui:resourceView element inside the ui:forEach element tells the SWP engine to display the resource according to whatever view was specified for it. In this case, the resource is a kennedys:Person instance, because that's what the SPARQL here query binds to the ?p variable, so it will use the view defined earlier.

To test this, I sent a browser to the URL http://localhost:8083/tbl/uispin?_resource=http://topbraidlive.org/mobileKennedys. (URLs for SPARQL Web Page applications often include a &_base parameter to identify the graph of data to use—in this case, it would be &_base=http://topbraid.org/examples/kennedys—but that was unnecessary here because one of the first steps of creating the mobileKennedys model was dragging the Kennedys data onto its Include tab, so it already knew which data to use.) The _resource parameter tells it which resource to render, so I used my file's base URI here because that's where I attached the markup and SPARQL code to display the full web page. These and other parameters are described in the SWP documentation.

This should work with any browser. (I recently discovered that picking User Agent from Safari's Develop menu lets you set Safari to emulate a variety of other browser, including the mobile versions that run on the iPhone and iPad, which helped me to debug some early problems I had with getting the jQuery Mobile code right.) Because you can't access TopBraid Composer's built-in copy of the TopBraid Live Personal edition from a different computer, there's no way for a phone's browser to access this application when running it on TopBraid Composer, so I uploaded the project storing this application to a copy of TopBraid Live to do the test shown in the photograph above.

Next week, I'll show how I extended this application to save a static HTML file of the mobile web display of Kennedys data as an alternative to the TopBraid Live server's dynamic display. I could then copy that file to a web server that doesn't necessarily have TopBraid Live installed on it. Then, any computer or phone web browser can display it. For a preview of how it looks, send your phone's browser to http://www.topquadrant.com/resources/blog/k/—or, if you want a shorter URL to type on your phone, http://bit.ly/topqkm.

Ontologies and Data Models – are they the same?

2011-09-30T10:32:00.000-07:00

Yesterday a question about how ontologies may be different from logical data models was asked by a newcomer on TopBraid Users Forum. As to be expected on the TopBraid Forum, by ontologies he meant specifically ontology models expressed in RDFS/OWL. Because we frequently hear this or similar questions in our trainings, workshops and in conversations with customers, I decided to respond in a blog post instead of writing an e-mail.

Data modeling was invented more than thirty years ago to help with the design of databases, specifically, relational databases. As quoted below, ANSI definition from 1975 differentiated between three data models – conceptual, logical and physical. Data modeling quickly became recognized as a tool for analyzing the semantics of an organization with the respect to the structure and flow of the information used in carrying out organization’s activities. Wikipedia offers the following definition of Data Modeling:

Data modeling is a method used to define and analyze data requirements needed to support the business processes of an organization. The data requirements are recorded as a conceptual data model with associated data definitions. Actual implementation of the conceptual model is called a logical data model.
<…>
In 1975 ANSI described three kinds of data-model instance:
Conceptual schema: describes the semantics of a domain (the scope of the model). For example, it may be a model of the interest area of an organization or of an industry. This consists of entity classes, representing kinds of things of significance in the domain, and relationships assertions about associations between pairs of entity classes. A conceptual schema specifies the kinds of facts or propositions that can be expressed using the model. In that sense, it defines the allowed expressions in an artificial "language" with a scope that is limited by the scope of the model.
Logical schema: describes the structure of some domain of information. This consists of descriptions of (for example) tables, columns, object-oriented classes, and XML tags.
Physical schema: describes the physical means used to store data. This is concerned with partitions, CPUs, tablespaces, and the like.
According to ANSI, this approach allows the three perspectives to be relatively independent of each other. Storage technology can change without affecting either the logical or the conceptual model. The table/column structure can change without (necessarily) affecting the conceptual model.

These definitions describe a clear progression from conceptual to logical to physical data models. SInce their origin is in the 70s, they reflect certain technology assumptions than no longer hold true.

When information modeling is done to create a relational database, conceptual model must be different from a logical model because there is no place in a relational database structure to capture, for example, business rules, create subsumtion relationships and describe other key aspects of a conceptual model. This semantic information collected and documented as part of the initial modeling is left behind when modelers and designers move on to define a logical data model. The "left behind" parts are used by software developers as they encode business semantics directly into custom programs.

Logical data model is a subset of a conceptual model that can be expressed using a particular technology. However, there are always some performance considerations that require additional changes to the logical data model before it can be implemented in a relational database. Hence, some of the aspects of a logical model are left behind as it gets translated into a physical data model.

Since an ontology is a model of a domain describing objects that inhabit it, all three types of data models can be thought of as ontologies. They range from the most expressive one that describes business concepts and processes (the conceptual model) to less expressive and progressively moving from describing business semantics to describing physical structures of the data as it is stored in the databases (the logical and physical data model). Physical model can be thought of as an ontology of a particular database. Wikipedia goes on to note

Early phases of many software-development projects emphasize the design of a conceptual data model. Such a design can be detailed into a logical data model. In later stages, this model may be translated into physical data model. However, it is also possible to implement a conceptual model directly.

Semantic Web standards (governed by the W3C, the World Wide Web Consortium) make it possible to implement conceptual models directly. This is possible due to the layered architecture of the Semantic Web technology stack consisting of:

RDF – a canonical data model that is like relational data model in its ability to connect related objects and unlike relational data model in that the data objects (or resources in RDF-speak) are highly granular.

The smallest unit of information in RDF is not a table or a row in a table, but individual statements – a single fact about a resource.

These statements are called RDF triples. For example, “Atlantis decommission-date July, 2011” is a triple where Atlantis is a subject of a triple, decommission date is a predicate of a triple and July, 2011 is an object of a triple. Atlantis and decommission date are RDF resources and July, 2011 is XML literal. Subjects and predicates of a triple are always RDF resources. An object can be either a resource or a literal value. Predicates that connect two resources are relationships or associations in the data modeling speak. Predicates connecting a resource to a literal value are attributes. In RDF they are called respectively object and data properties.

Because RDF model is highly canonical, RDF data is schema-less. There are no constraints that require it to fit into tables or hierarchies. RDF data is simply a network of connected triples. As such, it can be used to represent, if needed, both - table structures and hierarchies. Standard mappings have been defined from relational tables and XML hierarchies into RDF.

Another key differentiating factor of RDF is that it was “born on the web”. Each RDF resource has a globally unique identity, a URI (uniform resource identifier). For example, the URI for Atlantis may be http://www.nasa.gov/shuttle/Atlantis and the URI for a decommission date may be http://www.nasa.gov/lifecycle#decommissionDate . As a result, it is possible to link RDF data over web in a way similar to how documents can be hyperlinked over the web. By web we mean all HTTP based networks including intranets and extranets.

RDF databases store and provide query access to RDF data. Just like there are standard languages for query of relational and XML data, there is a standard for querying RDF. It is called SPARQL. True to the web-native nature of RDF, SPARQL is not only a query language, but also a protocol that makes it possible to access RDF data over HTTP.

RDFS (RDF Schema) and OWL (Web Ontology Language) – RDF-based languages for expressing business semantics.

Jointly RDFS and OWL offer ability to define classes or groups of resources that share common characteristics such as Vehicles and Space Shuttles. The richness of RDFS/OWL makes it possible to fully express the meaning of the business concepts. Data models in RDFS/OWL are stored in the same way as the data, in RDF triples. For example, we can have triples stating that Space Shuttle is a Class and it is a sub class of a Vehicle class and that a vehicle can have only one decommission date (cardinality = 1) and its value must be xsd:date. And you can go beyond cardinality and use the Semantic Web standards to represent a variety of business rules.

Since the data and the schema are stored in the same way, it is possible to query schemas the same way data is queried and to combine search criteria about schemas with the search criteria about data. For example, we can create SPARQL queries to ask for all vehicles that have been decommissioned, all subclasses of a vehicle class, all relationships and attributes a vehicle should have and, when returning decommissioned vehicles, to provide only data values for the fields that have cardinality = 1.

The use of RDF means that the modeling constructs and definitions can be linked and connected. Organizations can refer to each other’s business definitions. Models can be modularized and re-used where appropriate. Differences between related, but not identical concepts can be described. All of this can now be done in a standard compliant and interoperable way.

A growing number of standards bodies and communities of interest are publishing RDF/OWL data models for their particular domains. For example:

SKOS – provides a way to represent taxonomies and thesauri
ISO 15926 – offers a data model for sharing life-cycle data for process plants including oil and gas production facilities
Ontology for Media Resources - defines a core set of metadata properties for multimedia resources
SIOC - defines information about online communities
QUDT - provides models describing measurable quantities, units for measuring different kinds of quantities and the data types used to store and manipulate these objects in software
Provenance Vocabulary - defines provenance-related metadata

There is much more that can be added to this post including a discussion on the best practices for ontology modeling, ontology architecture, approaches for connecting and mapping models, using rules and constraints, publishing, versioning and governing models. Each of these topics, however, deserves an exploration in its own right.

I will end by pointing to a few relevant related blogs and web pages we have published before:

How to extend an ontology http://topquadrantblog.blogspot.com/2011/03/how-to-extend-ontology.html
Ontology Mapping with SPINMap http://topquadrantblog.blogspot.com/search/label/SPINMap
Training on RDF, OWL and ontology modeling http://www.topquadrant.com/training/training_overview.html
Transforming XML Schemas and XML into RDF/OWL http://topquadrantblog.blogspot.com/2011/09/living-in-xml-and-owl-world.html
Converting UML models to OWL http://topquadrantblog.blogspot.com/2011/02/converting-uml-models-to-owl-part-1.html

Living in the XML and OWL World - Comprehensive Transformations of XML Schemas and XML data to RDF/OWL

2011-09-28T23:03:00.000-07:00

Many enterprise information models are expressed using XML Schemas. Data between applications is commonly exchanged in XML, compliant with those schemas. Connecting XML data from different systems in a coherent aggregated way is a challenge that confronts many organizations. Capabilities of RDF/OWL to describe semantics of different data models and aggregate disparate data are a natural fit for addressing these challenges.

For a number of years now, TopBraid Composer included the ability to convert XSDs and associated XML files to RDF/OWL. However, for some XML Schemas our converter did not work as well as customers needed. For the upcoming TopBraid Composer 3.6.0 release, it was significantly improved to have a more comprehensive coverage of XSD constructs and more meaningful conversion to OWL. In parallel, we improved our XML data conversion to RDF so that transformations automatically happen based on the generated OWL models. And we have improved performance of the transformations.

An overview of the approach is illustrated in the following figure:

Since, the conversion occurs automatically, users do not have to worry about writing any rules for commonly needed mappings. However, those users that need to make further transformations can use SPARQL Rules and SPARQLMotion to customize their generated OWL ontology or further transform RDF triples representing the XML data.

The content of this blog is organized as follows:

XML Schemas converted as part of our tests
Some challenges in converting XML Schemas to OWL
Illustrative example of transformation rules
Another example of transformation rules
Complete table of supported transformations
A SPARQL Metric Query
Concluding remarks

XML Schemas converted as part of our tests

We tested the importer on a broad range of complicated and large-scale industry standard XSD files, and converted many XML instances with impressive results. The XSDs we have tested with the new importer include:

Banking
- FpML, the Financial products Markup Language
- ISO 20022, a standard for Universal financial industry message scheme

Energy and Utilities
- MultiSpeak, de-facto standard for defining data needed to be exchanged between software applications in order to support the business processes commonly applied at utilities

Government
- DoDAF, the Department of Defense Architecture Framework
- NIEM, the U.S. National Information Exchange Model

Oil and Gas
- ISO 15926, a standard for integration of life-cycle data for process plants including oil and gas production facilities
- WITSML, Wellsite Information Transfer Standard Markup Language

Healthcare
- HL7

Electronics
- IP-XACT, the XML Schema for meta-data documenting Intellectual Property (IP) used in the development, implementation and verification of electronic systems

Other
- ATML, the Auto-Test Markup Language

Some of the converted schemas will be published at LinkedModels.org. To get an early access to converted models or for any other questions, contact us at TopQuadrant.

The examples we use in this blog are mainly from the Financial products Markup Language (FpML). All FpML 5.2 XSD and XML instance files were tested. An example is transparency standard under FpML, for which the transparency.ttl ontology was generated from 23 XSD files.

Some challenges in converting XML Schemas to OWL

Some of the challenges in converting XSD to OWL that were addressed are:

Transforming of anonymous types
Converting complex types with simple contents

Resolving conflicting nested element and attribute names during OWL property generation

When and how to distinquish global elements from complex types with similar names during OWL class generation

Generating enumerations

Handling substitution groups both at the XSD and XML levels

Handling the overriding of an XSD type with xsi:type in XML

The example that follows shows the approaches that we have used for the transformation.

Illustrative example of transformation rules

The basic transform for a Complex Type in XSD follows these rules:

An OWL class is generated for a complex type.

The URI of the class is generated in three different ways. If the complex type is global and named, then the name attribute is used. If the complex type is local and named, then the name attribute of the owner element is used. If the complex type is anonymous, then the names of its owner element and its parent element are used.

The xsd:annotation and attribute annotations describing the complex type get generated as dc:description, rdfs:comment and/or skos:definition OWL annotations.

Nested or reference children elements of the complex type become OWL allValuesFrom restrictions on the class. If the element has a simple type, then a restriction with an OWL datatype property and an XSD datatype range or a user-defined RDFS datatype range is generated. If the element has a complex type, then a restriction with an OWL object property and an OWL class range is generated. The minOccurs and maxOccurs values become OWL cardinality restrictions.

Element group and attribute group references are generated as super classes.

Attributes become restrictions with an OWL datatype property and an XSD datatype range or a user-defined RDFS datatype range.

An example of a Complex Type, Trade, in fpml-doc-5-2.xsd of transparency standard is displayed below:

<xsd:complexType name="Trade">
  <xsd:annotation>
    <xsd:documentation xml:lang="en">
    A type defining an FpML trade.</xsd:documentation>
  </xsd:annotation>
  <xsd:sequence>
    <xsd:element name="tradeHeader" type="TradeHeader">
      <xsd:annotation>
        <xsd:documentation xml:lang="en">
         The information on the trade which is not 
         product specific, e.g. trade date.
         </xsd:documentation>
      </xsd:annotation>
    </xsd:element>
    <xsd:group ref="TradeEconomics.model">
      <xsd:annotation>
        <xsd:documentation xml:lang="en">
        The economics of the trade. In the case of an 
        OTC trade, this is the OTC derivative product.
        In the case of a trade of a security,
        it is the instrument trade economoics.
        </xsd:documentation>
      </xsd:annotation>
    </xsd:group>
  </xsd:sequence>
  <xsd:attribute name="id" type="xsd:ID" />
</xsd:complexType>

The following is the graph of the OWL class generated for Trade complex type, which shows the OWL class, restrictions, annotations and superclass.

The following class diagram shows a more sophisticated view of Trade and its related classes downstream in the generated ontology (click on the diagram to open a window with a bigger image).

The diagram highlights these advanced features in generation:

A superclass relation exists between Trade, generated from an XSD complex type and TradeEconomics.model, generated from an XSD element group.

In the XSD, Swap element has the substitutionGroup Product element. Thus, A_Global-Swap becomes a subclass of A_Global-Product. A_Global- prefix is used to distinguish the element-derived classes from similarly named complex-type-derived classes.

dtype:value restrictions are generated to hold the simple contents occuring in complex types. The complex content part of the type become other restrictions.

The generated object properties have a Ref suffix to distinguish them from datatype properties with same names. Both types of properties can be used in restrictions on different classes as they may be generated from nested or reference children elements under different complex types.

The instance file, "msg_ex001_new_trade.xml" was imported into the transparency ontology. Here is a peek into that XML file:

...
<trade>
<tradeHeader>
  <partyTradeIdentifier>
    <tradeId tradeIdScheme=
        "http://fpml.org/universal_swap_id">123</tradeId>
    <tradeId tradeIdScheme=]
        "http://fpml.org/submitter_trade_id">456</tradeId>
  </partyTradeIdentifier>
  <tradeInformation>
    ...
    <cleared>true</cleared>
    <nonStandardTerms>false</nonStandardTerms>
    <offMarketPrice>false</offMarketPrice>
    <largeSizeTrade>false</largeSizeTrade>
    ...
  </tradeInformation>
  <tradeDate>2011-02-04</tradeDate>
</tradeHeader>
<swap>
  <productType>InterestRateSwap</productType>
  <assetClass>InterestRates</assetClass>
  <swapStream>
   ...
  </swapStream>
  <swapStream>
    ...
  </swapStream>
</swap>
</trade>
...

The above XML constructs were mapped into the following RDF graph, where you can see how the instances, their relationships and their types are generated with respect to the Trade class diagram (click on the graph to open up a window for a more detailed view).

Another example of transformation rules

The basic transform for an Enumeration in XSD follows these rules:

An OWL class is generated from an XSD simple type having XSD enumeration facets. The localname of the class has Enum suffix to distinguish it from classes generated with similar names.

This class becomes a subclass of EnumeratedValue in the same namespace as the OWL class, which itself becomes a subclass of dtype:EnumeratedValue.

Each XSD enumeration facet becomes an instance of the generated class. dtype:value holds the enumeration value. dtype:order is the order in which the enumeration facet occurs.

An Enumeration class in the same namespace as the OWL class is also generated. This class becomes subclass of dtype:Enumeration. An instance of this class is generated as a container to refer to all the instances generated from the current simple type.

Enumerated value instance URIs are generated using a concatenation of the abbreviation of the class localname's upper case letters and the dtype:value literal.

The following figure shows a graph for PremiumQuoteBasisEnum class and the OWL constructs generated from the related XSD enumeration facets (click on the diagram to open a window with a bigger image):

Complete table of supported transformations

For the reader interested in more details a full overview of the mapping transformations is given in the following table:

*Table: Conversion from XSD Constructs to OWL Constructs*
#	XSD/XML Constructs	OWL Constructs
1	`xsd:simpleType`	`owl:Datatype`
2	`xsd:simpleType` with `xsd:enumeration`	Becomes an `owl:Class` as a subclass of `EnumeratedValue`. Instances are created for every enumerated value. An instance of `Enumeration`, referring to all the instances, is created as well as the `owl:oneOf` union over the instances.
3	`xsd:complexType` over `xsd:complexContent`	`owl:Class`
4	`xsd:complexType` over `xsd:simpleContent`	`owl:Class`
5	`xsd:element` (global) with complex type	`owl:Class` and subclass of the class generated from the referenced complex type
6	`xsd:element` (global) with simple type	`owl:Datatype`
7	`xsd:element` (local to a type)	`owl:DatatypeProperty` or `owl:ObjectProperty` depending on the element type. OWL Restrictions are built for the occurrence.
8	`xsd:group`	`owl:Class` and subclass of `A_AbstractElementGroup`
9	`xsd:attributeGroup`	`owl:Class` and subclass of `A_AbstractAttributeGroup`
10	`xsd:minOccurs` and `xsd:maxOccurs`	Cardinality specified in minimum cardinality, maximum cardinality and universal (allValuesFrom) OWL restrictions.
11	Anonymous Complex Type	As for Complex Type except a URI is constructed from the parent element and the nested element reference. Also, the class is defined as a subclass of `A_Anon`.
12	Anonymous Simple Type	As for Simple Type except a URI is constructed from the parent element and the nested element reference.
13	`xsd:default` on an attribute	Uses `dtype:defaultValue` to attach a value to the OWL restriction representing the associated property.
14	Substitution Groups	Subclass statements are generated for the members. Instance files resolve their types by consulting the OWL model at import-time.
15	Annotation attributes on elements	OWL Annotation properties are created and placed directly on the relevant class.
16	Annotations using `xsd:annotation`	Become, based on user selection, `dc:description`, `rdfs:comment` and/or `skos:definition` OWL annotations.
17	`xsi:type` on an XML element	Overrides the schema type with the specified type.

A SPARQL Metric Query

As a quick check on the generated OWL models, the following is a useful SPARQL query that counts the number of properties on each OWL class.



 SELECT ?class (COUNT(DISTINCT ?p) AS ?properties)
 WHERE {
     ?class a owl:Class .
     FILTER( afn:namespace( ?class ) = 
        "http://www.fpml.org/FpML-5/transparency#") .
     OPTIONAL {
         ?class rdfs:subClassOf ?r .
         ?r a owl:Restriction .
         ?r owl:onProperty ?p .
     }
 }
 GROUP BY ?class
 ORDER BY DESC( ?properties )

The classes for the transparency ontology have the distribution of properties shown in the following figure (click on the diagram to open a window with a bigger image). For example, TradeInformation has 12 properties:

Concluding remarks

The new capability is easy to use. As before, a convenient import wizard will guide the user. The dialog has a number of new options. XML conversion will happen automatically when users open XML files in TBC or use XML import modules in SPARQLMotion. As long as an XML file is valid against an XSD that it is based on, the XML will be transformed in accordance to the schema. Parts of the XML files that do not validate against a schema will continue to be converted using the default Semantic XML structure. There is also a new option to specify which OWL file to use as a schema when mapping a specific XML file to triples. This feature is also available for spreadsheets and will be covered in a separate blog.

We believe that the importance of this work is not only in its value to harvest XML Schemas. Ability to use the automatic creation of triples from XML instance files directly in applications is proving to be key to a number of customers. For example, TopQuadrant is currently using this approach in a project for the North Sea Oil and Gas industry.

The functionality we have described will be released in TopBraid Composer 3.6.0. This release entered internal beta this week. Please contact us if you want a try these capabilities before general availability, which is currently planned for November.

Putting your drag-and-drop SPINMap vocabulary mappings into production

2011-07-21T15:25:00.000-07:00

The "Composing the Semantic Web" blog entry SPINMap: SPARQL-based Ontology Mapping with a Graphical Notation describes TopBraid 3.5's new tool for mapping between vocabularies or ontologies. (It also points to a handy video that demonstrates both simple and sophisticated uses of SPINMap.) Once you've created a mapping, though, how do you use it to convert data? As it turns out, no new technology is necessary; SPINMap just creates SPIN rules that you can apply in a SPARQLMotion script.

Let's look at an example. Imagine that I'm a publisher who receives images and metadata about those images from ExampleCo every month, and I load these images and metadata into my company's Digital Asset Management system. ExampleCo uses their own vocabulary to describe the metadata, but I prefer to use the NEPOMUK vocabulary for describing image metadata, because I know that by taking advantage of a vocabulary used by other systems around the world, my data can more easily interoperate with other data and tools.

Following the steps described in the blog posting mentioned above, I create the mapping from ExampleCo's pd:Image class and its associated properties to the NEPOMUK equivalents. Because the NEPOMUK image vocabulary's nexif:Photo class has so many properties associated with it, the diagram of it doesn't all fit on the screen at once, but it was easy enough scroll up and down as I mapped the pd:Image properties on the left to various NEPOMUK nexif:Photo properties.

I saved the mapping in its own file, which I called ExampleCo2Nepomuk.ttl. At this point, I could convert a set of ExampleCo metadata by importing a file of that data and ExampleCo2Nepomuk.ttl into the same model and then picking Run Inferences from the Inference menu, assuming that Configure Inferencing on the same menu had TopSPIN configured as the inferencing engine.

I wanted this to be more automated, though, so I put it in a SPARQLMotion script that could be called as a web service or from a TopBraid Ensemble interface. This would make it easier to re-use this mapping every month on each new batch of ExampleCo image data as it comes in:

The script's first module prompts for the input filename, because it will be a new dataset each month. This module hands the filename to the "Get ExampleCo RDF" module, an Import RDF From Workspace module that reads in the ExampleCo data.

At the same time, another Import RDF From Workspace module named "Get mapping rules" reads in the ExampleCo2Nepomuk.ttl file storing the SPIN-based mapping rules. Both of these modules feed their triples to an Apply TopSPIN module named "Apply mapping rules," which has its sml:replace value set to true so that it only passes along the new triples that it creates and not the input triples. The script's last module saves the result in a disk file, but could easily send it off for addition to a triplestore in a Digital Asset Management system.

There's nothing especially new or unusual in this script; what's new is that the rules that it applies to the data were created by a graphical drag-and-drop tool instead of being coded by hand. (Rest assured that the rules stored by the tool are still expressed using standard SPARQL.) With easy data aggregation being one of the great advantages of semantic web applications, it's nice to know that SPINMap lets you define data transformations with less trouble than ever before, making your application development (and application maintenance) even faster.

As an added bonus, because the mappings are stored as SPIN rules (also known as SPARQL Rules), they can easily be combined with other SPARQL Rules that you can run with the same script. These other rules might perform validation to ensure that the data being read conforms to certain data quality standards, or they could calculate new values based on a combination of the incoming data and existing stored data.

Comparing SPIN with RIF

2011-06-26T17:50:00.000-07:00

Since SPIN (SPARQL Inferencing Notation) aka SPARQL Rules became W3C member submission,we find ourselves responding to the growing interest to it.

With this, a question some may ask is how SPIN is different from or similar to RIF - W3C's standard for rules interchange.

While I have heard this asked a couple of times, I was pleasantly surprised that it was is not a very common question. Pleasantly, because a certain level of confusion is to be expected about new things and, both, SPIN and RIF are relatively new. If so few people ask this question, then SPIN specification did a good job explaining and positioning it and people easily grasp the unique and important needs it serves. Still, I thought it was worth while to write up my thoughts on comparing SPIN with RIF.

The goal of RIF was to create an interchange format for use between rules engines. As such, unlike SPIN, RIF is not an idea that is specifically or particularly aligned with RDF. This is why RIF was created as XML (although there is now work on RDF serialization). I am not pointing this out as a shortcoming of RIF, but rather to put in perspective the origin and the reason for RIF. In its goals, RIF is similar to OMG's XMI which also uses XML and was created to be an interchange format between different tools.

Given this similarity, XMI’s failure in being a reliable interchange format becomes relevant when considering RIF's future. Will RIF succeed in reaching its goal? One can easily argue that with the variety of available rules languages and engines, RIF’s job is harder than what XMI needed to do to succeed.

As noted here, different rules languages exist because there are different algorithms and formalisms for rules. Furthermore, different rule products have different sets of capabilities. RIF dialects are intended to be the least common denominators for a given type of a rule engine. This means that in order to effectively use the same set of RIF rules in the ‘rules engine A’ and in the ‘rules engine B’, the following needs to happen:

1. RIF dialect used to express the rules, needs to be supported by both rules engines.

Checking the implementation page, one will see that currently the overlap between any two engines is not that great. Some support BLD, some support PRD + Core, others support BLD partial or PRD minus something, etc.

2. RIF dialect used to express the rules, must be enough for the task at hand.

As mentioned above, RIF by design is somewhat of a least common denominator. This means that a user could always do more with a given rules engine than they can express in a dialect of RIF.

For example (as noted here), SPARQL is more expressive than what is possible with RIF. This is not unique to SPARQL, it is true for pretty much any rules technology.

3. The interchange must work

Given well known XMI issues, I am quite keen to see RIF test cases as well as test case results from the implementers

Attitude of the major rule engine vendors towards RIF is currently, at best, lukewarm. For example, on the Oracle forum, support engineers recommend against attempting to interchange rules by saying:

“In a hybrid environment I'd recommend that rules authored in ILOG be executed in the ILOG engine, and that rules authored in OPA be executed in the OPA engine, rather than attempt to interchange rules between the two products. As long as there is a clear scope boundary between what the rule sets are used for, then there wouldn't be any duplication or interchange of rules.”

Having considered the design goals and challenges of RIF, it is easy to see that the design goals of SPIN are quite different. SPIN is not about capturing rules that can then be translated for execution by different types of rule engines. Rather it is about capturing rules that can be executed directly over RDF data and about having rules that are intimately connected to the Semantic Web models.

With these goals in mind, we identified the following three things as important principles in SPIN's design:

1. Rules can be expressed in a familiar language. People working with RDF must know SPARQL. Using SPARQL for rules means that they don’t need to use another language

2. Rules can be executed by any RDF database. Since they are in SPARQL, rules are portable – not across rules engines, but across RDF stores

3. Evolution of the models does not unnecessarily break the rules. For example, let’s say we change the URI of a resource used in a rule. If a rule uses some other format (XML) and is not connected to the underlying RDF in a way other than a blob, it becomes hard to maintain these two different sets of information

Finally, SPIN takes an object-oriented approach to rules. It is about programming and about associating behavior with classes while RIF takes a model-theoretic view on how the rules may relate to ontologies. This is a key difference as noted in W3C comments on SPIN submission.

In short, SPIN and RIF address different needs and have different design goals. They can be considered complimentary.

What about using SPIN and RIF together? Given the key role SPARQL plays in the architecture of Semantic Web solutions, I am certain that should RIF get traction in its adoption, someone will create a RIF profile for SPARQL and write a RIF to SPARQL translation.

How to: extend an ontology

2011-03-30T07:04:00.000-07:00

When people work with ontologies, XML schemas and software development, it's almost a cliché to say that re-use of existing work is better than creating something new from scratch. Existing work, though, is not always a perfect fit to your needs, and the ease of customizing it for your needs often depends a lot on how the original work was designed—when you're reusing XML schemas or software source code. Customizing OWL ontologies and RDF schemas, on the other hand, is pretty simple nearly all of the time, especially when you use TopBraid Composer.

For example, let's say that you have a taxonomy of business terms to track, and the W3C's SKOS standard defines all the properties you need to maintain metadata about these terms, with two exceptions: SKOS has nothing about the last person to edit a term and it has no slot for the editor's department code, which is a special bit of metadata within your enterprise. Customizing SKOS to include these is just two steps:

Create an empty new ontology and import SKOS into it.
Define your two properties in this new ontology and give them a domain of skos:Concept so that SKOS tools such as TopBraid Enterprise Vocabulary Net(EVN) know that they're potential properties of your SKOS concepts.

To do this with TopBraid Composer, start by creating a new RDF/OWL file in one of your projects. On the wizard dialog box for creating this file, enter the Base URI of your new customized version of the SKOS ontology. If I worked for The Example Company, I might create a baseURI of http://example.com/ns/exskos.

On the same dialog box, click the checkbox next to SKOS under "Initial imports" and click Finish. (If you forget to click the checkbox first, you can always drag the skos-core.rdf file from the Navigator View's /TopBraid/SKOS folder to the Imports tab. In fact, to start creating a customized version of any ontology, drag a copy of it into your custom ontology's Imports tab.)

Once the file is created, having a namespace prefix associated with the base URI makes it easier to create the new properties, so on your new ontology's Overviewtab add an ex prefix for the http://example.com/ns/exskos# namespace. Don't forget the pound sign; the prefix will be standing in for this URI, and you wantex:editor to represent http://example.com/ns/exskos#editor, not http://example.com/ns/exskoseditor.

Now we're ready to add the customizations. Instead of creating a new editor property from scratch, it's better to define it as a subproperty of the Dublin Coredc:creator property so that applications that don't know about our new property but do know about Dublin Core properties will have some clue what it's for.

Drag the dc1-1.rdf Dublin Core ontology file from the /TopBraid/Common folder in TopBraid Composer's Navigator view to the Imports tab to import the Dublin Core ontology. You'll see several new properties will join the SKOS ones on the Properties view.
Right-click on dc:creator there and pick Create subproperty. In the Create subproperty wizard, replace the dc:Property_1 value that appears as the default Name of new instance value with ex:editor, which uses the ex prefix that you defined earlier.
Click the checkbox next to rdfs:label on the dialog box's Annotations Template so that an rdfs:label property gets automatically set for this property. (One nice thing about how the RDF data model lets you assign properties to properties is that you can associate human readable names to substitute for the actual property names on forms and reports.)
Click the OK button and you'll see a property form in the middle of your screen for your new property, which will be selected in the Properties view. (It's on the left of the screenshot below because the Classes view is not shown there.)
To show that your ontology is defining this new property for SKOS concepts, click the small white triangle next to the word domain on the ex:editor property form and pick Add existing to indicate that this property's domain will be an existing class. Click skos:Concept, which will be under owl:Thing on the wizard's class tree, and click OK.

You're finished defining your editor property. The department code one will be even quicker to create, because it won't be a subproperty of something else:

Click the Properties view's small white triangle to display its menu and pick Create rdf:Property. Name the new property ex:deptCode on the Create Property wizard dialog box. The Annotations Template's rdfs:label checkbox should already be checked, so click OK.
Set your newest property's domain to skos:Concept the same way you did for ex:editor and save your file.

You're done! You know have a customized version of the SKOS ontology. If you now use TopBraid Composer or EVN to create a new instance of the Concept class in this file, you would see editor and dept code fields on your new concept's resource form along with has broader and all the other standard SKOS properties. (If you create concept instances with TopBraid Composer and the labels are toggled to show qnames instead of human-readable labels, they'll say ex:editor andex:deptCode.) Instead of creating concept instances in this file, though, you would more likely create a new taxonomy file that imports your customized ontology the same way that your ontology imported the SKOS and Dublin Core ontologies, and then you would store your taxonomy's concepts in this new file.

The modularity of this approach brings another benefit that isn't as easy when customizing typical XML schemas and other software resource files: when a SKOS upgrade is released, you can simply delete the import of the current SKOS ontology in your customization of it and import the new one instead, and all of your applications that use your custom ontology should be able to go on using it the same way the did before.

It's nice to know that customization of a standard ontology that nearly meets your needs is so easy, and many organizations are doing this with the SKOS ontology to create a better fit with their vocabulary management requirements. This isn't limited to customizing SKOS, though; the same principle works with any OWL ontology or RDF schema. As an added benefit, if you create a customized version of a particular standard for your enterprise, you can follow these same steps to create customizations of your customization for individual departments within your enterprise.

Converting UML Models to OWL - Part 1: The Approach

2011-02-03T17:12:00.000-08:00

Convert UML to OWL - why would you ever want to do this? One reason suffices: many enterprise models, that serve as either standards or enterprise schemas, are specified in UML. Increasingly, there is interest in having content of UML models re-purposed in RDF/OWL and the need for RDF/OWL to interoperate with systems built from UML Models.

UML Models are notoriously hard to exchange between UML tools, let alone be transformed into OWL. The exchange format XMI is not only is difficult to understand but also has vendor-specific extensions. The vagaries of MOF, CMOF and EMOF create their own challenges. Nonetheless we have done transformations of UML to OWL. Using a model-based transformation approach, based on SPARQL Rules, XMI models of UML models can be converted to OWL. UML class diagrams can be represented in OWL without information loss. The inverse, however, is not true and will require another blog series.

UML to OWL - Part 1 Contents

Part 1 of the series explains the basis of the approach. The complete series of blogs, as currently conceived, is as follows:

Converting UML Models to OWL - Part 1: The Approach
Converting UML Models to OWL - Part 2: Transforming UML Models to OWL Using SPARQL
Converting UML Models to OWL - Part 3: Examples of Industry UML Model Transformations

The content of this blog is organized as follows:

Goals, Objectives and Requirements
Backgrounder on XMI
Backgrounder on MOF
Solution Outline
Overview of Semantic XML
OCMOF - the OWL Representation of CMOF
How the Transformations from UML to OWL Work
Generation of UML Metaclasses
Generation of UML Classes
Generation of UML Class Superclass Relationships
Generation of UML Packages
Generation of UML Package Relationships
Performance
Concluding Remarks

Readers who are very interested in the detailed technical approach, should read all sections of this blog in order. Those who just need to have an overview of the approach could skip sections 9 through 12. Those who have deep knowledge of XMI and MOF may want to skip sections 2 and 3, but I would welcome their feedback on the accuracy of my statements.

Note that some diagrams may be too small to be viewed in the body of the document. Clicking on such a diagram will open a new window with a larger depiction of the diagram.

Goals, Objectives and Requirements

The OWL Models must faithfully represent packages and the logical models or class diagrams. Out of scope, currently, are all of the other UML models such as Interaction Diagrams and State Diagrams. The approach must be able to convert UML by processing XMI files from specific tools. This requires a strategy for converting from the XML structures of XMI to OWL models.

Backgrounder on XMI

XMI, the XML Metadata Interchange standard is a serialization format for UML Models. The main purpose of XMI is to define how the XML elements are organized within an XMI file. The XMI spec also defines a mechanism for how one XMI element references another, within and across XMI files. Such a mechanism is needed as it is a legal scenario for a single UML model to be serialized to more than one XMI file.

Top

Backgrounder on MOF

MOF began at the time of CORBA and the need for IDL interfaces. MOF 1.4 resulted in its mapping to Java being codified in the Java Community Process (JCP) as the Java Metadata Initiative (JMI). MOF 2.0 was developed in tandem with UML 2.0. The separation of MOF into EMOF and CMOF was motivated by the influence of EMF's Ecore, and model-driven Java development. CMOF was more the motivation of meta model developers. CMOF stands for Complete Meta Object Facility and is an OMG standard for the UML 2 model interchange. More information can be found at this page on the OMG Website.

CMOF includes fully fledged associations, association generalization, property subsetting and redefinition, derived unions, and package merge. Typical XMI container structures look like the example below, from the CMOF UML Infrastructure Model. The basic idea is that a packagedElement owns other elements. A type attribute specifies the type of the packagedElement.
Things get a little busy with how IDs are used for associations and their member ends. That complication, we can leave for Part 2.

<?xml version="1.0" encoding="UTF-8"?>
  <xmi:XMI xmi:version="2.1" xmlns:xmi="http://schema.omg.org/spec/XMI/2.1"
   xmlns:cmof="http://schema.omg.org/spec/MOF/2.0/cmof.xml">
   <cmof:Package xmi:id="_0" name="InfrastructureLibrary">
    <ownedMember xmi:type="cmof:Package" xmi:id="Core" name="Core">
     <ownedMember xmi:type="cmof:Package"
      xmi:id="Core-Abstractions" name="Abstractions">
      <packageImport xmi:type="cmof:packageImport"
      xmi:id="Core-Abstractions-_packageImport.0"
      importedPackage="Core-PrimitiveTypes"
      importingNamespace="Core-Abstractions"/>
      <ownedMember xmi:type="cmof:Package"
       xmi:id="Core-Abstractions-Ownerships" name="Ownerships">
       <packageImport xmi:type="cmof:packageImport"
        xmi:id="Core-Abstractions-Ownerships-_packageImport.0"
        importedPackage="Core-Abstractions-Elements"
        importingNamespace="Core-Abstractions-Ownerships"/>
        <ownedMember xmi:type="cmof:Class"
         xmi:id="Core-Abstractions-Ownerships-Element" name="Element" isAbstract="true">
         <ownedComment xmi:type="cmof:Comment"
          xmi:id="Core-Abstractions-Ownerships-Element-_ownedComment.0"
          annotatedElement="Core-Abstractions-Ownerships-Element">
          <body>An element is a constituent of a model.
          As such, it has the capability of owning other elements.</body>
         </ownedComment>
        <ownedRule xmi:type="cmof:Constraint"
         xmi:id="Core-Abstractions-Ownerships-Element-not_own_self"
         name="not_own_self" constrainedElement="Core-Abstractions-Ownerships-Element"
         namespace="Core-Abstractions-Ownerships-Element">
         <ownedComment xmi:type="cmof:Comment"
          xmi:id="Core-Abstractions-Ownerships-Element-not_own_self-_ownedComment.0"
          annotatedElement="Core-Abstractions-Ownerships-Element-not_own_self">
          <body>An element may not directly or indirectly own itself.</body>
         </ownedComment>
         <specification xmi:type="cmof:OpaqueExpression"
          xmi:id="Core-Abstractions-Ownerships-Element-not_own_self-_specification">
          <language>OCL</language>
          <body>not self.allownedElements()->includes(self)</body>
         </specification>
        </ownedRule>
        ...
        <ownedAttribute xmi:type="cmof:Property"
         xmi:id="Core-Abstractions-Ownerships-Element-ownedElement"
         name="ownedElement" type="Core-Abstractions-Ownerships-Element"
         upper="*" lower="0" isReadOnly="true" isDerived="true"
         isDerivedUnion="true" isComposite="true"
         association="Core-Abstractions-Ownerships-A_ownedElement_owner">
         <ownedComment xmi:type="cmof:Comment"
          xmi:id="Core-Abstractions-Ownerships-Element-ownedElement-_ownedComment.0"
          annotatedElement="Core-Abstractions-Ownerships-Element-ownedElement">
          <body>The Elements owned by this element.</body>
         </ownedComment>
        </ownedAttribute>
        ...

Figure 1: A sample of XMI

For more background on the history of MOF the following references may be of value: MOFLON, and Wikipedia.

Top

Solution Outline

Model-based transformation is the central idea of the approach. To implement it we have developed a metamodel of CMOF in OWL. Our strategy is to get out of XML into RDF Triples as soon as possible. Using an ontology of XML we convert XMI into a composite model of triples. XML is a simple enough structure for the composite object pattern - elements contain elements and elements have attributes. XML elements and attributes that make up the XMI file are transformed into OWL instances of the CMOF metamodel. Once we have the XMI in triples we can map constructs to classes and properties of a CMOF metamodel. This model then serves as the generator for model-based transformations to an OWL model of the UML.

Once these instances are loaded as "raw" RDF, rules fire to perform the transformations. Rules are associated with classes to ensure that instances of those classes are processed in an execution sequence. Using SPARQL Rules (SPIN), instances of a class are each processed through a binding mechanism specified by ?this variable. SPARQL Rules can be considered an approach that is similar to, or can be compared with, UML's Object Constraint Language (OCL) and the Query/View/Transformation (QVT) approach to transformations.

The benefits of the OWL and SPARQLRules model-based approach to transformation are:

Intimacy of the rules with RDF/OWL - triples are evaluated directly
Understandability - rules are smaller and expressed in the relevant contexts of the model
Enhanced Performance - evaluation of rules is localized to relevant instances
Customizability and Evolvability – transformations can be changed by modifying models and/or SPARQL rules
Ease of maintenance - rules are associated with the constructs they operate over

Top

Overview of Semantic XML

XMI is imported into the CMOF metamodel using TopBraid Composer's Semantic XML as a mapping method. With Semantic XML, TopBraid can automatically generate an OWL/RDF ontology from any XML file. Each distinct XML element name is mapped into a class, and the elements themselves become instances of those classes. A datatype property is generated for each attribute. Nesting of XML elements is represented in OWL using a composite:child property - an object pattern in OWL that is described at this blog entry.

The key idea of Semantic XML is that each of the generated OWL classes and datatype properties is annotated with an annotation property, sxml:element and sxml:attribute, respectively. These properties relate the OWL concepts to the XML serialization. Note that these annotations are also used if an OWL model needs to be serialized back to XML format.

If you import an XML file into an ontology that already contains classes and properties with Semantic XML annotations, then the loader will reuse those. The mapping is bi-directional and loss-less so that files can be loaded, manipulated and saved without losing structural information.

A video explaining how Semantic XML works is available at this link.

Top

OCMOF - the OWL Representation of CMOF

The strategy for the transformation can be summarized as follows:

Use OWL classes to represent XMI Element Types
Use SPARQL Rules on those classes to generate CMOF Metaclasses
Use Metaclasses to make OWL Classes that represent the UML Model

An OWL metamodel of CMOF represents the kinds of containers, elements and attributes shown above. The metamodel was built by studying the UML Metamodel of UML 2.0 - the original motivation for this was to have an automated way of dealing with changes to UML. That will be a future consideration, for now this has proven to be a valuable way of doing verification and validation. The UML metamodel will be covered in Part 2 of this blog series, for Part 1, it is instructive perhaps to show a small piece of the XMI. Below is the XMI for Basic-Property from the UML model infrastructure.cmof.xmi.

<ownedAttribute xmi:type="cmof:Property"
  xmi:id="Core-Basic-Class-ownedAttribute" name="ownedAttribute"
  type="Core-Basic-Property" isOrdered="true"
  upper="*" lower="0" isComposite="true"
  association="Core-Basic-A_ownedAttribute_class">
  <ownedComment xmi:type="cmof:Comment"
    xmi:id="Core-Basic-Class-ownedAttribute-_ownedComment.0"
    annotatedElement="Core-Basic-Class-ownedAttribute">
    <body>The attributes owned by a class.
    These do not include the inherited attributes.
    Attributes are represented by instances of Property.</body>
  </ownedComment>
 </ownedAttribute>

Figure 2: A fragment of the XMI for the UML metamodel

As a example of XMI element mappings, the sxml:element maps the XMI element for ocmof:ownedAttribute as shown in the Turtle extract from the OWL model below.

ocmof:ownedAttribute
  a  owl:Class ;
  rdfs:label "Attribute"^^xsd:string ;
  rdfs:subClassOf ocmof:TypedThing , ocmof:NamedThing ;
  rdfs:subClassOf
    [ a owl:Restriction ;
      owl:maxCardinality "1"^^xsd:nonNegativeInteger ;
      owl:onProperty ocmof:isComposite
    ] ;
  rdfs:subClassOf
    [ a owl:Restriction ;
      owl:maxCardinality "1"^^xsd:nonNegativeInteger ;
      owl:onProperty ocmof:type
    ] ;
  rdfs:subClassOf
    [ a  owl:Restriction ;
      owl:maxCardinality "1"^^xsd:nonNegativeInteger ;
      owl:onProperty ocmof:isDerivedUnion
    ] ;
  rdfs:subClassOf
    [ a  owl:Restriction ;
     owl:maxCardinality "1"^^xsd:nonNegativeInteger ;
     owl:onProperty ocmof:isReadOnly
    ] ;
  rdfs:subClassOf
    [ a  owl:Restriction ;
      owl:maxCardinality "1"^^xsd:nonNegativeInteger ;
      owl:onProperty ocmof:default
    ] ;
  sxml:element "ownedAttribute"^^xsd:string .

Figure 3: ocmof:ownedAttribute in Turtle

The last line, sxml:element "ownedAttribute"^^xsd:string, is the mapping.

As a example of XMI attribute mappings, the sxml:attribute maps the XMI attribute for ocmof:isOrdered as shown in the Turtle extract from the OWL model below.

ocmof:isOrdered
  a owl:DatatypeProperty ;
  rdfs:domain ocmof:ownedAttribute ,
  ocmof:ownedParameter ,
  ocmof:OwnedEnd ;
  rdfs:label "is ordered"^^xsd:string ;
  rdfs:range xsd:boolean ;
  sxml:attribute "isOrdered"^^xsd:string .

Figure 4: ocmof:isOrdered in Turtle

The last line, sxml:attribute "isOrdered"^^xsd:string, is the mapping.

The transformation to OWL results in the following class for uml:Core-Basic-Property.
Clicking on the image will open a larger image in a new window.

Figure 5: A Generated Metaclass Example - uml:Core-Basic-Property

The diagram shows how the datatype properties of the class uml:Core-Basic-Property correspond to the XMI attributes given in the above fragment. For example isComposite becomes the property hasBooleanIsComposite. The prefix hasBoolean is customizable.

First an OWL model of CMOF XML Elements is used to generate instances of metaclasses to build OWL Classes for XMI Elements. The namespace prefix of ocmof has been used to denote all modeling constructs that makeup the CMOF metamodel. The prefix cmof is the namespace for all constructs generated from the import of the XMI files.

In the diagram below, we show the main classes of the metamodel. Classes like NamedThing and TypedThing have been introduced to optimize the work of the transformers. Constructs in XMI can typically be both named and typed. This kind of multiple inheritance is no problem for the transformations. The diagram is a partial view only. Clicking on the image will open a larger image in a new window.

Figure 6: Some of the classes of the CMOF OWL model

As an alternate view, the diagram that follows is an HTML report of NamedThing in TopBraid Composer. This is automatically generated using SPARQL Web Pages (aka UISPIN)

Figure 7: OCMOF NamedThing - an abstract class for the transformations

The diagram below shows more details of some ownedElements. Note how attributes of each of these classes relate to CMOF constructs.

Figure 8: Some "ownedElement" OWL Classes in the OCMOF model

These ocmof classes serve as the starting point for generating ocmof meta-classes and instances of these classes that become the UML model transformed into OWL. The figure below shows the main metaclasses that are generated by rules on the ocmof classes.

Figure 9: The key Meta-classes of the CMOF OWL model

Top

How the Transformations from UML to OWL Work

Model-based transformations use rules associated with OWL Classes. OWL Metaclasses are built using a SPARQL rule for instances of TypedThing. The names of the metaclasses are determined from the value of the xmi:type attribute. A number of SPARQL Rules are defined on TypeThing. Priorities are set by the alphabetic ordering given by the first comment line of the rule. These rules look after the generation of:

UML Metaclasses
UML Classes
UML Class Superclass Relationships
UML Packages
UML Package Relationships

Each rule will now be described.

Top

Generation of UML Metaclasses

The first task is to create a metaclass and class for every type of element in the ingested XMI file. This is done using the SPARQL Rule below:


# STEP CMOF-SR-001  make UML Metaclass from CMOF type
CONSTRUCT {
  ?metaClassURI a rdfs:Class .
  ?metaClassURI rdfs:subClassOf cmof:MetaClass .
  ?metaClassURI rdfs:label ?metaClassLabel .
  ?typeURI a owl:Class .
  ?typeURI a ?metaClassURI .
  ?typeURI rdfs:subClassOf uml:Construct .
  ?typeURI rdfs:label ?classLabel .
 }
WHERE {
  ?this xmi:type ?type .
  FILTER (?type != "cmof:Property") .
  BIND (o2o:localNameOfQName(?type) AS ?name) .
  BIND (fn:concat("CMOF ", ?name) AS ?metaClassLabel) .
  BIND (fn:concat("UML ", ?name) AS ?classLabel) .
  BIND (xmi.common:makeUML-URI(?name) AS ?typeURI) .
  BIND (xmi.common:makeCMOF-URI(?name) AS ?metaClassURI) .
}

Figure 10: The SPARQL Rules that make the metaclasses in the OCMOF model

What is going on in these rules? First we explain the "where" clause.

?this xmi:type ?type binds ?this to an instance of TypedThing. For each instance the rule is evaluated.

FILTER (?type != "cmof:Property") blocks further evaluation of the rule if the instance is of type cmof:Property. The reason for this will be explained in Part 2.

BIND (o2o:localNameOfQName(?type) AS ?name) extracts the name of the type from the QName.

BIND (fn:concat("CMOF ", ?name) AS ?metaClassLabel ) builds a label for the metaclass. The function fn:concat is from the JENA SPARQL Library. We use it here to prepend "CMOF" to the name we get from the type of the XMI Element.

BIND (fn:concat("UML ", ?name) AS ?classLabel) makes a class label from the name. We will be constructing both a metaclass and a class from the XMI type. We build a metaclass in order to say what kind of things can happen on the classes. In other words, the generated OWL model is a 3-level ontology. Likewise here we build a label for the UML Class.

BIND (xmi.common:makeUML-URI(?name) AS ?typeURI) builds a URI for the UML Class corresponding to type. This uses a function call to xmi.common:makeUML-URI whose job it is to build the correct namespace path for a UML construct URI. The implementation is shown below.

SELECT ?uri
WHERE {
  BIND (xmi.common:baseURI() AS ?baseURI) .
  BIND (smf:buildURI("{?baseURI}#{?arg1}") AS ?uri) .
}

where,
smf:buildURI("{?baseURI}#{?arg1}")) builds a URI for the name given in ?arg1 with a base URI supplied by the function xmi.common:baseURI().

BIND (xmi.common:makeCMOF-URI(?name) AS ?metaClassURI ) builds a URI for the metaclass corresponding to type. Likewise this constructs a namspace path for CMOF constructs.

Next we explain what is happening in the head of the rule with the Construct statements. These statements use the generated URIs to create instances of meta-classes and classes.

?metaClassURI a rdfs:Class
gives the metaClass its type.

?metaClassURI rdfs:subClassOf cmof:MetaClass
specifies that the metaclass is a sub-class of cmof:MetaClass - an abstract metaclass for all cmof classes.

?metaClassURI rdfs:label ?metaClassLabel
gives the metaclass a human label.

?typeURI a owl:Class
gives the UML Class a type

?typeURI a ?metaClassURI
gives the UML Class a more specific type so that it can have more properties than owl:Class provides.

?typeURI rdfs:subClassOf uml:Construct
specifies that the UML Class is a subclass of the abstract OWL Class uml:Construct/

?typeURI rdfs:label ?classLabel
gives the UML Class a human label.

Top

Generation of UML Classes

Once we have the necessary metaclasses we can begin the work of creating instances of those classes. These instances will, of course, be classes (the meta-world can get confusing). This work is done the the SPARQL Rule below.


# STEP CMOF-SR-002  make UML Classes from CMOF elements
CONSTRUCT {
  ?type a rdfs:Class .
  ?type rdfs:subClassOf cmof:MetaClass .
  ?class a ?type .
  ?class rdfs:label ?name .
  ?class ocmof:hasCMOFbasis ?this .
  ?superURI a owl:Class .
  ?superURI a cmof:CategoryClass .
  ?superURI rdfs:label ?super .
  ?subURI a owl:Class .
  ?subURI a cmof:CategoryClass .
  ?subURI rdfs:subClassOf ?superURI .
  ?subURI rdfs:label ?sub .
  ?class rdfs:subClassOf ?mySuperClass .
}
WHERE {
  ?this xmi:type "cmof:Class" .
  ?this xmi:id ?name .
  BIND (o2o:pathPart(?name, "-") AS ?path) .
  OPTIONAL {
    ?path o2o:pairHyphenIncrementally ( ?super ?sub ) .
    BIND (xmi.common:makeUML-URI(smf:buildString("CLASSES_{?super}")) AS ?superURI) .
    BIND (xmi.common:makeUML-URI(smf:buildString("CLASSES_{?sub}")) AS ?subURI) .
    BIND (xmi.common:makeCMOF-Resource("cmof:Class") AS ?type) .
    BIND (xmi.common:makeUML-URI(?name) AS ?class) .
    } .
  BIND (xmi.common:makeUML-URI(smf:buildString("CLASSES_{?path}")) AS ?mySuperClass) .
  }

Figure 11: The SPARQL Rules that make UML Classes

More details of this transformation will be given in Part 2 of this blog series. An interesting aspect of this particular rule to mention now is how it builds deep inheritance structures by using Property Functions to recurse over hyphenated names (more on the use of Property Functions, also known as Magic Properties, with TopBraid Composer can be found at this blog entry). These hyphenated names occur throughout the XMI metamodel of UML. For example Core-Basic-Class looks like:

<ownedMember xmi:type="cmof:Class" xmi:id="Core-Basic-Class" name="Class" superClass="Core-Basic-Type">
  <ownedComment xmi:type="cmof:Comment" xmi:id="Core-Basic-Class-_ownedComment.0"
    annotatedElement="Core-Basic-Class">
    <body>A class is a type that has objects as its instances.</body>
  </ownedComment>
  <ownedAttribute xmi:type="cmof:Property" xmi:id="Core-Basic-Class-isAbstract"
    name="isAbstract" type="Core-PrimitiveTypes-Boolean" default="false">
    <ownedComment xmi:type="cmof:Comment" xmi:id="Core-Basic-Class-isAbstract-_ownedComment.0" annotatedElement="Core-Basic-Class-isAbstract">
    <body>True when a class is abstract.</body>
   </Attribute>

Figure 12: Example of Hyphenated Names in the UML Metamodel

How this is done in the SPARQL Rule is explained briefly below.

In the SPARQL Rule shown above in figure 11, the statement in the tail: ?path o2o:pairHyphenIncrementally ( ?super ?sub ) is a Property Function that returns two results: ?super and ?sub for every hypenated pair.
for each pair the statement: ?subURI rdfs:subClassOf ?superURI in the head of the rule builds superclass relationships.

Top

Generation of UML Class Superclass Relationships

Once we have all of the UML Classes, the next rule can build the rdfs:subClassOf relationships.

# STEP CMOF-SR-005 - fixup the superclass of the root Classes
CONSTRUCT {
  ?class rdfs:subClassOf uml:Class .
}
WHERE {
  ?class a cmof:CategoryClass .
  BIND (afn:localname(?class) AS ?className) .
  FILTER fn:starts-with(?className, "CLASSES_") .
  NOT EXISTS {
  ?class rdfs:subClassOf ?superClass .
 } .
}

The result of executing the preceding UML Class rules is the UML Class Hierarchy shown in the diagram below.

Figure 13: Generated UML Metamodel Class Hierarchy

Top

Generation of UML Packages

# STEP CMOF-SR-020 - make Packages
CONSTRUCT {
  ?package rdfs:label ?name .
  ?package ocmof:hasCMOFbasis ?this .
  ?mySuperClass a owl:Class .
  ?mySuperClass rdfs:label ?path .
  ?superURI a owl:Class .
  ?superURI a cmof:CategoryClass .
  ?superURI rdfs:label ?super .
  ?subURI a owl:Class .
  ?subURI a cmof:CategoryClass .
  ?subURI rdfs:subClassOf ?superURI .
  ?subURI rdfs:label ?sub .
  ?package a ?mySuperClass .
  ?package a uml:Package .
}
WHERE {
  ?this xmi:type "cmof:Package" .
  ?this xmi:id ?name .
  BIND (xmi.common:makeUML-URI(?name) AS ?package) .
  BIND (o2o:pathPart(?name, "-") AS ?path) .
  OPTIONAL {
    ?path o2o:pairHyphenIncrementally ( ?super ?sub ) .
    BIND (xmi.common:makeUML-URI(smf:buildString("PACKAGES_{?super}")) AS ?superURI) .
    BIND (xmi.common:makeUML-URI(smf:buildString("PACKAGES_{?sub}")) AS ?subURI) .
  } .
  BIND (xmi.common:makeUML-URI(smf:buildString("PACKAGES_{?path}")) AS ?mySuperClass) .
}

Top

Generation of UML Package Relationships

# STEP CMOF-SR-024 - fixup the superclass of the root Packages
CONSTRUCT {
  ?packageClass rdfs:subClassOf uml:Package .
}
WHERE {
  ?this xmi:type "cmof:Package" .
  ?package ocmof:hasCMOFbasis ?this .
  ?package a ?packageClass .
  NOT EXISTS {
  ?packageClass rdfs:subClassOf ?superClass .
  } .
 }

The result of executing the preceding UML Package rules is the UML Package Hierarchy shown in the diagram below.

Figure 14: Generated UML Metamodel Package Hierarchy

Top

Performance

As a measurement of the performance with the TopBraid Composer release 3.4.0, the conversion of the UML Infrastructure XMI took 38.611 seconds and generated 19,575 statements (RDF triples) on a DELL Studio XPS Laptop with 4GB of memory, running Windows 7. This translates to an inference speed of 507 TPS (Triples per second).

Top

Concluding Remarks

Part 1 of this blog has introduced the power of model-based transformation using SPARQL Rules as a means to transform XMI to OWL. Our experience in doing this work confirms the extensibility and flexibility of this approach. The subject is a complex one requiring a grounding in the intricacies of UML Metamodeling, and a knowledge of SPARQL and SPARQL Rules. We have attempted to do that briefly in this blog - not an easy matter.

Part 2 of this blog series will discuss transforming UML Models to OWL Using SPARQL.

Running TopBraid Live in the Amazon EC2 cloud

2011-01-19T05:12:00.000-08:00

A recent Dilbert strip inspired me to go through Dave Winer's EC2 for Poets tutorial as a geeky weekend project. It was surprisingly easy and inexpensive to get a computer image running in Amazon's "Elastic Computing Cloud" (EC2) and to then get a copy of TopQuadrant's TopBraid Live running in that image.

These images are cheap to run, as you can see on their price list. Note that the cost per hour of running a default Linux image is not eighty-five cents an hour when using servers in northern Virginia, but eight and a half cents. (It's an additional penny an hour when using their servers in California, Ireland, or Singapore.) If you're willing to spend a dollar or two an hour, you can get full control of some really large-scale computing power without spending much money unless you're planning some long-term use of it, in which case you'll want to compare the options with your requirements more closely than I did. You'll want one of the EBS images, which still offers you a wide choice of platforms. Otherwise, the only way to stop a running image is to terminate it, in which case it (and all your configuration of it) is gone. With an EBS image, once you stop running it, you can always restart it.

I started off with an Ubuntu Linux image, but I had trouble installing Sun Java on it. After starting up a Fedora image, installing Sun Java and Tomcat were easy, and once Tomcat was installed, the installation of TopBraid Live under Tomcat according to the TBL installation instructions was simple and straightforward.

The ease with which I switched my efforts from Ubuntu to Fedora was an important lesson in cloud use; it sure was easier than installing Fedora over Ubuntu on a laptop hard disk, and if I decided to switch back, going back to the Ubuntu EC2 image would take seconds, unlike reverting a hard disk once I'd overwritten the Ubuntu image with Fedora. You can initialize several different images in the EC2 cloud, and if you're not running any of them, Amazon only bills you for storage, which is pennies, so you can set up several to wait for you and start and stop them whenever you like. If you want more memory, cluster processing, 64-bit instead of 32-bit, or other additional resources, you can just try it and see how it goes. This means that when you plan out the deployment of TopBraid applications using 32- or 64-bit TBL, you don't have to worry too much about having an available machine with the best possible operating system and hardware configuration, because you can experiment with different cloud images until you find the setup that's best for you. (I never tried a Windows image or SUSE or one of the other Linux images, but if I was going to roll out a production TBL application I'd explore these further as part of my planning.)

It's pretty impressive to think about the kind of power and flexibility that you can get when you deploy cloud-based semantic web applications, and as I found out with my little experiment, the barriers to entry for trying it out are extremely low. After 16 hours of running one instance, 3 hours of running another, and 3.1 GB of data transfer in the last few weeks, I currently owe Amazon Web Services a total of $2.29. So, if you're holding off on trying TBL because you don't have the appropriate box on which to run it, Amazon's EC2 offers some nice options to try out.

How to: convert a spreadsheet to SKOS

2010-12-29T11:10:00.000-08:00

In an earlier entry, we learned how SPARQL Rules can increase the quality of taxonomies and other controlled vocabularies stored using the W3C SKOS ontology. (As I wrote there, the Simple Knowledge Organization Systemvocabulary management specification is gaining popularity because, as a standard, it makes it easier to share taxonomies and thesaurii between different systems. It also guards investments in vocabulary development against the potential problems of dependence on a proprietary vendor format.)

TopQuadrant's Enterprise Vocabulary Net (EVN) vocabulary manager uses SKOS as its default format for storing data. Whether you use EVN or not, a first step in systematic management of vocabularies is often the conversion of vocabularies stored in ad hoc spreadsheets—an unfortunately very popular way to store them—to SKOS, so today we'll look at how TopBraid makes this conversion easy.

Below is an Excel spreadsheet with some data about a few Caniformia animals. (In the Linnaeus classification of animals, Caniformia is the suborder of Carnivora, which is an order of the Mammalia class.) It shows two families of this suborder and a few genuses and species of each family, with both the Latin and common name of each species.

Using a SPARQLMotion script, the basic steps of converting a spreadsheet like this to SKOS are:

Read in the spreadsheet as a set of RDF triples.
Use a CONSTRUCT query to convert the spreadsheet triples to SKOS triples. This is the step that varies the most from one conversion to another, because people can arrange spreadsheets any way they want, so the logic of the CONSTRUCT query has to infer the correct relationships between the values on the spreadsheet.
Save the SKOS triples as an RDF file or in whatever format is appropriate to your applications that will use this data.

The following shows the SPARQLMotion script that I used to convert the spreadsheet above.

It has a module for each of the three steps listed above and an additional SetBaseURIStr module to set a ?baseURIStr variable. The script refers to the base URI of the output several times, and instead of hardcoding it in all those places, I decided to use this module to set this variable and to then reference the variable from other places so that resetting the base URI could be done in one place. The "set BaseURIString" module has a very simple SELECT query:

SELECT ?baseURIStr
WHERE {
    LET (?baseURIStr := "http://example.com/taxonomies/animals") .
}

When you import an Excel file into TopBraid, the "Import Excel Cell Instances" SPARQLMotion module can pull triples from the spreadsheet with information such as the fact that a given cell has a row value of 7 (using zero-based counting), a column value of 0, a type value of "xsd:string", and "giant panda" as its contents. This level of detail can be useful for picking apart complex spreadsheets, but for simpler ones, if you instead use an "Import RDF from Workspace" module (in other words, if you have the script open the spreadsheet as if it were an RDF file), TopBraid uses the headings of the spreadsheet to identify more of the semantics of the data. For example, it would create triples saying that the thing identified as Row-6 has a commonName value of "giant panda" and a genus value of "Ailuropoda". This will be easier to convert to SKOS with a CONSTRUCT query.

There are five basic tasks that the conversion module must perform, all through the creation of triples:

Declare that the dataset being created is an ontology.
Import the standard W3C SKOS ontology so that we can reference its classes and properties.
Declare a concept scheme. A SKOS vocabulary can have as many concept schemes as you like, but we'll just create one for our example.
Declare concepts for each species, genus, and family found in the input triples, with a skos:broader property pointing from each one to either the appropriate broader concept or, if there is none, to the concept scheme created in the previous step.
Create triples that attach any additional metadata to the appropriate concepts—in this case, to assign the common name value to each species concept. SKOS is very flexible, so if you had additional non-SKOS properties specific to your own applications that you wanted to assign to each concept, the steps would be similar to the ones for attaching the common name values from this spreadsheet to each concept.

Whe the "Import RDF from Workspace" module reads in a spreadsheet such as caniformia.xls, it uses the spreadsheet's filename to define a prefix for the spreadsheet's properties so that it can refer to those properties with names like caniformia:genus. After opening the spreadsheet directly in TopBraid Composer, I saw that the base URI created for the data and associated with the caniformia: prefix was file:///xls2skos/caniformia.xls, because I had it in a project named xls2skos. I wanted to use this prefix in my SPARQLMotion script's CONSTRUCT query, so I associated this URI with the caniformia: prefix in the Overview tab of the xls2skos.n3 file that stored the SPARQLMotion script.

The actual conversion takes place in the Apply Construct module that I named "convert XLSData". These modules can store multiple CONSTRUCT queries, so I used two. The first does the basic setup of the taxonomy being created, which are the first three of the five tasks listed above:

CONSTRUCT {
    ?baseURI a owl:Ontology .
    ?baseURI owl:imports <http://www.w3.org/2004/02/skos/core> .
    <http://example.com/taxonomies/animals/caniformia> a skos:ConceptScheme .
    <http://example.com/taxonomies/animals/caniformia> rdfs:label "Caniformia" .
}
WHERE {
    LET (?baseURI := smf:buildURI("<{?baseURIStr}>")) .
}

The second query performs steps 4 and 5:

CONSTRUCT {
    ?speciesURI a skos:Concept .
    ?genusURI a skos:Concept .
    ?familyURI a skos:Concept .
    ?speciesURI skos:prefLabel ?speciesName .
    ?speciesURI skos:altLabel ?commonName .
    ?speciesURI skos:broader ?genusURI .
    ?genusURI skos:broader ?familyURI .
    <http://example.com/taxonomies/animals/caniformia> skos:hasTopConcept ?familyURI .
}
WHERE {
    ?row caniformia:commonName ?commonName .
    ?row caniformia:species ?speciesName .
    LET (?species := smf:encodeURL(?speciesName)) .
    LET (?speciesURI := smf:buildURI("<{?baseURIStr}#{?species}>")) .
    ?row caniformia:genus ?genusName .
    LET (?genus := smf:encodeURL(?genusName)) .
    LET (?genusURI := smf:buildURI("<{?baseURIStr}#{?genus}>")) .
    ?row caniformia:family ?familyName .
    LET (?family := smf:encodeURL(?familyName)) .
    LET (?familyURI := smf:buildURI("<{?baseURIStr}#{?family}>")) .
}

In addition to creating a SKOS concept for each species, it creates one for each genus and family as well, using the skos:broader property to identify the connections between these concepts that make up the hierarchical taxonomy of terms.

One big decision to make with this query was how to create URIs that provided unique identifiers for each new concept being created. I knew that the species, family, and genus names must be unique, so I added those to the base URI after passing them to smf:encodeURL(), a SPARQLMotion extension function that escapes any characters that won't work well in a URI. If you have taxonomy data in a spreadsheet, there may already be a unique number or other form of ID assigned to some or all taxonomy terms that your conversion can grab so that you don't have to create URIs from the names on the spreadsheet like I did.

I also decided to use species names like "Canis lupus" as the skos:preferredLabel value in the output and to use labels from the spreadsheet's "common name" column like "gray wolf" as skos:altLabel values. If you wanted to to use common name values as preferred labels and species names as alternative labels, it would be a simple change to the query above.

My "convert XLSData" module also has its sml:replace value set to True so that it doesn't pass along the input triples to the final module, which saves the conversion result. This last module, which I named "Save as TDB", is an "Export to TDB" SPARQLMotion module that saves the conversion results using the Jena TDB format. I could have used a SPARQLMotion "Export to RDF" module, which saves triples as a Turtle or RDF/XML disk file, but I wanted to use the results of my conversion in EVN. EVN requires that you use Jena's TDB or SDB formats so that it can attach metadata to your work to support reporting and workflow tracking.

After running the conversion, here is one view of it in EVN:

This is a minimal example using a small spreadsheet with no non-SKOS metadata. If your spreadsheet includes columns for data that don't fit easily into the SKOS model, you can use TopBraid Composer to create a customized version of SKOS that includes your own properties, and your SPARQLMotion script's conversion module can then add triples for those properties to the result. Viewing them in EVN, they would appear under Custom Properties on the right. And of course, the screen shot above only hints at all that EVN lets you do with your controlled vocabulary once you convert it to SKOS.

Getting started with SPARQL Web Pages

2010-11-10T06:59:00.000-08:00

In earlier entries on this blog, we've seen how SPARQL Rules attached to classes can let you identify constraint violations in instances of those classes and implement other kinds of business logic, all using the SPARQL standard. With release 3.4, which is now in beta, TopQuadrant products have another new application of SPARQL that lets you attach useful metadata to class definitions: descriptions of how you want the class's instances to look in a browser. We call this SPARQL Web Pages.

TopQuadrant vice president of product development Holger Knublauch has written several blog entries introducing SPARQL Web Pages under its original name, UISPIN, such as UISPIN: Creating HTML and SVG Documents with SPARQL, Charts and Business Reports with UISPIN, and UISPIN Example: Documenting SPIN Functions. In this blog entry, we'll see how to get started with a simple but useful example.

Address book information is not as simple to represent in RDF as one might think. A street address, city name, and postal code should be shown in a specific order, but RDF facilities for ordering the property values for a given instance can add annoying layers of complexity to a data model. Without ordering, though a simple address book entry can be difficult to read, like the following fake address shown in Turtle format:

a:Entry_1
      rdf:type a:Entry ;
      a:city  "San Diego" ;
      a:email "jerry122@hotmail.com" ;
      a:firstName "Jerry" ;
      a:homePhone "(738) 610-2019" ;
      a:lastName "Snyder" ;
      a:mobile "(702) 382-4712" ;
      a:postalCode "39248" ;
      a:region "CA" ;
      a:streetAddress "3137 11th Ave." .

Using TopBraid Composer Maestro Edition, I created an RDF file with an Entry class for address book entries. I declared the properties shown above, with the Entry class as their domain, and then I added a few fake address book instances to the file. The file has a base URI of http://example.org/addressBook, which will be important later.

Next, I imported the html.rdf model from the UISPIN folder of the TopBraid project that is automatically added to every workspace. (In a production application, especially if I was working with a standard ontology instead of a hand-crafted little model defining an address book entry, I'd create a new file that imported the standard class and property definitions as well as html.rdf instead of adding the SPARQL Web page definitions directly into the file that defines the address book model.) The UISPIN folder also includes models to generate SVG, charts, and more; the html.rdf file lets you add HTML-generating SPARQL code to your classes. Below, you can see how I've added the ui:instanceView property from this file's model to the definition of my Entry class:

When I scroll the Class View down a little, you can see what I entered as the ui:instanceView property value. It's an HTML template, with instructions for plugging in instance data, that will be output whenever TopBraid sees an instance of this class:

It lays out an div element that begins with the address entry's mailing address and then has a small table showing the entry's email address and phone numbers, with these property names bolded in the output. You can take advantage of the full power of SPARQL in these templates, as you'll see in Holger's blog entries, but I kept things simple by mostly just using the spl:object() function to insert specific property values into various places in the HTML.

When I view the form tab of the Entry class and click on a row in the Instances view, I see a Resource Form for that instance, like I always did:

You can see a new Browser tab next to the Form tab, though, and it lets me see the instance view formatted according to the HTML template that I created in the ui:instanceView value:

Even better, TopBraid Live can serve up the data using these HTML templates outside of TopBraid Composer, so that sending a browser to the URLhttp://localhost:8083/tbl/uispin?_resource=http://example.org/addressBook%23Entry_1&_baseURI=http://example.org/addressBook displays the result in the browser (note the use of the base URI to specify the model with the data to display and the escaping of "#" as "%23"):

Of course, this HTML can also have links and reference CSS, Javascript, web services, and other applications on the TBL server where it's hosted, so you can build some very sophisticated user interfaces. Check out Holger's blog entries for further ideas on where you can take this, especially when you start incorporating SPARQL queries and their results into the templates.

TopQuadrant's new Enterprise Vocabulary Manager

2010-10-27T08:44:00.000-07:00

The TopBraid platform can be used to build all kinds of applications and solutions. We've recently noticed one particular area where more and more customers needed help, and where semantic technology and our tools were a great fit: the management of multiple connected vocabularies spread out across an enterprise. To meet this need, we've created TopBraid Enterprise Vocabulary Net (EVN), a solution that works out of the box while having all the power of TopBraid Suite behind its customization capabilities.

The EVN product page has a long list of its features, which provide everything you need to manage taxonomies and thesaurii (and even create simple ontologies) in multi-user environments. The ability to review proposed changes before rolling them into production, with a choice of reports and other options for analyzing those changes and their potential impact, will be especially useful in larger organizations.

The use of EVN requires no knowledge of SKOS, RDF, or the related W3C standards, but the use of these standards behind EVN's graphical user interface is what makes EVN both flexible and scalable. The use of public standards for data, models, and application logic makes it much easier to integrate EVN with other systems than any other vocabulary management solutions we've seen in the marketplace. They also make it easier for EVN to let you set up an environment where different vocabularies in different parts of a large organization can work cooperatively with no need to merge those vocabularies into a single large, central vocabulary.

EVN is included in TopBraid Composer Maestro Edition release 3.4, which is now in beta, so you can try it without purchasing a separate product. For a quick overview of the features and what the product looks like, start with thescreenshot tour, or jump right in to the tutorial included with EVN's documentation.

How to: read RSS and RDFa from the web with a SPARQLMotion script

2010-10-04T06:54:00.000-07:00

How do you get a SPARQLMotion script to read an RSS or Atom feed as RDF triples? How do you get a SPARQLMotion script to read triples that have been embedded into web pages using RDFa? The answer to both questions is the same: use the specialized SPARQLMotion module for the task. All you have to do is specify the URL of the file with the information you want to read.

To demonstrate both, we'll put together a short script that:

Reads the RSS feed about technology news from Newsweek magazine
Pulls the triples from the RDFa embedded in the Newsweek articles described in the feed
Saves the extracted triples in a Turtle file

Along with Dublin Core properties such as dc:title and dc:description, RDFa attributes in Newsweek articles store additional RDF metadata using the Open Graph vocabulary developed by Facebook. This makes it easier for Facebook to incorporate additional information about news articles in their applications—for example, if people click the Facebook button next to a Newsweek article in order to share it with their Facebook friends.

It also makes it easier for you to use information about these articles in your own applications. The sample application below just saves the retrieved triples in a file, but you could also pass them to other SPARQLMotion modules that could have OpenCalais analyze the text, combine the triples with data from another source, create a new, specialized RSS feed or SPARQL endpoint, or send an email message based on the results of your processing. Retrieving the data is just the beginning.

To create this application, start by creating a new SPARQLMotion file called getnewsweektech. (For more detailed background on the steps involved in creating and running a SPARQLMotion script, see the PDF tutorial TopBraid Application Development Quickstart Guide.)

Create a new SPARQLMotion script in your getnewsweektech.n3 file. For its first module, select sml:ImportNewsFeed from thesml:ImportFromRemoteModules category and name it GetNewsweekTechNewsFeed. To configure it, you only need to set its sml:url value tohttp://feeds.newsweek.com/newsweek/technology?format=xml, a URL I learned about from Newsweek's web page about their RSS feeds.

Once this module pulls down the RSS data and TopBraid converts it to triples, your script will look through these triples for web page URLs provided as RSS link values and then retrieve the triples that are stored as RDFa in those web pages. The script can't pull the triples from all those web pages at once, so we'll use an IterateOverSelect module to drive the next step. We'll specify a SPARQL SELECT query in the IterateOverSelect module to find the RSS link values, and then for each result that this SELECT query finds, another module will retrieve the triples from the web page named by the link value.

Drag an Iterate over select module from the Control Flow section of the SPARQLMotion palette and name it GetArticleLinks. Paste the following query in as the value for its sml:selectQuery property:

PREFIX rss: <http://purl.org/rss/1.0/> 
SELECT ?articleURLString
WHERE {
  ?s a rss:item .
  ?s rss:link ?articleURL .
  LET (?articleURLString := xsd:string(?articleURL)) .
}

The module that retrieves the RDFa needs a string version of the URL to specify where it should look for the RDFa, so the query above assigns a string version of each rss:item resource's rss:link value to the variable ?articleURLString. The script will execute the body of the IterateOverSelect (a separate module that we haven't created yet) once for each value bound to this variable. You're done configuring this module.

Next, we'll create the body of the IterateOverSelect. This can be a series of modules, but for this application we'll only need one. Drag anImport RDFa module from the Import from Remote section of the SPARQLMotion palette and name it ReadArticleRDFa. When configuring this new module, click the white triangle for its sml:url property's context menu and select Add SPARQL expression. This lets you add any combination of SPARQL keywords, symbols, function calls, and operators that returns a single value; for this, all you need here is the variable reference ?articleURLString. Each time this module retrieves triples from the RDFa in the web page at this URL, it will pass along the triples that it found to the next module. If this module has an sml:needsTidy property, set it to True to make it easier to read RSS that isn't well-formed XML.

For our script's last module, drag an Export to RDF File module from the palette's Export to Local section and call it SaveArticleTriples. Set its sml:targetFilePath value to newsweekTech.n3; it will write this file to the directory that holds the SPARQLMotion file with your script. Set the module's sml:baseURI to http://example.com/newsweek/tech/metadata or to any URI that you like.

All that's left is to connect up the four modules as shown below. When you add a connector out of your Get Article Links Iterate Over Selectmodule, TopBraid Composer will ask you whether your new connector is pointing at the body of the loop (the part to execute for each binding of the selected variable) or at the module that should take control of the script when the iteration is finished. Connect Get Article Links to the Read Article RDFa module with an sm:body link, because that's the part we want executed for each iteration, and connect Get Article Links to Save Article Triples with an sm:next link to transfer control (and the collected triples) there when the iteration is all done.

Select the Save Article Triples module and click the green triangle at the top of the workspace to execute the script up to that final module, and you should end up with a newsweekTech.n3 file in the same directory as your getnewsweektech.n3 file that holds the script. This new file will hold triples extracted from the various web pages named in the Newsweek tech news feed.

To branch out, you could substitute the names of other Newsweek feeds, or additional ones, and then collect all the triples together. You could drive the whole thing with a TopBraid Ensemble interface where an end user picks the category of Newsweek news (for example, their technology, politics, business, or entertainment categories) whose metadata should be retrieved. You could also find other publications that store RDFa metadata in their articles, or other websites, such as TopQuadrant's. And, as I mentioned earlier, you could combine this with other features of SPARQLMotion and TopBraid to make a very powerful application.

How to: Find SKOS constraint violations in AGROVOC with SPARQL Rules

2010-08-16T07:24:00.000-07:00

The Simple Knowledge Organization System (SKOS) vocabulary management specification is gaining popularity because, as a standard, it makes it easier to share taxonomies and thesaurii between different systems. It also guards investments in vocabulary development against the potential problems of dependence on a proprietary vendor format.

The W3C makes an OWL ontology for SKOS available, which makes it easier to ensure that your vocabulary conforms to the standard. As a comment near the beginning of it tells us, though,

A number of semantic conditions are *not* expressed formally in this schema. These are:

S12
S13
S14
S27
S36
S46

For the conditions listed above, rdfs:comments are used to indicate the conditions.

The comment for S13 says that "skos:prefLabel, skos:altLabel and skos:hiddenLabel are pairwise disjoint properties". In plain English, this means that for a given concept, you can't use the same term for any two of these properties. For example, you shouldn't say that "dog" is both the preferred label and the alternate label for a given concept—it should be one or the other.

If these constraints are in the ontology as comments and not as something that can be implemented by executable code, how do you find violations of these constraints? The simplest way I've found is to use SPARQL Constraints (with SPIN). We've built on Paul Hermans' work to implement these constraints in the ontology at http://topbraid.org/spin/skosspin. It imports the W3C SKOS ontology and adds one rule to the skos:OrderedCollection class for constraint S36 and rules for the other five constraints to the skos:Concept class. For example, it adds the following three rules for constraint S13:

# Constraint S13: skos:prefLabel, skos:altLabel and skos:hiddenLabel 
# are pairwise disjoint properties.
ASK WHERE {
    ?this skos:prefLabel ?label .
    ?this skos:altLabel ?label .
}

ASK WHERE {
    ?this skos:prefLabel ?label .
    ?this skos:hiddenLabel ?label .
}

ASK WHERE {
    ?this skos:hiddenLabel ?label .
    ?this skos:altLabel ?label .
}

If any of these returns a boolean true, then we know that constraint S13 has been violated. (If you look at the skosspin ontology, you'll see these queries represented as triples, which is more difficult to read but easier to implement than rules expressed as SPARQL queries as shown above. TopBraid Composer can convert between the two formats, and so can a SPARQL Text to SPIN RDF Syntax Converter that Holger Knublauch has made available on the web.)

I tested this with the Food and Agriculture Organization of the United Nation's popular AGROVOC thesaurus, a vocabulary "designed to cover the terminology of all subject fields in agriculture, forestry, fisheries, food and related domains", and found over 1600 violations of constraint S13. Because this thesaurus has almost 29,000 concepts and preferred and alternate labels in multiple languages for most concepts, it's easier to violate these constraints than you might think, and I never would have found them without the ability to automate this search. For example, concept http://www.fao.org/aos/agrovoc#c_1135 has an English preferred label of "Buds", 14 preferred labels for other languages such as Farsi and Thai, and 24 alternate labels. Among these, the Slovak skos:prefLabel value and the Sloval skos:altLabel value are both "púèiky", so this concept violates constraint S13.

How do we find the violations? I tried it with the free edition of TopBraid Composer, because it has everything you need to define and use SPARQL Rules. (TopBraid Composer Maestro Edition's ability to use these rules from within applications has made it possible for me to add several nice features to applications for some of our clients.)

The screenshot below of TopBraid Composer's Navigator and Imports views shows that I created a checkAGROVOC project and added a checkAGROVOC.n3 ontology file to it. This ontology only does two things: it imports the ag_skos_080422.rdf file that I downloaded from the fao.org web site and it imports the skosspin ontology described above for its SPARQL Rules. (It imports a web version of the skosspin ontology and, because of its 62 meg size, the local copy of the actual AGROVOC thesaurus.)

With these two files imported, I opened TopBraid Composer's Problems view and clicked that view's "Refresh all problems of current TopBraid file" icon . A "Progress Information" message box told me that TopBraid Composer was "Checking SPIN constraints on skos:Concept", which took a few minutes because there were plenty to check.

After it finished checking, the Problems view said that there were Warnings and had a plus sign that I expanded to see the first few constraint violations:

(You may want to play with the column widths a bit, because the Location column is the one you really want to see.) Double-clicking anywhere on a specific warning line shows the details about that concept on the Resource Form, like this:

If you don't see the little yellow warning symbols on the Resource Form that show where the problems are, click the little "Display constraint violation warnings" icon at the top of TopBraid Composer . In this case, a bit of scrolling down when viewing concept c_1002 shows that the same term appears as both Hungarian skos:altLabel and skos:prefLabel property values for this concept.

The Problems view above only shows the first 100 of the 2,972 warnings. Clicking the context menu white triangle in the view's upper-right lets you configure Preferences for the view so that you can reset the number of items to be displayed; I had no problem with a figure of 3,000.

Now that we know what constraint has been violated, there are other ways to list the concepts that need to be corrected. For example, the following query in TopBraid Composer's SPARQL view lists the identifiers that have the same label for these two properties:

SELECT ?s ?label
WHERE {
  ?s skos:prefLabel ?label . 
  ?s skos:altLabel ?label . 
}

Once you execute this query you can export its results to a file and then use that as a reference point to address the issues in the vocabulary.

We could have started off by executing this query on the AGROVOC SKOS file, but remember, at the time we didn't know which constraints had been violated. Using SPARQL Rules as extra metadata for class definitions helps to automate the identification of quality issues with the data, letting us use other techniques to focus on the specific problems and how to fix them.

Are your SKOS vocabularies violating any of the six extra constraints described in the SKOS specification? As I mentioned, this all works with the free version of TopBraid Composer, which is available for Windows, Mac, and Linux, so you can try it yourself to find out. With the TopBraid Composer Maestro edition, you can build applications for end users who can then maintain these vocabularies with a web-based interface instead of using TopBraid Composer. The user interface for notification of constraint violations then becomes one of the many things you can customize to the needs of your end users. You can also define new constraints around your own shop's business rules—for example, to require that all labels begin with an upper-case letter—and you can set TopBraid Composer or your application to highlight these violations as soon as they occur, instead of checking in batch mode like I did above.

To summarize, what we've seen here is really just a starting point, and there are all kinds of places where you can take it to improve the consistency and value of your vocabularies.

How to: use the SPARQLMotion debugger

2010-07-27T06:38:00.000-07:00

Since release 3.3, TopBraid Composer has included an interactive debugger for SPARQLMotion scripts that can make your development go much faster. TopQuadrant VP of Product Development Holger Knublauch wrote a nice overview of the debugger's features in his blog; below is a short hands-on tutorial in the use of the debugger.

We're going to put together a short SPARQLMotion script with a problem that prevents it from running properly. Experienced SPARQLMotion developers may notice the problem when we add it, but leave it in there—we'll see how the SPARQLMotion debugger helps us locate it.

Creating our script

Our script will prompt the user for a string to search for and then list the first and last names of everyone in the sample kennedy data file included with TopBraid Composer who has that string as part of their first name.

First, create a new SPARQLMotion file and give it a base URI of http://www.topquadrant.com/debugdemo and a file name of debugdemo. (For a more detailed description of how to create a SPARQLMotion file, see the "Creating and running a SPARQLMotion script" chapter of the TopBraid Application Development Quickstart Guide.)

Once your new file is open, create the script by selecting Create SPARQLMotion Script from the TopBraid Composer Scripts menu, and for its initial module type select sml:EnterLiteral. This is the module that will ask the user to enter a query string, and you'll find it under sml:ImportModules -> sml:ImportFromVariousModules. For the name of your new module instance, enter GetQueryString. Click the OK button and you'll see your script on the SPARQLMotion workspace with its one module.

Double-click the GetQueryString module icon and enter queryString as the sm:outputVariable value and Enter query string as the sml:text value. When the script runs, this module will display a message box that prompts the user with this message. After the user enters a value into the field on that message box, the value will be stored in the variable queryString for use by later modules in the script. You are now done configuring this module.

There are two more modules to add. Drag an Import RDF From Workspace module from the Import from Local section of the SPARQLMotion palette onto your workspace and name it GetKennedysData. Double-click it to configure it, and set the sml:sourceFilePath value to /TopBraid/Examples/kennedys.rdf. For the third and final module, drag an Apply Construct module from the RDF Processing section of the Palette and name it FindMatchingNames. Set its sml:replace value to true so that the module passes along only the triples that it creates. Set its sml:constructQuery property to the following query:

PREFIX k: <http://topbraid.org/examples/kennedys#>
CONSTRUCT {   
 ?s k:firstName ?first .   
 ?s k:lastName ?last .
} WHERE {   
 ?s k:firstName ?first .   
 ?s k:lastName ?last .   
 FILTER regex(?first, ?searchString, "i") .
}

(Instead of setting the prefix for the kennedys data at the beginning of this query, you could also do it on the script file's Ontology Overview screen.) This query passes along the firstName and lastName triples for anyone in the data file who has the value of the searchString as part of their firstName value. (The "i" provided as the third parameter to the regex() function tells it to do a case-insensitive comparison.)

Connect up the three modules so that the script look like this:

To test the script, select the FindMatchingNames module and click the green arrow at the top of your workspace to run all the modules up to the selected one. When the GetQueryString module displays a message box asking you for a query string value, enter Carol.When the script finishes running and you see the SPARQLMotion Script Executed message box, make sure that the Display result triples checkbox is checked before continuing so that you can see what data the script found.

Although the kennedys data includes a Caroline and a Carolyn, you won't see any triples in the SPARQLMotion Results view when the script is finished running. Let's use the debugger to find out why.

Using the debugger

The small blue circle icon at the top of your SPARQLMotion workspace toggles whether the selected icon is a debug breakpoint. On the SPARQLMotion workspace, make sure that your FindMatchingNames module is selected and then click that small blue circle. This will add a blue circle to the module icon to indicate that it is now a breakpoint. You can set as many modules you like as breakpoints, but for this exercise we'll just set this one.

Click the green arrow to run the script again, enter "Carol" as the query string. When the script reaches the module with the breakpoint, it displays the debugger window:

The left part of the window has the Execution Plan, which lists the modules in the order that they will be executed, with a check mark next to each module that's been set as a break point. As you debug, you're free to check and uncheck any of these. The bottom of the window displays the arguments to the module and their values, and the top shows three tabs: Variables, Query, and Input Graph. Let's start with the last tab: click Input Graph.

On the tab's Input Graph panel, click the plus sign, and you'll see the that kennedys data is definitely being provided as input to the FindMatchingNames module. Click that line, as shown below, and you'll see some of its data appear in the lower panel. Scrolling down there shows that CarolineKennedy and CarolynBessette are in the data being passed along, so we can't blame our script's retrieval of data for its inability to show the expected results.

Now select the debugger window's Query tab. This lets you execute test queries on the data passed to the module. Because FindMatchingNames is an Apply Construct module, the default query shown on the Query tab is a SELECT version of the CONSTRUCT query that you entered when you created this module.

You can enter any SELECT query you want in in this panel of the debugger window and run it without affecting the state of the running script. Commenting and uncommenting lines of this query and then re-running it is a particularly valuable technique for exploring what information is available to your SPARQLMotion script at this point in its execution and what your application logic has done with that data.

Perhaps the problem is in the query's FILTER expression. Add a pound sign at the beginning of that line to comment it out, and click the green triangle in the upper-right to run the SELECT query shown in the debugger window. You'll see plenty of s, first, and last values appear in the lower panel, including Caroline Kennedy and Carolyn Bessette, so it looks like a problem in the FILTER expression prevented the query from passing along the requested data. The ?first variable is clearly being set properly, because we saw plenty of output when we commented out this line, so maybe the problem is with the ?searchString variable referenced in the FILTER line's regex() function call.

Click the debugger window's Variables tab to display it, and you'll see the problem: the GetQueryString module stored the entered value in a variable called queryString, and the query's FILTER expression was checking values against a non-existent searchString variable. (In a real debugging session, the Variables tab is probably the first one you'd check, which is why it displays first.) Go back to the Query tab, uncomment the FILTER line, change ?searchString to ?queryString, and click the green arrow. You should now see Caroline Kennedy and Carolyn Bessette and no one else show up under the query in lower part of the debugger window.

At this point, you've only fixed the debugger window's temporary query used to explore the workings of the script, and the application itself still needs to be fixed. Click theContinue button to resume execution of the script past the breakpoint. If there were more modules and any of them were set as break points, the Continue button would stop at each one and display the debugger window, but your script has no more breakpoints. (If there were more modules after FindMatchingNames module, the Step Into button would execute them one at a time so that you could review the three debugger tabs for information about those modules as they executed.) Once the script completes, change ?searchString to ?queryString in the FindMatchingNames module's query, click the debugger breakpoint icon while the FindMatchingNames icon is selected to turn off its breakpoint indicator, and run the query again with the same "Carol" input string. It should run with the expected results appearing in theSPARQLMotion Results view.

Debugging your own applications

Although we ran this SPARQLMotion script from within the SPARQLMotion editor, you can still set breakpoints and check all the same information about a SPARQLMotion script that is invoked from somewhere else—for example, from a TopBraid Ensemble Application or a from web service. (This assumes that the script is running under the TopBraid Live Personal Edition Server included with the TopBraid Composer Maestro Edition, which you use to develop these scripts.) This makes the debugger invaluable for just about all kinds of TopBraid development, and you'll find more uses for it as you use it more. Again, review Holger's blog posting for additional ideas.

How to: Publish your Linked Data with TopBraid Live SPARQL Endpoints

2010-05-07T08:36:00.000-07:00

SPARQL endpoints are an increasingly popular way to expose linked data. Invoking SPARQL Endpoints from TopBraid Composer's SPARQL view was the subject of a previous TQ blog on SPARQL Endpoints.In this entry we will discuss how TopBraid Live can be used to implement a SPARQL Endpoint using TopBraid Live. SPARQL Endpoints are Web services that conform to the SPARQL protocol. SPARQL queries are passed to a URL where a SPARQL service processes the query and returns results in a defined XML format. A number of SPARQL Endpoints exist for Web data (see the W3C list of current SPARQL Endpoints) and have become important sources for linked data.

A SPARQL Endpoint service implementation is packaged with TopBraid Live and is available out-of-the box for both TopBraid Live Personal Server (TopBraid Composer-ME running on localhost:8083), and TopBraid Live Enterprise Server (for more information, see TBL Home page). Creating a SPARQL Endpoint for your data is therefore an easy three-step process:

Load the model you wish to query into your TBL/TBC-ME workspace.
Use the GRAPH SPARQL keyword to access any named graph in the workspace.
Send a SPARQL query in the query string of a url that access the TBL SPARQL endpoint.

For example, if you have TBC-ME running, the TopBraid Live Personal Server is automatically available. Open a browser window and enter the following URL:

http://localhost:8083/tbl/sparql?query=SELECT DISTINCT ?p WHERE {GRAPH <http://topbraid.org/countries> {?s ?p ?o} }

This URL passes a query string that is applied to the specified graph, the countries.owl example included in the TopBraid library. The query is passed to TopBraid Live and executed using TBL's SPARQL engine. The results are converted to the SPARQL Endpoint format and returned via HTTP. The above URL specifies the TBL Personal Server (via TBC-ME's localhost:8083) as the endpoint. If you have TopBraid Live Enterprise Edition running on a server, just substitute the server address for your Enterprise server.

To further explore the ease of creating SPARQL Endpoints with TopBraid Live,
click here to access a page that defines an HTML form that submits a query to the TBL Personal Server SPARQL Endpoint. Copy and paste the following queries that use some of the example models included in the TopBraid library.

This query finds all countries and their abbreviations from the countries model in TopBraid/Examples:

# Get all countries and abbreviations from countries model
PREFIX countries: <http://topbraid.org/countries#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
SELECT ?name ?abbrev
WHERE
{ GRAPH <http://topbraid.org/countries>
  {  ?country a countries:Country .
     ?country rdfs:label ?name .
     ?country countries:abbreviation ?abbrev .
  }
}

This query finds all children of Joseph Kennedy from the kennedys model in TopBraid/Examples:

# Find Joe Kennedy's children in kennedys model
PREFIX k: <http://topbraid.org/examples/kennedys#>
SELECT ?cname
WHERE
{ GRAPH <http://topbraid.org/examples/kennedys>
  {  k:JosephKennedy k:child ?child .
     ?child k:name ?cname .
  }
}

Again, substitute your Live server address for "localhost:8083" in the action tag of the HTML file to apply queries to your Live server.

Using SPIN functions in SPARQL Endpoints

TopBraid SPARQLMotion Functions and user-defined SPIN functions registered in a Live workspace can also be used in SPARQL Endpoint queries. For example, the following query uses the TopBraid SPARQLMotion Function smf:if() to compute the age of all persons at death or their current age using the example kennedys model. Instead of returning variable bindings via SELECT, this query returns a RDF graph via CONSTRUCT. Since the graph is in RDF/XML format, the file returned by the endpoint can easily be imported into existing RDF/OWL models.

# infer age at death or age as of 2010
PREFIX k: <http://topbraid.org/examples/kennedys#>
PREFIX smf: <http://topbraid.org/sparqlmotionfunctions#>
CONSTRUCT {?person k:age ?age}
WHERE
{ GRAPH <http://topbraid.org/examples/kennedys>
  {  ?person k:birthYear ?byear .
     OPTIONAL {?person k:deathYear ?dyear}
     LET (?age := smf:if(bound(?dyear), ?dyear-?byear, 2010-?byear))
  }
}

Note that the age computation is hardcoded for 2010. A SPARQL query that returns the current year can be defined with a few statements. An example is shown in the kennedysSPIN model in the TopBraid Library, see TopBraid/Examples/kennedysSPIN.rdf in the Composer workspace. If you look at the SPIN function getCurrentYear (defined as a subclass of spin:Functions, which is a subclass of spin:Modules), it finds the current year as the first four characters returned in xsd:dateTime format returned from the function afn:now().

Instead of copying this code into the query, let's register this as a SPIN function so it can be called by any model in the workspace, including SPARQL Endpoints. Do the following:

Re-name the file kennedysSPIN.rdf to kennedysSPIN.spin.rdf. Adding the .spin extension registers all of the SPIN functions in this model with the workspace, allowing SPIN functions to be called without importing or opening the files.

From the TBC-ME menu, select Scripts > Refresh/Display SPARQLMotion functions... This will register the functions for the current session. When Live or Composer is started, the system will scan the files in the workspace for .spin files and register all functions. The extra step is needed here only if the file name was changed without stopping the Composer session. A Deploy (Export... Deploy in Composer) to a Live server will automatically refresh scripts.

Now try the same query with the following changes:

# infer age at death or age from current year
PREFIX k: <http://topbraid.org/examples/kennedys#>
PREFIX smf: <http://topbraid.org/sparqlmotionfunctions#>
PREFIX kspin: <http://topbraid.org/examples/kennedysSPIN#>
CONSTRUCT {?person k:age ?age}
WHERE
{ GRAPH <http://topbraid.org/examples/kennedys>
  {  ?person k:birthYear ?byear .
     OPTIONAL {?person k:deathYear ?dyear}
     LET (?age := smf:if(bound(?dyear), ?dyear-?byear, kspin:getCurrentYear()-?byear))
  }
}

Note the use of the user-defined SPIN function getCurrentYear(). This feature can be used to call any SPIN function including those that are defined by SPARQLMotion scripts. This raises the potential of using SPARQL endpoints for a wide range of processing capabilities, including importing models from outside of a Live workspace, processing triples before querying, applying queries to inference results, integrating models from different file types, and other kinds of SPARQL and RDFS/OWL processing. For example, a SPARQL Endpoint request could call a SPARQLMotion script that runs standard RDFS or OWL inferences before submitting the query, thus returning results from both inferred and asserted triples.

Advanced SPARQL Protocol: Federated SPARQL Queries

The SPARQL SERVICE keyword sends a query to remote service endpoint. Since TopBraid Live supports the SERVICE keyword, SPARQL endpoint queries to TopBraid Live can call other SPARQL Endpoints! Try the following query in the example query form.


PREFIX k: <http://topbraid.org/examples/kennedys#>
PREFIX smf: <http://topbraid.org/sparqlmotionfunctions#>
CONSTRUCT {?child k:birthDate ?birthdate}
WHERE
{ GRAPH <http://topbraid.org/examples/kennedys>
  {  k:RoseFitzgerald k:child ?child .
     ?child k:firstName ?fname .
     ?child k:lastName ?lname .
     ?child k:gender k:female .
     ?child k:spouse ?spouse .
     ?spouse k:lastName ?slname .
     LET (?dbpRsc := smf:buildURI("http://dbpedia.org/resource/{?fname}_{?lname}_{?slname}"))

     SERVICE <http://dbpedia.org/sparql>
     {  ?dbpRsc <http://dbpedia.org/ontology/Person/birthDate> ?birthdate .
     } .
  }
}

This query is applied to the kennedys example model to query for female children of Rose Fitzgerald and sends a query to the DBPedia SPARQL Endpoint to find their birth dates. The buildURI() function will generate a URI that is known in DBPedia, such as <http://dbpedia.org/resource/Eunice_Kennedy_Shriver>. The results from DBPedia bind the birth date to ?birthdate, which is returned in the TopBraid Live SPARQL endpoint response. As long as DBPedia is up and running, the result federates data from two SPARQL Endpoints, realizing the potential of linked data sources.

Conclusions

SPARQL endpoints are a complement to TopBraid Live's ability to create
RESTful Web services. While Web services are more flexible, allowing data to be returned in any text-based format, SPARQL endpoints can be used in a variety of applications expecting SPARQL result sets in an XML format. TopBraid Live significantly improves on existing SPARQL Endpoints with capabilities to federate queries and design functions and scripts that process data for external usage.

These examples demonstrate the power of TopBraid Live as an RDF back-end. Using a straightforward HTML form, one can access to full power of TopBraid Live and advanced SPARQL queries. These examples can be directly applied against the Personal Server version of TopBraid Live, packaged in TopBraid Composer-Maestro Edition (TBC-ME), which is freely available for a 30-day trial. TopBraid Live Enterprise Edition is deployed as a Tomcat servlet for Web-enabled access. For more information, see the TopBraid Live web page.

TopBraid release 3.3: three things that have already made my application development easier

2010-04-18T14:04:00.000-07:00

TopBraid Suite Release 3.3 has many features that let you add new things to the semantic web applications that you develop. As I've worked lately on some client application development, three new features in particular have made this work go more quickly and easily.

SPARQLMotion debugger

The SPARQLMotion debugger lets you set breakpoints and examine variables like any other debugger. It also lets you enter SPARQL queries against the data available to the SPARQLMotion module where you set the breakpoint, a feature that has made it much easier for me to understand why a certain module was doing what it was doing instead of what I thought it would be doing. A future entry in this blog will describe the SPARQLMotion debugger in more detail; for now you can get a nice tour from Holger Knublauch's introduction to it.

Base URI Management

Applications typically read and write files, but I sometimes forget that in semantic web application development, the real identifier for a resource such as a file is not its filename, but a URI. After all, you can have several files with the same name in different folders or directories, so a URI should provide a truly unique identifier. If you somehow have the same URI representing more than one file (for example, if you made an alternative copy of your application development folders and used both in the same workspace), this can lead to trouble.

Release 3.3's new "Base URI Management" screen for both the Enterprise Server and Personal Server editions of TopBraid Live lists the URIs of the files and other data resources being tracked in each project and puts a warning icon next to any URI being used for more than one resource. Once you straighten these out by assigning a new base URI to one of the files (or, more likely, by deleting an old file that you had forgotten was there) you'll have much better control over the use of those resources.

Application Event Dashboard

I recently needed to use TopBraid Ensemble's drag and drop capability for an application I was working on. I had never used it before, but instead of pesteringing co-workers to show me an example of drag and drop in action, I opened up the Default TBE application with the kennedys.owl data model, clicked the wrench icon to display the Application Configuration screen, and went to the new Event Wiring screen of the Application Configuration dialog box. After clicking its Display All button, in the top list I saw that for this application the Graph Editor and Query components had Drop listener events defined, and in the bottom list I saw that the Grid and Tree components had Dragging post events defined, so I knew I could drag from a Grid or Tree component to the Graph Editor and Query components in this TBE application. I tried dragging a few resources between these components, and it worked; then I could examine how this was configured and reproduce it in my own application.

It's only been a few weeks since release 3.3 has been out, so I'm sure I'll be growing to appreciate other new features more in the coming months.

What's new in TopBraid Suite 3.3

2010-04-06T16:30:00.000-07:00

The new 3.3 release of the TopBraid Suite includes significant improvements to TopBraid Composer, TopBraid Ensemble, and TopBraid Live. For more details on the new features, see the Release Notes and detailed change list.

TopBraid Composer

Release 3.2 let you could run a SPARQLMotion script up to a certain point and then examine the triples that it would have passed along, which was handy for debugging. Release 3.3 brings a real SPARQLMotion debugger, which lets you examine all kinds of useful information at the script breakpoint: variable bindings, the arguments passed to the current module, the result of test queries entered at the break point, and the engine's execution plan. This will make development of more sophisticated SPARQLMotion scripts much easier.
More automated handling of taxonomies and other controlled vocabularies stored using the SKOS standard.
Improved OWL 2 support.
The introduction of UISPIN, a SPARQL-based framework for describing user interfaces. Just as SPIN let you define business rules and application logic, UISPIN lets you describe an application's visual presentation, all with RDF and standard SPARQL underneath, and TopBraid 3.3 lets you see that definition in action. A recent blog posting by Holger Knublauch describes a sample application.
For customers with complex policies on the use of open source software, the new TBC Clear and TBC-ME Clear versions of TopBraid Composer come with open source components (outside of Eclipse and Jena) unbundled, letting you select the components to add back in according to your needs and policies.

TopBraid Ensemble

The new Application Event Dashboard makes it much easier to track which events are used by which form components.
Release 3.3 gives you greater control over form appearance. You can hide component tabs and resizing dividers, and new components make it easier to add white space and formatted text.
The form component's "Edit Mode" lets TopBraid Ensemble application developers offer their end users new options in how they edit data in forms.
When SPARQLMotion scripts triggered from TopBraid Ensemble generate HTML documents, they can be displayed in the browser automatically.
When someone is using an Ensemble application, the displayed URL can be used to record and reproduce the state of the application at that point.
A new Flex Developer's Guide shows how to make your own components to incorporate into your TopBraid Ensemble applications.
The new Basket component works like TopBraid Composer's Basket view, serving as a "scratch pad" for resources.

TopBraid Live

Release 3.3 of TopBraid Live gives server administrators greater control over running sessions and the relationship of file resources to the URIs used to identify with them. URI conflicts and missing imports are easier to identify and resolve, and multiuser change propagation and file indexes are also easier to manage from the Server Administration Console.

SPARQLMotion

The SPARQLMotion debugger mentioned above is built into TopBraid Composer. You can execute SPARQLMotion scripts to drive the execution of application logic on a TopBraid Live server from both TopBraid Composer and from TopBraid Ensemble, so improvements to SPARQLMotion capabilities benefit the entire TopBraid Suite. Release 3.3 makes several new things possible with SPARQLMotion scripts:

A spellchecker accepts blocks of text and returns triples that identify words not found in its configurable dictionary, along with suggested corrections.
For added flexibility in the use of SPARQLMotion modules, arguments passed to them can now be SPARQL expressions, SELECT queries, or SPIN templates.
New SPIN functions let you build unique URIs and compute AVG, MIN, MAX aggregations and more from the SPARQL queries in your scripts.
The cache-all option speeds the use of queries against disk-based databases.

Considering that it's a minor upgrade, there are a lot of new toys to play with!

How to: deploy your TopBraid applications on a TopBraid Live Enterprise server

2010-03-21T08:32:00.000-07:00

An updated version of this blog post is now available as a chapter in the TopBraid Application Development Quickstart Guide (pdf).

So far, as we've learned how to create a TopBraid Ensemble application, how to create and run a semantic web service, and how to call a SPARQLMotion script from TopBraid Ensemble, we've always used the TopBraid Live (TBL) Personal Server that's built into TopBraid Composer (TBC) to test our applications. However, the reason developers create these applications on TBC is for others to use, and that's where the TBL Enterprise Server (purchased separately from TBC) comes in.

When you deploy your TopBraid applications on a Enterprise TBL Server, many users can use your applications simultaneously, and none of them need a copy of TopBraid Composer. In this posting, we'll see how simple it is to copy your applications to a TBL Enterprise Server where you end users have access to them. (And remember, those "end users" aren't always people; when you deploy a semantic web service to a TBL Enterprise Server, it can be a first-class member of any number of Service-Oriented Architecture applications, serving as a building block for larger applications by fulfilling requests from automated processes elsewhere in the architecture.)

Referencing your server

In the following example, I'm going to deploy an application to a TBL Enterprise Server running on my local machine and configured as a Tomcat servlet listening to port 8080, making the URL of the server http://localhost:8080/tbl/tbl. If the Enterprise Server was installed on a host named www.domain.com and used the same port, the URL would be http://www.domain.com:8080/tbl/tbl, so you would follow the same steps, substituting that name.

Deploying apps to the server

If you package all the files needed to run an application in a single project, which is a root file in TBC's Navigator view, it's a convenient way to upload a collection of related files to the TBL Enterprise Server. So, for example, if the personScripts.sms.n3 file that you created in the last posting is not in its own project, create one called personProject and move it there.

Right-click on the project folder icon in TBC's Navigator view and pick Export. On the Export dialog box, select "Deploy Project to TopBraid Live Server" from the TopBraid Composer section of the menu and click the Next button.

On the Deploy Project to TopBraid Live Server dialog box that appears, enter the URL of the server where you want to send your app (for example, http://localhost:8080/tbl/tbl, as described above) in the Server URL field. The remaining fields are self-explanatory: enter a user name and password if the server was set up to require it (always a good idea for serious production applications) and check "Overwrite existing project with the same name" if there is any chance that a project with that name already exists.

Click the Finish button, and TBC will show you upload progress and then let you know it's finished:

If you sent a project with the personScripts.sms.n3 file created in the How to: call a SPARQLMotion script from TopBraid Ensemble posting to the Enterprise server, and a different computer can access that one as http://www.domain.com:8080/tbl/tbl/ or http://192.168.1.2:8080/tbl/tbl/, then sending a browser on the other computer to one of these URLs will display the TopBraid Suite Console screen. On that computer you can then follow all the steps described in that posting's "Testing It" section, even though you're doing it remotely, right up through the use of the "Age at Death" menu choice that you implemented.

You can upload a web service to the Enterprise TBL server the same way. For example, in How to: create and run a semantic web service we saw that after defining a web service in the file ws1.sms.n3 and registering the function that invokes the service by selecting "Refresh/Display SPARQLMotion functions" from the TBC Script menu, you could test this web service by entering the following URL in a browser:

http://localhost:8083/tbl/actions?action=sparqlmotion&id=searchKennedys&arg1=Rob

This sent the request to the TBL Personal Server included with TBC. To deploy the web service on the Enterprise Server, upload the project that holds the ws1.sms.n3 file file as described above. Then, if the server is running on the local machine and listening to port 8080, you can test it with the following URL:

http://localhost:8080/tbl/tbl/actions?action=sparqlmotion&id=searchKennedys&arg1=Rob

To call this web service from another computer on the same network (which could be another one on your intranet, or could be a computer on the public internet if the machine running the TBL Enterprise Server is accessible there), substitute the appropriate computer name or IP address for "localhost" in that URL.

Installing TopBraid Ensemble Apps on the Enterprise Server

As we've seen in an earlier posting, when you create a TopBraid Ensemble (TBE) application, you save your screen layout with the Save App button. After clicking this button, the Save To field on the Save Application As dialog box shows server.topbraidlive.org as the default project in which to save the application, but you can select another one. (You can't create a new project from within TBE, so if you want to save the application in a particular project, make sure you've already created that project you need in TBC's Navigator view.)

If you named your TBE application MyTBEApp and saved it in the project MyTBEProj, in TBC's Navigator view you'll see that it stored MyTBEApp.n3 in a subdirectory of MyTBEProj named user-applications. Once you upload the MyTBEProj project to the TBL Enterprise Server the same way you uploaded the other projects described above, you'll see MyTBEApp listed on the right side of TopBraid Live's TopBraid Suite Console screen at http://localhost:8080/tbl/tbl/ under the column heading "Applications Under Development":

When you click the application name on the TopBraidSuite Console, if it wasn't saved with a particular set of data, it will prompt you to pick one. Once you see the application screen you designed, your browser's navigation toolbar will show the URL that you would use to send users directly to that application once you substituted the computer name for "localhost" as described above. It will be a long URL, so instead of actually giving that URL to your users, you'd be more likely to use it as the href value of a link on a web page that takes them to the application.

TopBraid Composer puts a lot of capabilities at your fingertips. When you copy the web services, TBE applications, and any SPARQLMotion scripts called by those TBE applications from TBC to a TBL Enterprise Server, you can make the power of your semantic web applications available to a wide range of people and applications throughout your enterprise or even throughout the World Wide Web. That's a lot of power.

How to: call a SPARQLMotion script from TopBraid Ensemble

2010-03-05T15:18:00.000-08:00

An updated version of this blog post is now available as a chapter in the TopBraid Application Development Quickstart Guide (pdf).

In earlier entries, we've seen how to create a TopBraid Ensemble application and how to create and run a semantic web service. When you combine these, the user interface that you create with TopBraid Ensemble (TBE) can take advantage of all the power of SPARQLMotion scripts, which can accept parameters from a TBE app, run in the background, and pass the results back to be incorporated into the TBE user interface.

In this posting, we'll see how to add an "Age at death" choice to the TBE gear menu, which triggers SPARQLMotion scripts. When your end user selects a person in the kennedy data and picks this menu choice, a SPARQLMotion script will return either a message about the person's age at death (assuming that they've already had their birthday in the year of their death) or the phrase "Still alive" for display in TBE.

Defining the function to call

As we saw in the posting on creating web services, the first step is to define a SPIN function that points to a SPARQLMotion script, and then we create the script. (The following description assumes that you've read the more detailed description of the same steps in How to: create and run a semantic web service.) Start by creating a new SPARQLMotion file called personScripts, and remember to check "Script will declare (Web) Services or Functions (.sms extension)" on the Create SPARQLMotion File dialog box.

Next, in the Classes view, create a subclass of spin:Functions called AgeAtDeath. The rdfs:label value for this function is important, because that's the text that will appear on the TBE gear menu; drag this property from the Properties view onto the class form for your new function and enter "Age at death" there.

The next step is to use the function's spin:constraint property to identify the argument being passed to the function. For our new application, we want the argument to be the identifier for the currently selected member of the Kennedy family in the TBE application—or, in TBE development terms, the URI for the selected resource—so that the function knows whose age at death to calculate. To make the selected resource the argument, on the AgeAtDeath class form click the spin:constraint property's white triangle context menu and pick "Create from SPIN template". On the Create from SPIN template dialog box, pick sml:SelectedResourceArgument from the selection of "Available Ask/Construct Templates". Click the plus sign next to the predicate field and pick sml:selectedResource. You're now finished specifying the spin:constraint.

Remember, a SPIN function identifies the SPARQLMotion script to call by naming the script's last module in the function's sm:returnModule property. The module that we need to point to doesn't exist yet, so pick "Create and add..." from this property's context menu, and then pick sml:ReturnText from the sml:ExportToRemoteModules section of the sml:ExportModules choices and name your new module ReturnComputedAge. It's exporting to a remote module because a script on the server is returning the value to a calling process on the client, and when a TBE application gets text passed by a Return Text module, it displays that text in a message box, which is what we want for this application.

Creating the SPARQLMotion script

You don't need to create a new SPARQLMotion script, because when you defined the sml:selectedResource and sm:returnModule modules as part of the AgeAtDeath function, TopBraid Composer put them into a new script for you. Select "Edit SPARQLMotion Script" from the Scripts menu, and you'll see see this script with icons representing these two modules waiting for you to add new modules and connect them up.

Drag the Selected Resource icon to the top of the screen, because it will be the beginning of your script. Under it, create a new module to perform the query that will compute the age at death of the selected resource by dragging a Bind by select icon from the Control Flow section of the SPARQLMotion workspace palette onto the workspace. Name this module ComputeAge and set its sml:selectQuery property to the following SPARQL query:

PREFIX k: <http://topbraid.org/examples/kennedys#>
SELECT ?deathAge
WHERE {
  ?selectedResource k:birthYear ?birthYear .
  OPTIONAL {
      ?selectedResource k:deathYear ?deathYear .
  } .
  LET (?deathAge := smf:if(bound(?deathYear),
                           (?deathYear - ?birthYear),
                           "Still alive")) .
}

It uses the SPARQLMotion extension function sml:if to calculate and return the person's age at death if the ?deathYear variable is bound and to return the string "Still alive" if it's not bound.

The sml:selectQuery property is the only one to set for this module, so you're ready to add a connection from the Selected Resource module icon to the ComputeAge one.

The ComputeAge module knows what SPARQL query to execute, but it doesn't know what data to execute it with. Drag an Import current RDF module, which passes all of the triples of the currently open model to the script, from the Import from Various section of the palette to the workspace. Name it ImportTBETriples and add a connection from this module to the ComputeAge one. You don't need to set any of its properties; it will know what to do.

The query above binds the computed answer to the variable ?deathAge. This is what we want to return to the TBE application that calls the AgeAtDeath function, so set the sml:text property of the returnComputedAge module that's been waiting for you to use it to {?deathAge} and connect the ComputeAge module icon to the returnComputedAge one. Your completed script should look like this:

The gear menu in TBE won't know about the AgeAtDeath function until you register it with the server, so select "Refresh/Display SPARQLMotion functions..." from from the TopBraid Composer Script menu, and after a few seconds you'll see the updated list of registered functions on the Console view.

Testing it

From the TBL main screen, pick "Default Application (configurable)," and on the "TopBraid Live Data Selection" screen pick kennedys from the "Data from Project TopBraid" section. When the application appears, click Person in the Tree component in the upper-left to list data about members of the Person class on the Results Grid. Once they appear there, click a row that includes both a "Year of birth" and a "Year of death" value to select it.

Click the gear menu in the upper-right, and you should see your new "Age at death" menu choice:

Select it, and you'll see a message box showing the result of the calculation from the SPARQLMotion script:

Try again with a row of the Results Grid that has a "Year of birth" but no "Year of death" value, and you should see a similar message box appear with the "Still alive" message.

Taking it further

If you created ten different applications like this, the menu choices for all ten would appear on all gear menus in all of your TBE applications. To limit the menu choice to only appear on the gear menu for applications showing instances of a particular class, such as members of the Person class from the kennedys data, you can name that class in the ValueType property of the sml:SelectedResourceArgument when you configure the constraint in the function you're defining.

The SPARQL query used in your SPARQLMotion script set the return value of ?deathAge to a very simple value: either (?deathYear - ?birthYear) or the string "Still alive". Browse through the "TopBraid SPARQLMotion Functions Library" help panel and you'll see an extensive choice of functions that you can use to assemble a much more complex string to return, with substring and case manipulation functions, regular expression matching, concatenation functions to assemble multiple pieces together, and more.

This SPARQLMotion script also had very limited input: the URI of the resource to query about and the data from the kennedys file. As we've seen in other postings, a SPARQLMotion script can open up many other kinds of both local and remote data, so the URI passed from TBE can be used in much more sophisticated kinds of processing. And, as we saw in How to: create a TopBraid Ensemble application, you can custom-design your own TBE screens to be the front end of your application. It's a great combination of front end client-side design flexibility and back end server-side scripting capabilities.

How to: use XSLT in a semantic web application

2010-02-19T06:36:00.000-08:00

An updated version of this blog post is now available as a chapter in the TopBraid Application Development Quickstart Guide (pdf).

XML-based tools form the plumbing of modern electronic publishing systems, so when your semantic web application can create XML, it can easily feed content into one of these systems. The W3C standard XSLT is the most popular tool for converting one kind of XML into another, so in this posting we'll see how SPARQLMotion scripts can use XSLT to create customized XML from your triples. (TopBraid Suite is also very good at reading XML; see the TopBraid Composer help panel "Creating, Importing, Querying, Saving XML documents with Semantic XML" for more on this.) Our example script will save its output as an XHTML file, but you can create XML conforming to any DTD or schema you like and save it as a file or return it via a web service.

Creating XML

TopBraid offers several ways to create XML. The Semantic XML feature lets you read arbitrary XML and gives you fine-grained control over XML that you create, but for a quick conversion of data to XML, there are simpler options. The W3C's RDF/XML serialization standard is the most well-known (if not the most popular) format for representing triples in XML, but its flexibility in the representation of triples can add complexity to the XSLT stylesheet, which must account for several possible locations of a given triple's subject, predicate, and object in various combinations of XML elements and attributes.

The simplest, most straightforward XML format that a semantic web application can create is the W3C standard SPARQL Query Results XML Format. For example, with the following query against the kennedys.owl file included with TopBraid Composer,

PREFIX k: <http://topbraid.org/examples/kennedys#>
SELECT ?last ?first ?birth
WHERE {
?s k:lastName ?last .
?s k:firstName ?first .
?s k:birthYear ?birth .
}
ORDER BY (?birth)

the XML representation of the result will have the following structure, with a head element listing the variables followed by a results element that contains a result child for each row of results returned by the SPARQL engine:


<sparql xmlns="http://www.w3.org/2005/sparql-results#">
<head>
<variable name="last"/>
<variable name="first"/>
<variable name="birth"/>
</head>
<results>
<result>
  <binding name="last">
    <literal>Kennedy</literal>
  </binding>
  <binding name="first">
    <literal>Joseph</literal>
  </binding>
  <binding name="birth">
    <literal datatype="http://www.w3.org/2001/XMLSchema#integer">1888</literal>
  </binding>
</result>
<result>
  <binding name="last">
    <literal>Fitzgerald</literal>
  </binding>
  <binding name="first">
    <literal>Rose</literal>
  </binding>
  <binding name="birth">
    <literal datatype="http://www.w3.org/2001/XMLSchema#integer">1890</literal>
  </binding>
</result>
<!-- more result elements -->
</results>
</sparql>

As we'll see, a very brief XSLT stylesheet can handle this.

Creating an XML version of SPARQL query results in a SPARQLMotion script

In our sample application, the XSLT stylesheet will create an XHTML version of the data returned by the query above.

Start by creating a SPARQLMotion File in TopBraid Composer file called xsltdemo. Create a SPARQLMotion script in this file with an ImportRDFFromWorkspace initial module named GetKennedyData, and set its sml:sourceFilePath property to /TopBraid/Examples/kennedys.owl. (For more details on following these steps, see How to: create and run a SPARQLMotion script.)

For your script's second module, drag a SerializeSPARQLResults module from the SPARQLMotion palette's Text Processing section onto the workspace and name it SelectRptData. Set the following three properties to configure it:

Set its sml:selectQuery property to the SPARQL query shown above.
Set sm:outputVariable, a property that names the variable that will hold the results of the query, to queryResults.
For the sml:serialization property, click the white triangle to display the context menu and select Add Existing, because you want to pick from the predefined list of legal values. On the Add existing dialog box, select sm:XML on the right. (If you click on sml:RDFSerialization on the left of this dialog box, you'll see that this module can also output N3, NTriples, RDF/XML, and Turtle RDF.)

Close the Edit SelectRptData dialog box and connect the GetKennedyData module's icon to the SelectRptData one.

When you develop an XSLT stylesheet, you want some sample input XML data handy to test it, so let's have this script create a file for this. Add an Export to XML File icon from the Export to Local section of the palette and call it SaveTestXML. Set its sml:xml property to {?queryResults} to get the data that your SelectRptData module stored in this variable, and set sml:targetFilePath to testdata.xml. That's all you need to set, so connect your SelectRptData icon to it as shown here:

Select that third icon and click the debug icon at the top of the SPARQLMotion workspace to run it, and you should see a testdata.xml file appear in the same directory as your script. This file will have XML that follows the structure of the example shown above.

Applying an XSLT stylesheet in a SPARQLMotion script

First, you need to create the XSLT stylesheet file. If you're using an Eclipse-based XML editor such as oXygen or XML Spy, go ahead and use one of those editors. Otherwise, after selecting the folder in the Navigator view where you want to store the file, pick New from the File menu. Because you're not creating one of the specialized files that TopBraid Composer typically deals with, pick Other from the cascade menu. Pick XML from the XML section of the New dialog box and click next. Call the file createKennedyHTMLRpt.xsl, and click Finish on the New XML File dialog box. (Clicking Next on that dialog would lead to wizards that make the process more complicated than this example requires.) Paste the following stylesheet, which I wrote to process the testdata.xml file created above, onto the editor and save the stylesheet:


<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:sp="http://www.w3.org/2005/sparql-results#"
    xmlns="http://www.w3.org/1999/xhtml">

<xsl:template match="sp:sparql">
  <html>
    <head><title>XSLT demo</title></head>
    <body>
      <table border="1">
        <tr>
          <th>Last Name</th>
          <th>First Name</th>
          <th>Birth</th>
        </tr>
        <xsl:apply-templates/>
      </table>
    </body>
  </html>
</xsl:template>

<xsl:template match="sp:result">
  <tr>
    <td><xsl:value-of select="sp:binding[@name='last']"/></td>
    <td><xsl:value-of select="sp:binding[@name='first']"/></td>
    <td><xsl:value-of select="sp:binding[@name='birth']"/></td>
  </tr>
</xsl:template>

</xsl:stylesheet>

(If you create the stylesheet outside of TopBraid Composer, make sure to put it in the same folder as your script, then right-click the folder's icon in the Navigator view and select Refresh so that Eclipse knows that the new file is there.) The stylesheet converts the XML into an XHTML file with a table that has a row for each row of the result set.

The next module to add to the SPARQLMotions script is the one that applies the stylesheet to the XML. Drag a Convert XML by XSLT module from the XML Processing section of the palette, name it CreateHTML, and configure it by setting the following three properties:

Set sm:outputVariable to HTMLResult. This variable will store the result of the XSLT transformation.
Set sml:template to {?stylesheet}. We haven't added the script module that sets this variable yet, but will soon.
Set xml:xml to {?queryResults}. As we saw above, the SelectRptData module stores the XML of the query results in this variable; the SaveTestXML module also used the contents of this variable.

Connect the SelectRptData module to your new CreateHTMLRpt module. SelectRptData is already connected to SaveTestXML, but sending a module's output to multiple modules is a common technique in SPARQLMotion scripts.

To read the createKennedyHTMLRpt.xsl stylesheet disk file and send its contents to the CreateHTMLRpt module for use in the XSLT transformation, drag an ImportTextFile module from the Import from Local section of the palette and name it ReadXSLTStylesheet. Tell it where to find the stylesheet file by setting its sml:sourceFilePath property to createKennedyHTMLRpt.xsl, and then set its sm:outputVariable property to stylesheet—the variable that the CreateHTMLRpt module will expect to find the text of the stylesheet that it applies to the output of the SelectRptData module. Connect the ReadXSLTStylesheet module to the CreateHTMLRpt module.

The last module will save the results of the XSLT transformation in a disk file. Drag another ExportToXMLFile module from the Export to Local section of the palette and name it SaveHTMLFile. Set its sml:xml property to {?HTMLResult} so that it reads the variable set by the CreateHTML module, and then indicate where to save this variable's value by setting the module's sml:targetFilePath property to KennedyRpt.html. Connect the CreateHTML module to the new SaveHTMLFile module. Your completed script should look something like this:

To test it, select the CreateHTML icon and click the debug icon. You should see a KennedyRpt.html file appear in the same directory as your xsltdemo script; in a browser, this HTML file will look like this:

Now that the script is applying the XSLT stylesheet to the XML and creating the HTML, you can right-click your SaveTestXML script module and select Delete to remove it from your script if you want to, but maybe you don't want to—if you ever need to refine the SELECT query in the SelectRptData module or the XSLT stylesheet that the ReadXSLTStylesheet module feeds to the CreateHTML module, the ability to create another disk file of test data may prove handy in the future.

This also demonstrates another technique of sophisticated SPARQLMotion scripts: the ability to share common processing modules for different output modules. You essentially have two SPARQLMotion scripts here—SaveTestXML and SaveHTMLFile—which share several modules. That's why, when you pick Edit SPARQLMotion Script or Execute SPARQLMotion Modules from the Scripts menu, you'll see these two "scripts" listed as choices, even though they're graphically represented as a single flow chart with a choice of end points.

Taking it further

If you had created this script as a web service, then instead of ending with an Export to XML File module from the Export to Local section of the palette, you could route the XML from the Convert XML by XSLT module to a Return XML module from the Export to Remote section of the palette. (Or, that could be a third endpoint added to the two discussed above!) If you set the Return XML module's sml:mimetype property to text/html and had it return XHTML like the xsltdemo script above does, you'd be setting up the dynamic creation of a web page, so that users browsing to the URL that invokes the service would think that they were just linking to a web page.

Of course, it doesn't have to return HTML. It can return anything that an XSLT script can create, and you can set sml:mimetype to any valid HTTP MIME type, and any application that can do an HTTP GET can request delivery of this data, which opens up a wide range of possibilities for the contributions that this application can make to a larger system.

How to: read triples from a relational database

2010-02-12T08:23:00.000-08:00

An updated version of this blog post is now available as a chapter in the TopBraid Application Development Quickstart Guide (pdf).

TopBraid includes the D2RQ interface to relational database management systems and automates most of the configuration for you, giving your TopBraid applications easy access to relational data. In this entry, we'll see how easily a SPARQLMotion script can use data stored in a MySQL database. You would use a similar process to get at data stored using Oracle, Sybase, SQL Server, and PostgreSQL. Our sample uses MySQL's sample "world" database, a collection of data about countries around the world. See the Other MySQL Documentation page to download the database and for information on installing it.

Configuring TopBraid to read a MySQL database

Once you have a running copy of MySQL with the world database installed, the next step is to tell TopBraid Composer to create the configuration files that let you treat this relational database as a virtual triplestore. Start by selecting the project or folder in the Navigator window where you want TBC to put these files, and then select Import from the File menu and pick "RDF/OWL View on Relational Database via D2RQ" from the list of formats that can be imported. Click the Next button, and you'll see the Specify database connection dialog box.

For the database connection file name, enter worlddb, and then change the Base URI of each of the configuration files to something that reflects this filename, as shown in the screen shot below.

The last four fields are where you specify the key information for giving TopBraid access to the database. The drop-down list for the Database URL field offers a choice of templates for different RDBMS systems; for our MySQL database, select jdbc:mysql://<server>/<database> and change it to jdbc:mysql://localhost/world. (This assumes that MySQL is running on the same machine as your copy of TBC. If not, substitute the appropriate name for localhost. The part after the last slash is the database that you want to read from the MySQL server on that system.) In the next two fields, enter a user name and password for an account that has access to the worlddb database when using MySQL tools, and then in the Driver Class field select com.mysql.jdbc.Driver.

Click the Finish button, and TBC will create the configuration files and display a message box about it:

The "Importing Relational Databases with D2RQ" online help panel has good background on this whole process, especially on the role that each of these configuration files play. Among other things, it points out that the property names assigned to the columns of the relational database tables are just a starting point that can be reconfigured by editing these files. For example, the tableName_columnName format of the property names that you use to access the data is only a default and can be changed.

Reading the data from a SPARQLMotion script

To see how SPARQLMotion can use this data, we're going to create a simple script that gets the database's value for the president of the United States. After running it, we'll update the data using a MySQL tool and then run the SPARQLMotion script again to see the update reflected in the data that TopBraid is using.

Start by creating a new SPARQLMotion file as described at How to: create and run a SPARQLMotion script and call it mysqltest. Create a new SPARQLMotion script for this file, and for its initial module pick sml:ImportRDFFromWorkspace from the sml:ImportFromLocalModules section of the sml:ImportModules category. Call it GetWorldData, and when its icon appears, set this module's sml:sourceFilePath property to worlddb-Test.owl. This was one of the files created earlier by the configuration process, and it imports two others: the worlddb-Schema.owl schema file and the worlddb.d2rq file that serves as a placeholder for the actual data in the MySQL world database. This is the only property that you need to set for this module.

Next, we'll add a module that queries the database for the name of the president of the United States. Add a Bind by select module from the Control Flow section of the SPARQLMotion palette and call it PresidentQuery. Set its sml:selectQuery property to the following query (note how the namespace for the property names is the "Base URI of generated Schema file" value you assigned when configuring how TopBraid would import the MySQL data):

PREFIX w: <http://example.org/worlddb/myschema.owl#>
SELECT ?headOfState
WHERE {
 ?s w:country_Name "United States" .
 ?s w:country_HeadOfState ?headOfState .
}

That's the only property to set for this module, so close the Edit PresidentQuery dialog box, click Add connect on the SPARQLMotion palette, and connect the GetWorldData module icon to the new icon.

When you work with a database like this, TopBraid keeps cached copies of the data to speed up later queries on the same data. This has obvious advantages, but it can be a disadvantage if you're more interested in data currency than in quick retrieval times. Our database is small, and we want to see changes reflected as soon as possible, so we're going to add a module that clears out this cache each time the script is run.

Drag a Collect garbage module from the Control Flow section of the palette to your workspace and name it EmptyCache. Set its sml:baseURI property to the URI for the database you want to clear from the cache: http://example.org/worlddb/myDatabase.owl, which is the base URI for the worlddb.d2rq placeholder file. (This is the URI you entered in the "Base URI of Instances (Database)" field when you set up the initial import and configuration.) After configuring this one property, connect the President query module icon to the Empty cache icon.

Select the Empty cache icon and click the debug icon to run this short script. You'll see in the Console view (a handy view when developing and debugging scripts) that the headOfState property has the value "George W. Bush", so the MySQL sample database is a bit out of date:

Let's fix this. Update the data using your favorite MySQL tool. For example, from MySQL's command line interface, you could enter this:

mysql> USE world
Database changed
mysql> UPDATE country SET HeadOfState="Barack Obama"
 -> WHERE HeadOfState="George W. Bush";

Run the SPARQLMotion script again, and you should see the update reflected in the Console view. If it wasn't for the script's Empty cache icon, your SPARQL query might have used a cached copy of the data that did not include this updated value.

As the online help tells us, it's a good idea to be careful with the sml:CollectGarbage module, especially if you're developing a multi-user application where data is frequently updated. If quick access to updated data is not a priority in your application, omitting this module will mean faster response times for your users. It's up to the needs of your application, and it's good to know that you can have it either way.

Instead of opening the worlddb-Test.owl file with your application, you're more likely to open a file that you created that imports one or more of the configuration files generated above (again, read the "Importing Relational Databases with D2RQ" help panel to learn more about what each file can contribute to your application) and perhaps some other files as well. You could add additional data files and RDFS or OWL files describing relationships between the various imported files so that you can do interesting things with the combinations. For example, you could declare the properties that reference MySQL database columns to be subproperties of others declared for your application or declared in a standard vocabulary such as Dublin Core.

Something else that can make your script development easier is importing the generated schema into your script—in this case, dragging the worlddb-Schema.owl file onto the Import view of the mysqltest file. This way, the namespace prefixes will already be defined for you and you can use autocompletion when you enter class and property names in your SPARQL queries.

The sample application here didn't do much on its own, but when you combine the imported data with other data and metadata and take advantage of the other techniques described in this How-to series, the ability to incorporate live relational data in your semantic web applications adds some great possibilities to what your applications can do.

How to: get data from SPARQL endpoints and save data in a spreadsheet

2010-02-06T09:04:00.000-08:00

An updated version of this blog post is now available as a chapter in the TopBraid Application Development Quickstart Guide (pdf).

This entry covers two useful SPARQLMotion techniques: how to retrieve and use data from SPARQL endpoints, and how to save data so that you can open it with a spreadsheet program such as Excel or OpenOffice Calc. While this may look like a lot of material to cover, both techniques are so simple that this is the shortest How to blog entry yet.

Retrieving data from a SPARQL endpoint

A SPARQL endpoint is a server that accepts SPARQL queries and returns the result using the SPARQL Protocol for RDF, a W3C standard XML-based format that is much simpler than RDF/XML. This endpoint can be part of your intranet, providing access for others there to data behind your firewall, but more and more SPARQL endpoints are available on the public internet to provide you with data to use in your applications, and excitement about the new possibilities provided by this data is a big driver of the Linked Data movement. (In a future entry, we'll see how you can create your own SPARQL endpoint with TopBraid Live.)

The ability to retrieve SPARQL endpoint data from within a SPARQLMotion script is not a feature specific to SPARQLMotion, but actually an extension to the Jena framework that will hopefully become part of SPARQL 1.1: the SERVICE keyword. This lets you name a data source right in your query, and the source can be a remote SPARQL endpoint.

To try it out, enter the following query into TopBraid Composer's SPARQL view and execute it. Instead of pulling data from the currently open data graph, it gets it from the SPARQL endpoint specified by the URI http://dbpedia.org/sparql, which is the endpoint for the huge DBpedia collection of triples harvested from the structured infobox data in Wikipedia pages:


SELECT ?subsidiaryName ?description
WHERE {
 {
     SERVICE <http://dbpedia.org/sparql> {
         <http://dbpedia.org/resource/IBM> <http://dbpedia.org/ontology/Company/subsidiary> ?ibmSub .
         ?ibmSub <http://dbpedia.org/property/abstract> ?description .
         ?ibmSub rdfs:label ?subsidiaryName .
         FILTER (lang(?description) = "en")
         FILTER (lang(?subsidiaryName) = "en")
     }
 }
}

This query retrieves names and descriptions of IBM subsidiaries from DBpedia and only passes along the English language ("en") ones.

You can use this technique anywhere that you can use a SPARQL SELECT query in your TopBraid development—including in the sml:selectQuery property of a SPARQLMotion Create Spreadsheet module, as we'll see.

Saving data in a spreadsheet

If you search the TopBraid Composer online help for the word "spreadsheet," you'll see that TopBraid offers a variety of ways to read and write spreadsheet data. The simplest way to create a spreadsheet from a SPARQLMotion script is the Create Spreadsheet module. After you enter a SELECT query in its sml:selectQuery property, it sends a tab-delimited spreadsheet to the next SPARQLMotion module with a column for each variable in the select query. (Any spreadsheet program can read a tab-delimited file. OpenOffice Calc doesn't offer "tab-delimited file" as a choice of file format to open, but if you tell it to open a "Text CSV" file and point it at a tab-delimited file, it will find that tab characters are the delimiters and import it properly.) For example, with the SELECT query shown above, this module will create a two-column spreadsheet with subsidiaryName values in the first column and description values in the second.

Let's create a SPARQLMotion script that does this. Create a new SPARQLMotion file as described at How to: create and run a SPARQLMotion script, calling the file endpointtsv. Next, select Create SPARQLMotion script from the Scripts menu, and when it asks you to select an initial module type, select sml:CreateSpreadsheet from the sml:TextProcessing modules choices under sml:ProcessingModules. Name the module GetIBMSubs and click OK.

When the icon for your new module appears on the SPARQLMotion workspace, double-click it to configure it and set sml:selectQuery to the query shown above. The sml:CreateSpreadsheet module creates a spreadsheet in memory, not on disk, to provide you with the flexibility of doing other things with its output before you send it to a specific output destination such as a disk file, so set the sm:outputVariable property to the name spreadsheetData so that subsequent modules can reference it. You're finished configuring this module, so click the Close button.

Our SPARQLMotion script will only have one subsequent module: the one that saves the spreadsheet to a disk file. Drag an Export to text file module from the Export to local section of the palette onto the workspace and name it SaveSpreadsheet. Double-click its icon to configure it and set the following properties:

Set sml:targetFilePath to ibmsubs.txt.
The sml:text property is where you assign the text to store in the file being created; set it to {?spreadsheetData} so that it uses the value of the variable being set in the GetIBMSubs module.
Set sml:replace to true so that running the script more than once replaces the existing disk file instead of appending the data on as new lines. (Of course, setting sml:replace to false can be very handy in other scripts that you write.)

Click the Close button, and then click on Add connection on the palette. Connect your two icons, and you're finished with your simple little script:

Save it, select the SaveSpreadsheet icon, and click the debug icon to run your script. You should see an ibmsubs.txt file appear in the same directory as the script itself, because you didn't specify a path name when you configured the sml:targetFilePath property. (You're welcome to include a pathname with the file in that property.)

Open up this new file in a spreadsheet program, and then you can use any of the program's formatting features that you like before saving the spreadsheet using the program's native format. Here are the first few rows of the spreadsheet in Microsoft Excel:

If you combine these two techniques with others covered in this How-to series, you could have all of this triggered by a web service call, or the script could combine the data retrieved from a SPARQL endpoint with other data from other sources before saving the spreadsheet, or it might output the data in another of the destinations available in the "Export to" sections of the SPARQLMotion workspace palette. Take another look at those palette choices, or better yet, review the help page for them; I'm sure you'll get some great ideas.

How to: create a TopBraid Ensemble application

2010-01-29T12:06:00.000-08:00

An updated version of this blog post is now available as a chapter in the TopBraid Application Development Quickstart Guide (pdf).

TopBraid Ensemble (TBE) is a Flex-based web application for assembling and displaying TopBraid applications. The user interface runs on web browsers with Adobe Flash installed. When developing applications, the data behind the application is hosted by the TopBraid Live Personal Server included with TopBraid Composer Maestro Edition (TBC-ME); applications can then be deployed for multi-user use on a server running the TopBraid Live Enterprise Server product.

Today we're going to see how to use TBE to create an editor that lets an end user edit a thesaurus using the SKOS vocabulary management ontology. This exercise is excerpted from the TopBraid Ensemble Application Development and Reference Guide (pdf), which you can download and use to find more detailed background on TBE's features and architecture.

TopBraid Ensemble: the basics

TBE lets developers quickly configure applications around semantic web models by:

Assembling pre-packaged graphical user interface components such as search forms and tree-based displays into desired configurations that let the end user view or edit data
Adding custom business actions through SPARQLMotion scripts
Building and using new custom components

Once the application developer designs forms and event logic around a data model and deploys this TBE application on a server running the Enterprise edition of TopBraid Live, the developer's end users can use this application with any web browser that supports Adobe Flash.

The nine user interface component types offered in release 3.2 include trees, grids, forms and maps. A developer can arrange these across multiple pages that function like a tabbed interface to reduce the clutter than can result from squeezing too much onto a single screen.

Each UI component can:

Contain one or more widgets that can display or provide an edit mode for information from the connected information source
Include its own menu actions and icons
Broadcast events to other components and listen for and react to events broadcast by other components

TBE also provides non-UI, or invisible components, which work behind the scenes to perform data transformation and to relay events.

A developer designs one or more input screens with TBE components and specifies the data graph that it will work with. These applications can call SPARQLMotion scripts, combining sophisticated back-end triples processing with quick development of application user interfaces to create powerful, intuitive semantic web applications.

Creating an application consists of four basic steps:

Select whether you want to begin with a New Application or a Default Application
Identify the data to use with your application
Select and configure the components to go on your screens
Test your application

Starting with a New Application means starting with a blank form and then adding and configuring the components that you need. This is especially useful for experienced TBE developers creating simple applications. New TBE developers often find it easier to begin with a Default Application, which includes several components arranged in a default layout that you can modify by deleting, adding, moving, and reconfiguring the components.

The following shows a Default Application before any modifications have been made, with kennedyfamily.owl specified as the data to use with it:

A configured TBE application specifies what components are shown when it is launched, how components are arranged, and how they interact with each other. Each application also defines what data to use with it. Some applications, such as the default apps and the new app, are configured to work with any data. Other applications are configured to work with specific datasets.

When creating a TBE application, components can be rearranged, resized, removed and added. For example, you can have have two Results Grid components on the screen instead of one, like the default applications have. You can also customize the menu options for each component and the way the components interact with each other.

TopBraid Ensemble's Default configurations are general TBE applications included in your installation. These configurations include most of the built-in components, laid out and wired for interaction with each other so that when an event such as a resource selection happens in one component, information displayed in some of the other components changes. For example, when you select a class in the tree, the members of the class appear in the grid, and when you select a resource in the grid, it populates the form with information about that resource.

You can learn a lot about the potential power of these forms by studying how the components in the default configurations generate and listen to the events that make this kind of component interaction possible.

Creating the sample application

Our sample SKOS thesaurus editing application will have only two components:

A tree displaying resources connected by skos:broader property.
A form showing details of each resource. When the end-user clicks a node of the tree or a class name on the form, the form will refresh to display all the details about that class.

TBE applications typically work with ontologies created using TBC, so before creating our TBE app, we need to create the data model.

Step 1: Create Ontology

In TBC, create a new RDF/OWL file called thes1 and import skos-core.rdf from the Common folder of the TopBraid library into your new model. Create an instance of skos:Concept (a subclass of owl:Thing) in this model and call it Taxonomy Root. Remember which project it's stored in; we will need it as a root resource for the tree created on your TBE form.

Save your work.

Step 2: Create New Application

To get started with developing and configuring a TBE application, first send your browser to the TopBraid Suite Console, which can be accessed at http://localhost:8083/tbl when running the TopBraid Live Personal Server included with TBC Maestro Edition. This will take you to the TopBraid Administration Console.

In the TBS Administration Console, select "New Application".

On the "Data Graphs Compatible With New Application" screen, select the thes1 model that you just created from the project where it was stored.

Step 3: Select Components

In TBE, click the plus sign icon and select Add Components to add each of two new components to the application: tree and form. Drag their tabs to lay them out side by side as shown in the following:

Note how the default configuration of the tree component is to show the class structure of the data model; you'll be reconfiguring it to show the tree of taxonomy terms being edited by your new application's end user.

Step 4: Configure Application

Launch the configuration console by clicking the wrench icon or pressing F2. You began with a copy of "New Application"; Click on "New Application" at the top of the navigation pane on the left, and then under General on the right change the application label to be "SKOS Editor". (Hover to the left of the existing value and click the "edit" button that appears there to change the value. )

Step 5: Configure Tree

Now click on the Tree component on the left side of the Application Configuration console (you may need to first click the Components node of the tree to expand it into a list of available components) and make the following modifications:

Rename the Tree's label to read Concept Hierarchy.
Under Tree/Attributes, change the "Show the root of the tree in the display" setting to true.
Change the Root Node to the Taxonomy Root (the new resource you created in TBC).
Change the Transitive Property that identifies tree node child/parent relationships to "has broader" (the label assigned to the skos:broader property).
Under Events, create a new Post Event for Single-click Selection and call it Tree Selection.

Your configuration should look similar to the one shown here:

Step 6: Configure Form

Now select the Form component on the left of the Application Configuration console and make the following modifications:

Wire the Form to listen to the Tree Selection event. Click on Add existing entry next to Refresh Display (Listen Event) and start typing "Tree". When "Tree Selection" shows up in the drop down, select it and click OK.
Create a new Post Event for the Request Resource Navigation listener. Call it Form Selection.
The form is already configured to refresh when the end user clicks on a tree node, which sends a Tree Selection event. You also want the form to refresh if the end user clicks on the name of a concept on the currently displayed form. For example, if the form for the concept Sparrow shows that Sparrow's broader term is Bird, and the end user clicks on Bird, you want it to display data for the concept Bird. Click on Add existing entry next to the Refresh Display listen even and start typing Form. When "Form Selection" shows up in the drop down list, select it. It will join Tree Selection as the second value for the Tree's Refresh Display listen event.

Your configuration should look similar to the one shown here:

Step 7: Test and Save

Your new application is done.

You can close the configuration console by clicking the X at the upper right and test your new application by creating new nodes in the tree with the buttons in the upper-right of the tree component and entering additional information in the forms as shown here:

As you add and select nodes in the Concept Hierarchy, the tab title and many of the fields in the form on the right will update automatically to provide information about the selected tree node.

To save your application, click the Save App button. On the Save Application As dialog box, enter new values for Application Name and Application Base URI. (If this is not the first time you've saved this application, you'll also want to click the Overwrite check box.) The next time you go to the TopBraid Suite Console, you'll see your new application listed where you earlier saw New Application, Default Application, and the other choices.

To save the data you've created using your new editor, such as the bird information shown in the screen shots, you or your application's end users would click the Save Data button.

Applications like this can be hosted on the TopBraid Live server, where any users with browser access to that server can run your application and browse and edit the data you choose to make available. They don't need TopBraid Composer, and they don't need to know about RDF or any data modeling issues; they only need to know how to use the graphical, interactive application that you design for them using TopBraid Ensemble.

So far, we've only seen a small part of what a TBE application can do for your end users. Take a look at the TopBraid Ensemble Application Development and Reference Guide (pdf) to learn more.

How to: create and run a semantic web service

2010-01-22T07:49:00.000-08:00

An updated version of this blog post is now available as a chapter in the TopBraid Application Development Quickstart Guide (pdf).

When you know how to define a new function and create a SPARQLMotion script in TopBraid Composer, you can put these two skills together to create a RESTful web service that can be called from any computer with HTTP access to the server running your web service. In a production environment where many people and systems might call your web service, TopBraid Live would be the best way to host it, but for developing and testing you can use the HTTP server built into TopBraid Composer Maestro Edition. Today we're going to see how to create a web service that searches the database about the Kennedy family included with TopBraid Composer and returns RDF/XML that lists everyone with a particular first name.

But first, what makes this a "semantic" web service? TopBraid Composer puts all the power of semantic web technology, such as SPARQL, OWL, and more, behind your development efforts, and it lets you read and write both semantic web data formats and more traditional ones. By delivering RDF, your web service can plug into other semantic web applications, although you're certainly not limited to RDF as a delivery format—you can deliver XML, a spreadsheet, or even HTML, so that a browser hitting the URI you designate for your service would actually be displaying a dynamically generated web page.

If you've never defined your own functions or created a SPARQLMotion script in TopBraid Composer, you'll want to review the earlier entries How to: write new SPARQL functions with SPIN and How to: create and run a SPARQLMotion script before continuing.

Creating your script

Start by creating a SPARQLMotion File, but on the Create SPARQLMotion File dialog box, remember to check the "Script will declare (Web) Services or Functions (.sms extension)" checkbox. If you name your file ws1, you'll see that TopBraid Composer then saves it with the full name ws1.sms.n3.

Once you've created this file, turning it into a web service script involves two steps:

Defining the function that the web service URI will call.
Defining the script that will be triggered by the function call.

Defining the function to call

Create a new subclass of spin:Functions and name it searchKennedys. Add an sp:arg1 argument as a spin:constraint value using "Create from SPIN template" (as described in How to: write new SPARQL functions with SPIN) with a predicate value of sp:arg1. For the argument's comment field, enter "String to search for." Also, set its valueType to xsd:string so that the function receiving the passed argument value knows to treat it as a string and not as a URI. With that, you'll be finished configuring your spl:Argument and can click the OK button.

A given SPARQLMotion script can have several possible endpoints return different variations on how the data is processed, so identifying a set of script modules to run means identifying a specific endpoint in a script. On the class form for your new searchKennedys function, set the sm:returnModule property by clicking the white triangle to display its context menu and then selecting "Create and add..." You will create this return module before adding it because you haven't created your script yet, so there's no endpoint module to point to.

On the Create and add dialog box, pick sml:ReturnRDF as the first module to add, as shown below. (It's a subclass of ExportToRemoteModules because your web service will return its results to a remote caller; you'd pick something from ExportToLocalModules if it were going to save the results in a local file.) Name your new module ReturnSearchResults.

Click the OK button.

Creating the SPARQLMotion script

Next, select "Edit SPARQLMotion script" from the Scripts menu. You'll see that two modules have already been added to your new script: your ReturnSearchResults module and an Argument module to represent the arg1 argument being passed to your script.

Note the gray "arg1" with a little arrow on a circle in the lower-right of the Argument module, which shows the name of the variable that's being set and passed along from it. Your script will use this variable to specify what to search for in the Kennedy family data.

The next step is to add something to the script to read the data that it will search. Drag an Import RDF from Workspace icon from the Import from Local section of the palette onto the workspace and name it GetKennedysData. To configure it, double-click its icon, set its sml:sourceFilePath property to /TopBraid/Examples/kennedys.owl, and then click the Edit GetKennedyData dialog box's Close button.

Now we'll add the module that searches through the data and extracts the subset that we need. Drag an Apply Construct module from the palette's RDF Processing section onto the workspace and name it SelectData. Set its sml:replace property to True so that it only passes along the selected data and not the input data as well. Set the sml:constructQuery value to the following:

PREFIX k: <http://topbraid.org/examples/kennedys#>
CONSTRUCT {
?s k:firstName ?first .
?s k:lastName ?last .
}
WHERE {
?s k:firstName ?first .
?s k:lastName ?last .
FILTER regex(?first, ?arg1, "i" )
}

(After you press Enter, TopBraid Composer will replace the "k" namespace prefixes with the namespace URIs.) If you wanted this query to always search for firstName values with "John" in them, you could put that as the second argument of the regex function, but this query has something more flexible: a reference to the arg1 variable that will be passed in from the Argument module. (The third argument of "i" to this function tells it to ignore case when searching.) The use of the regex function means that it will search for substring matches as well, so that a search for "Car" will turn up Caroline Kennedy, Carolyn Bessete, and Cart Hood.

You're finished configuring this module, so click the Close button and connect the two input modules to it like this (don't worry about connecting up the Return Search Results module just yet, and remember that at any time you can drag the module icons around to make the script's flow easier to understand):

Select the SelectData module and then click the debug icon near the upper-left of the workspace to test it. Because no value has been passed for the Argument, a SPARQLMotion Script Input dialog box prompts you for one. Enter "jean" as a test, press Enter, and then click the Next button. When it's finished, you should see triples storing the firstName and lastName values for Jean Olssen, Jean Kennedy, and Jeannie Ripp appear in the SPARQLMotion Results view.

Your web service script can now do everything except return the data that it extracted from the data source to the process that called the service. Connect the Select data module to the Return search results module, which still needs one bit of configuration: to tell it to return the data as RDF/XML, double-click the icon and click the context menu for the sml:serialization property. Don't pick "Add empty row" this time; you want to add one of the predefined values from the sml:RDFSerialization class, so pick "Add Existing." On the Add existing dialog box, click on sml:RDFSerialization, select sml:RDFXML on the right (note the other choices available to you), click this dialog box's OK button, and then click the Edit ReturnSearchResults dialog box's Close button.

Save your work. The final script should look something like this:

Registering and calling your web service

Before something can tell TopBraid Live or the server built in to TopBraidComposer to call the searchKennedys function that calls this script, the function must be registered with the server. To do this, select "Refresh/Display SPARQLMotion functions..." from the Script menu, and after a few seconds you'll see the updated list of registered functions on the Console view.

To test your web service, enter the following into any web browser running on the same computer as your copy of TopBraid Composer; note how its two parameters identify the function to call and the argument value to pass to it:

http://localhost:8083/tbl/actions?action=sparqlmotion&id=searchKennedys&arg1=Rob

After making this call, you should see RDF/XML data in your web browser about the six people in the database who have "Rob" in their first name: five Roberts and a Robin. (For some browsers, you may need to to a View Source to see all the XML.)

Congratulations! You now have a working, RESTful, semantic web service up and running. Any tool that can send a URI to the HTTP server built into TopBraid Composer and then parse the result can use it. (Most modern program languages include libraries that make this simple.) If you install your web service and the appropriate data files on a computer running TopBraid Live, multiple systems can retrieve data from that service at once.

When you consider the possibilities of using data from more diverse, large-scale data sources and creating SPARQLMotion scripts that take advantage of a wider range of the modules available on the workspace palette, you'll start to see the tremendous possibilities of what you can do with your TopBraid semantic web services.

How to: create and run a SPARQLMotion script

2010-01-14T20:38:00.000-08:00

An updated version of this blog posting is now available as a chapter in the TopBraid Application Development Quickstart Guide (pdf).

SPARQLMotion is a visual scripting language that lets you tie together triples from diverse data sources (whether they're natively stored as triples or not), process them using SPARQL, and then save the results in a wide choice of output formats. One of these scripts can be run from TopBraid Composer, as a semantic web service, or as the back end to a user interface created with TopBraid Ensemble.

You create a SPARQLMotion script by dragging icons representing different kinds of modules from a palette onto a workspace and then filling out a dialog box for each to configure it. Here, we're going to look at the creation of a simple script, and in future entries on this blog we'll learn about more modules and what they can add to your application development: reading and writing spreadsheet data, reading and writing from large, disk-based triplestores, checking of data constraints, HTML and XML processing, and much more.

Our sample script will import the triples from the FOAF ("Friend of a Friend") files of some well-known semantic web advocates, extract the names and nicknames of their friends who are listed there, and save the results in a new RDF file. Much more complex scripts are possible, but the creation of those scripts will always follow the basic steps described here.

To create a SPARQLMotion file, create a new file in TopBraid Composer the same way you would create any other file. When you indicate that you want to create a file, you'll see that "SPARQLMotion File" is one of the options; picking it creates a file that already has the sparqlmotionfunctions and sparqlmotionlib libraries imported; these have what you need to add a script to your new file. Assign the file any Base URI and File name you like (for example, http://www.example.com/sm1 and sm1) and click the Finish button to indicate that you're done with the Create SPARQLMotion File dialog box.

Next, select "Create SPARQLMotion Script" from TopBraid Composer's Scripts menu. A Select initial module type dialog box will ask you about the first module to add to your script. Because the script will import RDF files from the web, start with an ImportRDFFromURL module and name it GetTimBLData:

After filling out the dialog box as shown and clicking the OK button, the script appears with its single module:

Double-click the icon representing the module to configure it. The only information you must add to an ImportRDFFromURL module is the URL of the RDF to import. To set the value of a property such as sml:url on one of these dialog boxes when there is no existing value, display its context menu by clicking the white triangle next to it and select Add Empty Row. To to set the property value to retrieve Tim Berners-Lee's FOAF file, enter the value http://www.w3.org/People/Berners-Lee/card.rdf, as shown below. (I've also tweaked the rdfs:label value, and note from the "Ok" to the right of the sm:url that the data entry of this field is not complete—TopBraid Composer will not know about this new value until you either press the Enter key with the cursor in that field or click on "Ok" to make it go away.)

We'll see shortly why you don't have to add a value for the sm:next property on this dialog box. Click the Close button to return to the script workspace.

At this point, your script only has one module, but you can still run it. Clicking a module icon selects it, and clicking the little bug icon ("Debug selected SPARQLMotion module") at the top of your workspace tells TopBraid Composer to run the script up to the selected module. Do this with GetTimBLData, and after it retrieves the triples from the designated URL, TopBraid Composer displays a message box offering to display the result triples in TopBraid Composer's SPARQLMotion Results view.

As with so many semantic web applications, this one becomes more interesting when you add more data sources and then select a subset of the combined data that meets your needs. If you don't see the module palette on the right side of the script workspace, click the small white triangle in the upper-right of the workspace to display it. To add a second module, click "Import from Remote" on the palette and click the small darker triangle to scroll through the choices there. When you see Import RDF from URL, add a new one of these modules to your workspace by dragging it there and name it GetJimHendlerData. Enter http://www.cs.umd.edu/~hendler/2003/foaf.rdf as the data's location. Then, add a third Import RDF from URL module named GetDanBrickleyData and set http://danbri.org/foaf.rdf as the sml:url value. (At any point in the creation and editing of your script modules, you can rearrange the icons by selecting and dragging them.)

Once the script retrieves data from these three sources, it will send it to a module that only passes along the triples that identify the name and nickname values found the source data. Drag an Apply Construct module from the RDF Processing section of the palette onto the workspace and name it ExtractData. After this module appears, double-click it and set the sml:constructQuery's value to the following:

CONSTRUCT {
?s a <http://xmlns.com/foaf/0.1/Person> .
?s <http://xmlns.com/foaf/0.1/name> ?name .
?s <http://xmlns.com/foaf/0.1/nick> ?nick .
}
WHERE {
?s <http://xmlns.com/foaf/0.1/name> ?name .
?s <http://xmlns.com/foaf/0.1/nick> ?nick .
}

You can use SPARQL CONSTRUCT queries to rearrange and cross-reference data to infer and create new information based on the input, but our simple query merely finds and passes along triples that match two patterns and declares that the subject that has these properties is a Person as defined by the FOAF vocabulary.

After entering this CONSTRUCT query, set the same dialog box's sml:replace value to True so that only the constructed triples get passed along to the output without the input data. Now you're finished with this dialog box, so click its Close button.

We haven't identified the input of this CONSTRUCT module yet. Instead of writing code, we can do it by just pointing and clicking. To make your first connection, select the Add connection icon on the Palette, click on the GetTimBLData module, and then click the ExtractData module. An arrow will appear to show that data will flow from one to the other when the script is run:

Follow the same steps to connect the other two data retrieval modules to ExtractData.

Instead of waiting until the application is finished to test it, we can do more incremental testing here. Select the ExtractData module, click the debug icon at the top, and you should end up with a list of names, nicknames, and foaf:Person type declarations in the SPARQLMotion Results view.

Our final step has two parts:

Tell the script to save the results of the ExtractData module to an RDF file. From the Export to Local section of the Palette, drag an Export to RDF file module onto the workspace and call it SaveOutput. Double-click the new module and set the sml:baseURI property to http://mytest/ (or any URI you like; it will be used as the base URI of the saved file) and set sml:targetFilePath to sm1out.rdf. If you don't specify a path for this file, the SPARQLMotion script will save the file in the same directory as the script itself. After setting these two values, click the Close button.
By including the FOAF ontology in your output, applications that read the file created by your script will have more context about the meaning of the data. You can import this ontology from the web, but the TopBraid Composer distribution includes many popular ontologies so that importing them will happen more quickly. From the Import from Local section of the SPARQLMotion palette, drag an Import RDF from workspace module onto the workspace and name it FOAFOntology. Double-click it, set its sml:sourceFilePath property to /TopBraid/Common/foaf.owl, press Enter, and then click the dialog box's Close button.

Use Add connection to connect the ExtractData and the FOAFOntology modules to the Save Output module. Your completed script should look like this:

After saving your application, test it by selecting the Save Output module and clicking the debug icon. As it runs, watch the Console view, which has informational messages about your script as it runs. This and the Error Log view are valuable for debugging complex scripts.

The script should create the sm1out.rdf file described above. Open it up, and then in TopBraid Composer's Classes view, drill down from owl:Thing to foaf:Agent and select foaf:Person so that the instances you've created appear in TopBraid Composer's Instances view. Selecting any of them there will display details about that person in the Form view, where you'll see the name and nickname assigned to that person somewhere on that form. (You may need to scroll down a bit to see those particular properties.)

If you look through the many module choices on the SPARQLMotion palette you'll get some idea of the wide variety of possibilities you have for the kinds of data that a script can read and write and the sophisticated options for processing that data. We'll look at many of them in future entries here. Meanwhile, take a look at this short video, which shows you how to create another simple script that uses a few more interesting modules.

How to: write new SPARQL functions with SPIN

2010-01-05T12:44:00.000-08:00

An updated version of this blog post is now available as a chapter in the TopBraid Application Development Quickstart Guide (pdf).

This is the first of a series of short pieces on how to use various features of the TopBraid Suite application development tool set. For example, we'll see how to create a SPARQLMotion script, how to create a user interface with TopBraid Ensemble, how to create a web service, how to convert triples to XML and run an XSLT stylesheet on them, and how to use data from spreadsheets, databases, public SPARQL endpoints, and more. These postings will focus mostly on application development, and assume that you already know the basics of TopBraid Composer itself, which you can learn from the "TopBraid Composer Getting Started Guide" (pdf). Each posting will be assigned to the How to category of this blog to make it easier for readers to use them as a set.

This first entry shows how you can define your own functions using the SPARQL Inferencing Notation, because it's such an important building block for many of the techniques that we'll learn about in future entries. This material is excerpted from the "Getting Started with SPIN" (pdf) document, which is available for you to download. The complete "Getting Started with SPIN" document also covers inferencing, constructors, constraints, and more.

Before we cover the definition of new functions, let's review the calling of functions in TopBraid, because there are so many useful ones available to you.

Calling built-in functions

The "TopBraid SPARQLMotion Functions Library" page of TopBraid Composer's online help lists over 70 functions available for you to call from your applications, including string functions, mathematical functions, logical functions, and ontology functions. Because TopBraid Composer uses the Jena SPARQL engine, it supports the LET keyword for assignment of variables, which gives you a way to experiment with built-in variables using a short, simple SPARQL query.

For example, to try the mathematical function smf:random() (a TopBraid Composer extension to the Jena function library), enter the following query in the SPARQL view's Query Editor tab:

SELECT ?x
WHERE {
LET (?x := smf:random()).
}

To execute it, either click the "Execute SPARQL" green triangle button or press Ctrl+Enter, and you'll see a random number between 0 and 1 appear in the [x] column of the SPARQL query result. Because this function returns a different value each time you call it, go ahead and execute it a few more times.

Another built-in extension function is smf:parseDate(), which converts a string in a semi-structured date format into an xsd:date, xsd:dateTime, or xsd:time value. (To learn more about smf:random(), smf:parseDate(), and other available functions, select Help Contents from the Help menu and then TopBraid Composer -> Reference -> SPARQLMotion Functions Reference.) This function takes two parameters: a date string and a template showing which pieces of the date are where in that string. Try pasting the following query into the SPARQL

view and executing it:

SELECT ?x
WHERE {
LET (?x := smf:parseDate("12/3/09","MM/dd/yy")).
}

The value displayed under [x] is "2009-12-03". (The TopBraid Composer online help for this function includes the URL of a web page with greater detail about options for the format string in the smf:parseDate() function's second parameter.)

We're going build on this function to create a simpler function to convert dates in the MM/dd/yy format.

Defining your own functions

Before we use the SPIN vocabulary, we need to import it into our vocabulary. Open your Import

s view and drag the spin.owl file from the TopBraid/TBC folder in the Navigator view into the Import view. You'll see several new nodes appear in the Properties view, each with an sp: or spin: prefix and a plus sign next to it that expands the node to show the full range of SPIN properties. For now, you won't need to use these properties from this view, because TopBraid Composer gives you forms to fill out to specify everything you need to define a new function.

The Classes view also shows a few new classes. Click the plus sign next to spin:Modules to expand it, and you'll see that one of the subclasses is spin:Functions.

To create your new function,

Right-click on spin:Functions and pick Create subclass from the context menu.
On the "Create Classes" dialog box, click on the default new function name of "Functions_1", replace it with the name "mmddyy2ISO8601", and press Enter.
Click the OK button. You'll see that your new subclass of spin:Functions is currently selected in the Classes view, and the class form will have the URI and rdfs:subClassOf values already filled out.

Your new function will have an argument passed to it: the date string to convert. The body of the function must reference this argument, so let's define the argument before creating the function body. Like everything else in SPIN, this definition will ultimately be stored with a series of triples, but because argument definition is so common in application development, SPIN includes a template to make it easier.

SPIN models arguments to functions as constraints on those functions. On the mmddyy2ISO8601 class form, click the spin:constraint widget (the small white triangle to the right of the name "spin:constraint") and pick "Create from SPIN template".
Select spl:Argument from the Available Ask/Construct Templates list, and then fill in the following two fields in the Arguments panel on the right of the "Create from SPIN template..." dialog box:
- In the comment field, enter Convert mm/dd/yy formatted date to xsd:date type and format. (Remember to press the Enter key after entering a value on one of these forms. If you tab to another field and the comment field has a heavier border around it and and the "Create from SPIN template" dialog box's OK button is grayed out so that you can't click it, go back tot he comment field and press the Enter key.)
- Instead of typing a value directly into the predicate field, click the plus sign to the right of it. This displays a "Select Resource..." dialog box; because we're defining our first (and only) argument to pass to the new mmddyy2ISO8601 function, click sp:arg1 on the right panel and then the OK button.
Click the OK button to finish with the "Create from SPIN template" dialog box, and you'll see that the spin:constraint field of the class form for your new mmddyy2ISO8601 function has been filled out.
All that remains is to define the function body. Click the spin:body widget on the class form and select Add empty row.

Enter the following as the body:

SELECT ?x
WHERE {
LET (?x := smf:parseDate(?arg1, "MM/dd/yy")) .
}

The body of this function uses ?x as the variable name, but you can use any name you want. If you have more than one variable, and it binds to more than one value, the function will return the first value of the first variable, so there's no point in naming more than one variable after the SELECT keyword.

Note how, in this SPARQL query, the first argument to the smf:parseDate() function is arg1, which we defined on the spin:constraint part of the form earlier.

Your new functions should be all ready. Test it by entering and executing the following in the SPARQL view:

SELECT ?x
WHERE {
LET (?x := :mmddyy2ISO8601("9/6/09")).
}

When you make calls to your own functions, don't forget the colon at the beginning of the function call and, if your function was not defined in the default namespace, the namespace prefix before that. All function calls are in a particular namespace (for example, parseDate() is in the namespace represented by the smf prefix), and your new mmddyy2ISO8601() function is in the default namespace for this project. For production application development, it's better to identify the function's namespace with a specific prefix, because it will be easier to import the code into another file without confusion.

mmddyy2ISO8601() is now a function that you can call anywhere, including from the body of new functions that you create. Test it further in the SPARQL view by passing other dates as a parameter.

RDFa on topquadrant.com

2009-12-16T16:17:00.000-08:00

RDFa is the W3C standard for embedding RDF triples in arbitrary XML using attributes. Its most popular use is in HTML, because it makes it easy to add machine-readable versions of a web page's information with minimal new markup.

You could add this markup to just about any web page, but it's especially useful on pages with information that is good for redistribution and can be described with popular vocabularies. We've just added RDFa to our products, management, and contact pages using the FOAF, Dublin Core, vcard, and GoodRelations vocabularies.

GoodRelations is the newest of these vocabularies, and it's getting popular quickly. A recent Semantic Technology Blog posting described a talk at the 2009 Search Engine Strategies conference in which BestBuy's Lead Web Development Engineer described how "GoodRelations + RDFa improved the rank of the respective pages in Google tremendously." In addition to improving page rank when selling products, RDFa makes it easier to share other kinds of information for use by applications on web pages where the data is already present for human consumption, whether it's research data, government data, or airline flight schedules.

An increasing number of tools are available for extracting triples from web pages with embedded RDFa, and the TopBraid Suite has supported this for a while. A TopBraid application can define a particular internet or intranet web page as an RDFa data source so that each time you run the application it will check for the latest set of triples in that page and then incorporate that data with the other data and logic used for that application.

And now, applications like this can pull data about TopQuadrant's products, people, and places from our HTML web pages!

What's new in TopBraid Suite 3.2

2009-11-02T16:21:00.000-08:00

3.2 is a minor release of TopBraid Suite, but it adds more than just background improvements such as faster performance and improvements to both memory management and 64-bit support. Each of TopBraid Suite's components includes new features that will ease the development of Semantic Web applications and offer a wider range of features to include in those applications. The following lists some highlights:

TopBraid Composer

TopBraid Composer makes it easy to develop and integrate Semantic Web data models, SPARQL Inferencing Notation (SPIN) inferencing rules, and SPARQLMotion data processing pipelines into a standards-compliant application. New TopBraid Composer features in release 3.2 include:

OWL 2 support, with features such as property chain axioms, user-defined datatypes, and OWL 2 class axioms. Read more at VP of Product Development Holger Knublach's blog posting OWL 2 Support in TopBraid Composer.
New SPIN features such as spin:fix to implement suggestions for constraint violation fixes, easier control over rule execution, and the ability to distribute rules across devices. An important new improvement is support for user-defined magic properties in SPIN, allowing users to define rules with backward chaining. You can define a magic property, or property function, using SPARQL and then use it as the predicate in another SPARQL expression to compute new values based on the data being queried. Holger's Magic Properties with SPIN blog posting has some good examples.
A new SPARQL debugger that lets you set break points, display intermediate variable bindings, collect statistics, and see the internal algebra of SPARQL queries as they run. Holger's recent blog posting on this topic walks the reader through the underlying logic, the use of the debugger view, and the use of profiling.
More options for the display of long lists in the Properties and Classes view, making it easier to find what you're looking for quickly.
Easier packaging of a project developed with TopBraid Composer for deployment to a server running TopBraid Live for use by multiple users.
Support for the latest major release of the Eclipse platform, release 3.5.

TopBraid Ensemble

With TopBraid Ensemble, you can create web-based user interfaces for the applications you develop using Composer. End users, without using Composer, can interact with your application's data using a wide choice of graphical interface components that you customize for them. New Ensemble features in 3.2 include:

Support for pop-up windows, so that you can build wizards that lead your application's users through a series of steps.
Support for multi-page applications, letting you spread the interface for more complex applications across a series of pages, as with a tabbed interface.
A new, customizable button component that lets your application's users trigger events with a mouse click.
A new SPARQL Relay component, which can listen for events and trigger SPARQLMotion scripts in the background of your application.

TopBraid Live

TopBraid Live lets you deploy your application for use by hundreds of users with the screens that you designed with Ensemble, with screens that were custom-built using Adobe Flex, or with an automated web services interface. New features in 3.2 include:

User and group management, including optional integration with LDAP roles.
Access Control Lists for projects, folders and graphs.
The ability to host an application as a SPARQL end point.
Significant performance improvements when used with Oracle.

For a complete list of new features in TopBraid Ensemble release 3.2, see the 3.2 Release Notes.

SPIN Tutorial Available

2009-10-14T16:53:00.000-07:00

I recently wrote in my personal blog about how, since joining TopQuadrant, I've grown to appreciate how well SPARQL can serve as a rules language. SPARQL Inferencing Notation, or SPIN, lets you associate rules and constraints expressed in SPARQL with classes of triples. While you don't need to use TopQuadrant products to take advantage of SPIN, they sure make it much easier, especially if you want to use those rules as part of an application. I just finished writing a tutorial (pdf) on how to implement SPIN rules and constraints with your models using TopBraid Composer, and except for one optional detail of the tutorial, it all works with the free edition, so it's available for anyone with a Mac or Windows machine to try.

Using a small collection of data about service contracts and materials purchases, the tutorial walks you through the creation of:

your own functions, written in SPARQL and returning values of whatever type you like
inferencing: the generation of new triples based on other data (in the case of the tutorial, the generation of ISO 8601 yyyy-mm-dd format invoiceDate values from "mm/dd/yy" date values stored in the original data)
constructors: the automatic generation of a postingDate value when a new MaterialsPurchase or ServiceContract instance is created
constraints: setting up the system to alert the user to unpaid materials purchases that are more than 90 days old or unpaid service contracts that are more than 60 days old. Instead of using a lot of redundant code to achieve these two different but similar goals, the tutorial shows how to define a reusable template with a SPARQL query and pass parameters to it (in this case, the numbers 60 or 90) when using the template.

I hope the tutorial demonstrates the potential connections between SPIN technology and real-world business issues to its readers, as well as the ease of implementing it all with TopBraid Composer.

Creating and Managing Metadata about RDF statements

2009-09-07T16:51:00.000-07:00

Metadata about RDF statements can be quite useful and even required for a number of purposes. For example, let’s consider questions one may have about a statement “Washington, DC is a capital of the United States”:

What is the provenance of this statement – who said this, when did they say this?
What is the temporal scope of this statement – when did DC became a capital of the US, is it still the capital?
What is the access control for this statement – who can see it, who can change it?

Interest in this topic is evidenced by Paul Hermans' blog summarizing recent discussions on approaches to implementing such metadata http://www.proxml.be/users/paul/weblog/9d47d/A_must_read__Temporal_Scope_for_RDF_Triples.html.

We often get asked what is TopQuadrant’s recommended approach to supporting statements about statement – for versioning, for governance, etc? TopBraid Suite is fully flexible in this respect and can be used to implement any number of approaches. However, in our work, we found an approach based on RDF reification to be particularly useful.

TopBraid Composer provides a Change History view where one can see every added and deleted triples. The view is based on a small ontology called change.owl and available as part of TopBraid library, It contains a class Change and a handful of properties – added, deleted, graph and timestamp. The class Change is described as follows: “A change to an RDF Graph, encapsulating lists of added and/or deleted rdf:Statements. Additional metadata such as a timeStamp (or author or whatever) can be added”,

We often extend this model for particular applications to add the metadata required by the app, for example, author or scope. TopBraid Composer inserts change statements automatically. Every time there is a triple is added or deleted, there is a new change statement. In web applications deployed under TopBraid Live, we use a SPARQLMotion scripts that start with sml:TrackChanges module.

Sml:TrackChanges is used to implement services that shall be executed as a side effect of a change to an RDF model. In TopBraid, any script containing an instance of this module will be executed as part of each change. The output of this module is using the http://topbraid.org/change ontology, with triples describing the changes that have happened.

In other words, TopBraid listens for the changes and, when a change happens, it will trigger execution of a script(s) containing TrackChanges module. One can provide a filter to specify what type of changes a script should react to. And, as with any SPARQLMotion script, what happens when the script is triggered is up to the script designer. For example, in the Enterprise Vocabulary Management solution we use this approach to stamp every change with the author id and a timestamp and also to trigger the governance processes – send e-mails about the changes to the appropriate parties, promote approved changes, etc.

Linked Data and what it really takes to create it

2009-07-25T14:17:00.000-07:00

Recently there has been a number of popular discussions (some heated) on whether RDF is necessary for the Linked Data. For example:

http://cloudofdata.com/2009/07/does-linked-data-need-rdf/

http://www.semanticsincorporated.com/2009/07/if-linked-data-is-a-brand-it-has-big-problems-to-address.html

I believe that for the Linked Data to happen, we need a standard for representing semantic information. URIs are just an addressing scheme and do not carry any semantics.

Once one moves beyond the high level marketing statements and starts to consider specific technical details of how the data linking could work, it quickly becomes apparent that, at minimum, RDF (or something very much like RDF) is required. This is often lost on bloggers without a sufficiently deep grasp of the underlying technologies and results in somewhat unfocused and meandering discussions. Questions on how exactly URIs alone can bring the data together are kept unanswered and glossed over in a rhetoric of high level statements.

Unfortunately, this makes it hard to conduct substantive discussions. Perhaps, this is why I found the recent post on Data Reconciliation Strategies and Their Impact on the Web of Data more insightful and useful than discussions above. It makes a number of valid points and raises important questions. Having said this, I disagree with author's conclusion:

"I cannot help but think that any effort tasked to promote and increase such density will look and feel just like Freebase: carefully selecting datasets and painfully trying to reconcile as much data as possible right when it enters the system and entice a community of volunteers to maintain it, curate it and clean up eventual mistakes."

There are the following "connectivity" problems in the query proposed by the author - “the height of all towers located in Paris”:

1. Identity of a tower

2. Identity of predicate height

3. Identity of Paris

I believe it is important to encourage people to provide mappings from their vocabulary/schema to some established vocabularies. These will (and are) appear over time. Foaf:Person is a good example. Now, you do not have to and may not be able to use it directly in your data, but as you expose your data to the Linked Data Cloud (LOD) it would be a good practice to provide such mappings. Otherwise, someone else will have to do it.

This is where dbpedia becomes very useful. It has a URI for height. NASA/TopQuadrant units ontology (to be shortly released here http://www.oegov.org/) has that as well. We linked units of measure, quantities and dimensions to dbpedia. If a broader community finds this work useful, it may be that over time this ontology becomes the de-facto standard and people will link to it. But even today there is a possibility to connect between it and other relevant ontologies/datasets - through dbpedia.

The same premise applies to Paris. There are some authoritative sources that have emerged - the geonames and, again, dbpedia. If they both cross reference each other, then linking to either one should do the trick.

Tower is a trickier problem because it is a type and can be identified in a number of ways - may be you have a table (class) Tower. May be you have a column (predicate) indicating the type. May be you have a table of controlled values for building types and there is a link to the appropriate resource (this will be the case, if something is more than one type - let's say both, a tower and a lighthouse). A similar issue may apply to the height if you reify it to provide a unit of measure, for example, or a measurement date.

Some mechanisms are needed to describe less simplistic mappings. In our work SPARQL-based SPIN rules have proven to be an effective standard-based solution for more complex mappings. Overall, I believe there are only a handful of patterns that will constitute 80% of the cases. A good number of these are described above.

The key benefit of using such an approach is that people can start with their own vocabularies and then, at some later stage, add the links to dbpedia. Or they don't, and someone else does it. This freedom is lost if a system (such as Freebase) forces users to do the mapping up front. With the units ontology it was quite easy to add the mappings, and likewise it will be for most other existing data models.

Data Transformation using Semantic Web Standards

2009-07-09T19:05:00.001-07:00

I have created the presentation below in response to a recent discussion about converting XML to RDF.

A person I was talking to assumed that there was a mapping process one needed to go through before a translation of XML (or relational databases, spreadsheets, etc.) into RDF could take place.

Indeed mapping often happens, but it happens after translation. First the non RDF information is represented in RDF. Any mappings that are created are also captured in RDF/OWL - either by using constructs such as rdfs:subClassOf and owl:sameAs or, for more complex mappings, by using SPIN (SPARQL rules).

I am always surprised how often people find this approach novel and need time to understand what is going on. I guess this is because RDF is so flexible - it is quite easy to represent any data structures in RDF. And because both, data and models are represented in RDF, once imported, structural transformations are very straightforward. RDF is built for change. Other data models do not have this advantage. Hence, they require mappings before importing external data.

The presentation below explains in detail how RDF import and transformations are done including a step by step example. The benefits of the approach are also discussed.

Data Transformation using Semantic Web Standards

View more presentations from irenetq.

Presentations from the Second TopBraid User Group Meeting - SemTech2009

2009-07-06T14:08:00.000-07:00

The Second TopBraid Suite Open User Group Meeting was conducted at the Semantic Technology Conference 2009 in San Jose, CA, Thursday, June 18, 2009, 09:45 AM - 12:45 PM. The Agenda for the meeting and links to the presentations used by the speakers is given below.

User Group Meeting Agenda and Presentations
•9:45am - 9:55am
Welcome (Robert Coyne, TopQuadrant)

•9:55am - 10:30am
Keynote User Talk 1:
"Using SPARQLMotion to Execute Task Networks among Distributed Cyber Physical Systems",
Cyber-physical systems define networks of interactive sensors and actuators, grounded in the physical world. Such systems require a high degree of interoperation to achieve the system's objective. SPARQLMotion, a model-driven scripting language, has been used to achieve that level of interoperation. In addition, this approach results in greater operational redundancy among networks through distributed control. Three recent extensions to SPARQLMotion will be shared along with a motivating example for their use.
John T. Carson, Software Engineer, Lockheed Martin Aeronautics
(slides)

•10:30am - 11:00am
TopBraid - New Capabilities - Sampler 1
Presentation/demo of key aspects of TopQuadrant's Enterprise Vocabulary Management (EVM) Solution Package (just announced. Customers across a range of industries are building EVM solutions on top of the TopBraid Suite platform. In response to customer requirements, TopQuadrant is offering an EVM Solution Package of commonly needed, high-value components including models, scripts (e.g. for workflow management with approvals), and application configuration templates.
Irene Polikoff, CEO and Co-founder, TopQuadrant, Inc.
(slides)

•11:00am - 11:35am
Keynote User Talk 2:
"Managing Your Online Social Graph with TopBraid Composer"
In this short user experience session Marco will show you how he makes use of TopBraid Composer to keep track of the Semantic Web Meetup community (http://www.swnyc.org) and how he manages events, rsvps and security. In addition we will take a look at some of the editing and reporting features readily available and build into TopBraid Composer to visualize community data for evaluation and the identification of trends.
Marco Neumann, Information Scientist and CEO & Founder, KONA
(slides)

•11:35am - 12:05pm
TopBraid - New Capabilities - Sampler 2
For QA of ontologies, many users have interest in knowing key statistics or metrics regarding their models, such as the number of properties that reference each class. A simple, flexible, convenient to use solution will be demonstrated within TopBraid. By importing a special ontology, and running provided scripts, desired metrics fields get populated, and a reporting script is used to pretty-print the results to html.
Ralph Hodgson, CTO and Co-founder, TopQuadrant, Inc.
(slides)

•12:05pm - 12:40pm
User Feedback Session (conducted through lunch),
moderated by Tom Fitzgerald, Director of Sales, TopQuadrant
Notes from the session:
1. Education –tremendous demand for online training materials. Customers recommend we provide more training material (tutorials, videos, examples). Training is a key to success for many TopBraid customers.
2. Help Menu – context search and more examples –need a “Getting Started” tutorial. The tutorial would include how to use the Help Menu effectively.
3. User’s Forum – “great asset” – would like to see it expanded to include wiki format and possibly all of our products. Discsussion of how the forum can be expanded and provide more extensive support, e.g., examples of customer applications.
4. Site Spin – Tim Smith – new capability
5. Mind Mapping integration – Tim Smith talked about the value of providing a easy tool for business users to map out model requirements. The mapping could then be integrated into TBC.
6. Customer Use Cases – request to expand website to include tab for Use Cases and customer scroll

TopQuadrant appreciated and valued this input and is responding quickly, in particular by making the extensive Help provided in TopBraid Composer more accessible, and in providing more support assets through our web site. See links on the TopBraid Composer page, Support and other product pages, and this recent post. for further details and links. Additionally, two new slide sets are being provided for download the give an extensive tour of TopBraid Composer capabilities, and in depth details on importing data into RDF and transforming data with TopBraid utilities and power tools.

•12:40pm - 12:45pm
Announcements and Closing Remarks
Robert Coyne, TopQuadrant

(See also Reflections from the First TopBraid Suite User Group Meeting, May 21, 2008,San Jose, CA)

The Meaning of "semantic" - post II

2009-07-04T11:18:00.001-07:00

After writing this earlier post, I've decided to expand on my thoughts about the intersection between the Semantic Web and text mining, natural language processing, etc.

The write up was ready just in time for TopQuadrant's submission to the monthly Semantic Universe column, so I've ended up publishing it there, but here is a link http://www.semanticuniverse.com/articles-using-semantic-web-standards-improved-text-mining.html.

RDFa - a good way to provide access to your data?

2009-07-04T10:07:00.000-07:00

I have been thinking about RDFa recently. With the announcement from Google and continued support from Yahoo/Search Monkey there is an increased buzz around RDFa. So, why RDFa and what is it good for?

TopBraid have had support for RDFa as long as I can remember – at least two years now. A user can point to a page with RDFa markups and TopBraid will import them. I remember getting existed about this and wanting to mark up all our web pages with RDF. This did not happen. At least partially because RDFa’s interaction with HTML formatting tags is pretty funky – the pages become harder to maintain. Then, there was also a persistent question on why do it at all. If one wants to provide data in RDF, why not do exactly that?

Each web page on a site, could have a corresponding N3 page. There is a standard tag in HTML that can be used to refer to related information. It can be used to point to the N3 page and/or the naming convention could be the same as for the given HTML page, but with the N3 extension. In TopQuadran’t case this would be an only alternative solution since the information on our web site is not in a database (at least not yet, this is changing). If it was in a database, then a way to go would be to provide a SPARQL endpoint.
I looked at the RDFa presentation by Mark Birbeck at the Semantic Technologies conference. I did not get a chance to attend – 7:30 AM is way too early for me , but I browsed through slides. Here is an example of RDFa markup (from the presentation):

This says that there is a dc:creator relationship between the header “RDFa: Now everyone can have an API” and a string “Mark Birbeck”.

Good, but we have not given a URI to the thing we are talking about – a presentation entitled “RDFa: Now everyone can have an API”.

Absence of the URI makes it somewhat hard to talk about the presentation. Any RDFa crawler/importer would have to generate some kind of URI for it. If we used the URI to begin with, we could have simply put the triple {:RDFa_presentation dc:creator “Mark Birbeck” } into an RDF file.

One issue may be the maintenance – having 2 files to maintain. But, embedding RDFa into HTML arguably creates even worse maintenance problems. And, if RDFa markup was automatically generated (most serious publishing happens by generation, not hand crafting), then the maintenance issue is not there – it is easier to generate RDF file in addition to HTML file that it is to generate and insert markups. Not to mention that automatic generation means there is a database that could be exposed through SPARQL.

There must be something I am missing here. While I could not attend Mark Birbeck’s presentation, I just discovered he is giving a webinar on July 12^th: http://skillsmatter.com/event/ajax-ria/the-possibilities-of-rdfa-and-the-semantic-web/ng-94 . I think I will sign up and see if some of my questions get answered.

I’ll report what I learn here, so stay tuned.

Reflections on the User Feedback Panel or More Help, please ...

2009-06-24T18:19:00.000-07:00

One striking difference between user input this year compared to the last, is that this year we received considerably smaller number of new functionality requests. It seems that our users are still digesting all the new features that came out in 3.0 and in the ramp up to 3.0 - SPARQLMotion, SPIN, the new TopBraid Ensemble with the end-user composable web applications, etc.

User feedback is very important to us as a key input to the product development plans. Of course, many requirements come on a regular basis through the web user forum, but in-person interaction adds different dimensions. It is immediately clear if a requirement voiced by one user is shared by the rest. You can also quickly explore requirements in more depth. For example, at the last year User Group meeting, we received repeated requests for the support of version control and governance. It took some time to fully understand the requirements and design the solution to address them, but after close interactions with several users, version control is now available as part of the new Enterprise Vocabulary Management solution.

This year most of the requests were for better documentation and educational resources - a SPIN tutorial, more videos, example applications for TBE, etc. It seems that the richer the product suite becomes, the more we need to work on providing resources explaining how to use it.

TopBraid Composer already has a pretty good help facility. However, we've learned that not all its features are well understood by the users. After returning from the conference we have created a new page dedicated to Help http://www.topquadrant.com/products/ComposerHelp.html. It is accessible from a number of places including the download page.

Other requests already in the works include a Powerpoint tour of TBC features, example TBE application for download with, probably, a video explaining how it was developed and, yes, a SPIN tutorial (once I get a chance). Help is being updated as well, in preparation for the 3.1 release.

TopBraid User Group Meeting at SemTech2009

2009-06-18T22:15:00.000-07:00

TopQuadrant conducted its second TopBraid Open User Group Meeting at the Semantic Technology Conference 2009, on Thurs., June 18, San Jose, CA.

Topics presented and discussed included:

• User Talk 1: Using SPARQLMotion to Execute Task Networks among Distributed Cyber Physical Systems

• TopBraid - New Capabilities Sampler 1: Key aspects of TopQuadrant's Enterprise Vocabulary Management (EVMS) Solution package (just announced)

• User Talk 2: Managing Your Online Social Graph with TopBraid Composer

• TopBraid - New Capabilities - Sampler 2: Ontology Metrics & Report Generation for Quality Assurance of Knowledge Models

• User Feedback Session

More details to follow!

The Meaning of "semantic"

2009-06-18T11:27:00.000-07:00

Going through the exhibit hall of the Semantic Technologies conference one quickly notices that there are two types of vendors:

Providers of the middleware and tools that leverage Semantic Web standards (RDF, RDFS, OWL and SPARQL) to support a variety of business applications.This is an area where TopQuadrant plays.
Providers of the search and text mining products.

The word "semantic" is used when talking about both types of products. The reason for this is quite clear. Before W3C coined the term "Semantic Web", the word "semantics" was used to describe "smart software" capable of extracting some meaning from text. With the development of the Semantic Web standards, it is increasingly being used to describe semantics (schemas) of the data and to support integration across different sources and formats - databases, XML, spreadsheets, etc. As well as to support new, model driven ways to develop applications.

I wonder if this creates confusion. In fact, I know it does. At a recent customer meeting, we gave a two hour product presentation. There were many good questions afterward. I did not think there was any confusion until one of the attendees asked if our software had multi-lingual support. I explained that yes, language specific labels can be provided and information can then be displayed in a selected language. He looked puzzled, thought a little and then said "I am not talking about labels. Different languages have different semantics, like sentence structure, etc. How do you address that?"

I explained that TopBraid Suite does not directly provide text extraction. Instead, we integrate with software that has these capabilities. It can be a product like Calais which provides the extracted concepts directly in RDF or a product that provides results in XML. We can convert it to RDF in an automated way. Integration with the text mining software makes it possible to bring together (and, consequently, query) structured and unstructured information using a common RDF infrastructure.

With this, I wander if there should be two very distinct tracks at the conference? Is there some other space where these two different streams of technologies come together? If so, can the intersection be better explained and positioned?

Starfleet Day One

2009-06-16T23:07:00.000-07:00

Well, not exactly day one... We have been on a journey for quite some time now. And a few of us at TopQuadrant have been blogging for awhile. However, until today we did not have a shared company blog. Our hope is to make the blog be a place for publishing tips on using the TopBraid Suite, discussing best practices for working with the Semantic Web standards, sharing thoughts and ideas, learning from each other and from our customers.

Today is not only the first day for the blog, but also the first day of the exhibit at the Semantic Technologies 2009. A busy day, many familiar faces, many new ones as well. Some reflections:

No one asked me today to explain RDF. Such a difference from just a year ago!
Almost no one asked me "what is this technology good for". Everyone I talked to had a pretty clear idea what problems they wanted to solve. Integration was probably the most common theme.
The simplest demos seem to be the most effective ones. We've build a few fairly elaborate demos, but there is only so much one can show and absorb at a busy conference. Luckily, the demos we have are pretty versatile. I plan to post at least one of the demo apps for everyone to download and try.

Tomorrow a good amount of demoing will be done by our partners - Oracle and CTG. Both of the partner demos are focusing on medical informatics.

There is clear evidence that people are ready to adopt the technologies. Many spoke of pilot projects and the need to make IT people comfortable with new infrastructure. For TQ this was good confirmation of our TopBraid Ensemble/Live approach and product direction.

At the same time, some of the things we take for granted in our product are not always known to the market. We will therefore need to do some more communication of the "out-of-the-box" capabilities of the suite.