Stream WatDiv Benchmark Suite

Generating streaming RDF queries involves two steps. First, the query generator will generate a set of SPARQL queries that are as diverse as possible. Second, these SPARQL queries will be equipped with time windows, and then translated to equivalent streaming RDF queries in different language.

In the first step, Stream WatDiv reuses the query generator of WatDiv to generate SPARQL queries. The query generator will perform a random walk among the datasets and generate a set of query templates with various structure and selectivity. These query templates normally contain some placeholders and will be instantiated by replaceing the placeholders with real RDF terms. Here are some query template examples.

Once the SPARQL queries are generated, they will be equipped with time windows and translated to equivalent streaming RDF queries. Currently there is no standard streaming RDF query language. Different engine proposes its own language. Stream WatDiv covers C-SPARQL [2] and CQELS [3] query language. Other languages can be added as needed.

The query generator will generate two types of queries. Pure stream queries refer to the queries that only involve streaming data, and hybrid queries refer to the queries that involve both streaming data and static data.

The reason why providing a diversified workload is important is that it can help detect engines' unusual behaviors that are not detected before. For example, following is one generated CQELS query that makes CQELS crash during query registeration. We find other similar failed queries which share the same query pattern: the static part of the query is disjoint, and triple patterns in the static part of the query have high result cardinality, so the intermediate result of the static part of the query is large. When CQELS registers a query, it will pre-calculate and cache the intermediate result of the static part of the query. This mechanism makes CQELS run out of memory when the engine registers this type of query. At the same time, this query makes C-SPARQL crash during query execution. We monitor the memory and CPU usage, after around first 15 query executions, C-SPARQL has used up all heap memory, and the CPU is busy with garbage collection. At some point, no more memory is available and the engine crashes.

SELECT ?v0 ?v1 ?v2 ?v3 ?v4 ?v5
FROM NAMED <http://dsg.uwaterloo.ca/watdiv/knowledge>
WHERE{
        STREAM <http://ex.org/streams/test> [RANGE ${WSIZE} SLIDE ${WSLIDE}] {
                ?v0     <http://db.uwaterloo.ca/~galuc/wsdbm/follows>     ?v2 .
                ?v2     <http://db.uwaterloo.ca/~galuc/wsdbm/follows>     ?v3 .
                ?v4     <http://db.uwaterloo.ca/~galuc/wsdbm/makesPurchase> ?v5 .
        }
        GRAPH <http://dsg.uwaterloo.ca/watdiv/knowledge>{
                ?v0     <http://schema.org/email>     ?v1 .
                ?v3     <http://db.uwaterloo.ca/~galuc/wsdbm/friendOf>        ?v4 .
        }
}
                

The source code of Stream WatDiv can be downloaded from here.

The steps to compile Stream WatDiv is same with WatDiv. Once the executable file is generated, following commands can be used to generate data and workload.

To run the data generator, issue the following command.

• ./watdiv -sd <model-file> <static-scale-factor> <stream-scale-factor> <rand-seed>

You will find a model file in the model sub-directory where Stream WatDiv was installed. The static scale factor and stream scale factor control the size of static dataset and stream dataset seperately. Running the data generator with same random seed, together with same model file, static and stream scale factor will generate the same datasets. The stream dataset is written to a file named stream.txt. At this stage, triples in the stream.txt have not been annotated with timestamps.

To annotate streaming data with timestamps, issue the following command.

• ./watdiv -ts <source-file> <dest-file> <stream rate>

Source-file is the stream.txt generated in the previous step. Dest-file defines the name of the file where the stream dataset with timestamps is written to. Timestamps are determined by the stream rate. A stream rate of 1 means 1 triple per second. Higer stream rate results in smaller interval between two consecutive timestamps.

To generate the workload, issue the following command.

• ./watdiv -sq <model-file> <static-dataset> <stream-dataset> <max-query-size> <query-count> <constant-per-query-count>

Running the query generator needs to cite model file, static dataset and stream dataset. The query generator conducts a random walk in the stream dataset and static dataset. Each query may have different number of triple patterns. Max-query-size constrains the maximum number of triple patterns in each query. Query-count specifies the number of queries to be generated. Users can also specify the number of placeholders in a query through constant-per-query-count.

[1] Güneş Aluç, Olaf Hartig, M Tamer Özsu, and Khuzaima Daudjee. Diversified stress testing of RDF data management systems. In Proc. 13th International Semantic Web Conference, pages 197-212, 2014.

[2] Davide Francesco Barbieri, Daniele Braga, Stefano Ceri, Emanuele Della Valle, and Michael Grossniklaus. Querying RDF streams with C-SPARQL. ACM SIGMOD Record, 39(1):20-26, 2010.

[3] Danh Le-Phuoc, Minh Dao-Tran, Josiane Xavier Parreira, and Manfred Hauswirth. A native and adaptive approach for unified processing of linked streams and linked data. In Proc. 10th International Semantic Web Conference, pages 370-388, 2011.