Distributed Query Processing for Federated RDF Data Management
3 likes814 views
PhD defense talk about SPLENDID, a state-of-the-art implementation for efficient distributed SPARQL query processing on Linked Data using SPARQL endpoints and voiD descriptions.
More Related Content
What's hot (20)
Similar to Distributed Query Processing for Federated RDF Data Management (20)
Recently uploaded (13)
Distributed Query Processing for Federated RDF Data Management
- 1. Distributed Query Processing for Federated RDF Data Management Olaf Görlitz 07.11.2014
- 2. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 2 The Linked Open Data Cloud Use as one large database!
- 3. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 3 Life Science Scenario Find drugs for nutritional supplementation SELECT ?drug ?id ?title WHERE { ?drug drugbank:drugCategory category:micronutrient . ?drug drugbank:casRegistryNumber ?id . ?keggDrug rdf:type kegg:Drug . ?keggDrug bio2rdf:xRef ?id . ?keggDrug purl:title ?title . }
- 4. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 4 Linked Data Querying Paradigms Data Warehouse Link Traversal Federation
- 5. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 5 Linked Data Querying Paradigms Requirements Data Warehouse Link Traversal Federation Query Expressiveness Schema Mapping Data Freshness Result Completeness Scalability Flexibility Availability Performance
- 6. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 6 Contributions Large Scale Information Retrieval RDF Federation & Query Optimization Benchmarking RDF Federation Systems PINTS Peer-to-Peer Statistics Management SPLENDID Distributed SPARQL Query Processing SPLODGE Linked Data Query Generation Görlitz, Staab: SPLENDID: SPARQL Endpoint Federation Exploiting VOID Descriptions. COLD'11 Görlitz, Thimm, Staab: SPLODGE: Systematic Generation of SPARQL Benchmark Queries for Linked Open Data. ISWC'12 Görlitz, Sizov, Staab: PINTS: Peer- to-Peer Infrastructure for Tagging Systems. IPTPS'08
- 7. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 7 SPLENDID Federation Federated Databases Federated RDF ● Relational Schema ● Implicit Schema, Ontologies ● Specific Data Wrappers ● SPARQL endpoints ● Rich Data Statistics ● Limited Statistics (voiD) Execute complex SPARQL queries over federated RDF data sources
- 8. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 8 SPLENDID Federation SPARQL Query Source Selection Query Optimization Query Execution SELECT ?drug ?id ?title WHERE { ?drug drugbank:drugCategory category:micronutrient . ?drug drugbank:casRegistryNumber ?id . ?keggDrug bio2rdf:xRef ?id . ?keggDrug rdf:type kegg:Drug . ?keggDrug purl:title ?title . } ⋈?drug ⋈?id ⋈?keggDrug ⋈?keggDrug ? drugdrugbank :drugCategory category: micronutrient ? drugdrugbank :casRegistryNumber ?id ? keggDrugrdf : type kegg: Drug ? keggDrugbio 2rdf : xRef ?id ? keggDrugpurl: title? title
- 9. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 9 Source Selection Objectives SPARQL Query Source Selection Query Optimization Query Execution Determine all relevant data sources DARQ FedX SPLENDID ● Explicit 'capabilities' ● Query restrictions (bound predicates) ● ASK queries + caching many (initial) requests ● Sub query aggregation ● VoiD descriptions + ASK queries ● Sub query aggregation
- 10. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 10 voiD voiD voiDvoiD Source Selection Example SELECT ?drug ?title WHERE { ?drug drugbank:drugCategory category:micronutrient . ?drug drugbank:casRegistryNumber ?id . ?keggDrug rdf:type kegg:Drug . ?keggDrug bio2rdf:xRef ?id . ?keggDrug purl:title ?title . } → KEGG, DBpedia, ChEBI → KEGG → DrugBank SPARQL ASK → DrugBank, ChEBI → KEGG
- 11. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 11 Source Selection Result ⋈?drug ⋈?id ⋈?keggDrug ⋈?keggDrug ? drugdrugbank :drugCategory category: micronutrient ? drugdrugbank :casRegistryNumber ?id ? keggDrugrdf : type kegg: Drug ? keggDrugbio 2rdf: xRef ?id ? keggDrugpurl: title? title
- 12. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 12 Query Optimization SPARQL Query Source Selection Query Optimization Query Execution Find best (fastest) query execution plan DARQ FedX SPLENDID ● Dynamic Programming ● Custom Statistics ● Only bound predicates ● Bind Join ● Join Order Heuristics ● No Statistics ● Join Chains ● Bind Join ● Dynamic Programming ● Extended voiD statistics ● Bind + Hash Join
- 13. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 13 Dynamic Programming ● iterate over all possible execution plans ● compare cost (execution time) BindJoin, HashJoin ⋈?drug ⋈?id ⋈?keggDrug ⋈?keggDrug ? drugdrugbank :drugCategory category: micronutrient ? drugdrugbank :casRegistryNumber ?id ? keggDrugrdf : type kegg: Drug ? keggDrugbio 2rdf : xRef ?id ? keggDrugpurl: title? title Cost Model costsend−query costreceive−tuple card(R(qi ))
- 14. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 14 Cardinality Estimation ⋈?drug ⋈?id ⋈?keggDrug ⋈?keggDrug ? drugdrugbank :drugCategory category: micronutrient ? drugdrugbank :casRegistryNumber ?id ? keggDrugrdf: type kegg: Drug ? keggDrugbio 2rdf : xRef ?id ? keggDrugpurl: title? title
- 15. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 15 Cardinality Estimation (Triple Pattern) cardd (s, p,o) = |d|⋅seld(s)⋅seld (p)⋅seld(o), d∈D Assuming independence of s, p ,o cardd (?,p,?) cardd (s ,? ,?) cardd (?,?,o) cardd (s ,? ,o) cardd (s ,p,?) cardd (?,p,o) cardd (?,?,?) cardd (s,p,o)= voiDd →|d| = 1 = voiDd →p = voiDd→|d| voiDd →|s| = voiDd→|d| voiDd →|o| = 1 = voiDd →p voiDd→|sp| = voiDd →p voiDd→|op| cardd (?,rdf: type,T) = voiDd →T
- 16. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 16 Cardinality Estimation (Basic Graph Pattern) Star Pattern Path Pattern kegg:Drug ?keggDrug rn:R01786 ?title rdf:Type purl:title bio2rdf:xRef drugbank:Drug ?keggDrug rdf:Type owl:sameAs ?drug kegg:Drug rdf:Type cardd * (P1 ⋈ P2 ⋈ P3) = min(cardd (P1),cardd (P2)) ⋅ voiDd →p3 voiDd →|sp3 | cardd ,d ' ~ (P1 ⋈ P2) = cardd (P1)⋅cardd ' (P2) ⋅seld ,d ' (P1 ⋈ P2)
- 17. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 17 Query Optimization SPARQL Query Source Selection Query Optimization Query Execution ⋈?drug ⋈B(? id) ⋈?keggDrug ⋈H(? keggDrug) ? drugdrugbank :drugCategory category: micronutrient ? drugdrugbank :casRegistryNumber ?id ? keggDrugrdf : type kegg: Drug ? keggDrugbio 2rdf: xRef ?id ? keggDrugpurl: title? title
- 18. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 18 Evaluation Methodology Compare with state-of-the-art federation systems – Use Multiple linked datasets – With representative characteristics – Execute 'typical' SPARQL queries – In a reproducible benchmark setup FedBench
- 19. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 19 Evaluation Results
- 20. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 20 Conclusion ● Federation for Linked Open Data – Database + Semantic Web technology – Efficient Distributed Query Processing – Extension of voiD statistics ● Query generation for Federation Benchmarks ● Efficient statistics management in P2P networks
- 21. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 21 Thank You
- 22. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 22 VoiD Descriptions/Statistics } } } } General Information Basic statistics triples = 732744 Type statistics chebi:Compound = 50477 Predicate statistics bio:formula = 39555
- 23. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 23 VoiD statistics extension
- 24. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 24 State of the Art DARQ AliBaba FedX SPLENDID Statistics ServiceDesc – – VoiD Source Selection Statistics (predicates) All sources ASK queries Statistics + ASK queries Query Optimization DynProg Heuristics Heuristics DynProg Query Execution Bind join Bind join Bound Join + parallelization Bind Join + Hash Join
- 25. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 25 SPARQL limitations ● Query protocol ● Only SPARQL endpoints ● Endpoint limitations – SPARQL version – Result size – Data rate – Availability
- 26. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 26 Join Implementation R1 R2 R1 R2 ⋈B ⋈H Bind Join Hash Join ?id ?y 1 42 2 13 3 20 4 50 5 3 ?id ?x 1 'A' 1 'G' 4 'A' 7 'A' 7 'C'
- 27. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 27 Join Cost Model R(q1) R(q2 ') R(q1) R(q2) ⋈B ⋈H Bind Join Hash Join cost⋈B (q1, q2) = |R(q1)|⋅costtuple + |R(q1)|⋅costquery + |R(q2')|⋅costtuple cost⋈H (q1, q2) = |R(q1)|⋅costtuple + |R(q2)|⋅costtuple + 2⋅costquery
- 28. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 28 SPARQL Semi Join
- 29. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 29 SPLENDID Architecture
- 30. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 30 FedBench Datasets ● Cross Domain ● Life Science ● Linked Data
- 31. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 31 Data Source Selection: Requests
- 32. Olaf Görlitz: Distributed Query Processing for Federated RDF Data Management 07.11.2014 Slide 32 Conclusion Linked Open Data voiD Web-scale Query Processing SPLENDID