Reporting Summary Information of Spatial Datasets and Non-Compliance Issues Using FME Workspaces

Reporting Summary Information of Spatial Datasets and Non-
Compliance Issues Using FME Workspaces
Martin de Zuviria
GIS Architect
Mapping & Charting Establishment
April, 2016
1

Abstract
An overview of two groups of FME workspaces implemented at the Mapping and
Charting Establishment (MCE) that include the generation of reports in Excel format is
presented here.
The first group includes data validation and data compliance assessments. An example
showing Self Validation of Spatial Data Input from DND Bases using FME Server is
presented.
The second group, implemented using FME Desktop, includes the creation of summary
reports for some key datasets distributed by MCE. Two examples of FME workspaces
are presented here: the first one showing reports created for NRCan CanVec plus
charts, and the second one for Open Street Map (OSM) data delivered in FGDB format
for custom AOIs.

3
Self Validation of Spatial Data Input from DND Bases
Summary
The FME workspace presented here is part of a joint project between the Mapping and Charting
Establishment (MCE) and the Assistant Deputy Minister for Infrastructure and Environment (ADM(IE)).
This project involves managing DND real property and other spatial data provided by DND Bases across
Canada at MCE through a unique, integrated and standardized Real Property Spatial Data Warehouse
(RPSDW) containing a SQL Server database.
Data provided by DND Bases must meet the standards defined and documented by ADMIE, in terms of
data format accepted (ArcGIS FGDB), schema and attribute data types, domains and accepted values for
each feature class. An FME workspace was created to perform this data compliance assessment. This ETL
workspace was published to FME Server to make it available to DND GeoTechs from DND Bases through
the DWAN. This ETL was created together with a user’s guide, that allows GeoTechs to perform a self
validation of the DND real property and other spatial data before these data is sent to MCE to be loaded
into the RPSDW SQL Server database. This on-site QA will save time and resources and will prevent
loading of non-compliant data to the SQL Server database.
The FME workspace presented here outputs all errors, warnings and non-compliance issues to a
destination FGDB and a detailed report of all these issues to an Excel file.

ADMIE Standards: Feature Classes, attributes and
properties defined and documented

Overview of the Self-Validation Workspace

Errors and Warnings are reported to a FGDB with the same input
schema and format: A single feature may be output several times
showing only one error at a time (one error – one record) in a new
‘TypeError’ field and a timestamp is also added

Issues found are reported to an Excel file containing a summary
sheet (below) and two additional sheets for every feature class
where errors have been detected

Issues found for every feature class are reported in two separate
sheets: one containing a bar-type chart (see below) to allow a
fast view and another one showing the extent and shape of
features with errors (next slide)

Issues found for every feature class are reported in two separate
sheets: one containing a bar-type chart (previous slide) to allow
a fast view and another one showing the extent and shape of
features with errors (see below)

OpenStreetMap is the worldwide project that creates, daily updates and distributes
free geographic data for the world (OSM XML format)
OpenStreetMap represents physical features on the ground (e.g.roads or buildings)
using tags attached to its basic data structures (its nodes, ways, and relations). Each tag
describes a geographic attribute of the feature being shown by that specific node, way
or relation.
OpenStreetMap's free tagging system allows the map to include an unlimited number
of attributes describing each feature. The community agrees on certain key and value
combinations for the most commonly used tags, which act as informal standards.
However, users create new tags to improve the style of the map or to support analyses
that rely on previously unmapped attributes of the features. Short descriptions of tags
that relate to particular topics or interests can be found using the feature pages.
For more info, please visit www.openstreetmap.org
Open Street Map

OSM XML to ArcGIS Workflow
Note: this draft workflow is currently being tested for a selected CountrySteps
1. Define AOI
AOI could be a city, country, continent or the world
2. Define Source
Select source for OSM data (e.g. Planet OSM for the world or Geofabrik for a
country , continent or city)
3. Download Download data in OSM XML compressed format (*.osm.bz2)
4. Decompress Decompress downloaded data (Windows RAR may be used)
5. Run FME Workspace
Run on Desktop OSM FME Workspace (transforms OSM XML to FGDB and
Reports to an Excel file summary information of all feature’s timestamps,
individual contributors and failed -non OGC compliant – features)
6. Open MXD Open Map Document template with symbology and scale dependencies, repair
data sources, rename it and save it to folder where output FGDB is located
7. Analyze & Compare Open the Excel summary, analyze content and, if applicable, compare update of
The same AOI from previous reports
Deliver data Workflows are been tested using FME Desktop 2016.0 and ArcGIS 10.3.1
11
Tool
Web
Web
Web
Windows
FME
ArcGIS
Excel
8. Compress Compress output FGDB, Excel report and MXD fileWindows

Steps 1,2,3: Select an AOI and Export data from the Web (OSM XML
format)

Step 5: Run on FME the OSM tools transforms OSM XML to FGDB
(Tool1) and Reports to an Excel file summary (Tool 2) statistics of all
feature’s timestamps, individual contributors and failed -non OGC
compliant – features)

Step 5: The OSM workspace transforms OSM XML to FGDB and
Reports to an Excel file the summary statistics of all feature’s
timestamps, individual contributors and failed -non OGC
compliant – features)

Step 6: Open FGDB with template ArcGIS .MXD , analyze results
(including warning and errors reported in Log file) and rename *.MXD

Step 6: Open FGDB with template ArcGIS .MXD, analyze results and
rename *.MXD

Step 7: Open Excel output and analyze results (selected results for a
test AOI are illustrated below). If satisfied with all results, compress
files and deliver to User(s) (steps 8 and 9)
Year NrFeatures
2015 2972696
2014 3236623
2013 2202313
2012 2896236
2011 1935381
2010 198892
2009 104301
2008 52628
2007 11067
2006 14
Month NrFeatures
1 249865
2 225178
3 163145
4 297738
5 491939
6 398689
7 549378
8 298912
9 245497
10 52355
UniqueUser NrFeatures
jsimoes (uid = 451289) 1168479
topolusitania (uid = 380552) 779378
ViriatoLusitano_import (uid = 2330803) 603570
rtafav2 (uid = 602999) 564034
Lobinho (uid = 571263) 512478
Patrick Bous (uid = 223763) 473572
Maria Inês (uid = 1751259) 422466
oha (uid = 8204) 251820
TaedeT (uid = 57577) 241793
biscas (uid = 882362) 186720
bri g (uid = 20163) 172049
ecosdamontanha (uid = 1786621) 160102
Wolfram Sobotta (uid = 74983) 156806
UniqueUser NrFeaturesFail
topolusitania (uid = 380552) 329
TaedeT (uid = 57577) 226
chipsterxx (uid = 412095) 212
rtafav2 (uid = 602999) 121
bitormarques (uid = 702584) 102
Rui Oliveira (uid = 445220) 93
ch_stgt (uid = 834112) 89
oha (uid = 8204) 82
jpaf84 (uid = 458277) 59
biscas (uid = 882362) 54
Patrick Bous (uid = 223763) 54
Barrocas (uid = 1131231) 50
dcp (uid = 44221) 50

CanVec+ is a digital cartographic reference product produced by Natural Resources
Canada. CanVec+ originates from the best data sources covering Canadian territory and
offers quality topographical information in vector format that comply with international
geomatics standards.
CanVec+ is a key data source for many GeoTechs at DND and for other people as well.
New versions of charts are published on a periodic basis, but the user has to be aware
that this does not mean that all individual data layers in a chart are updated in the
same way. For example, a chart may have been updated in 2014 but it may contain
some power lines that were last validated long ago. Since every feature in CanVec+ has
a ‘Validity date’ (date of the data source used to create, revise or confirm an object), a
report was created for each CanVec+ 2014 chart grouping the features of each feature
class by their validation year. In addition to these individual reports, a unique table
(FGDB format) containing all individual reports was created. FME Workbench was used
for this task.
NRCan CanVec plus digital charts

Report of an individual CanVec+ 2014 chart (partial view)

FGDB table containing the reports of all individual CanVec+ 2014 charts

FGDB table containing the reports of all individual CanVec+ 2014 charts sorted
decreasingly by ‘Validation_Year’

FGDB table containing the reports of all individual CanVec+ 2014 charts sorted
increasingly by ‘Validation_Year’

Reporting Summary Information of Spatial Datasets and Non-Compliance Issues Using FME Workspaces

Recommended

More Related Content

What's hot (18)

Viewers also liked (7)

Similar to Reporting Summary Information of Spatial Datasets and Non-Compliance Issues Using FME Workspaces (20)

More from Safe Software (20)

Recently uploaded (20)

Reporting Summary Information of Spatial Datasets and Non-Compliance Issues Using FME Workspaces