| Exercise | ChangeDetector and ArcGIS Online |
| Overall Goal | To update an External AGOL layer with only those records that have been updated in our internal business system. |
| Demonstrates | To show how the ChangeDetector transformer can be used to publish only those records that have changed to an external AGOL layer. |
| Completed Workspace | C:\FMEData2020\Workspaces\IMGS_ArcGIS\ChangeDetection_complete.fmw |
This exercise will look at how we can reduce the volume of data that needs to be uploaded to ArcGIS Online from an internal business system. The ArcGIS Online layer represents the current external view of the data whilst the Geodatabase represents the latest information held internally to the department. A nightly replication process keeps the external view up dated but we do not need to publish all of the data if only a small number of records are changing.
The ChangeDetector transformer can be used to identify those records that are changing and update the external layer with just those records that have been modified, deleted or have been newly created.
1) Create a New Blank Workspace
Open FME Desktop and select New to create a new blank canvas.
2) Read in Source Data from AGOL and Local Data Mart
Add a File Geodatabase reader to read the local data into the workspace. Load in the following dataset setting the Coord System to EPSG:2157
C:\FMEData2020\Data\IMGS\Planning\planning.gdb
This will load in the latest planning data from our internal business system that we want to compare with the data on AGOL.
Having added the local data we can add the AGOL data. To do this add an ESRI ArcGIS Online (AGOL) Feature Service reader.
Under Parameters point the reader to the planning data that we uploaded to ArcGIS Online in the earlier exercise.
This will bring in the latest internal data alongside the currently published layer as below. It is best practise to book mark the feature types to make it easy to identify which is which for new user to your workspace.
Run the workspace to populate the feature cache for each dataset and review the datasets.
6) Add the ChangeDetector
Add the ChangeDetector transformer and connect the AGOL layer to the Original port and the local Geodatabase to the Revised port.
Open the ChangeDetector and set the Update Detection Key Attribute to Planning_Reference. This will use the unique planning reference identifier when comparing the two datasets to show any updates that have taken place to existing records.
Under Check Attributes set the Attribute Matching Strategy to Match Selected Attributes and from Selected Attributes select everything except ObjectID and pkid. These are both locally managed identifiers and liable to change between datasets they are also not part of the core data so we can ignore them.
Finally under the Generate Detailed Changes List for Update Detection set the List Name to _lstChanges.
7) Run the Change Detector and Review Output
Run the ChangeDetector transformer, how many features have changed?
As you can see from the above example all of our data is Updated this seems unlikely!
From reviewing the data in the Visual Preview panel and the Feature Information window we can see from the Updated list created by the transformer that the dates seem to be an issue.
As can be seen from the above the update is picking up differences in the date-time formatting. Even though the values are the same the AGOL layer includes time.
To correct this we can include a DateTimeConverter transformer on each input to format the datetime values into a consistent format.
In both the DateTimeConverters set the DateTime Attributes to Decision_Date and Registration_Date and the Output Format too %Y%m%d.
Run the Workspace again, but we still have a problem as can be seen below. The GeoDatabase still has null records in that we have not fixed. To ensure we are consistent with the AGOL layer we can set the NULL values to the same default date.
Note: We could also set the AGOL layer to match our GeoDatabase. Which method you use doesn't matter as long as they are consistent and typically the business logic will determine which is the better to align to.
Add a NullAttributeMapper between the GeoDatabase feature type and its DateTimeConverter as below.
Configure the parameters as shown below.
8) Round the Coordinates
Running the ChangeDetector again still throws up a number of updated records that is higher than expected. Investigating the data a number of these come from the geometry changing but the data looks to have 10 decimal places of precession. Given the nature of the data we know its only accurate to 1m so we can round the coordinates off to 0 dp.
Add a CoordinateRounder transformer to each feature stream and round the coordinates off to 0 in both the X & Y.
Set both as follows.
As the data we are using is in a meter based projection system any precision under around 2 decimal places is unlikely to be improving the locational accuracy of the data and can be rounded up.
9) Fix the Attribute Lengths
There is one last issue with our Geodatabase data, and that is the field length for the AGOL layer is set to a maximum of 254 characters whilst the Geodatabase is 2048 so some of the data will have been truncated. To mimic this truncation and stop it being falsely flagged as change we can truncate the source data for comparison.
Add an AttributeManager before the ChangeDetector.
Then configure it as per the table below.
| Output Attribute | Attribute Value | Action |
|---|---|---|
| Description | @Left(@Value(Description),254) | Set Value |
| Location | @Left(@Value(Location),254) | Set Value |
Running the ChangeDetector now should show just real change between the datasets.
The ChangeDetector has reduced the volume of data that needs to be updated from 8338 features to 572 which will optimise the remote writes required and make the update of the AGOL layer more robust.
10) Updating the AGOL Layer
Now we have identified the change we can update the AGOL layer with just the changed records.
The ChangeDetector adds the relevant fme_db_operation attribute to each feature output from the Updated, Inserted and Deleted ports. This is a special FME attribute that FME uses to determine at a feature by feature level what action to perform when populating a feature type. It over rides any feature type or dataset setting.
Having filtered the change we now need to prepare it for being loaded into the AGOL layer.
11) Preparing the Inserts
The data to be inserted is simple to add however it lacks a GlobalID and this is a required field on our AGOL layer so can not be missing. We can create a suitable ID using the UUIDGenerator transformer.
Add a UUIDGenerator transformer and connect it to the Inserted port of the ChangeDetector. Then configure its New UUID Attribute parameter as pkidas below.
12) Preparing the Updates
The data from the Updated port contains the latest values however it is missing the PKID column that AGOL uses to update the layer. In order for the update operation to work we need to add the correct PKID back on to our data.
As our data shares a unique key in Planning_Reference we can use this to join our two datasets together and add the PKID from the AGOL features to our GeoDatabase features.
Add a FeatureJoiner and connect the Left port to the Updated port on the ChangeDetector and the Right port to the CoordinateRounder on the AGOL feature stream as shown below.
Set the FeatureJoiner up so the JoinOn fields for the Left and Right are both Planning_Reference as shown below.
As we are joining data from the older AGOL features to the current Geodatabse features we want the geodatabase features to be preserved even though they share the same names. The Attribute Conflict Resolution does that here, where attribute names are common between the joining feature streams FME will keep the Left (Geodatabase) attribute values and discard the right (AGOL) values.
13) Add AGOL Writer and Output Feature Type
Add an Esri ArcGIS Online (AGOL) Feature Service writer and configure as shown below.
When prompted to Select Feature Type choose the ArcGIS Online layer.
Connect the three feature streams to the output layer. The Joined port from the FeatureJoiner, the Output port from the UUIDGenerator and the Deleted port from the ChangeDetector all connect to the planning_applications feature type.
14) Run the workspace to upload the changes
Run the workspace to make the changes to the planning_applications layer. If you review the data in AGOL you will now see there are 8338 features in the layer rather than the 8308 their originally were.
Try rerunning the workspace to see if it finds any change between the newly published layer and the Geodatabase. In theory as the two layers should now be in sync it will not find any difference between them.
When running the ChangeDetection make sure to turn off Feature Caching and be aware that the cache is obviously a snapshot of the data if it changes on the live systems the cache will not be in sync and can cause problems. So it is best in this case to turn it off and then run the comparison so FME rereads the AGOL layer and subsequent transformers cache are refreshed.
| CONGRATULATIONS |
|
The task above demonstrates how you can use the ChangeDetector to identify change between two datasets and to manage any changes that might be introduced because of the differences between two systems |