| Exercise | Publish to ArcGIS Online in ITM |
| Overall Goal | To show how to publish to ArcGIS Online in ITM from FME |
| Demonstrates | Connect to ArcGIS Online and Publish a Layer in Irish Transverse Mercator using FME |
| Completed Workspace | C:\FMEData2020\Workspaces\IMGS_ArcGIS\Publish_To_AGOL.fmw |
This exercise will look at adding a layer to ArcGIS Online and setting it to the correct WKID (Coordinate System) for Irish Transverse Mercator. As part of the publication step the data can be validated to ensure that it correct.
Note this exercise will require an ArcGIS Online account if you do not have one you can create a trial account here: Create Trial Account
1) Create a New Blank Workspace
Open FME Desktop and select New to create a new blank canvas.
2) Add CSV Reader
Add a CSV Writer to the workspace and add the following data layer. The data is in Latitude and Longitude so set the Coord. System to EPSG:4326
C:\FMEData2020\Data\IMGS\Planning\Planning_Applications_12082020.csv
FME will recognise the latitude and longitude fields in the data and automatically create a point geometry from these, but you can confirm this under Parameters.
3) Inspect the Data
Read the data using the Run Just This option on the CSV feature type and review the data.
Pay particular attention to the Data fields and sort the fields to identify any potential issues.
4) Fix Dates:
You will notice from your inspection above that both the Registration Date and Decision Date containing hard coded null values. As we want to treat these columns as date fields these will need to be fixed.
Add a TestFilter to the canvas connected to the CSV reader.
In the TestFilter configure two tests as follows
| Test | Output Port Name |
|---|---|
| Registration_Date = NULL | RegistrationDate_Null |
| Decision_Date = NULL | DecisionDate_Null |
As Unfiltered does not make much sense in this case rename the Else Output Port to Passed.
5) Reason for Failure
Whilst we expect data the has only recently been added to not have a decision date we would expect all records to have a RegistrationDate. These records we will discard, but will keep a record of those records not loaded for future investigation.
As we want to capture the reason for data being filtered out in our error record we will add an attribute containing the reason to the data.
To quickly add an attribute we can right mouse click on the RegistrationDate_Null name on the TestFilter transformer and select Add Attribute.
In the pop up dialogue create an attribute as shown below.
6) Resolve Null Decision Dates
As we need a date value for our output we will add a dummy date to represent a null date. In this case we will use 20990101.
Add an AttributeCreator transformer connected to the DecisionDate_NULL port of the TestFilter.
Within the transformer select the Decision_Date attribute and set the Attribute Value to 20990101 as shown.
7) Format Dates
To ensure the dates are in the correct format for AGOL to recognise we will format them to follow the ISO standard Year-Month-Day format. To do this add an DateTimeConverter and connect it to the Passed port of the TestFilter transformer and the Output port of the AttributeCreator as below.
Configure the transformer so the Datetime Attribute is set to Decision_Date and Registration_Date, and the Output Format is %Y-%m-%d. This can be selected from the drop down as ISO Date.
If you have not already done so run the workspace and ensure that the data filters correctly and passes the DateTimeConverter. Review the features and the new ISO formatted date structure will be shown.
8) Quality Check the Data
As this data will be used in future analysis we need to ensure that the data is valid and that its unique identifier is actually unique. To do this we can use the AttributeValidator transformer connected to the DateTimeConverter.
Within the AttributeValidator configure the following rules.
| Attributes to Validate | Validation Rule | Rule Configuration |
|---|---|---|
| Planning_Reference | Not Null | *NotUsed* |
| Planning_Reference | Unique | *NotUsed* |
| Decision_Date Registration_Date | Type | Date |
| Decision_Date Registration_Date | Maximum Length | 254 |
9) Reproject Data to Irish Transverse Mercator
As we want our layer to be in Irish Transverse Mercator in ArcGIS Online to match our corporate data repository but it is currently in Latitude/Longitude(WGS84) we need to convert it. To do this we can use the ProjReprojector transformer.
Configure the transformer as shown below.
10) Create ArcGIS Online Connection
In order to add the Output Feature Type we need a connection to ArcGIS Online as this is the first time we have connect to AGOL using this machine we need to create a new connection.
Go to FME Preferences and select Web Connections from the Menu. At the bottom left of the Web Connections panel click on the + icon to add a new connection.
From the Web Service menu select ESRI ArcGIS Online
Rename the Connection to something that will identify the connection. The name does not matter but the more descriptive it is the easier it is to identify in the future or for other users to identify. Having named the connection click Authenticate.
This will present you with ESRI's ArcGIS log-in OAuth page. Complete your username and Password and click sign in to complete creating the connection.
Having successfully completed the registration, FME will add the connection to the list of available web connections. Click OK to close the preferences window and return to the canvas.
11) Add an AGOL Writer and Feature Type
Having created our ArcGIS Online connection we can now add our AGOL Writer and feature type. Add an AGOL Writer.
Prior to creating the feature type we need to configure the parameters to ensure the correct WKID is set for Irish Transverse Mercator. Click on the Parameters button to open the parameters window.
Set the Feature Service to FMETraining this is the name of the database containing all of the layers uploaded to AGOL by FME within this workspace. As we have not published any data yet we can enter any name but if we want to publish to an existing feature service we can select it here using the select option to the left of the parameter.
The WKID for ITM 2157 so set this as the Target WKID value.
Finally under the Advanced settings set Use Global IDs to Yes this will configure the layer for use with GlobalIDs that can be used within the ArcGIS Online Platform.
Continue to set up the output feature type giving it a Layer Name of planning_applications and ensuring the Geometry is set to arcgisonline_point.
12) Configure Attributes
Configure the attributes as shown below, setting the date fields to use esriFieldTypeDate and creating a new pkid attribute of type esriFieldTypeGlobalID. Do not worry that this has nothing connected to it it is not required to populate it for this exercise.
13) Add an Error Log
We created the error attribute earlier but have so far not used it, and the AttriuteValidation has failed attributes we have not captured either. To fix this add a CSV writer and add a new CSV feature type layer. Name the CSV Feature Type planning_errors.
Configure the attributes as shown below. The ErrorID attribute is a unique id for each error so we can use the Count FME Feature Function. This can be selected from the Value Drop down and then the Arithmetic Editor and finally select Count from the right hand panel.
Having added the Feature Type connect it to the TestFilter Registration_Date_Null port, the Rejected Port on the DateTimeConverter and the Failed port of the AttributeValidator. This will ensure all errors in the data flagged up in the workspace will be captured in this log.
To avoid messy lines in the Workspace and to make it more legible its possible to hide the connections we've made by right mouse clicking on each line and selecting Hide as shown below.
As these lines have a clear purpose and end point we can use hide, however if the data was part of the main workflow it might be more appropriate to use the Create Tunnel option as this allows for a better described way of hiding lines that can indicate via annotation where data is flowing to in the workspace. This makes it easier to understand and manage. As the purpose of the rejected ports here is clear it is sufficient just to use Hide.
14) Run Workspace
Run the workspace and review the CSV file to inspect that the errors have been written to this file.
Then open ArcGIS Online and sign in using the same credentials used in the connection created above.
In ArcGIS Online go to the Content page and you should see you FMETraining output.
Clicl on the FMETraining Feature Layer (hosted) and from the second page click on view map in the top right to view the output.
From the Layer Overview page select the View option at the bottom right of the page by the Service URL to bring up the overview of the layer.
Within this page you can confirm that the data has been written as Irish Transverse Mercator (2157).
Lastly you can read the data back into FME using the View Written Data option on the AGOL feature type. Checking the features in the map and Feature Information window of the Visual Preview should show the data in ITM coordinates / CRS.
| CONGRATULATIONS |
|
Whilst a basic task commonly performed this exercise shows how we can manage the flow of data within FME to ensure that it is valid before publishing and also how we can make sure its in the required local coordinate system for use in our downstream applications. |