Web-Based Datasets
There can be no doubt of the trend towards storing data in the cloud, including spatial data. For that reason, FME has comprehensive tools for reading datasets that are web-based.
Simple URL Selection
The easiest way to read a web-based dataset is to simply paste the URL into the source dataset parameter.
Here the workspace author is adding a reader to read a MapInfo TAB dataset directly from an open data catalogue stored on Amazon S3.
| FME Lizard says… |
| A folder-based dataset must be compressed to a single zip file for FME to read it from the web like this; the above is a perfect illustration of that requirement. Datasets can be read from non-zipped datasets, but only when the dataset consists of a single file (such as an AutoCAD DWG file). |
The URL entered into a source dataset field may refer to any resource on a web-based file storage system. For example, here a user is reading the same MapInfo TAB dataset directly from a Dropbox link:
The key is to force the web service to provide a direct link to the file, rather than to their own web interface. For example, too cause Dropbox to render data, you should set the dl query parameter to 1 in the URL, as in the above screenshot, instead of the default value of 0 (zero).
However, there are better ways to read data from a web service...
| FME Lizard says… |
| If data is read from a website data needing authentication, most readers have parameters to enter such information: |
Web Services
Besides being able to read from a URL, FME can also directly access certain web services to read data. Direct access to a web service is achieved with functionality inside FME called Web Connections.
Web Connections are created by selecting Tools > FME Options > Web Connections on the FME Workbench menubar and clicking the plus button in the Web Connections dialog:
There are a large number of web services to which a connection can be made - and the Manage Services button allows you to integrate any other web service you require - but the key ones are:
| • Amazon S3 | • FME Server |
| • Autodesk A360 | • Google Drive |
| • Box | • HDSF |
| • Dropbox | • Microsoft OneDrive |
These are key types because they are capable of storing data in a way that can be accessed directly from the Add Reader dialogs:
In the above screenshot, an author is adding a MapInfo dataset from Amazon S3. This action opens a dialog in which they can browse Amazon S3 for the required file:
The reader is then added to the workspace and functions just as any other.
Note that if you wish to use a FeatureReader transformer instead of a reader, then the same "Select File From Web" option is available.
| WARNING |
| Usually there is an authentication step that needs to be carried out when adding a new web service connection into FME, and if the workspace is published to FME Server or simply copied to another FME Desktop installation, then the same authentication needs to be carried out in (or copied to) the new location. |
Connector Transformers
Besides being able to direct a reader to a web service, various transformers exist that can read files. These are similar to the list supported by readers:
| • AzureBlobStorageConnector | • DropboxConnector | • ProjectWiseWSGConnector |
| • AzureFileStorageConnector | • FMEServerResourceConnector | • S3Connector |
| • AutodeskA360Connector | • GoogleDriveConnector | • SalesForceConnector |
| • BoxConnector | • HDFSConnector | • SlackConnector |
| • CesiumIonConnector | • OneDriveConnector |
These transformers don't read data in the traditional FME sense. Instead, they copy a selected file and either add its contents to an attribute or download the file to the local filesystem.
Here, for example, an author is using a Creator transformer to trigger the reading of a file from Google Drive:
The contents of the file have been added to an attribute that can then be processed as required. For example, maybe it is a snippet of XML that can be decoded with an XML transformer.
| TIP |
| Connector transformers can also upload a file, list the contents of a web service, and delete files from that service. |
Given the "Select File From Web" tool on readers, Connector transformers are not generally intended for reading source data; although they could be used to download a file that is subsequently read using a FeatureReader.
Another use would be to retrieve a list of files that are then read directly using a FeatureReader.
However, perhaps the more likely use is to transfer written data to the web. Writers don't have an equivalent "Select File From Web" option, so the best alternative is to write data with a FeatureWriter transformer and then use a Connector transformer to transfer that data to a web service of choice.