FME and Web Service Security
As the web has grown the types of data shared and the value of that data has grown as well. Security is often a critical component of a modern web service and protects the value of the data stored within the service. The value and sensitivity of the data are the key drivers to how well protected a web service should be.
Web services may offer their data freely without any authentication and this is typically the case with services that provide data that is in the public domain. Conversely data that is highly sensitive such as banking information will be protected by many layers of security.
In practise this means that the more valuable or sensitive the information you are trying to retrieve from an API the more complicated the security will be to deal with.
FME supports a number of different methods for accessing secure web services as shown below including its own Web Connection method.
Password Based Authentication:
Basic (HTTP Basic authentication)
This is the default choice in FME, and the only method in widespread use and supported virtually everywhere.
In the context of an HTTP transaction, basic access authentication is a method for an HTTP user agent (e.g. FME or web browser) to provide a user name and password when making a request. In basic HTTP authentication, a request contains a header field in the form of Authorisation: Basic
Base64 is not a form of encryption and is considered the same as sending the user name and password in clear text, making it non-secure unless used in conjunction with TLS (HTTPS).
HTTP Basic makes the assumption that the connection between the client and server is trusted and secure, be aware that any credentials passed from client to server can be easily intercepted through an insecure connection.
Digest (HTTP Digest authentication):
Digest access authentication is one of the agreed-upon methods a web server can use to negotiate credentials, such as username or password, with a user's web browser. This can be used to confirm the identity of a user before sending sensitive information, such as online banking transaction history. It applies a hash function to the username and password before sending them over the network.
Digest authentication is therefore more secure over public networks than the Basic method.
It is a challenge-response scheme that is intended to replace Basic authentication. The server sends a string of random data called a nonce to the client as a challenge. The client responds with a hash that includes the user name, password, and nonce, among additional information. The complexity this exchange introduces and the data hashing make it more difficult to steal and reuse the user's credentials with this authentication scheme.
NTLM (HTTP NTLM authentication):
This is a proprietary protocol invented and used by Microsoft and typically not found outside of Microsoft services or environments. It uses a challenge-response and hash concept similar to Digest to prevent the password from being eavesdropped.
NTLM uses Windows credentials to transform the challenge data instead of the unencoded user name and password. NTLM authentication requires multiple exchanges between the client and server. The server and any intervening proxies must support persistent connections to successfully complete the authentication.
Web Connections:
Web connections are an FME specific functionality and are a sub type of FME's Named Connections. “Named connections” refers to the ability to define a database or web connection only once, then subsequently use it in your FME Workspaces—saving time and without needing to store the password within the workspace.
Web Connections can be a standard HTTP Authentication Based on the connection types above or can be configured for a specific service type such as ESRI ArcGIS Online, Microsoft Sharepoint, or FME Server.
They are managed through FME Options under Web Connections.
Whilst some of the connection like the HTTP Authentication the power in web connections is simplifying access to OAuth based services within FME. Rather than us as users having to understand and build into our workspaces the functionality to make a request based on OAuth a web connection can use an existing template and native FME functionality to handle the request.
FME ships with web connections pre-configured for a number of common service platforms like ESRI, and Microsoft however there is also the ability to add in community defined definitions that have been made available on the FME Hub.
OAuth Connections:
Whilst FME simplifies using OAuth connections it helps to have a basic understanding of how they function and how they work with FME Desktop and FME Server.
OAuth is a term people may have heard of but many people don't fully understand and there is a lot of confusion around what OAuth actually is.
Some people think OAuth is a login flow (like when you sign into an application with Google Login), and some people think of OAuth as a “security thing”.
What Is OAuth?
At a high level, OAuth is not an API or a service: it’s an open standard for authorisation and anyone can implement it.
More specifically, OAuth is a standard that apps can use to provide client applications with “secure delegated access”. OAuth works over HTTPS and authorises devices, APIs, servers, and applications with access tokens rather than credentials.
OAuth2 is the current standard and most modern web services will follow this specification.
Why OAuth?
OAuth was created as a response to the direct authentication pattern. This pattern was made famous by HTTP Basic Authentication, where the user is prompted for a username and password. Basic Authentication is still used as a primitive form of API authentication for server-side applications: instead of sending a username and password to the server with each request, the user sends an API key ID and secret. Before OAuth, sites would prompt you to enter your username and password directly into a form and they would login to your data (e.g. your Gmail account) as you. This is often called the password anti-pattern.
To create a better system for the web, federated identity was created for single sign-on (SSO). In this scenario, an end user talks to their identity provider, and the identity provider generates a cryptographically signed token which it hands off to the application to authenticate the user. The application trusts the identity provider. As long as that trust relationship works with the signed assertion, you’re good to go.
OAuth is a delegated authorisation framework for REST/APIs. It enables apps to obtain limited access (scopes) to a user’s data without giving away a user’s password. It decouples authentication from authorisation and supports multiple use cases addressing different device capabilities. It supports server-to-server apps, browser-based apps, mobile/native apps, and consoles/TVs.
You can think of this like hotel key cards, but for apps. If you have a hotel key card, you can get access to your room. How do you get a hotel key card? You have to do an authentication process at the front desk to get it. After authenticating and obtaining the key card, you can access resources across the hotel.
To break it down simply, OAuth is where:
- App requests authorisation from User
- User authorises App and delivers proof
- App presents proof of authorisation to server to get a Token
- Token is restricted to only access what the User authorised for the specific App
OAuth in FME:
One of the most common ways people come across OAuth in FME is through using ESRI's ArcGIS Online platform. When the user creates a connection they are presented with the log-in screen from ESRI who checks that the details are correct. If they are then ESRI provides FME with a token that it can use to access services provided by ESRI. FME never knows or stores the AGOL username and password and the scope of the authorisation provided is determined by ESRI.
Whilst the connection is provided out of the box the below is the connection template details that FME uses.
Note that the Client ID and Client Secret are provided by Safe and it is best practise to update these with your own organisations details.
The above authorises FME Desktop but when a connection is published up to FME Server the server is a different environment and the thus the token is technically no longer valid for that environment. It is therefore possible within FME Serve to reauthorise a connection to ensure that it is valid.
In practise FME Server will typically use a connection published from Desktop without any issue however if the token provided by ESRI becomes invalid or changes the connection would need to be authorised again.
Tokens:
The last part of the FME web service security puzzle is Tokens. Unlike the tokens provided by OAuth these are tokens that are explicitly created and provided by FME to use for WebHooks, and programatically talking to FME Server's APIs. A similar token system is also used by FME Cloud to manage the administrative side of an FME Cloud account programatically via the FME Cloud API.
