I know what your API did last summer 2019-08-21

Making an API is a thankless task. Most users aren't aware they are using it, and you rarely hear from the developers unless something isn't working. While our weather site Yr is hugely popular, our API doesn't get much publicity. So when we finally get some exposure, it's sad to find only negative publicity, and misguided criticism at that. But before addressing the issues, let's watch the presentation (starting about 8:44 into the video).

"...a company called met.no, seems to be a weather type service..."

To start with, met.no isn't a company, it's the Norwegian Meteorological Institute – a state-owned, non-profit government organisation which has as one of its missions to give free (both as in beer and in freedom) weather data to the whole world. So far we've managed to do this without any developer registration or other conditions than nicely asking not to overload our services. As a consequence we have no clue about most of the thousands of apps and sites using our services, and no way of contacting them unless they subscribe to our mailing list or read the API front page.

From the interview one would get the impression we are making apps ourself using unsecure communications. Such is not the case – the only app we have some influence over is the Yr weather app, which doesn't speak with our API at all. So already the (admittedly valid) criticism is targeted at the wrong party.

"... it was making an HTTP request, totally normal"

Disregarding that "it" in this case is an app we have no control over, we have actually supported (and recommended) HTTPS for many years (although in the old API this wasn't default because of compatibility issues). Since the rewrite and launch of WeatherAPI v.3 (which came out in beta in 2016 and was gradually phased in on a per-user basis over the next year) we have only supported HTTPS. All unencrypted HTTP traffic is either being redirected or blocked (more on this later).

To this day we still get complaints about this, e.g. when no longer being able to use Squid as a local caching proxy, or using client libraries which don't support SSL/TLS. Incredible as this may seem in 2019 it is still an issue, and when your application is a watering plant run by a Saia-Burgess PLC controller there is no sofware upgrade path.

About half of the requests to api.met.no are either throttled, blocked or invalid. Of the latter, most are for products which were removed years ago. We're still getting requests for locationforecast/1.8 which was removed in 2014(!). Most developers don't bother checking the response status codes, so lots of sites and apps go on blindingly ignoring 304, 404 and 429 errors for several years. And since many users don't update their mobile apps until they have to, we're getting a lot of legacy traffic, included unencrypted HTTP which we're unable to prevent people from sending us.

As an example, let's look at the HTTP traffic from Android (pre <5.0) apps where the developer haven't read the TOS and uses the default "Dalvik" User-Agent header which amounts to 13% of the total unencrypted HTTP traffic. These are the amount of requests sorted by status code between 12 and 13 today (UTC):

requests status protocol
62054 301 HTTP
44758 429 HTTPS
35019 200 HTTPS
1408 404 HTTPS
313 400 HTTPS
19 499 HTTPS
17 503 HTTPS

As the table shows, the unencrypted traffic amounts to a whooping 43% of the total identifying only as Android (and an obsolete version at that). How much of this actually handles the redirect is difficult to see as we don't track users, but we find it hard to believe that 76% percent of the HTTPS traffic comes from redirects. This means that a large portion of clients never follow the redirect.

Of those who use HTTPS, an incredible 55% of the HTTPS requests are being throttled for too much traffic or missing identification, while only 43% get a meaningful answer. The remaining requests (apart from a handful of timeouts) are either for products which been discontinued (sunrise/1.1 is still very popular) or they can't manage to construct a valid URL. That's a lot of developers who can't be bothered to RTFM.

"... trying to figure out when the sun goes up"

Now, this is where it gets interesting. Most of the HTTP traffic is actually for locationforecast; sunrise doesn't count for more than about 5%. Of this, 5 out of 6 sunrise requests are actually zombie apps calling version 1.1, which was discountinued in February and returns a 404. (Incredibly we also get some for version 1.0 which was EOL in 2016!)

Still, that leaves about 1% of HTTP traffic for sunrise/2.0, which has never worked with unencrypted HTTP. Every piece of documentation, example code and tests we have published have been using HTTPS, so why are people still using HTTP? These requests are (if identified at all) coming from Android, so either there is something amiss with the Dalvik HTTP client stack, or developers are intentionally breaking security by rewriting our examples from "https:" to "http:". As the saying goes, it's hard to make stuff fool-proff because the fools are so smart.

"Latitude and longitude, that's harmless, right?"

Many of the requests do indeed contain geocoordinates. However, the vast majority of these originate from servers located in data centers totally remote from the geolocation in the request. Even when an app enable location services on your phone, we have no way of knowing if the coordinates are your actual GPS location or if you're just checking the weather for your next trip to Paris. We're still getting ~25,000 different IP addresses from all over the world downloading the weather forecast for a cornfield in Poznan, PL every 5 minutes, amounting to 3.15% of the total api.met.no traffic.

"... but you've got the MAC address [...] and lat/lon"

I confess this claim has us stumped, and after putting our collective heads together we still can't figure out what they're getting at. Sure, the MAC address is an identificator which can be used to link with personal data, but that is never transmitted to us (what the app developers are doing with your personal data you must ask them about).

Now, the only third party getting access to both your MAC address and your unencrypted traffic is either the owner of your Wifi access point/router or your phone operator. Admittedly, both these can be spoofed and your traffic intercepted by evil middlemen. However, in both cases they can already pretty much tell your geolocation even without snooping your data traffic. When a phone is connected to (in this case) a 800mW faked wifi access point and indoors it is pretty much given that the person is within a 50 m radius. In fact this is how shops track customer behaviour, logging how much time each person spends before each isle. Now, having the GPS coordinates in addition would probably give you a better resolution, but then GPS reception isn't great inside buildings anyway.

"So now we can track that person when their phone requests updates"

Actually, no. The API doesn't collect any personal data, except for the ubiquitous access log. We don't even use cookies or Google Analytics, not do we write any mobile apps ourselves. The only potentially identifiable data logged by us are the IP address (which for vast majority is either a website server or a NATed 3G/4G address which changes every time you move into a new cell tower area), the User-Agent string (set by the app) and the URL itself. This log is used for monitoring resources, debugging errors and trying to stem abuse, which is necessary to avoid the tragedy of the commons.

Sure, the app developers can (and probably will) track you, and the same goes for your phone company or free Wifi provider. But that has nothing to do with HTTP encryption, and very little to do with geolocation (which they know already).

HTTP redirects

Now, you might ask why we didn't go directly for HTTPS in the first place. At the time api.met.no was lauched in 2007, this was not usually an option; Facebook didn't offer HTTPS until 2011, and many client libraries did not support it. While this now has changed, our main priority has always been backwards compatibility, something we're proud of having had for 12 years now, even though individual products and versions have come and gone.

As mentioned earlier, all of the unencrypted HTTP traffic is either blocked or redirected, in the latter case to a similar HTTPS URL. Now, since we're mirroring the original URL in the Location response header, it can be argued that we still "support HTTP" since legacy apps can still go on working if they handle the 301 properly. If they instead got a 404, the theory goes, they would see the error in their ways and change their code to use HTTPS instead.

Unfortunately, in practice that is not the case. The sites topping the list of HTTP have all been blocked for abuse and get a 403 Forbidden response. They have been blacklisted for years, but still continue to hammer our servers with 14k req/hour each with no end in sight. For these large websites there is no connection between the IP address and the geolocation received by us, so any "leak of personal data" here would be between the browser and the website if they're not using HTTPS.

If we can conclude that policing obsolete websites is futile, then this goes double for obsolete mobile apps where it doesn't matter what the developer does unless the user updates the app. As can be seen from the previous table, a lot of the traffic isn't only against services which no longer exist, but are also more than two years old and run on obsolete versions of Android. There is not much hope that these apps will be updated any time soon.

Anyway, as a precaution we tried disabling the redirect to HTTPS to see what happened. Not only did we get a lot of support tickets from developers who couldn't figure out what was wrong, it also broke the website. Whereas you previously could type e.g. "api.met.no/apis.json" in the browser location bar and get JSON back, the browser actually defaults to unencrypted HTTP which no longer works. Instead you actually have to type "https://" before the hostname. Can you think of any other sites in 2019 where this is necessary? Me neither.

In fact, all major sites we have tested (e.g. Google, GitHub, Paypal, eBay) redirect HTTP requests to corresponding HTTPS URLs, which is also recommended in all best practice documentation we have found, including the HTTP Strict Transport Security. Here is an example of GitHub doing exactly the same thing:

$ GET -Se http://api.github.com/users/octocat/orgs
GET http://api.github.com/users/octocat/orgs
301 Moved Permanently
Connection: close
Date: Fri, 23 Aug 2019 11:08:56 GMT
Location: https://api.github.com/users/octocat/orgs
Content-Length: 0
Client-Date: Fri, 23 Aug 2019 11:08:57 GMT
Client-Peer: 140.82.118.6:80
Client-Response-Num: 1

GET https://api.github.com/users/octocat/orgs
200 OK

So we've rolled back the change, and until we can find a way to differentiate zombies from actual live humans we're keeping it that way.

The bottom line

So, the next time you hear someone boldly stating that

"APIs leak personal data"

you would be wise to interpret that as

"abandoned zombie apps, written by unknown parties for a defunct API, who send unsolicited, not-very-personal data to non-existent services, can be intercepted by hackers who will learn nothing they don't already know, and there is nothing the API owner can do to prevent it".

Admittedly that won't make any headlines at a security conference. Instead, if there's anything to learn from this, it's that most traffic against APIs is actually noise, which you'll just have to learn to deal with. But admittedly that doesn't make for good soundbites on YouTube.


More changes to Radar and Radarlightning 2019-07-02

Due to increased workload by the new 5 min radar interval, we have had to reduce the number of different images available. In short, generating GIF animations from existing static GIF images continues as before. However, due to the 50% increase in images and 33% shorter production window, generating animations from PNG is no longer possible and has been terminated. We apologize for the lack of advance notice, but we had not foreseen this would be a problem.

Since we know this will cause existing functionality to break, we've come up with a temporary solution where we substitute the missing animations with similar products. In most cases the pixel dimensions will be different, but this can be scaled in the browser. Also note that content=image and content=animation now will return totally different images, so be wary of using size=small for static thumbnails for the animations.

reflectivity_hires and rainrate_with_phase

All animations of type reflectivity_hires and rainrate_with_phase will be substituted with type reflectivity. Where radarsite indicates an area these will be identical, but single radars will be substituted to the closest area as per the following table:

Old New
andoya nordland_troms
berlevaag finnmark
bomlo western_norway
hafjell east_norway
hagebostad southwest_norway
hasvik finnmark
hurum southeast_norway
rissa trlagnordland
rost nordland
somna trlagnordland
stad western_norway

lightning

All requests for lightning animations (including the Radarlightning product) now return the "norway" image. All other animations have been terminated.

We have some images currently used internally which could be substituted for each individual area, but they are currently not available on the API. We will be working on a possible replacement later this summer.

"medium" and "large" size static images

The little used "medium" and "large" sizes have been terminated. All requests for these sizes now return the "normal" image. If you scale this in the browser it should not be noticable to the end users. This does not affect radarlightning which already uses only "normal" size.

Also, the small GIF animations haven't been produced for a long time and will be removed from the API. This means only the following variations will be available in the future:

Content Size
animation normal (default)
image normal (default), small

Again, we're sorry for the inconvenience, but hopefully the faster radar scan rate should compensate for the reduced size options.

Future changes

In closing, we will like to inform you that later this year we'll be releasing a new beta version of the Radar service, with a coherent set of areas and types which should be much more understandable than the current mess. Also, GIF animations will be replaced by Javascript, which should be much more user-friendly as you can then pause or jump forwards and backwards in time to see details.

We strongly suggest you consider dropping GIF animation as soon as possible, since now that most partners are using Javascript or WMS we plan to phase out GIF completely by the end of 2019 sometime during early 2020.


Changes to Radar, Nowcast and other products 2019-06-07

Radar

From Wednesday June 12th, the radar images will be produced every 5 mins.

New features

  • new product "aviationforecast", see below
  • new beta product "volcanicashforecast"
  • sunrise now returns moon position and phase for each day
  • routeforecast have several new routes
  • upperwindweather now has a status function showing image update and transmission errors (if any)

End-of-life

Spotwind 1.0 has been replaced by 1.1 which delivers data for 24 hours instead of 6. Version 1.0 will expire 2019-09-02.

ExtremesWWC is expected to be removed some time in the near future, to be replaced with a similar feature in our Frost API for observations. We will come back with further news when this is available.

Textforecast 1.6 has been deprecated, and the land and sea forecasts have been re-implemented in 2.0 with a new, cleaner XML format which can now be validated. The aviation forecasts have been moved to a new product "aviationforecast" but remains otherwise unchanged. Since the migration is now complete we will terminate version 1.6 on 2019-09-02.


New API products 2019-03-04

We will shortly launching some new products, mainly of interest for Norwegian users. As described earlier, some obsolete products will also be removed.

textforecast 2.0

We have started the process of cleaning up the old text forecasts, with a new XML format (which can be validated), more consistent data structure and clearly defined areas. All of new forecasts share the same format and schema, including:

  • landoverview
  • coast_en, coast_no
  • seabank_en, seabank_no, seabank_wmo
  • seahigh_en, seahigh_no, seahigh_wmo

To aid in transition from textlocation, all areas are now identified with an id, which can be used to look up the corresponding geographical coordinate polygon. These files can be downloaded from the /areas endpoint, but we hope eventually to be able to include them directly in the forecast.

Textforecast 1.6

We have removed the "land" product (use "landoverview in 2.0) as well as "seaoslofjord" (use "coast_no" and "seabank_no" in 2.0).

Airqualityforecast

For some time we have been working on air quality forecasts in cooperation with several other public services. This is expected to go out of beta soon.

Routeforecast 1.0

This is a new image service for aviation, primarily offshore helicopter flights.

Textlocation 1.0

This product will expire March 15th. Please use the "landoverview" product in textforecast along with the area files.

Errornotifications 1.0

This service hasn't delivered any data for months and will expire on March 15th.

ExtremesWWC 1.2

This product will expire on April 1st, when we are turning off the WSklima service (delayed until further notice). There might be a similar service on frost.met.no in the future.


Discontinuation of text forecasts 2018-12-21

As mentioned earlier we are discontinuing some little used products from api.met.no, here is a complete list:

On February 1st we will remove the following products from the textforecast service:

  • the route_fbno products (these will still be available for professional users via NAIS which everybody seem to use anyway)

  • the landday/long, seabankday1 and metkyst products (which are used to generate more comprehensive forecasts)

  • the iga_fbno43, easter and miskred products (which haven't been produced for years)

We have checked the logs and all of these have little or no traffic, so we don't expect them to be missed. This is the full list of discontinued forecasts:

easter, landday0, landday1, landday2, iga_fbno43, landlong, metkyst0, metkyst1, miskred_north, miskred_south, route_fbno69, route_fbno70, route_fbno72, route_fbno74, route_fbno75, route_fbno76, route_fbno77, route_fbno78, route_fbno79, route_fbno80, seabankday1.

Also, on March 1st the textlocation service will be discontinued. We will also remove the "land" product from the textforecast service. There will be another forecast product to replace those two products, details of which are to follow later.

At the same time, the product "seaoslofjord" containing forecasts for Skagerrak and for the Norwegian coast from the Swedish border to Hordaland will be discontinued. Forecasts for Skagerrak can be found in the "seabank" product, while forecasts for the complete Norwegian coastline can be found in the "coast" product.

We will soon also remove the errornotifications service, which is no longer producing any data. This will be replaced by specific status messages for each product.


Changes to GeoSatellite and new privacy policy 2018-06-29

Due to licensing issues, problems with authentication and abuse of the service we have been forced to suspend delivery of non-restricted geosatellite images to the general public. Currently there are only 8 unrestricted pictures from a total of 2162 which we are allowed to distribute freely, but we have seen repeated attempts to download images which are clearly documented as not publicly available. Until further notice the service will only be available for our own sites yr.no and halo.met.no.

Also, as part of the European General Data Protection Regulations, we have updated the privacy policy for all our sites and services. You can find the new text under Privacy Policy Statement for MET Norway.


Planned maintainance and discontinuation of old domains 2018-05-09

On Monday May 14th, 2018 from 07:00 to 13:00 UTC we will be running at 50% capacity due to replacement of the cooling system in one of our data centers. All services are expected to continue running, but latency times and timeout errors may increase due to reduced capacity.

Also, as has been noted earlier, the domains api.yr.no and beta.api.met.no will be turned off on June 11th, 2018. Please switch your systems to use api.met.no instead.


Migration completed 2018-02-26

As most of you probably have noticed, we upgraded api.met.no about A week ago to the new version. After some initial performance problems we managed to tune the caches so it is now handling traffic well.

Some users have reported problems accessing the API, mostly related to not handling HTTPS and/or redirects. If you are getting "Failed to get data from weather api. Reason: end tag name </body> must match start tag name <hr> from line 5" this means you are trying to parse the HTML message as XML instead of following the 301 redirect. We are also working on recommendations for users using HTTPS via local proxies.

In order to ensure stability of our own services, we have also introduced traffic rate limiting (and in some extreme cases, blocking) for some users not abiding by our terms of use. If you are receiving 429 status codes it means you are being throttled, either due to a missing/generic User-Agent header and/or excessive traffic (>20r/s).

Finally, we have noticed there is a lot of production traffic against beta.api.met.no (which was only to be used for testing) as well as api.yr.no (which has been deprecated for years). We would like to inform you that these hostnames will be removed within a few months, so please update your applications accordingly.


Migration of api.met.no 2017-12-12

As previously mentioned on the mailing list, we will shortly be migrating the api.met.no domain to version 3. This has already been done for some users and should be relatively painless, provided you follow the documentation and terms of use. In particular we ask you to notice the following:

  • Use HTTPS only! Unencrypted requests are no longer supported
  • Honor redirects (301 and 303), which is used e.g. from HTTP to HTTPS
  • Semicolon as parameter separator is now deprecated, use ampersand instead
  • Make sure you have a unique, identifiable User-Agent string

Otherwise the main difference is a lot of new features and bugs that have been fixed. For more information on the changes to the interface, please see the release notes for version 3.

The migration is scheduled to take place some time during the first half of January 2018.