{"id":547,"date":"2018-06-14T20:14:15","date_gmt":"2018-06-15T00:14:15","guid":{"rendered":"https:\/\/micah.waldste.in\/?p=547"},"modified":"2018-06-24T20:15:03","modified_gmt":"2018-06-25T00:15:03","slug":"heatmaps-of-spherical-densities-in-r","status":"publish","type":"post","link":"https:\/\/micah.waldste.in\/blog\/blog\/2018\/06\/heatmaps-of-spherical-densities-in-r\/","title":{"rendered":"Heatmaps of Spherical Densities in R"},"content":{"rendered":"<p><em>DISCLAIMER: While I know a thing or two, there&#39;s a reasonable chance I got\r\nsome things wrong or at very least there are certainly more efficient ways to\r\ngo about things. Feedback always appreciated!<\/em><\/p>\r\n\r\n<p><a href=\"https:\/\/micah.waldste.in\/blog\/2018\/06\/introduction-to-spherical-densities-in-r\/\">Last time<\/a>\r\nwe made contour maps of densities of points on a globe, now it is\r\ntime to take another step and make heatmaps. We created all the data we needed\r\nwhen creating the contours, but heatmaps add new challenges of dealing with\r\nlarge amounts of raster and polygon data. Lets get to it.<\/p>\r\n\r\n<h1>Set-Up<\/h1>\r\n\r\n<p>First, we&#39;ll make use of a number of libraries and setup our plotting\r\nenvironment:<\/p>\r\n\r\n<pre><code class=\"r\">library(rgdal)       # For coordinate transforms\r\nlibrary(sp)          # For plotting grid images\r\nlibrary(sf)\r\nlibrary(lwgeom)\r\nlibrary(Directional) # For spherical density functions\r\nlibrary(spData)      # worldmap\r\nlibrary(raster)\r\n<\/code><\/pre>\r\n\r\n<p>We&#39;ll also use the same <code>vmf_density_grid<\/code> function we introduced in the Intro\r\npost.<\/p>\r\n\r\n<pre><code class=\"r\">vmf_density_grid &lt;- function(u, ngrid = 100) {\r\n  # Translate to (0,180) and (0,360)\r\n  u[,1] &lt;- u[,1] + 90\r\n  u[,2] &lt;- u[,2] + 180\r\n  res &lt;- vmf.kerncontour.new(u, thumb = &quot;none&quot;, ret.all = T, full = T,\r\n                             ngrid = ngrid)\r\n\r\n  # Translate back to (-90, 90) and (-180, 180) and create a grid of\r\n  # coordinates\r\n  ret &lt;- expand.grid(Lat = res$Lat - 90, Long = res$Long - 180)\r\n  ret$Density = c(res$d)\r\n  ret\r\n}\r\n<\/code><\/pre>\r\n\r\n<h2>Global Earthquakes Again<\/h2>\r\n\r\n<p>Global Earthquakes from <a href=\"http:\/\/www.ncedc.org\/anss\/catalog-search.html\">Northern California Earthquake Data\r\nCenter<\/a> is a great dataset we&#39;ll\r\ncontinue to use, so we start with a set of quakes since Jan 1, 1950 of\r\nmagnitude 5.9 or higher.<\/p>\r\n\r\n<p>For all our heatmaps, we&#39;ll start the same as we did for contours, calculating\r\nthe density map:<\/p>\r\n\r\n<pre><code class=\"r\">grid.size = 100\r\nearthquakes &lt;- read.csv(file.path(&quot;..&quot;, &quot;data&quot;, &quot;earthquakes.csv&quot;))\r\nearthquake.densities &lt;- vmf_density_grid(earthquakes[,c(&quot;Latitude&quot;,\r\n                                                        &quot;Longitude&quot;)],\r\n                                         ngrid = grid.size)\r\n<\/code><\/pre>\r\n\r\n<p>Once we have the densities, we need to coerce them into a spatial format &#8211; in\r\nthis case we&#39;ll create a <code>SpatialGridDataFrame<\/code>, matching the grid of densities\r\nwe calculated with <code>vmf_density_grid<\/code>.<\/p>\r\n\r\n<pre><code class=\"r\">density_matrix &lt;- matrix(earthquake.densities$Density, nrow = grid.size)\r\ndensity_matrix &lt;- t(apply(density_matrix, 2, rev))\r\ngridVals &lt;- data.frame(att=as.vector(density_matrix))\r\ngt &lt;- GridTopology(cellcentre.offset = c(-180 + 180 \/ grid.size,\r\n                                         -90 + 90 \/ grid.size),\r\n                   cellsize = c( 360 \/ grid.size, 180 \/ grid.size),\r\n                   cells.dim = c(grid.size, grid.size))\r\nsGDF &lt;- SpatialGridDataFrame(gt,\r\n                             data = gridVals,\r\n                             proj = &quot;+proj=longlat +datum=WGS84 +no_defs&quot;)\r\n\r\nplot(sGDF)\r\nplot(gridlines(sGDF), add = TRUE, col = &quot;grey30&quot;, alpha = .1)\r\nplot(st_geometry(world), add = TRUE, col = NA, border = &quot;grey&quot;)\r\n<\/code><\/pre>\r\n\r\n<p><img decoding=\"async\" src=\"https:\/\/i.imgur.com\/GW4RnsC.png\" alt=\"plot of chunk earthquake_plot\"\/><\/p>\r\n\r\n<p>Great, we have a heatmap! But it is in rectangular coordinates, we want to\r\nproject it to something nicer, like a Winkel triple. There&#39;s a problem\r\nthough&hellip; We can&#39;t just re-project our SpatialGridDataFrame &#8211; it gets\r\ninterpolated into points, losing our nice pretty smooth heatmap.<\/p>\r\n\r\n<p>There are two real options for us:<\/p>\r\n\r\n<ul>\r\n<li>Convert to raster data, then project the raster<\/li>\r\n<li>Convert to raster, convert to polygons, project the polygons<\/li>\r\n<\/ul>\r\n\r\n<h2>Projecting Raster Data<\/h2>\r\n\r\n<p>This is really slow, so we have to turn the resolution way down.<\/p>\r\n\r\n<pre><code class=\"r\">r &lt;- raster(sGDF)\r\ncrs1 &lt;- &quot;+proj=wintri&quot;\r\nworld.crs1 &lt;- st_transform_proj(world, crs = crs1)\r\n\r\npr1 &lt;- projectExtent(r, crs1)\r\nres(pr1) &lt;- 9e5\r\npr2 &lt;- projectRaster(r, pr1, method = &quot;bilinear&quot;, over = TRUE)\r\nplot(pr2)\r\nplot(st_geometry(world.crs1), add = TRUE, col = NA, border = &quot;grey&quot;)\r\n<\/code><\/pre>\r\n\r\n<p><img decoding=\"async\" src=\"https:\/\/i.imgur.com\/NJQ9UdX.png\" alt=\"plot of chunk earthquake_proj_raster\"\/><\/p>\r\n\r\n<p>I guess this works, but the low resolution suggests we can do better.<\/p>\r\n\r\n<h2>Using Polygons<\/h2>\r\n\r\n<p>We&#39;ll use raster data again, but we&#39;ll immediately convert it into a grid of\r\nsquare polygons which we can then project<\/p>\r\n\r\n<pre><code class=\"r\">r2 &lt;- raster(sGDF)\r\n# We&#39;ll manually colorize\r\nr2 &lt;- cut(r2,\r\n          pretty(r2[], 50),\r\n          include.lowest = F)\r\ncolor.vals &lt;- rev(terrain.colors(50))\r\npol &lt;- rasterToPolygons(r2)\r\ncrs1 &lt;- &quot;+proj=wintri&quot;\r\nworld.crs1 &lt;- st_transform_proj(world, crs = crs1)\r\npol.crs1 &lt;- spTransform(pol, crs1)\r\nplot(pol.crs1, col=color.vals[r2[]], border = NA)\r\n# plot(gridlines(sgdf.crs1), add = TRUE, col = &quot;grey30&quot;, alpha = .1)\r\nplot(st_geometry(world.crs1), add = TRUE, col = NA, border = &quot;grey&quot;)\r\n<\/code><\/pre>\r\n\r\n<p><img decoding=\"async\" src=\"https:\/\/i.imgur.com\/kTYDKf9.png\" alt=\"plot of chunk earthquakes_projected\"\/><\/p>\r\n\r\n<p>Now that looks good!<\/p>\r\n\r\n<p>One thing to keep in mind however &#8211; because our polygons are rectangular in equal\r\ncoordinates, they will warp and distort as a projection gets more severe. In\r\nour animation, you can see what I mean<\/p>\r\n\r\n<h2>Animating<\/h2>\r\n\r\n<p>We&#39;re projecting into an orthographic projection to simulate the rotating\r\nglobe. A few things you&#39;ll see in the code where I jump through hoops:<\/p>\r\n\r\n<ul>\r\n<li><strong>Cropping the top<\/strong> &#8211; If I leave the top polygons in place, they bunch up\r\nin an ugly fashion<\/li>\r\n<li><strong>Making features valid<\/strong> &#8211; Both for the world and our heatmap polygons I\r\njump through hoops to make sure only valid polygons get through to the final\r\nplot.<\/li>\r\n<\/ul>\r\n\r\n<pre><code class=\"r\">r3 &lt;- raster(sGDF)\r\n\r\n# Crop down because projecting the poles causes problems\r\nr.crop &lt;- res(r3)\r\nrc &lt;- crop(r3, extent(-180, 180,\r\n                      -90 + r.crop[2], 90 - r.crop[2]))\r\npol &lt;- rasterToPolygons(rc)\r\npol.breaks &lt;- pretty(pol$att, 20)\r\npol.colors &lt;- rev(terrain.colors(length(pol.breaks) - 1))\r\n# Make the lowest color transparent\r\nsubstr(pol.colors[1], 8, 9) &lt;- &quot;00&quot;\r\n\r\npar_old &lt;- par()\r\npar(mar = c(0, 0, 0, 0))\r\nn.frames &lt;- 30\r\ngrad &lt;- st_graticule(ndiscr = 1e4)\r\nfor (i in 1:n.frames) {\r\n  long &lt;- -180 + (i - 1) * 360 \/ n.frames\r\n  crs.ani &lt;- paste0(&quot;+proj=ortho +lat_0=0 +lon_0=&quot;, long)\r\n  grad.ani &lt;- st_geometry(st_transform(grad, crs.ani))\r\n\r\n  world.ani &lt;- st_transform(st_geometry(world), crs = crs.ani)\r\n  world.ani &lt;- st_make_valid(world.ani)\r\n  # We don&#39;t want the points\r\n  world.ani &lt;- world.ani[st_geometry_type(world.ani) %in% c(&#39;POLYGON&#39;,\r\n                                                            &#39;MULTIPOLYGON&#39;)]\r\n\r\n  # There are inevitable some bad polygons out of the transform\r\n  world.ani &lt;- world.ani[st_is_valid(world.ani)]\r\n\r\n  pol.ani &lt;- st_transform(as(pol, &quot;sf&quot;), crs.ani)\r\n  pol.ani.geo &lt;- lwgeom::st_make_valid(pol.ani)\r\n  pol.ani.geo &lt;- pol.ani.geo[st_geometry_type(pol.ani.geo) %in% c(&#39;POLYGON&#39;,\r\n                                                                  &#39;MULTIPOLYGON&#39;,\r\n                                                                  &#39;GEOMETRYCOLLECTION&#39;), ]\r\n  pol.ani.geo &lt;- pol.ani.geo[st_is_valid(pol.ani.geo), ]\r\n  pol.ani.geo &lt;- pol.ani.geo[!st_is_empty(pol.ani.geo), ]\r\n\r\n  plot(grad.ani, col = &quot;black&quot;)\r\n  plot(world.ani, add = TRUE, col = &quot;grey30&quot;, border = &quot;grey&quot;)\r\n  plot(pol.ani.geo, border = NA, breaks = pol.breaks, pal = pol.colors,\r\n       add = TRUE, main = NA, key.pos = NULL)\r\n}\r\n<\/code><\/pre>\r\n\r\n<p><img decoding=\"async\" src=\"https:\/\/i.imgur.com\/6Ge2sLD.gif\" alt=\"plot of chunk earthquake_ani\"\/><\/p>\r\n\r\n<pre><code class=\"r\">par(par_old)\r\n<\/code><\/pre>\r\n\r\n<p>Looks pretty good, but we do have some interesting world map problems with\r\ncountries popping out as they reach the edge&hellip; Something to investigate\r\nanother day.<\/p>\r\n\r\n<h2>Final Notes<\/h2>\r\n\r\n<p>In both these examples we&#39;ve used global data as it shows the problems of using\r\n&ldquo;traditional&rdquo; density estimators, but the same issue exists at all scales. It\r\nis just a question of when a simpler approximation is reasonable.<\/p>\r\n\r\n<p>You can also see a bit of blockiness which we could reduce with an increase in\r\ngrid size, but that will be very dependent on need.<\/p>\r\n\r\n<p>Next, some real data&hellip;<\/p>\r\n\r\n<h1>Appendix<\/h1>\r\n\r\n<h2>Spherical Density Function<\/h2>\r\n\r\n<p>This calculates a grid of densities which can then be used with <code>geom_contour<\/code>.\r\nThe code basically comes directly from <a href=\"https:\/\/rdrr.io\/cran\/Directional\/man\/vmf.kerncontour.html\">Directional&#39;s\r\nvmf.kerncontour<\/a>,\r\nonly returning a data.frame instead of actually plotting the output.<\/p>\r\n\r\n<pre><code class=\"r\">vmf.kerncontour.new &lt;- function(u, thumb = &quot;none&quot;, ret.all = FALSE, full = FALSE,\r\n                            ngrid = 100) {\r\n  ## u contains the data in latitude and longitude\r\n  ## the first column is the latitude and the\r\n  ## second column is the longitude\r\n  ## thumb is either &#39;none&#39; (default), or &#39;rot&#39; (Garcia-Portugues, 2013)\r\n  ## ret.all if set to TRUE returns a matrix with latitude, longitude and density\r\n  ## full if set to TRUE calculates densities for the full sphere, otherwise\r\n  ##   using extents of the data\r\n  ## ngrid specifies the number of points taken at each axis\r\n  n &lt;- dim(u)[1]  ## sample size\r\n  x &lt;- euclid(u)\r\n\r\n  if (thumb == &quot;none&quot;) {\r\n    h &lt;- as.numeric( vmfkde.tune(x, low = 0.1, up = 1)[1] )\r\n  } else if (thumb == &quot;rot&quot;) {\r\n    k &lt;- vmf(x)$kappa\r\n    h &lt;- ( (8 * sinh(k)^2) \/ (k * n * ( (1 + 4 * k^2) * sinh(2 * k) -\r\n    2 * k * cosh(2 * k)) ) ) ^ ( 1\/6 )\r\n  }\r\n\r\n  if (full) {\r\n    x1 &lt;- seq( 0, 180, length = ngrid )  ## latitude\r\n    x2 &lt;- seq( 0, 360, length = ngrid )  ## longitude\r\n  } else {\r\n    x1 &lt;- seq( min(u[, 1]) - 5, max(u[, 1]) + 5, length = ngrid )  ## latitude\r\n    x2 &lt;- seq( min(u[, 2]) - 5, max(u[, 2]) + 5, length = ngrid )  ## longitude\r\n  }\r\n  cpk &lt;- 1 \/ (  ( h^2)^0.5 *(2 * pi)^1.5 * besselI(1\/h^2, 0.5) )\r\n  mat &lt;- matrix(nrow = ngrid, ncol = ngrid)\r\n\r\n  for (i in 1:ngrid) {\r\n    for (j in 1:ngrid) {\r\n      y &lt;- euclid( c(x1[i], x2[j]) )\r\n      a &lt;- as.vector( tcrossprod(x, y \/ h^2) )\r\n      can &lt;- sum( exp(a + log(cpk)) ) \/ ngrid\r\n      if (abs(can) &lt; Inf)   mat[i, j] &lt;- can\r\n    }\r\n  }\r\n\r\n  if (ret.all) {\r\n    return(list(Lat = x1, Long = x2, h = h, d = mat))\r\n  } else {\r\n    contour(mat$Lat, mat$Long, mat, nlevels = 10, col = 2, xlab = &quot;Latitude&quot;,\r\n            ylab = &quot;Longitude&quot;)\r\n    points(u[, 1], u[, 2])\r\n  }\r\n}\r\n<\/code><\/pre>\r\n\r\n<h2>References<\/h2>\r\n\r\n<ul>\r\n<li>Earthquake data was accessed through the <a href=\"http:\/\/www.ncedc.org\/anss\/catalog-search.html\">Northern California Earthquake Data Center (NCEDC)<\/a>, doi:10.7932\/NCEDC.<\/li>\r\n<\/ul>","protected":false},"excerpt":{"rendered":"DISCLAIMER: While I know a thing or two, there&#39;s a reasonable chance I got some things wrong or at very least there are certainly more efficient ways to go about things. Feedback always appreciated! Last time we made contour maps of densities of points on a globe, now it is time to take another step &hellip; <p class=\"link-more\"><a href=\"https:\/\/micah.waldste.in\/blog\/blog\/2018\/06\/heatmaps-of-spherical-densities-in-r\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Heatmaps of Spherical Densities in R&#8221;<\/span><\/a><\/p>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[16],"tags":[],"class_list":["post-547","post","type-post","status-publish","format-standard","hentry","category-rstats","no-wpautop"],"_links":{"self":[{"href":"https:\/\/micah.waldste.in\/blog\/wp-json\/wp\/v2\/posts\/547","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/micah.waldste.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/micah.waldste.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/micah.waldste.in\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/micah.waldste.in\/blog\/wp-json\/wp\/v2\/comments?post=547"}],"version-history":[{"count":1,"href":"https:\/\/micah.waldste.in\/blog\/wp-json\/wp\/v2\/posts\/547\/revisions"}],"predecessor-version":[{"id":548,"href":"https:\/\/micah.waldste.in\/blog\/wp-json\/wp\/v2\/posts\/547\/revisions\/548"}],"wp:attachment":[{"href":"https:\/\/micah.waldste.in\/blog\/wp-json\/wp\/v2\/media?parent=547"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/micah.waldste.in\/blog\/wp-json\/wp\/v2\/categories?post=547"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/micah.waldste.in\/blog\/wp-json\/wp\/v2\/tags?post=547"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}