Postgres OnLine Journal: Jan 2018 - Dec 2018
An in-depth Exploration of the PostgreSQL Open Source Database
 

Table Of Contents

Basics
Using PostgreSQL Contribs

Basics

 

Unpivoting data using JSON functions



Most of our use-cases for the built-in json support in PostgreSQL is not to implement schemaless design storage, but instead to remold data. Remolding can take the form of restructuring data into json documents suitable for web maps, javascript charting web apps, or datagrids. It also has uses beyond just outputting data in json form. In addition the functions are useful for unraveling json data into a more meaningful relational form.

One of the common cases we use json support is what we call UNPIVOTING data. We demonstrated this in Postgres Vision 2018 presentation in slide 23. This trick won't work in other relational databases that support JSON because it also uses a long existing feature of PostgreSQL to be able to treat a row as a data field.

Unpivoting data is the process of converting columns to rows. It's the reverse of a cross tabluation (aka PIVOT). So why would you ever need to unpivot data?

Take for example if you had a spreadsheet where your HR person decided to input each employees hours in columns where each column represents a separate week.

The spreadsheet data might look something like this:

Employee2018-05-042018-05-112018-05-182018-05-25
Regina70403565
Leo50804565
Caitlin40253535
Jane20251518

If you are using a spreadsheet as your data entry and presentation tool, then this is a great setup, but for storing the data in a relational database and being able to do arbitrary date summary aggregations, it's easier to work with the data in a format such as below because it's more flexible how you rollup date ranges and you don't have to add a new column to your table every time a new week comes along.

EmployeePPEhours
Regina2018-05-0470
Regina2018-05-1140
Regina2018-05-1835
Regina2018-05-1865

Note this trick you can accomplish using the hstore extension as well as we covered a while back in Unpivoting data in PostgreSQL. Since JSON/JSONB functions are built-in no CREATE EXTENSION is required to use them.

INSERT INTO staff_hours(employee, ppe, hours)
SELECT hours_raw."Employee" As employee,  j.key::date AS ppe,  j.val::numeric As hours
FROM hours_raw, LATERAL jsonb_each_text(to_jsonb(hours_raw)) AS j(key,val)
WHERE j.key NOT IN('Employee');

The above query is composed for 4 steps

  • A row in a query (not just tables, though we are only showing a table here), can be output as a single thing. In this case we are going to use our hours_raw table as a set of elements of type hours_raw and feed it to the to_jsonb function.
  • to_jsonb (and it's companion to_json) can convert any element (including row elements) into json representations.
  • The jsonb_each_text function returns a set of key / value pairs. When used in conjunction with LATERAL (which is the default for functions when used in from), it generates a new set of rows for each row of hours_raw
  • For this example we assume all columns except for Employee are hour columns where the header is a pay period ending (PPE). We only want to return the Key/Values that do not correspond to employee name.

Using PostgreSQL Contribs

 

FDWS for PostgreSQL Windows 32 and 64-bit



We've updated our binaries for PostgreSQL 10 windows, both 32 and 64-bit. The 64-bit should work fine with EnterpriseDb windows as well as BigSQL.

Note this package does not include our favorite FDW ogr_fdw since that is packaged as part of PostGIS packages from EnterpriseDb Stackbuilder and also BigSQL distributions.

If you do not have PostGIS installed (and don't want to for some reason) and want to use ogr_fdw on windows, you can download from: Winnie's PG 10 Extras. ogr_fdw is a great FDW for querying not just spatial data, but also a ton of other file formats or relational since spatial is a superset.

These packages contain the following FDWs:

  • odbc_fdw (recently updated to work with PostgreSQL 10)
  • file_textarray_fdw, great and fast for working with arbitrary and pesky delimited data. Especially where they weren't considerate enough to give you the same number of columns per row.
  • WWW FDW exposes web services as queryable tables. I'll note we haven't really used this since 9.4 since the 9.5 beyond we had some random crashes we haven't investigated. That said the http extension, Windows binaries here, is a better alternative especially for newer PostgreSQL. We'll do another article soon on the new features of the http extension which is pretty sweet if you have a need to use web services (even ones that require authentication) . We've used it for pushing our client data to SalesForce as well as posting and pulling down files from other third-party sources our clients are subscribed to. Both the http and www_fdw rely on Curl for their heavy lifting.

Using PostgreSQL Contribs

 

PostgresVision 2018 Slides and Impressions



Leo and I attended PostgresVision 2018 which ended a couple of days ago.

We gave a talk on spatial extensions with main focus being PostGIS. Here are links to our slides PostgresVision2018_SpatialExtensions HTML version PDF.

Unfortunately there are no slides of the pgRouting part, except the one that says PGRouting Live Demos because Leo will only do live demos. He has no fear of his demos not working.

Side note, if you are on windows and use the PostGIS bundle, all the extensions listed in the PostGIS box of the spatial extensions diagram, as well as the pointcloud, pgRouting, and ogr_fdw are included in the bundle.

Postgres Vision 2018 was surprisingly good. We feared the talks would be mostly focused on upper management folks, but there turned out to be quite a few very technical ones to chew on as well such as sharding, JIT, WAL, and ZHeap to name some. Though Bruce Momjian's talk Why Postgres will live forever wasn't that technical it was the best we saw. It was inspiring, comical, and full of computer history tidbits. I love computer history.

Most exciting, I got to meet fellow package maintainer Devrim Gündüz for the first time. Devrim manages yum.postgresql.org and as such does packaging for PostGIS in addition to the other 100s of packages he maintains. I only do packaging for PostGIS bundle for windows and even that feels like a lot.