Planet PostgreSQL

Mark Wong: PDXPUG: February meeting

Fri, 10 Feb 2012 22:05:29 GMT

When: 7-9pm Thu Feb 16, 2012
Where: Iovation
Who: John Melesky
What: Locks… etc.

High-traffic systems have to deal with locks. That’s just the way it is. But that’s okay! Locks invisibly help manage concurrency! Properly structured transactions will never run into deadlocks with each other! Right?

Well, sometimes locks are very visible, and sometimes you end up with deadlocks that don’t make sense. I’m going to talk through a couple of row-locking problems i’ve run into, and explain how and why that should end up changing PostgreSQL’s SQL syntax.

John’s been poking computers since he was old enough to type, which was long enough ago that he debated whether to make a “POKEing” joke. He’s a Software Engineer at Janrain, working with the reporting and analytics infrastructure.

Our meeting will be held at Iovation, on the 32nd floor of the US Bancorp Tower at 111 SW 5th (5th & Oak). It’s right on the Green & Yellow Max lines. Underground bike parking is available in the parking garage; outdoors all around the block in the usual spots. No bikes in the office, sorry!

Building security will close access to the floor at 7:30.

After-meeting beer location TBD. See you there!

Jim Smith: Announcing the Arizona Postgres Users Group

Thu, 09 Feb 2012 23:10:30 GMT

As mentioned in an earlier post, we're announcing the start of a PUG for Arizona. The first meeting will be Thursday, March 29, from 5 to 7 at Bull's facility in Northwest Phoenix. Refreshments will be provided.

You can join and get onto the mailing list and RSVP for the first meeting at https://www.bigtent.com/groups/azpug

We look forward to meeting other PostgreSQL lovers and to help grow the PostgreSQL community in our part of the world. Please feel free to pass the above link to anyone that's interested.

Bruce Momjian: Virtualizing Postgres

Thu, 09 Feb 2012 18:00:01 GMT

Postgres is an ideal database to run in a virtual environment or public/private cloud — one reason is that Postgres relies heavily on the operating system, rather than using features like raw devices. Second, its license is obviously very flexible for virtual deployments.

I am often asked about running Postgres in virtual environments, and I usually answer that it runs just fine --- and it does. However, I am starting to realize that I am not answering the more complex questions of which visualization technology to choose, and what is the performance and reliability impact of virtualization.

Chris Travers: Further Response to Robert Young

Thu, 09 Feb 2012 01:47:46 GMT

Robert Young has responded to my previous post and clarified his position. However, he is still incorrect in his assessment.

The issue is lost cycles as one moves down to a single thread. It's more about the number of clients that can be supported "simultaneously". At one time I was also enamoured of thread count to execute queries in parallel; not so much now. For reporting generally, and BI/DW specifically, sure. But for OLTP, where my interest lies, not so much. There, client count is what matters. Clients run relatively discrete transactions on few rows; parallelism isn't much help for that sort of client. Yes, to some extent I've changed my mind.

I think this makes the mistake of assuming that single threaded multiple process models cannot utilize hyperthreading architectures to increase computational efficiency.

The first thing to note is that how a CPU cycles are utilized is primarily the job of the scheduler which is a part of any multitasking kernel, and different operating systems utilize very different process and thread models. These cannot be merely reduced to their hardware equivalents. If they were we would see process vs thread performance to be largely equivalent in Windows and Linux, but in fact while Windows is very thread-friendly and process-unfriendly, though my understanding is that applications on Windows running under SUA experience better multiprocess performance than under the Win32 subsystem.

On Linux and UNIX there is very little difference between a pthread and a process except for memory isolation. Individual processes can be scheduled on logical cores supporting hyperthreading, for example. The overhead for supporting multiple processes is low.

On the other hand, on Windows, threads are cheap and processes are expensive. This is one reason why I typically recommend that high-concurrency PostgreSQL instances are not run on Windows. My understanding is that SUA addresses this issue but I can't find solid comparisons at the moment and I don't know of a PostgreSQL port to that platform.

Think of the issue this way: if it were true that scaling down threads to one per processor/core yielded a scale up in performance equivalently (only one of two threads yields twice the performance on that last thread), then the question gets more difficult. But threads and cores don't scale that way. Such reverse scaling could only happen if the core's frequency increased as the inverse of threads active. That just doesn't happen, you get a bin or two; that nasty frequency brick wall is the reason vendors have turned to multi-core and threads in the first place.

I am not sure I buy that. Linux, for example, is quite happy to schedule different processes on different logical cores, thus meaning that two processes can share a hyperthreading core in the same way two threads can. Again, that behavior is specific to a combination of a CPU, OS and scheduler.

Unless you've been building database applications from before 1995 (or thereabouts), which is to say only webbie thingees, the efficiency and user friendliness of *nix/RDBMS/RS-232/terminal is a foreign concept; just as it still is for mainframe COBOL coders writing to 3270 terminals, same semantics (I'll grant that javascript is more capable than 3270 edit language, and Wikipedia does it again, with this write up). In the web age, AJAX was the first widely known (and possibly absolute first) attempt to get back to that future, then there is Comet. Now, we have WebSocket, discussed in earlier posts.

I don't see what this has to do with anything. I personally find the user friendliness of an RDBMS and *nix to be very important. I also primarily build accounting apps, and I will tell you that doing so often over the web is not very performance-friendly for reasons that I am more than happy to discuss in another post (it's one reason why long-run I want to move away from requiring that everything goes through a web interface). But basically, transactional handling is somewhat crippled when tied to a stateless network protocol like HTTP, meaning all transactions must be atomic to within a single http request, and where we would normally not do that, we have to invent ways to get around that limitation.

The Achilles heel of those *nix/RDBMS/RS-232/VT-X00 applications was the limit on the number of clients that could be accommodated since the client patch remained while the user kept a session. The emerging hardware/infrastructure that I'm discussing may/should remove that restriction. This also means that developers must heed the mantra, "the user doesn't determine transaction scope". With HTTP (un)connectedness, that isn't much of an issue.

This is the wrong way to look at it. HTTP unconnectedness actually in many cases makes the problem worse rather than better if you have to preserve db state across HTTP requests (which LedgerSMB btw does in some cases). It's one thing to have 15 connections updating payments in a db with 10 million transactions and relying on advisory locks. It's a very different thing to have 15 users hitting a db with 10 million transactions and having to store in the db which transactions they are temporarily holding for payment. now suddenly you go from 3 sec queries to 45 sec queries, and I would expect that we could support at least 10x the number of concurrent users in a large db if it wasn't for this unconnectedness. Now you are dealing with a large amount of I/O contention and row-locking (because we have to make sure we get the right info so invoices aren't double-paid), and while it works, it isn't pretty, nor does it perform particularly well.

This being said, the DB is usually even less of an issue in these cases than the web server.....

Moreover, web apps, particularly those that run public sites, can experience traffic spikes that can easily exceed the number of connections a db can be expected to gracefully handle. So in both cases: simple web site apps like blogs accessible to the public, and complex web-based business apps, rdbms-backed web apps are worse performance-wise than thick clients.

But the overall point remains the same: scheduling of processes and threads is the OS's job, not the CPU's. Wasted cycles in scheduling are the OS's problem, not the CPU's. If an OS can't schedule processes as efficiently as it schedules threads, that's the OS's constraint, and it is only the app's problem if you are writing with an OS in mind and your OS doesn't behave the way you would like it to. Windows, in particular, is heavily oriented towards threads. It isn't clear that most *nix platforms see the same benefit, as witnessed by Oracle's architecture which assumes one process per connection.

Also a note on DB2's connection multiplexing. It isn't clear that this actually requires multiple threads to do. The fact is you can only receive a string of data over a connection. This has to be parsed and handled. It's not at all clear that this needs threads when it is clear that regardless of how you do it, it needs a queue.

BTW, PostgreSQL parallel performance is getting a LOT of attention in the move from 9.1 to 9.2.

Robert Haas: My Patches Are Breeding

Wed, 08 Feb 2012 19:28:42 GMT

One of the great things about being a long-term contributor to an open source project like PostgreSQL is that you get to see other people take the stuff you've done and use it as a stepping stone to bigger and better things. One of my early PostgreSQL hacking projects was a patch to extend the syntax of EXPLAIN. Prior to 9.0, the grammar looked like this:

EXPLAIN [ ANALYZE ] [ VERBOSE ] statement

Now, there's nothing particularly wrong with that grammar from a usability perspective, but it turns out to be pretty terrible for extensibility. Let's suppose we want to add a new EXPLAIN option that does something new and different - say, omit the costing information from the output. Then we have to change the grammar to something like this:

EXPLAIN [ ANALYZE ] [ VERBOSE ] [ NOCOSTS ] statement

There are a couple of problems with this. One is that, as the number of options increases, it gets hard to remember the order in which they must be specified. You might think that it would be easy enough to recode the grammar to look like this:

EXPLAIN [ ANALYZE | VERBOSE | NOCOSTS ]... statement

...and it might be, but it's surprisingly easy, when using bison, to create situations that bison finds ambiguous, even though a human being might not. Another problem is that NOCOSTS has to become what's called a keyword, which has a very small but nonzero distributed cost across our entire grammar. Rightly or wrongly, a number of patches that proposed to enhance EXPLAIN in various ways got shot down because of these issues.

Read more »

Selena Deckelmann: Catching up with pgsql-hackers: CRCs, checkpoint performance, SKIP LOCKED ROWS, dry-run pg_archivecleanup

Tue, 07 Feb 2012 16:25:04 GMT

I’ve been sick for a few days, so I settled in with a nice cup of tea and started in on the tremendous backlog I’ve got on pgsql-hackers. I put patch status at the end of each paragraph.

The first thread I read was a patch for implementing 16-bit CRCs for all buffer pages. This is the most recent patch in a multi-year effort around CRCs. There are many small and large problems with implementing this (performance, hint bits and cramped page header space for starters). If you read the whole thread, you’ll see some interesting compromises, and the role that pg_upgrade now plays in the stewardship of page format changes. Patch is still under review.

That lead me over to performance testing results from what was titled as a double-write buffer patch from VMWare. The author of these patches doesn’t respond to comments in-line like everyone else on -hackers does, so it makes his responses pretty difficult to keep track of. Still WIP.

And, related a checkpoint performance tuning treatise from Greg Smith. This started from a patch to add a parameter that allows adding pause time before a checkpoint. He points out that to tune this usefully, you need to know what the number of files a checkpoint typically syncs. (and has a related patch that publishes this information). Patches are still under review.

There’s another discussion about SKIP LOCKED ROWS and a patch, apparently a commercial database feature. Early WIP.

In usability land, there was an addition of a “dry-run” feature to pg_archivecleanup. This one’s committed!

This all makes me so excited for the 9.2 release this year. Also, great to see a bunch of new folks submitting patches.

Chris Travers: Robert Young is Wrong about Threads and PostgreSQL

Tue, 07 Feb 2012 05:54:18 GMT

Robert Young has claimed, not once but twice, that a multi-threaded engine is necessary for database performance to be good enough. I am going to explain here why he is wrong.

Mr Young's argument more or less goes something like this: threads are more efficient, processor-power-wise to spin up than processes, and therefore if you insist on a single-threaded multi-process model, you can't have sufficient performance to make your database system relevant.

There are several important problems with this thinking. The first is that for it to be significant, the thread vs process overhead, plus related IPC, etc. must be a significant portion of actual CPU load. If this is not the case, the difference largely gets lost in the noise. In my experience in looking at databases, I have yet to identify one case where this is the case. Instead things like network connection/teardown, command parsing, and, well, actual work dwarf process/thread startup cycles by several orders of magnitude. This means that the actual startup processing overhead is likely insignificant in the overall scheme of things,

Of course actually starting up a process or thread is not all there is to the picture. There's communication between processes or threads to take into account. There is also the added complexity of locking shared resources in a threaded vs process model. And there are of course the costs that come with a lack of process isolation on the other side. These all have to be weighed against eachother, and it is not clear who the winner is. In fact even these costs will dwarf the startup costs by a significant margin.

Rather the reason why many RDBMS's have gone to a multi-threaded model is that it is one way, perhaps even an elegant way, to support intraquery parallelism. In other words if you can spin up a few more threads in your process, those threads can be conducting scans on different partitions in different table spaces, and thus overall increasing the resource utilization of a given query. This is an important feature and it is currently lacking on PostgreSQL. Postgres-XC however is a project that offers such a solution based on a multi-server, distributed model. It is perhaps not quite ready for prime time but is getting there.

But while a multi-threaded engine would make this sort of parallelism easier to implement, it isn't clear to me that either it is necessary or that the cost in complexity would be worth it compared to multi-process and even multi-server models. Threaded systems are far harder to troubleshoot, far less transparent, and far more brittle than process-oriented systems.

Note that each approach has strong points and weak points. Declaring a specific architecture as a clear winner forgets that these differences exist. Ideally eventually PostgreSQL would support multi-server and multi-process methods of intraquery parallelism. I am not convinced that we need to support a multi-threaded method in addition.

Steve Singer: 10 years of PostgreSQL replication

Mon, 06 Feb 2012 12:26:18 GMT

February 6′th 2012 marks the 10 year anniversary of the open source release of the DBMirror replication system. DBMirror was not the first PostgreSQL replication solution to be released but it was the first one I was involved with.

In the summer of 2001 I was working for Navtech System Support, an aviation software company. We were using PostgreSQL 6.x to store data for one of our applications. We needed to have an up to date copy of the database available on servers at a remote site. We also needed a standby database server in case our primary server failed.

PostgreSQL didn’t have any built in replication but it did have triggers. My colleagues at Navtech and myself figured that if we wrote a trigger that captured a record of all changes made to a database table we would then know what changes needed to be made on a replica to keep it up to date with the master.

I wrote the initial version of DBMirror over a few weeks during the summer of 2001 targetting PostgreSQL 7.1. I no longer have a copy of the original version but I think it only stored the primary key of each row and referenced the original table while replicating. In the September/October time-frame I spent a few more weeks reworked DBMirror into the form we released it in. This version stored the entire contents of a modified row. This is was the first version that saw production use.

In February of 2002 the management team at Navetch was kind enough to allow us too open-source DBMirror. DBMirror was included with the PostgreSQL source code as contrib/dbmirror where it remained as a contrib module until 2007. No one tracks how many users any particular PostgreSQL replication solution has (or had) but based on mailing list traffic DBMirror was far more popular than contrib/rserv, or any of the other options available until Slony become more mature almost 4 years later.

DBMirror was not particularly efficient, it stored each column of a replicated table in its own row of the pendingData table, the replication daemon was written in perl and the query to approximate the commit order taxed the database server. The cleanup script also created a lot of work for vacuum.

The biggest strength DBMirror had going for it was that it was simple to setup and it was flexible. People could install DBMirror and replicate their database after reading a three page README file. The replication daemon was written in perl and was easy to understand. Many people customized dbmirror.pl to perform site specific tasks like renaming tables or excluding rows that matched a particular regex.

I haven’t modified the dbmirror code in years and I would be surprised if there were many dbmirror installations still running. I am truly grateful that I was given the opportunity to develop and release DBMirror. I continue to be involved with PostgreSQL replication as a Slony developer. I also assist with built in replication where by skills and time allow.

In the past 10 years PostgreSQL replication has come a long way from the few thousand lines of C and perl code that made up dbmirror. I look forward to the advances in replication we will see in the next 10 years.

It has been years since I have heard from the other people at Navtech who were involved in the development, testing and deployment of DBMirror. If any of them are reading this they should send me a note.

Joe Abbate: Automated Database Augmentation

Mon, 06 Feb 2012 03:38:49 GMT

Suppose you have a PostgreSQL database like the Pagila sample with 14 tables, each with a last_update timestamp column to record the date and time each row was modified, and it is now a requirement to capture which user effected each change. Or perhaps you have several tables without such audit trail columns and need to add them quickly. Or maybe you have decided to denormalize your design by adding a calculated column, e.g., extended price = unit price times quantity ordered, or a derived column, e.g., carrying the customer name in the invoice table.

If you have some experience as a DBA, the word “drudgery” may have come to mind at the prospect of implementing the above features. It’s possible that, after a while, you’ve developed an approach for dealing with some of them but still wish there’d be some way to automate these thankless tasks.

You may have looked at the Andromeda project’s “automations” which provide some of these capabilities. However, in order to take advantage of the automations, you’ll first have to manually describe your database in a YAML format (and you’ll have to install Apache and PHP). Or you could have tried to use the follow-on project, Triangulum, but essentially you’d still have to create a YAML schema (no need for Apache, but you still need PHP).

Some relief is forthcoming. As a result of discussions resulting from my Business Logic in the Database post, I have been collaborating with Roger Hunwicks on a potential solution to these common DBA needs. The new Pyrseas tool is tentatively named dbextend¹ and its initial documentation is available in the Pyrseas extender branch. This is how I envision dbextend being used.

Consider the opening example. The DBA would create a simple YAML file such as the (abbreviated) one below, listing the tables and the needed features:

schema public:
  table actor:
    audit_columns: default
  table category:
    audit_columns: default
...
  table store:
    audit_columns: default

The DBA would then use this file, say audext.yaml, as input to dbextend, e.g.,

dbextend pagiladb audext.yaml

dbextend reads the PostgreSQL catalogs (using code shared with dbtoyaml and yamltodb), building its internal representation. It also reads the YAML extensions file and builds a parallel (albeit much smaller) structure. Thirdly, it reads extension configuration information, e.g., a definition of what columns need to be added for “audit_columns: default“, for example, modified_timestamp and modified_by_user, what trigger(s) to add, and what function(s) to be created.

The output of dbextend is a YAML schema file, just like the one output by dbtoyaml, which can be piped directly to yamltodb to generate SQL to implement the desired features.

In case you’re wondering, dbextend —like other Pyrseas tools— will require Python, psycopg2 and pyyaml.

What features would you like to see automated? What are your suggested best practices for automating these common needs?

Picture credit: Thanks to Mr. O’Brien, a fourth-grade teacher in Minnesota.

¹ We’re still receptive to some other suitable name.

Filed under: Database tools, PostgreSQL, Python

Andrew Dunstan: JSON for PG 9.2 ... and now for 9.1!

Sun, 05 Feb 2012 18:24:42 GMT

Robert Haas committed his bare bones JSON type five days ago, and three days later I committed my array_to_json and row_to_json constructors, so this is now definitely a go for 9.2. Kudos to Robert for persisting with this - I had just about given up on us getting it done for this release.

As an added bonus for those of you who are a bit impatient to use this stuff, I have back ported this to 9.1 as an extension. You can get the code from my bitbucket repository, and when I have polished it a bit more I will publish it on PGXN.

PLEASE NOTE: this should work fine if you dump and restore the database when moving to 9.2 - just suppress the extension creation. But it won't work with binary upgrade.

Leo Hsu and Regina Obe: Back from DDOS Attack

Sun, 05 Feb 2012 15:43:02 GMT

As many may have noticed, PostgresOnline.com has been down for the past week or so and probably is still not reachable from many parts of the world since our DNS server was also taken down as a result of a Denial of Service (DDOS) attack instigated by an Activision Call of Duty Game exploit that turned thousands of Call of Duty game servers into Zombies launching an attack on us.

We have a small confession to make. One of the businesses we co-own is an e-Commerce site that sells condoms. You never know how people will react when you say that in mixed company so we only mention it in closer company. Some people are glad we are in a business protecting against venereal diseases or unwanted pregnancies and some feel strongly we are violating a mother nature creed of conduct. WowCondoms was the site that was under attack on a UDP port and we are not sure if it was a malicious intent or not since the root instigator has not been found yet. The attack was higher up from our servers so it knocked our ISP who in turn blamed us for their outage. We never saw the traffic.

The tragic thing is that it can happen to any site and does all the time. It really hit home when it happened to us.

Details of our fight are described here: WowCondoms plugs hole in Activision's Call of Duty Game Servers

Chris Travers: LedgerSMB talk at FOSDEM

Sun, 05 Feb 2012 10:54:01 GMT

Hi,

I just arrived at FOSDEM - Brussels. Due to last minute arrangements, my talk about LedgerSMB isn't in the schedule. So, here's the announcement:

There will be a LedgerSMB talk in the Perl devroom (AW1.121)

between 12.25 and 12.45

The talk is on the fosdem site: http://fosdem.org/2012/schedule/event/ledgersmb_perl

With kind regards,

Erik.

Hubert 'depesz' Lubaczewski: Waiting for 9.2 – pg_basebackup from slave

Fri, 03 Feb 2012 14:08:48 GMT

On 25th of January, Simon Riggs committed patch: Allow pg_basebackup FROM standby node WITH safety checking. Base backup follows recommended PROCEDURE, plus goes TO great lengths TO ensure that partial page writes are avoided. Jun Ishizuka AND Fujii Masao, WITH minor modifications In PostgreSQL 9.1 we got pg_basebackup – it is a tool to [...]

Jared Watkins: What I Do – Broadsoft CDR Files to Radius Accounting Records

Fri, 03 Feb 2012 03:40:17 GMT

As part of a larger project I needed to generate real time radius records from the CDR accounting files of several cluster pairs of Broadsoft application servers. So I wrote a perl script to do just that. It maps the CDR fields to radius attribs and encodes the accounting packet using the Net::Radius::Packet CPAN module. In my case I’m using the Radiator radius server from OSC Software on the other end with lots of custom ‘hook code’ to clean up and store the call data coming off our network into a Postgresql database. This is my first time doing any development with radius.. but I’ve been running this script on several servers for a few weeks now and it appears to be quite stable.

Get it on the Project Page

Bruce Momjian: Let's See work_mem

Thu, 02 Feb 2012 15:45:01 GMT

Having shown memory allocation (and deallocation) in my previous blog post, I would like to show work_mem in action.

First, I ran the following database session using a ~6k RPM drive:

Hubert 'depesz' Lubaczewski: Waiting for 9.2 – Trigger Depth

Wed, 01 Feb 2012 17:36:50 GMT

On 25th of January, Alvaro Herrera committed patch: ADD pg_trigger_depth() FUNCTION This reports the depth level OF triggers currently IN execution, OR zero IF NOT called FROM inside a TRIGGER. No catversion bump IN this patch, but you have TO initdb IF you want access TO the NEW FUNCTION. Author: Kevin Grittner [...]

Jim Smith: Why aren't more people/companies using PostgreSQL?

Wed, 01 Feb 2012 16:58:55 GMT

Why aren't more people/companies using PostgreSQL? Why aren't they moving from proprietary databases to PostgreSQL?
Possible answers: I haven't heard of PostgreSQL. I don't think PostgreSQL will meet my needs. I don't know how to get away from my database vendor.
I'd like to hear some of the reasons for not considering PostgreSQL when creating or migrating a database.

Bruce Momjian: Postgres Memory Surprises

Wed, 01 Feb 2012 15:15:01 GMT

In my previous blog entry, I analyzed how various tools (ps and smem) report memory usage. In summary:

ps columns TRS, DRS, and VSZ report virtual address space allocated, not actual RAM allocated.
smem's USS reports a process's private (unshared) memory allocated.
smem's PSS is a sum of process's private memory allocated and a proportional amount of shared memory (both System V shared memory, like Postgres's shared_buffers, and shared libraries).
RSS shows actual RAM allocated, private and shared.

With these issues understood, let's look at a running Postgres cluster:

Hubert 'depesz' Lubaczewski: Waiting for 9.2 – ALTER IF EXISTS

Wed, 01 Feb 2012 13:14:11 GMT

On 23th of January, Simon Riggs committed patch: ALTER <thing> [IF EXISTS] ... allows silent DDL IF required, e.g. ALTER FOREIGN TABLE IF EXISTS foo RENAME TO bar Pavel Stehule This adds important capability – change object of it exists, and not raise exception if it doesn’t. Conditional DDL was always big point on [...]

Pavel Golub: Factorial using CTE in PostgreSQL

Wed, 01 Feb 2012 09:42:26 GMT

Not so long ago I used Common Table Expressions for Fibonacci Numbers calculation.

Today I had a conversation with one client about SQL in general and about PosgreSQL dialect in particular. We talked about SQL’s Turing completeness also. Well my opponent is sure that SQL (Postgres dialect either) is not Turing Complete. But I know for sure that if SQL supports CTE it is Turing Complete. Well, I’m sure about it because some time ago at Oscon 2009 David Fetter said so. And my confidence in this man is boundless.

Anyway, my client proposed to implement Factorial calculation on a pure SQL. I choose Postgres dialect. He agreed.

That was his first mistake! He didn’t know that Postgres has built in “!” and “!!” operators for this purpose.

But to be more convincing, I have wrote this code:

WITH RECURSIVE fact(i, f) AS (
    VALUES (2, 1)
UNION ALL
    SELECT i + 1, i * f FROM fact
)
SELECT f FROM fact LIMIT 10;

Filed under: Coding, PostgreSQL Tagged: CTE, factorial, oscon, PostgreSQL, SQL, trick

Damien Clochard: Write a Foreign Data Wrapper in 15 minutes (Part 1/2)

Tue, 31 Jan 2012 22:49:00 GMT

Among the long list of new features of PostgreSQL 9.1 the new SQL/MED implementation is probably of the most underrated. SQL/MED is an extension of the SQL standard defines foreign data wrappers (FDW) that allow you to reach data located outside of your database, with regular SQL request (MED stands for Management of External Data). In a nutshell, SQL/MED is cool.

As far as i know, PostgreSQL and DB2 are the only major RDBMS providing an implementation of this standard. The beauty with PostgreSQL is that you can write your own data wrappers to connect your database to any storage you want ; other RDBMS, NoSQL storages, web services, whatever system that holds data and can deliver it.

Basically to Write a Foreign Data Wrapper you need to code six callback routines in C to explain how to plan and execute queries toward your new source of data.

PostgreSQL 9.1 has been released a few month ago, but there's already a bunch of FDW available (even though most of them are still beta). At the time, I'm writing this we already have http://wiki.postgresql.org/wiki/For..., so you can connect your PostgreSQL server to ; Oracle, MySQL, SQLlite, anything providing an ODBC driver, LDAP, couchdb, redis, Amazon S3 storage, CSV files, Twitter, Google Search, HTML pages, RSS feeds, ....

For an almost complete list, check out the PostgreSQL wiki ; http://wiki.postgresql.org/wiki/For...

And of course these wrappers are packaged as extension and most of them are available on PGXN and quite easy to install : http://pgxn.org/tag/fdw/

But there's more ! Among these FDW there's one called Multicorn. This particular extension that allow you to write FDW in Python. A wrapper over another wrapper. Basically this means you don't have to know C to write your own FDW and you can use high-level libraries from the Python world.

Ok let's take a simple example. I'd like to query data contained in a web calendar like Google Calendar. Those data are available in ical files, with a specific format called iCalendar. So want i need to do is explain PostgreSQL how to reach and parse the files.

With Python and Multicorn, this is done with 20 lines of code : see icalfdw.py

And that's it.

Of course this is a basic implementation. We can do better with some optimizations and a few log handling. It's also important to keep in mind that foreign data wrappers have strong limitations... I'll talk about all that and other things in my next blog post.

In the meantime, if you're in Brussels this week-end for FOSDEM you can come to the PostgreSQL Devroom, I'll make a presentation of Multicorn along with its creator.

Greg Smith: Using the PostgreSQL System Columns

Tue, 31 Jan 2012 22:02:06 GMT

There are a few parts of the PostgreSQL internals that poke out usefully if you look in the right place for them. One useful set to know about are the System Columns, which you can explicitly request but don't see by default. For example:

psql -x -c "SELECT oid,* FROM pg_class LIMIT 1"

There is no column named oid in the pg_class table, but it's there if you ask for it. The oid used to be relied on more heavily in PostgreSQL as a way to identify rows. That's not true for regular tables anymore, and you really don't want to start doing that for your own tables. OIDs are mainly useful now when joining parts of the System Catalog together. A good example is the Disk Usage query. If you want to find the namespace a table is in, you need to know you can ask for its OID. It's possible to get some of this data out of more portable views like information_schema.tables. But many of the useful things in this area are PostgreSQL specific. Sometimes I see people starting with the information_schema views and joining against other tables using its text name fields, such as the listed table_name. That approach has several edge cases that don't work out correctly; not handling TOAST columns is a common example. That makes them more prone to breaking on you later, probably after your system has gone into production, than an OID based join.

There is also a tableoid system column. As described in the documentation, its main use case is identifying which partition a row come from. That's not a great thing to be driving application logic from, but it can be useful for monitoring or troubleshooting purposes. For example, if you SELECT rows from the parent table in a partitioning inheritance scheme, it's normally expected that no rows will actually be stored there. Checking the tableoid is one way to confirm that. You might confirm that your INSERT/UPDATE trigger is moving rows to the right place using tableoid as well. It's possible to do that for each individual partition section, but running a query against the parent will make sure you hit every row in the table.

Another internal column related to uniquely identifying rows is the ctid. The ctid is a direct pointer to the physical block (using PostgreSQL's 8K page size) and position of a row. ctids are a pair of numbers, and the first row will be (0,1). While this is the fastest way to find a row more than once in the same transaction block, these numbers are not stable in the long term. Any UPDATE and some maintenance operations will change them. One thing you can use these for is finding duplicate data in a table. Let's say you're trying to add a unique constraint, but one row in the table is duplicated 3 times, which blocks the unique index from being created. When rows are identical in every column, you can't write any simple SELECT statement to uniquely identify them. That means deleting all of them but one copy requires some annoying and fragile SQL code, combining DELETE with LIMIT and/or OFFSET--which is always scary. If you use the ctid instead, the implementation will be PostgreSQL specific, but it will also be faster and cleaner. See Deleting Duplicate Records in a Table for an example of how that can be done.

The other system columns all relate to transaction visibility: xmin, cmin, xmax, cmax. When you delete a row in PostgreSQL, it isn't eliminated from disk immediately. It's possible that some other query that's executing at the same time will still need to see that row, and the transaction isolation in PostgreSQL worries about such things. If you ever want to learn how that isolation works, the way the Multiversion Concurrency Control (MVCC) implementation is handled, you can watch parts of it happen. Just open transactions in two different sessions, UPDATE/DELETE in one of them, and then look at those rows in the other. You can still see them in the session where they weren't touched, but they'll be marked to expire in the future via their xmax being set. To really pull that all together, you also need to know about some of the System Information Functions. txid_current() is the most useful for this sort of learning experience, it provides a reference point for the always increasing system transaction ID. You can find a more detailed exploration of using these functions and system columns in Bruce's MVCC Unmasked talk. The "Routine Maintenance" chapter of my book also shows examples how how MVCC works through the perspective of the system columns.

Hubert 'depesz' Lubaczewski: Change in anonmymization of plans on explain.depesz.com

Tue, 31 Jan 2012 12:10:56 GMT

As you perhaps know, explain.depesz.com has anonymization feature. Couple of days ago Filip contacted me and sent a patch that stopped anonymization of typecasts. I thought about the patch, and what it achieves, changed it’s internals, but kept the effect. And today, it got released. Normal plan looks like this. But you might want to [...]

Greg Sabino Mullane: Protecting and auditing your secure PostgreSQL data

Tue, 31 Jan 2012 03:55:59 GMT

span{font-family:sans-serif;}span.p{color:red;}span.d{color:gray;}span.c{color:green;font-family:monospace;}span.o{font-family:monospace;color:#0022cc;}span.oo{font-family:monospace;font-weight:bolder;}

PostgreSQL functions can be written in many languages. These languages fall into two categories, 'trusted' and 'untrusted'. Trusted languages cannot do things "outside of the database", such as writing to local files, opening sockets, sending email, connecting to other systems, etc. Two such languages are PL/pgSQL and and PL/Perl. For "untrusted" languages, such as PL/PerlU, all bets are off, and they have no limitations placed on what they can do. Untrusted languages can be very powerful, and sometimes dangerous.

One of the reasons untrusted languages can be considered dangerous is that they can cause side effects outside of the normal transactional flow that cannot be rolled back. If your function writes to local disk, and the transaction then rolls back, the changes on disk are still there. Working around this is extremely difficult, as there is no way to detect when a transaction has rolled back at the level where you could, for example, undo your local disk changes.

However, there are times when this effect can be very useful. For example, in a recent thread on the PostgreSQL "general" mailing list (aka pgsql-general), somebody asked for a way to audit SELECT queries into a logging table that would survive someone doing a ROLLBACK. In other words, if you had a function named weapon_details() and wanted to have that function log all requests to it by inserting to a table, a user could simply run the query, read the data, and then rollback to thwart the auditing:


BEGIN;

SELECT weapon_details('BFG 9000'); -- also inserts to an audit table

ROLLBACK;                          -- inserts to the audit table are now gone!

Certainly there are other ways to track who is using this query, the most obvious being by enabling full Postgres logging (by setting log_statement = 'all' in your postgresql.conf file.) However, extracting that information from logs is no fun, so let's find a way to make that INSERT stick, even if the surrounding function was rolled back.

Stepping back for one second, we can see there are actually two problems here: restricting access to the data, and logging that access somewhere. The ultimate access restriction is to simply force everyone to go through your custom interface. However, in this example, we will assume that someone has psql access and needs to be able to run ad hoc SQL queries, as well as be able to BEGIN, ROLLBACK, COMMIT, etc.

Let's assume we have a table with some Very Important Data inside of it. Further, let's establish that regular users can only see some of that data, and that we need to know who asked for what data, and when. For this example, we will create a normal user named Alice:


postgres=> CREATE USER alice;
CREATE ROLE

We need a way to tell which rows are suitable for people like Alice to view. We will set up a quick classification scheme using the nifty ENUM feature of PostgreSQL:


postgres=> CREATE TYPE classification AS ENUM (
 'unclassified',
 'restricted',
 'confidential',
 'secret',
 'top secret'
);
CREATE TYPE

Next, as a superuser, we create the table containing sensitive information, and populate it:


postgres=> CREATE TABLE weapon (
  id              SERIAL          PRIMARY KEY,
  name            TEXT            NOT NULL,
  cost            TEXT            NOT NULL,
  security_level  CLASSIFICATION  NOT NULL,
  description     TEXT            NOT NULL DEFAULT 'a fine weapon'
);
NOTICE:  CREATE TABLE will create implicit sequence "weapon_id_seq" for serial column "weapon.id"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "weapon_pkey" for table "weapon"
CREATE TABLE

postgres=> INSERT INTO weapon (name,cost,security_level) VALUES
 ('Crowbar',  10,  'unclassified'),
 ('M9',  200,  'restricted'),
 ('M16A2',  300,  'restricted'),
 ('M4A1',  400,  'restricted'),
 ('FGM-148 Javelin',  700,  'confidential'),
 ('Pulse Rifle',  50000,  'secret'),
 ('Zero Point Energy Field Manipulator',  'unknown',  'top secret');
INSERT 0 7

We don't want anyone but ourselves to be able to access this table, so for safety, we make some explicit revocations. We'll examine the permissions before and after we do this:


postgres=> \dp weapon
                           Access privileges
 Schema |  Name  | Type  | Access privileges | Column access privileges 
--------+--------+-------+-------------------+--------------------------
 public | weapon | table |                   | 

postgres=> REVOKE ALL ON TABLE weapon FROM public;
REVOKE

postgres=> \dp weapon

                               Access privileges
 Schema |  Name  | Type  |     Access privileges     | Column access privileges 
--------+--------+-------+---------------------------+--------------------------
 public | weapon | table | postgres=arwdDxt/postgres |

As you can see, what the REVOKE really does is remove the implicit "no permission" and grant explicit permissions to only the postgres user to view or modify the table. Let's confirm that Alice cannot do anything with that table:


postgres=> \c postgres alice
You are now connected to database "postgres" as user "alice".
postgres=> postgres=> SELECT * FROM weapon;
ERROR:  permission denied for relation weapon
postgres=> postgres=> UPDATE weapon SET id = id;
ERROR:  permission denied for relation weapon

Alice does need to have access to parts of this table, so we will create a "wrapper function" that will query the table for us and return some results. By declaring this function as SECURITY DEFINER, it will run as if the person who created the function invoked it - in this case, the postgres user. For this example, we'll be letting Alice see the "cost and description" of exactly one item at a time. Further, we are not going to let her (or anyone else using this function) view certain items. Only those items classified as "confidential" or lower can be viewed (i.e. "confidential", "restricted", or "unclassified"). Here's the first version of our function:


postgres=> CREATE LANGUAGE plperlu;
CREATE LANGUAGE

postgres=> CREATE OR REPLACE FUNCTION weapon_details(TEXT)
RETURNS TABLE (name TEXT, cost TEXT, description TEXT)
LANGUAGE plperlu
SECURITY DEFINER
AS $bc$

use strict;
use warnings;

## The item they are looking for
my $name = shift;
## We will be nice and ignore the case and any whitespace
$name =~ s{^\s*(\S+)\s*$}{lc $1}e;

## What is the maximum security_level that people who are 
## calling this function can view?
my $seclevel = 'confidential';

## Query the table and pull back the matching row
## We need to differentiate between "not found" and "not allowed",
## by comparing a passed-in level to the security_level for that row.
my $SQL = q{
SELECT name,cost,description,
  CASE WHEN security_level <= $1 THEN 1 ELSE 0 END AS allowed
FROM weapon
WHERE LOWER(name) = $2};

## Run the query, pull back the first row, as well as the allowed column value
my $sth = spi_prepare($SQL, 'CLASSIFICATION', 'TEXT');
my $rv = spi_exec_prepared($sth, $seclevel, $name);
my $row = $rv->{rows}[0];
my $allowed = delete $row->{allowed};

## Did we find anything? If not, simply return undef
if (! $rv->{processed}) {
   return undef;
}

## Throw an exception if we are not allowed to view this row
if (! $allowed) {
   die qq{Sorry, you are not allowed to view information on that weapon!\n};
}

## Return the requested data
return_next($row);

$bc$;
CREATE FUNCTION

The above should be fairly self-explanatory. We are using PL/Perl's built-in database access functions, such as spi_prepare, to do the actual querying. Let's confirm that this works as it should for Alice:


postgres=> \c postgres alice
You are now connected to database "postgres" as user "alice".

postgres=> SELECT * FROM weapon_details('crowbar');
  name   | cost |  description  
---------+------+---------------
 Crowbar | 10   | a fine weapon
(1 row)

postgres=> SELECT * FROM weapon_details('anvil');
 name | cost | description 
------+------+-------------
(0 rows)

postgres=> SELECT * FROM weapon_details('pulse rifle');
ERROR:  Sorry, you are not allowed to view information on that weapon!
CONTEXT:  PL/Perl function "weapon_details"

Now that we have solved the restricted access problem, let's move on the auditing. We will create a simple table to hold information about who accessed what and when:


postgres=> CREATE TABLE data_audit (
  tablename TEXT         NOT NULL,
  arguments TEXT             NULL,
  results   INTEGER          NULL,
  status    TEXT         NOT NULL  DEFAULT 'normal',
  username  TEXT         NOT NULL  DEFAULT session_user,
  txntime   TIMESTAMPTZ  NOT NULL  DEFAULT now(),
  realtime  TIMESTAMPTZ  NOT NULL  DEFAULT clock_timestamp()
);
CREATE TABLE

The 'tablename' column simply records which table they are getting data from. The 'arguments' is a free-form field describing what they were looking for. The 'results' column shows how many matching rows were found. The 'status' column will be used primarily to log unusual requests, such as the case where Alice looks for a forbidden item. The 'username' column records the name of the user doing the searching. Because we are using functions with SECURITY DEFINER set, this needs to be session_user, not current_user, as the latter will switch to 'postgres' within the function, and we want to log the real caller (e.g. 'alice'). The final two columns tell us then the current transaction started, and the exact time when an entry was made inside of this table. As a first attempt, we'll have our function do some simple inserts to this new data_audit table:


postgres=> CREATE OR REPLACE FUNCTION weapon_details(TEXT)
RETURNS TABLE (name TEXT, cost TEXT, description TEXT)
LANGUAGE plperlu
SECURITY DEFINER
AS $bc$

use strict;
use warnings;

## The item they are looking for
my $name = shift;
## We will be nice and ignore the case and any whitespace
$name =~ s{^\s*(\S+)\s*$}{lc $1}e;

## What is the maximum security_level that people who are 
## calling this function can view?
my $seclevel = 'confidential';

## Query the table and pull back the matching row
## We need to differentiate between "not found" and "not allowed",
## by comparing a passed-in level to the security_level for that row.
my $SQL = q{
SELECT name,cost,description,
  CASE WHEN security_level <= $1 THEN 1 ELSE 0 END AS allowed
FROM weapon
WHERE LOWER(name) = $2};

## Run the query, pull back the first row, as well as the allowed column value
my $sth = spi_prepare($SQL, 'CLASSIFICATION', 'TEXT');
my $rv = spi_exec_prepared($sth, $seclevel, $name);
my $row = $rv->{rows}[0];
my $allowed = delete $row->{allowed};


## Log this request
$SQL = 'INSERT INTO data_audit(tablename,arguments,results,status)
  VALUES ($1,$2,$3,$4)';
my $status = $rv->{rows}[0] ? $allowed ? 'normal' : 'forbidden' : 'na';
$sth = spi_prepare($SQL, 'TEXT', 'TEXT', 'INTEGER', 'TEXT');
spi_exec_prepared($sth, 'weapon', $name, $rv->{processed}, $status);


## Did we find anything? If not, simply return undef
if (! $rv->{processed}) {
   return undef;
}

## Throw an exception if we are not allowed to view this row
if (! $allowed) {
   die qq{Sorry, you are not allowed to view information on that weapon!\n};
}

## Return the requested data
return_next($row);

$bc$;

However, this fails the case pointed out in the original poster's email about viewing the data within a transaction that is then rolled back. It also fails to work at all when a forbidden item is requested, as that insert is rolled back by the die() call:


postgres=> \c postgres alice
You are now connected to database "postgres" as user "alice".

postgres=> SELECT * FROM weapon_details('crowbar');
  name   | cost |  description  
---------+------+---------------
 Crowbar | 10   | a fine weapon
(1 row)

postgres=> SELECT * FROM weapon_details('pulse rifle');
ERROR:  Sorry, you are not allowed to view information on that weapon!
CONTEXT:  PL/Perl function "weapon_details"

postgres=> BEGIN;
BEGIN
postgres=> SELECT * FROM weapon_details('m9');
 name | cost |  description  
------+------+---------------
 M9   | 200  | a fine weapon
(1 row)
postgres=> ROLLBACK;
ROLLBACK

postgres=> \c postgres postgres
You are now connected to database "postgres" as user "postgres".
postgres=> SELECT * FROM data_audit \x \g
Expanded display is on.
-[ RECORD 1 ]----------------------------
tablename | weapon
arguments | crowbar
results   | 1
status    | normal
username  | alice
txntime   | 2012-01-30 17:37:39.497491-05
realtime  | 2012-01-30 17:37:39.545891-05

How do we get around this? We need a way to commit something that will survive the surrounding transaction's rollback. The closest thing Postgres has to such a thing at the moment is to connect back to the database with a new and entirely separate connection. Two such popular ways to do so are with the dblink program and the PL/PerlU language. Obviously, we are going to focus on the latter, but all of this could be done with dblink as well. Here are the additional steps to connect back to the database, do the insert, and then leave again:


postgres=> CREATE OR REPLACE FUNCTION weapon_details(TEXT)
RETURNS TABLE (name TEXT, cost TEXT, description TEXT)
LANGUAGE plperlu
SECURITY DEFINER
VOLATILE
AS $bc$

use strict;
use warnings;
use DBI;

## The item they are looking for
my $name = shift;
## We will be nice and ignore the case and any whitespace
$name =~ s{^\s*(\S+)\s*$}{lc $1}e;

## What is the maximum security_level that people who are 
## calling this function can view?
my $seclevel = 'confidential';

## Query the table and pull back the matching row
## We need to differentiate between "not found" and "not allowed",
## by comparing a passed-in level to the security_level for that row.
my $SQL = q{
SELECT name,cost,description,
  CASE WHEN security_level <= $1 THEN 1 ELSE 0 END AS allowed
FROM weapon
WHERE LOWER(name) = $2};

## Run the query, pull back the first row, as well as the allowed column value
my $sth = spi_prepare($SQL, 'CLASSIFICATION', 'TEXT');
my $rv = spi_exec_prepared($sth, $seclevel, $name);
my $row = $rv->{rows}[0];
my $allowed = defined $row ? delete $row->{allowed} : 1;

## Log this request
$SQL = 'INSERT INTO data_audit(username,tablename,arguments,results,status)
  VALUES (?,?,?,?,?)';
my $status = $rv->{rows}[0] ? $allowed ? 'normal' : 'forbidden' : 'na';
my $dbh = DBI->connect('dbi:Pg:service=auditor', '', '',
  {AutoCommit=>0, RaiseError=>1, PrintError=>0});
$sth = $dbh->prepare($SQL);
my $user = spi_exec_query('SELECT session_user')->{rows}[0]{session_user};
$sth->execute($user, 'weapon', $name, $rv->{processed}, $status);
$dbh->commit();

## Did we find anything? If not, simply return undef
if (! $rv->{processed}) {
   return undef;
}

## Throw an exception if we are not allowed to view this row
if (! $allowed) {
   die qq{Sorry, you are not allowed to view information on that weapon!\n};
}

## Return the requested data
return_next($row);

$bc$;
CREATE FUNCTION

Note that because we are making external changes, we marked the function as VOLATILE, which ensures that it will always be run every time it is called, and not cached in any form. We are also using a Postgres service file with the 'db:Pg:service=auditor'. This means that the connection information (username, password, database) is contained in an external file. This is not only tidier than hard-coding those values into this function, but safer as well, as the function itself can be viewed by Alice. Finally, note that we are passing the 'username' directly into the function this time, as we have a brand new connection which is no longer linked to the 'alice' user, so we have to derive it ourselves from "SELECT session_user" and then pass it along.

Once this new function is in place, and we re-run the same queries as we did before, we see three entries in our audit table:


postgres=> \c postgres postgres
You are now connected to database "postgres" as user "postgres".
Expanded display is on.
-[ RECORD 1 ]----------------------------
tablename | weapon
arguments | crowbar
results   | 1
status    | normal
username  | alice
txntime   | 2012-01-30 17:56:01.544557-05
realtime  | 2012-01-30 17:56:01.54569-05
-[ RECORD 2 ]----------------------------
tablename | weapon
arguments | pulse rifle
results   | 1
status    | forbidden
username  | alice
txntime   | 2012-01-30 17:56:01.559532-05
realtime  | 2012-01-30 17:56:01.561225-05
-[ RECORD 3 ]----------------------------
tablename | weapon
arguments | m9
results   | 1
status    | normal
username  | alice
txntime   | 2012-01-30 17:56:01.573335-05
realtime  | 2012-01-30 17:56:01.574989-05

So that's the basic premise of how to solve the auditing problem. For an actual production script, you would probably want to cache the database connection by sticking things inside of the special %_SHARED hash available to PL/Perl and Pl/PerlU. Note that each user gets their own version of that hash, so Alice will not be able to create a function and have access to the same %_SHARED hash that the postgres user has access to. It's probably a good idea to simply not let users like Alice use the language at all. Indeed, that's the default when we do the CREATE LANGUAGE call as above:


postgres=>  \c postgres alice
You are now connected to database "postgres" as user "alice".

postgres=> CREATE FUNCTION showplatform()
RETURNS TEXT
LANGUAGE plperlu
AS $bc$
  return $^O;
$bc$;
ERROR:  permission denied for language plperlu

Further refinements to the actual script might include refactoring the logging bits to a separate function, writing some of the auditing data to a file on the local disk, recording the actual results returned to the user, and sending the data to another Postgres server entirely. For that matter, as we are using DBI, you could send it to other place entirely - such as a MySQL, Oracle, or DB2 database!

Another place for improvement would be associating each user with a security_level classification, such that any user could run the function and only see things at or below their level, rather than hard-coding the level as "confidential" as we have done here. Another nice refinement might be to always return undef (no matches) for items marked "top secret", to prevent the very existence of a top secret weapon from being deduced. :)

Bruce Momjian: Revisiting Memory Reporting

Mon, 30 Jan 2012 21:45:01 GMT

Memory is very important to databases — much more so than for typical applications (presentation). Unfortunately, because memory allocation is so complex, it is often hard to figure out how physical RAM is being used. There are several reasons for the complexity:

Virtual Memory: CPUs in virtual memory mode don't access RAM directly, but rather through page tables.
Segmentation: Memory is allocated in specific segments: text(code), data, and stack.
Sharing: physical RAM is often shared by multiple processes, either in read-only mode (program instructions), shared mode (read/write of share memory), or copy-on-write (create a new copy on write; used by fork).

Robert Haas's excellent blog post highlighted much uncertainty about how to analyze memory usage for specific processes, especially Postgres. I commented on his blog, as did others, and now have a much clearer idea of how to study memory usage. A blog post by Chris Siebenmann directly addresses some of my and Robert's questions, and suggests smem as a way to analyze memory, especially the sharing of memory. It was interesting to learn that smem was designed specifically to address the problems Robert outlined (2007, 2009).

David Wheeler:
SQL Change Management Sans Duplication

Mon, 30 Jan 2012 16:00:00 GMT

In the previous episode in this series, I had one issue with regard to SQL change management that I wanted to resolve: duplication of code between deploy and revert scripts sucks. Worse still is the duplication of code to change just one line of a procedure. Here’s how I propose to eliminate the dupes.

Satoshi Nagayasu: PostgreSQL Conference 2012 on February 24 in Japan

Mon, 30 Jan 2012 00:54:14 GMT

Japan PostgreSQL Users Group will be having an annual 1day technical conference, PostgreSQL Conference 2012, on February 24 in Tokyo. There will be two keynote sessions and 13 sessions on PostgreSQL. (The photo was taken by @koyhoge at PostgreSQL Conference 2011) - PostgreSQL Conference 2012 - NPO法人日本PostgreSQLユーザ http://www.postgresql.jp/events/pgcon2012/ (Google translated) The opening

Josh Berkus: pgCon CfP and other events

Sun, 29 Jan 2012 16:27:11 GMT

The dates for pgCon have been set and this year we have a bunch of exciting new events to make pgCon even more worth attending. First, though, I wanted to remind everyone that the Call for Presentations ends Wednesday, so please make some talk submissions!

Peter Geoghegan: Power consumption in Postgres 9.2

Sun, 29 Jan 2012 00:10:15 GMT

One of the issues of major concern to CPU vendors is optimising the power consumption of their devices. In a world where increasingly, computing resources are purchased in terms of fairly abstract units of work, and where, when selecting the location of a major data-centre, the local price of a kilowatt hour is likely to be weighed just as heavily as the wholesale price of bandwidth, this is quite understandable.

Globally, data centres consumed between 1.1 and 1.5 percent of electricity in 2010 (Source: Koomey). The economic and ecological importance of minimizing that number is fairly obvious.

The broad trend towards increasing amounts of computing being performed within large data centres, with consolidated infrastructure, sold as a service rather than a product is undeniable. Of course, the term “cloud computing” is often applied to this phenomenon. That’s a term that I try to avoid, as it’s fairly ambiguous.

There has been considerable effort to reduce wake-ups when idle in software in general, including everything from web browsers to word processors, which is related to the increasing importance of mobile and embedded platforms. However, this effort is most pronounced among developers of software that is expected to be deployed in virtualised environment on many servers, as wake-ups prevent CPUs from entering various idle states that allow them to save electricity, and when these wakeups are multiplied by thousands of VM instances, they add up very quickly.

As part of 4CaaSt, a research project funded by the European Commission's Seventh Framework programme, that brings together members of industry and academia with the collective goal of producing an innovative platform-as-as-service offering, I spent time reducing the idle wake-ups per second in PostgreSQL. Postgres services firm 2ndQuadrant, where I work as a database architect, has had the development of several PostgreSQL features sponsored by 4CaaSt in furtherance of that goal, of which this is only one.

Historically, PostgreSQL has been weak in this particular area. With a standard Postgres server, with no special configuration, I have measured the wake-ups when idle at 11.5 per second, using Intel’s powertop utility, as of the current 9.1 release. This was thought to be unacceptably high, for 4CaaSt, other solutions that leverage virtualisation extensively, and for embedded systems too.

CPUs have a number of methods of reducing power consumption. These are specified by the ACPI standard (which covers discoverability, configuration and power-management), which in case you hadn't heard, is an open specification that makes minimal assumptions about the architecture or platform in use, and was written to help authors of operating system kernels.

Briefly, ACPI describes the following states (I’ve avoided mentioning other states that have more to do with things like managing laptop hibernation):

Performance states P0 through to PN (i.e. the exact number of states is implementation-defined). Dynamic CPU frequency scaling states. This might be better known under marketing names for specific implementations, like “Intel SpeedStep technology”. Ever notice how the frequency reported for your CPU under /proc/cpuinfo varies from one moment to the next on Linux? This is why! This state tends to be a bit sticky, in that it might take a few seconds to observe changes in frequency, as it is increased to meet demand.

Processor states C0 through to C3. Processors will change this state very quickly, and we basically want to keep this as high as possible, as higher values are associated with using less power.

C0 is the operating state.
C1, or the halt state, is a state where the processor is not executing instructions, but can return to an executing state essentially instantaneously.
C2, or the Stop-Clock state, is a state where the processor maintains all software-visible state, but may take longer to wake up.
C3, or the sleep state, is a state where the processor does not need to maintain cache coherency, but does maintain some other state. There can even be graduations of how deep a sleep this state represents, depending on the implementation - the Intel Core i5 chip in my laptop has a C4 state, for example.

Postgres has a multi-process architecture, which includes at a minimum a number of “auxiliary processes”: processes that perform a single, well defined task across the installation. There is also a process associated with each connection, and autovacuum daemon. Out-of-the-box, you’ll see just the following processes, once the PostgreSQL server becomes idle:

Postmaster. A “clearing-house process”, that manages all other processes, and is minimally exposed to installation-wide failures, so that it has a good chance of recovering the server in the event of an unanticipated failure. To simulate this, you can kill another auxiliary process, and watch as the postmaster starts it again.

Background writer. A process that is charged with writing out “dirty”, or unwritten buffers, in the hope of preventing individual connection backends from ever having to.

WAL Writer. A process that writes out WAL, log files that describe changes made to data in PostgreSQL databases. This is part of a whole subsystem through which the server efficiently maintains its crash-safety/durability guarantees.

Autovacuum launcher. This process notices if there is a need to vacuum dead rows, which are an artifact of the Postgres MVCC implementation. It launches autovacuum worker processes as needed, to perform this garbage collection.

Statistics collector. This process collects statistics on tables, that influence the planner’s subsequent choice of plans, as the distribution and volume of data in tables can radically alter what the optimal plan is for most queries.

Checkpointer (new to 9.2). This process is responsible for managing checkpoints - smoothed writing of all data to disk, so that WAL files that describe those changes in sequence before a certain point can finally be truncated. This used to be an additional responsibility of the background writer.

The reason that all these wake-ups had to occur within each auxiliary process was because they needed to check if the Postmaster was still alive very regularly, or if they had work assigned to them. If they took too long to notice that the Postmaster was dead (a major failure that neccessitates all processes immediately exiting), they would take too long to detach from shared memory, which would prevent the DBA from starting a new instance, as Postgres will refuse to start when it notices this to avoid data corruption.

The solution was to amend the latch, a low-level facility to wait-sleep on an event that was already used for synchronous replication, to also monitor Postmaster death. This infrastructure was committed first. I then proceeded to write patches for each auxiliary process, most recently the Background writer, which was particularly tricky, though accounted for most of the wake-ups when idle among auxiliary processes - usually 5 per second.

Some considerable progress has been made. Additional variability has been added to the number of wake-ups per second, but if you monitor the wake-ups per second using powertop at a sufficiently high granularity, it stabilises at:

  3.8% ( 35.0)   SignalSender
  3.0% ( 27.2)   [kernel scheduler] Load balancing tick
  2.8% ( 25.6)   kworker/0:0
  0.8% ( 7.6)   postgres
  0.6% ( 5.7)   [TLB shootdowns] <kernel IPI>
  0.6% ( 5.6)   [kernel core] hrtimer_start (tick_sched_ti

To give you some notion of how this relates to CPU states, this is an account of the time my laptop’s CPU spends in each of the states at one moment in time, according to powertop:

Cn                Avg residency       P-states (frequencies)
C0 (cpu running)        ( 1.5%)       Turbo Mode     3.0%
polling           0.0ms ( 0.0%)         2.00 Ghz     0.1%
C1 mwait          1.0ms ( 1.3%)         1.80 Ghz     0.1%
C2 mwait          1.5ms ( 1.8%)         1200 Mhz     0.2%
C3 mwait          1.4ms ( 0.4%)          800 Mhz    96.7%
C4 mwait          7.9ms (95.0%)

There is still some more work to do though. Simon Riggs and I submitted a patch to add group commit to PostgreSQL, which is being reviewed in the ongoing commitfest. This feature is anticipated to be very valuable to workloads that are bound by their commit rate, and a number of benchmarks that have been performed are very promising. That patch included support for allowing the WAL Writer to sleep. However, the exact details of group commit’s implementation have yet to be agreed upon, and it is not yet completely clear how effectively we will be able to reduce the WAL writer's idle wake-ups. However, I am hopeful that we will be able to eliminate them entirely, bringing the total number down to 2.6 per second for an idle Postgres 9.2 installation with standard settings. The WAL writer, much like the background writer, accounts for a relatively large 5 wake-ups per second (assuming default settings), and is similarly a bit tricky to adjust in this way.

I’d previously measured the idle wake-ups per second for my distro’s mysqld at 2.2 (mysql-server version 5.1.56, Fedora 14), though when I check now, with mysql-server 5.5.19 on Fedora 16, that’s way up at consistently over 20 wake-ups per second. I’m not sure why that might be, but I welcome input as to what a fair, objective comparison would look like. I have made every effort to be fair here, and I'd speculate that this may have something to do with the storage engine in use in each case.

Andrew Dunstan: Microsoft C extension building

Sat, 28 Jan 2012 14:18:57 GMT

Someone asked on IRC how to build an external module using Microsoft C. I have done a lot of building core Postgres with various versions of Microsoft C, and written or committed substantial parts of the build system we use for it, but sadly I don't know the answer, and couldn't find one anywhere in our docs or on the wiki. Maybe there isn't a simple answer. If so, it's long since time we should remedy that. Unix people tend to look down on Windows, but in fact some of the largest deployments of Postgres anywhere are on Windows. It should not be beyond our wits to support this better.

Planet PostgreSQL

Mark Wong: PDXPUG: February meeting

Jim Smith: Announcing the Arizona Postgres Users Group

Bruce Momjian: Virtualizing Postgres

Chris Travers: Further Response to Robert Young

Robert Haas: My Patches Are Breeding

Selena Deckelmann: Catching up with pgsql-hackers: CRCs, checkpoint performance, SKIP LOCKED ROWS, dry-run pg_archivecleanup

Chris Travers: Robert Young is Wrong about Threads and PostgreSQL

Steve Singer: 10 years of PostgreSQL replication

Joe Abbate: Automated Database Augmentation

Andrew Dunstan: JSON for PG 9.2 ... and now for 9.1!

Leo Hsu and Regina Obe: Back from DDOS Attack

Chris Travers: LedgerSMB talk at FOSDEM

Hubert 'depesz' Lubaczewski: Waiting for 9.2 – pg_basebackup from slave

Jared Watkins: What I Do – Broadsoft CDR Files to Radius Accounting Records

Bruce Momjian: Let's See work_mem

Hubert 'depesz' Lubaczewski: Waiting for 9.2 – Trigger Depth

Jim Smith: Why aren't more people/companies using PostgreSQL?

Bruce Momjian: Postgres Memory Surprises

Hubert 'depesz' Lubaczewski: Waiting for 9.2 – ALTER IF EXISTS

Pavel Golub: Factorial using CTE in PostgreSQL

Damien Clochard: Write a Foreign Data Wrapper in 15 minutes (Part 1/2)

Greg Smith: Using the PostgreSQL System Columns

Hubert 'depesz' Lubaczewski: Change in anonmymization of plans on explain.depesz.com

Greg Sabino Mullane: Protecting and auditing your secure PostgreSQL data

Bruce Momjian: Revisiting Memory Reporting

David Wheeler: SQL Change Management Sans Duplication

Satoshi Nagayasu: PostgreSQL Conference 2012 on February 24 in Japan

Josh Berkus: pgCon CfP and other events

Peter Geoghegan: Power consumption in Postgres 9.2

Andrew Dunstan: Microsoft C extension building

David Wheeler:
SQL Change Management Sans Duplication