Planet PostgreSQL (short)

Mark Wong: PDXPUG: February meeting

Fri, 10 Feb 2012 22:05:29 GMT

When: 7-9pm Thu Feb 16, 2012
Where: Iovation
Who: John Melesky
What: Locks… etc.

High-traffic systems have to deal with locks. That’s just the way it is. But that’s okay! Locks invisibly help manage concurrency! Properly structured transactions will never run into deadlocks with each other! Right?

Well, sometimes locks are very visible, and sometimes you end up with deadlocks that don’t make sense. I’m going to talk through a couple of row-locking problems i’ve run into, and explain how and why that should end up changing PostgreSQL’s SQL syntax.

John’s been poking computers since he was old enough to type, which was long enough ago that he debated whether to make a “POKEing” joke. He’s a Software Engineer at Janrain, working with the reporting and analytics infrastructure.

Our meeting will be held at Iovation, on the 32nd floor of the US Bancorp Tower at 111 SW 5th (5th & Oak). It’s right on the Green & Yellow Max lines. Underground bike parking is available in the parking garage; outdoors all around the block in the usual spots. No bikes in the office, sorry!

Building security will close access to the floor at 7:30.

After-meeting beer location TBD. See you there!

Jim Smith: Announcing the Arizona Postgres Users Group

Thu, 09 Feb 2012 23:10:30 GMT

As mentioned in an earlier post, we're announcing the start of a PUG for Arizona. The first meeting will be Thursday, March 29, from 5 to 7 at Bull's facility in Northwest Phoenix. Refreshments will be provided.

You can join and get onto the mailing list and RSVP for the first meeting at https://www.bigtent.com/groups/azpug

We look forward to meeting other PostgreSQL lovers and to help grow the PostgreSQL community in our part of the world. Please feel free to pass the above link to anyone that's interested.

Bruce Momjian: Virtualizing Postgres

Thu, 09 Feb 2012 18:00:01 GMT

Postgres is an ideal database to run in a virtual environment or public/private cloud — one reason is that Postgres relies heavily on the operating system, rather than using features like raw devices. Second, its license is obviously very flexible for virtual deployments.

I am often asked about running Postgres in virtual environments, and I usually answer that it runs just fine --- and it does. However, I am starting to realize that I am not answering the more complex questions of which visualization technology to choose, and what is the performance and reliability impact of virtualization.

Chris Travers: Further Response to Robert Young

Thu, 09 Feb 2012 01:47:46 GMT

Robert Young has responded to my previous post and clarified his position. However, he is still incorrect in his assessment.

The issue is lost cycles as one moves down to a single thread. It's more about the number of clients that can be supported "simultaneously". At one time I was also enamoured of thread count to execute queries in parallel; not so much now. For reporting generally, and BI/DW specifically, sure. But for OLTP, where my interest lies, not so much. There, client count is what matters. Clients run relatively discrete transactions on few rows; parallelism isn't much help for that sort of client. Yes, to some extent I've changed my mind.

I think this makes the mistake of assuming that single threaded multiple process models cannot utilize hyperthreading architectures to increase computational efficiency.

The first thing to note is that how a CPU cycles are utilized is primarily the job of the scheduler which is a part of any multitasking kernel, and different operating systems utilize very different process and thread models. These cannot be merely reduced to their hardware equivalents. If they were we would see process vs thread performance to be largely equivalent in Windows and Linux, but in fact while Windows is very thread-friendly and process-unfriendly, though my understanding is that applications on Windows running under SUA experience better multiprocess performance than under the Win32 subsystem.

On Linux and UNIX there is very little difference between a pthread and a process except for memory isolation. Individual processes can be scheduled on logical cores supporting hyperthreading, for example. The overhead for supporting multiple processes is low.

On the other hand, on Windows, threads are cheap and processes are expensive. This is one reason why I typically recommend that high-concurrency PostgreSQL instances are not run on Windows. My understanding is that SUA addresses this issue but I can't find solid comparisons at the moment and I don't know of a PostgreSQL port to[...]

Robert Haas: My Patches Are Breeding

Wed, 08 Feb 2012 19:28:42 GMT

One of the great things about being a long-term contributor to an open source project like PostgreSQL is that you get to see other people take the stuff you've done and use it as a stepping stone to bigger and better things. One of my early PostgreSQL hacking projects was a patch to extend the syntax of EXPLAIN. Prior to 9.0, the grammar looked like this:

EXPLAIN [ ANALYZE ] [ VERBOSE ] statement

Now, there's nothing particularly wrong with that grammar from a usability perspective, but it turns out to be pretty terrible for extensibility. Let's suppose we want to add a new EXPLAIN option that does something new and different - say, omit the costing information from the output. Then we have to change the grammar to something like this:

EXPLAIN [ ANALYZE ] [ VERBOSE ] [ NOCOSTS ] statement

There are a couple of problems with this. One is that, as the number of options increases, it gets hard to remember the order in which they must be specified. You might think that it would be easy enough to recode the grammar to look like this:

EXPLAIN [ ANALYZE | VERBOSE | NOCOSTS ]... statement

...and it might be, but it's surprisingly easy, when using bison, to create situations that bison finds ambiguous, even though a human being might not. Another problem is that NOCOSTS has to become what's called a keyword, which has a very small but nonzero distributed cost across our entire grammar. Rightly or wrongly, a number of patches that proposed to enhance EXPLAIN in various ways got shot down because of these issues.

Read more »

Selena Deckelmann: Catching up with pgsql-hackers: CRCs, checkpoint performance, SKIP LOCKED ROWS, dry-run pg_archivecleanup

Tue, 07 Feb 2012 16:25:04 GMT

I’ve been sick for a few days, so I settled in with a nice cup of tea and started in on the tremendous backlog I’ve got on pgsql-hackers. I put patch status at the end of each paragraph.

The first thread I read was a patch for implementing 16-bit CRCs for all buffer pages. This is the most recent patch in a multi-year effort around CRCs. There are many small and large problems with implementing this (performance, hint bits and cramped page header space for starters). If you read the whole thread, you’ll see some interesting compromises, and the role that pg_upgrade now plays in the stewardship of page format changes. Patch is still under review.

That lead me over to performance testing results from what was titled as a double-write buffer patch from VMWare. The author of these patches doesn’t respond to comments in-line like everyone else on -hackers does, so it makes his responses pretty difficult to keep track of. Still WIP.

And, related a checkpoint performance tuning treatise from Greg Smith. This started from a patch to add a parameter that allows adding pause time before a checkpoint. He points out that to tune this usefully, you need to know what the number of files a checkpoint typically syncs. (and has a related patch that publishes this information). Patches are still under review.

There’s another discussion about SKIP LOCKED ROWS and a patch, apparently a commercial database feature. Early WIP.

In usability land, there was an addition of a “dry-run” feature to pg_archivecleanup. This one’s committed!

This all makes me so excited for the 9.2 release this year. Also, great to see a bunch of new folks submitting patches.

Chris Travers: Robert Young is Wrong about Threads and PostgreSQL

Tue, 07 Feb 2012 05:54:18 GMT

Robert Young has claimed, not once but twice, that a multi-threaded engine is necessary for database performance to be good enough. I am going to explain here why he is wrong.

Mr Young's argument more or less goes something like this: threads are more efficient, processor-power-wise to spin up than processes, and therefore if you insist on a single-threaded multi-process model, you can't have sufficient performance to make your database system relevant.

There are several important problems with this thinking. The first is that for it to be significant, the thread vs process overhead, plus related IPC, etc. must be a significant portion of actual CPU load. If this is not the case, the difference largely gets lost in the noise. In my experience in looking at databases, I have yet to identify one case where this is the case. Instead things like network connection/teardown, command parsing, and, well, actual work dwarf process/thread startup cycles by several orders of magnitude. This means that the actual startup processing overhead is likely insignificant in the overall scheme of things,

Of course actually starting up a process or thread is not all there is to the picture. There's communication between processes or threads to take into account. There is also the added complexity of locking shared resources in a threaded vs process model. And there are of course the costs that come with a lack of process isolation on the other side. These all have to be weighed against eachother, and it is not clear who the winner is. In fact even these costs will dwarf the startup costs by a significant margin.

Rather the reason why many RDBMS's have gone to a multi-threaded model is that it is one way, perhaps even an elegant way, to support intraquery parallelism. In other words if you can spin up a few more threads in your process, those threads can be conducting scans on different partitions in different table spaces, and thus overall increasing the resource utilization of a given query. This is an important [...]

Steve Singer: 10 years of PostgreSQL replication

Mon, 06 Feb 2012 12:26:18 GMT

February 6′th 2012 marks the 10 year anniversary of the open source release of the DBMirror replication system. DBMirror was not the first PostgreSQL replication solution to be released but it was the first one I was involved with.

In the summer of 2001 I was working for Navtech System Support, an aviation software company. We were using PostgreSQL 6.x to store data for one of our applications. We needed to have an up to date copy of the database available on servers at a remote site. We also needed a standby database server in case our primary server failed.

PostgreSQL didn’t have any built in replication but it did have triggers. My colleagues at Navtech and myself figured that if we wrote a trigger that captured a record of all changes made to a database table we would then know what changes needed to be made on a replica to keep it up to date with the master.

I wrote the initial version of DBMirror over a few weeks during the summer of 2001 targetting PostgreSQL 7.1. I no longer have a copy of the original version but I think it only stored the primary key of each row and referenced the original table while replicating. In the September/October time-frame I spent a few more weeks reworked DBMirror into the form we released it in. This version stored the entire contents of a modified row. This is was the first version that saw production use.

In February of 2002 the management team at Navetch was kind enough to allow us too open-source DBMirror. DBMirror was included with the PostgreSQL source code as contrib/dbmirror where it remained as a contrib module until 2007. No one tracks how many users any particular PostgreSQL replication solution has (or had) but based on mailing list traffic DBMirror was far more popular than contrib/rserv, or any of the other options available until Slony become more mature almost 4 years later.

DBMirror was not particularly efficient, it stored each column of a replicated table in its own row of the pendingData table, the replication daemon was written in perl and the query to

[...]

Joe Abbate: Automated Database Augmentation

Mon, 06 Feb 2012 03:38:49 GMT

Suppose you have a PostgreSQL database like the Pagila sample with 14 tables, each with a last_update timestamp column to record the date and time each row was modified, and it is now a requirement to capture which user effected each change. Or perhaps you have several tables without such audit trail columns and need to add them quickly. Or maybe you have decided to denormalize your design by adding a calculated column, e.g., extended price = unit price times quantity ordered, or a derived column, e.g., carrying the customer name in the invoice table.

If you have some experience as a DBA, the word “drudgery” may have come to mind at the prospect of implementing the above features. It’s possible that, after a while, you’ve developed an approach for dealing with some of them but still wish there’d be some way to automate these thankless tasks.

You may have looked at the Andromeda project’s “automations” which provide some of these capabilities. However, in order to take advantage of the automations, you’ll first have to manually describe your database in a YAML format (and you’ll have to install Apache and PHP). Or you could have tried to use the follow-on project, Triangulum, but essentially you’d still have to create a YAML schema (no need for Apache, but you still need PHP).

Some relief is forthcoming. As a result of discussions resulting from my Business Logic in the Database post, I have been collaborating with Roger Hunwicks on a potential solution to these common DBA needs. The new Pyrseas tool is tentatively named dbextend¹ and its initial documentation is available in the Pyrseas extender branch. This is how I envision dbextend being used.

Consider the opening example. The DBA would create a simple YAML file such as the (abbreviated) one below, listing the tables and the needed features:

schema public:
  table actor:
    audit_columns: default
  table category:
    audit_columns: default
...
  table store:
    audit_columns: default

The DBA would then use this file, say audext.yaml, as input to dbexte

[...]

Andrew Dunstan: JSON for PG 9.2 ... and now for 9.1!

Sun, 05 Feb 2012 18:24:42 GMT

Robert Haas committed his bare bones JSON type five days ago, and three days later I committed my array_to_json and row_to_json constructors, so this is now definitely a go for 9.2. Kudos to Robert for persisting with this - I had just about given up on us getting it done for this release.

As an added bonus for those of you who are a bit impatient to use this stuff, I have back ported this to 9.1 as an extension. You can get the code from my bitbucket repository, and when I have polished it a bit more I will publish it on PGXN.

PLEASE NOTE: this should work fine if you dump and restore the database when moving to 9.2 - just suppress the extension creation. But it won't work with binary upgrade.

Leo Hsu and Regina Obe: Back from DDOS Attack

Sun, 05 Feb 2012 15:43:02 GMT

As many may have noticed, PostgresOnline.com has been down for the past week or so and probably is still not reachable from many parts of the world since our DNS server was also taken down as a result of a Denial of Service (DDOS) attack instigated by an Activision Call of Duty Game exploit that turned thousands of Call of Duty game servers into Zombies launching an attack on us.

We have a small confession to make. One of the businesses we co-own is an e-Commerce site that sells condoms. You never know how people will react when you say that in mixed company so we only mention it in closer company. Some people are glad we are in a business protecting against venereal diseases or unwanted pregnancies and some feel strongly we are violating a mother nature creed of conduct. WowCondoms was the site that was under attack on a UDP port and we are not sure if it was a malicious intent or not since the root instigator has not been found yet. The attack was higher up from our servers so it knocked our ISP who in turn blamed us for their outage. We never saw the traffic.

The tragic thing is that it can happen to any site and does all the time. It really hit home when it happened to us.

Details of our fight are described here: WowCondoms plugs hole in Activision's Call of Duty Game Servers

Chris Travers: LedgerSMB talk at FOSDEM

Sun, 05 Feb 2012 10:54:01 GMT

Hi,

I just arrived at FOSDEM - Brussels. Due to last minute arrangements, my talk about LedgerSMB isn't in the schedule. So, here's the announcement:

There will be a LedgerSMB talk in the Perl devroom (AW1.121)

between 12.25 and 12.45

The talk is on the fosdem site: http://fosdem.org/2012/schedule/event/ledgersmb_perl

With kind regards,

Erik.

Hubert 'depesz' Lubaczewski: Waiting for 9.2 – pg_basebackup from slave

Fri, 03 Feb 2012 14:08:48 GMT

On 25th of January, Simon Riggs committed patch: Allow pg_basebackup FROM standby node WITH safety checking. Base backup follows recommended PROCEDURE, plus goes TO great lengths TO ensure that partial page writes are avoided. Jun Ishizuka AND Fujii Masao, WITH minor modifications In PostgreSQL 9.1 we got pg_basebackup – it is a tool to [...]

Jared Watkins: What I Do – Broadsoft CDR Files to Radius Accounting Records

Fri, 03 Feb 2012 03:40:17 GMT

As part of a larger project I needed to generate real time radius records from the CDR accounting files of several cluster pairs of Broadsoft application servers. So I wrote a perl script to do just that. It maps the CDR fields to radius attribs and encodes the accounting packet using the Net::Radius::Packet CPAN module. In my case I’m using the Radiator radius server from OSC Software on the other end with lots of custom ‘hook code’ to clean up and store the call data coming off our network into a Postgresql database. This is my first time doing any development with radius.. but I’ve been running this script on several servers for a few weeks now and it appears to be quite stable.

Get it on the Project Page

Bruce Momjian: Let's See work_mem

Thu, 02 Feb 2012 15:45:01 GMT

Having shown memory allocation (and deallocation) in my previous blog post, I would like to show work_mem in action.

First, I ran the following database session using a ~6k RPM drive:

Hubert 'depesz' Lubaczewski: Waiting for 9.2 – Trigger Depth

Wed, 01 Feb 2012 17:36:50 GMT

On 25th of January, Alvaro Herrera committed patch: ADD pg_trigger_depth() FUNCTION This reports the depth level OF triggers currently IN execution, OR zero IF NOT called FROM inside a TRIGGER. No catversion bump IN this patch, but you have TO initdb IF you want access TO the NEW FUNCTION. Author: Kevin Grittner [...]

Jim Smith: Why aren't more people/companies using PostgreSQL?

Wed, 01 Feb 2012 16:58:55 GMT

Why aren't more people/companies using PostgreSQL? Why aren't they moving from proprietary databases to PostgreSQL?
Possible answers: I haven't heard of PostgreSQL. I don't think PostgreSQL will meet my needs. I don't know how to get away from my database vendor.
I'd like to hear some of the reasons for not considering PostgreSQL when creating or migrating a database.

Bruce Momjian: Postgres Memory Surprises

Wed, 01 Feb 2012 15:15:01 GMT

In my previous blog entry, I analyzed how various tools (ps and smem) report memory usage. In summary:

ps columns TRS, DRS, and VSZ report virtual address space allocated, not actual RAM allocated.
smem's USS reports a process's private (unshared) memory allocated.
smem's PSS is a sum of process's private memory allocated and a proportional amount of shared memory (both System V shared memory, like Postgres's shared_buffers, and shared libraries).
RSS shows actual RAM allocated, private and shared.

With these issues understood, let's look at a running Postgres cluster:

Hubert 'depesz' Lubaczewski: Waiting for 9.2 – ALTER IF EXISTS

Wed, 01 Feb 2012 13:14:11 GMT

On 23th of January, Simon Riggs committed patch: ALTER <thing> [IF EXISTS] ... allows silent DDL IF required, e.g. ALTER FOREIGN TABLE IF EXISTS foo RENAME TO bar Pavel Stehule This adds important capability – change object of it exists, and not raise exception if it doesn’t. Conditional DDL was always big point on [...]

Pavel Golub: Factorial using CTE in PostgreSQL

Wed, 01 Feb 2012 09:42:26 GMT

Not so long ago I used Common Table Expressions for Fibonacci Numbers calculation.

Today I had a conversation with one client about SQL in general and about PosgreSQL dialect in particular. We talked about SQL’s Turing completeness also. Well my opponent is sure that SQL (Postgres dialect either) is not Turing Complete. But I know for sure that if SQL supports CTE it is Turing Complete. Well, I’m sure about it because some time ago at Oscon 2009 David Fetter said so. And my confidence in this man is boundless.

Anyway, my client proposed to implement Factorial calculation on a pure SQL. I choose Postgres dialect. He agreed.

That was his first mistake! He didn’t know that Postgres has built in “!” and “!!” operators for this purpose.

But to be more convincing, I have wrote this code:

WITH RECURSIVE fact(i, f) AS (
    VALUES (2, 1)
UNION ALL
    SELECT i + 1, i * f FROM fact
)
SELECT f FROM fact LIMIT 10;

Filed under: Coding, PostgreSQL Tagged: CTE, factorial, oscon, PostgreSQL, SQL, trick

Damien Clochard: Write a Foreign Data Wrapper in 15 minutes (Part 1/2)

Tue, 31 Jan 2012 22:49:00 GMT

Among the long list of new features of PostgreSQL 9.1 the new SQL/MED implementation is probably of the most underrated. SQL/MED is an extension of the SQL standard defines foreign data wrappers (FDW) that allow you to reach data located outside of your database, with regular SQL request (MED stands for Management of External Data). In a nutshell, SQL/MED is cool.

As far as i know, PostgreSQL and DB2 are the only major RDBMS providing an implementation of this standard. The beauty with PostgreSQL is that you can write your own data wrappers to connect your database to any storage you want ; other RDBMS, NoSQL storages, web services, whatever system that holds data and can deliver it.

Basically to Write a Foreign Data Wrapper you need to code six callback routines in C to explain how to plan and execute queries toward your new source of data.

PostgreSQL 9.1 has been released a few month ago, but there's already a bunch of FDW available (even though most of them are still beta). At the time, I'm writing this we already have http://wiki.postgresql.org/wiki/For..., so you can connect your PostgreSQL server to ; Oracle, MySQL, SQLlite, anything providing an ODBC driver, LDAP, couchdb, redis, Amazon S3 storage, CSV files, Twitter, Google Search, HTML pages, RSS feeds, ....

For an almost complete list, check out the PostgreSQL wiki ; http://wiki.postgresql.org/wiki/For...

And of course these wrappers are packaged as extension and most of them are available on PGXN and quite easy to install : http://pgxn.org/tag/fdw/

But there's more ! Among these FDW there's one called Multicorn. This particular extension that allow you to write FDW in Python. A wrapper over another wrapper. Basically this means you don't have to know C to write your own FDW and you can use high-level libraries from the Python world.

Ok let's take a simple example. I'd like to query data contained in a web calendar like Google Calendar. Those data are available in ical files, with a specific format called iCalendar. So want i need to do is explain P

[...]

Greg Smith: Using the PostgreSQL System Columns

Tue, 31 Jan 2012 22:02:06 GMT

There are a few parts of the PostgreSQL internals that poke out usefully if you look in the right place for them. One useful set to know about are the System Columns, which you can explicitly request but don't see by default. For example:

psql -x -c "SELECT oid,* FROM pg_class LIMIT 1"

There is no column named oid in the pg_class table, but it's there if you ask for it. The oid used to be relied on more heavily in PostgreSQL as a way to identify rows. That's not true for regular tables anymore, and you really don't want to start doing that for your own tables. OIDs are mainly useful now when joining parts of the System Catalog together. A good example is the Disk Usage query. If you want to find the namespace a table is in, you need to know you can ask for its OID. It's possible to get some of this data out of more portable views like information_schema.tables. But many of the useful things in this area are PostgreSQL specific. Sometimes I see people starting with the information_schema views and joining against other tables using its text name fields, such as the listed table_name. That approach has several edge cases that don't work out correctly; not handling TOAST columns is a common example. That makes them more prone to breaking on you later, probably after your system has gone into production, than an OID based join.

There is also a tableoid system column. As described in the documentation, its main use case is identifying which partition a row come from. That's not a great thing to be driving application logic from, but it can be useful for monitoring or troubleshooting purposes. For example, if you SELECT rows from the parent table in a partitioning inheritance scheme, it's normally expected that no rows will actually be stored there. Checking the tableoid is one way to confirm that. You might confirm that your INSERT/UPDATE trigger is moving rows to the right place using tableoid as well. It's possible to do that for each individual partition section, but running a query against the[...]

Hubert 'depesz' Lubaczewski: Change in anonmymization of plans on explain.depesz.com

Tue, 31 Jan 2012 12:10:56 GMT

As you perhaps know, explain.depesz.com has anonymization feature. Couple of days ago Filip contacted me and sent a patch that stopped anonymization of typecasts. I thought about the patch, and what it achieves, changed it’s internals, but kept the effect. And today, it got released. Normal plan looks like this. But you might want to [...]

Greg Sabino Mullane: Protecting and auditing your secure PostgreSQL data

Tue, 31 Jan 2012 03:55:59 GMT

span{font-family:sans-serif;}span.p{color:red;}span.d{color:gray;}span.c{color:green;font-family:monospace;}span.o{font-family:monospace;color:#0022cc;}span.oo{font-family:monospace;font-weight:bolder;}

PostgreSQL functions can be written in many languages. These languages fall into two categories, 'trusted' and 'untrusted'. Trusted languages cannot do things "outside of the database", such as writing to local files, opening sockets, sending email, connecting to other systems, etc. Two such languages are PL/pgSQL and and PL/Perl. For "untrusted" languages, such as PL/PerlU, all bets are off, and they have no limitations placed on what they can do. Untrusted languages can be very powerful, and sometimes dangerous.

One of the reasons untrusted languages can be considered dangerous is that they can cause side effects outside of the normal transactional flow that cannot be rolled back. If your function writes to local disk, and the transaction then rolls back, the changes on disk are still there. Working around this is extremely difficult, as there is no way to detect when a transaction has rolled back at the level where you could, for example, undo your local disk changes.

However, there are times when this effect can be very useful. For example, in a recent thread on the PostgreSQL "general" mailing list (aka pgsql-general), somebody asked for a way to audit SELECT queries into a logging table that would survive someone doing a ROLLBACK. In other words, if you had a function named weapon_details() and wanted to have that function log all requests to it by inserting to a table, a user could simply run the query, read the data, and then rollback to thwart the auditing:


BEGIN;

SELECT weapon_details('BFG 9000'); -- also inserts to an audit table

ROLLBACK;                          -- inserts to the audit table are now gone!

Certainly there are other ways to track who is using this query, the most obvious being by enabling full Postgres logging (by setting log_statement = 'all' in your postgresql.conf file.) Howe

[...]

Bruce Momjian: Revisiting Memory Reporting

Mon, 30 Jan 2012 21:45:01 GMT

Memory is very important to databases — much more so than for typical applications (presentation). Unfortunately, because memory allocation is so complex, it is often hard to figure out how physical RAM is being used. There are several reasons for the complexity:

Virtual Memory: CPUs in virtual memory mode don't access RAM directly, but rather through page tables.
Segmentation: Memory is allocated in specific segments: text(code), data, and stack.
Sharing: physical RAM is often shared by multiple processes, either in read-only mode (program instructions), shared mode (read/write of share memory), or copy-on-write (create a new copy on write; used by fork).

Robert Haas's excellent blog post highlighted much uncertainty about how to analyze memory usage for specific processes, especially Postgres. I commented on his blog, as did others, and now have a much clearer idea of how to study memory usage. A blog post by Chris Siebenmann directly addresses some of my and Robert's questions, and suggests smem as a way to analyze memory, especially the sharing of memory. It was interesting to learn that smem was designed specifically to address the problems Robert outlined (2007, 2009).

David Wheeler:
SQL Change Management Sans Duplication

Mon, 30 Jan 2012 16:00:00 GMT

In the previous episode in this series, I had one issue with regard to SQL change management that I wanted to resolve: duplication of code between deploy and revert scripts sucks. Worse still is the duplication of code to change just one line of a procedure. Here’s how I propose to eliminate the dupes.

Satoshi Nagayasu: PostgreSQL Conference 2012 on February 24 in Japan

Mon, 30 Jan 2012 00:54:14 GMT

Japan PostgreSQL Users Group will be having an annual 1day technical conference, PostgreSQL Conference 2012, on February 24 in Tokyo. There will be two keynote sessions and 13 sessions on PostgreSQL. (The photo was taken by @koyhoge at PostgreSQL Conference 2011) - PostgreSQL Conference 2012 - NPO法人日本PostgreSQLユーザ http://www.postgresql.jp/events/pgcon2012/ (Google translated) The opening

Josh Berkus: pgCon CfP and other events

Sun, 29 Jan 2012 16:27:11 GMT

The dates for pgCon have been set and this year we have a bunch of exciting new events to make pgCon even more worth attending. First, though, I wanted to remind everyone that the Call for Presentations ends Wednesday, so please make some talk submissions!

Peter Geoghegan: Power consumption in Postgres 9.2

Sun, 29 Jan 2012 00:10:15 GMT

One of the issues of major concern to CPU vendors is optimising the power consumption of their devices. In a world where increasingly, computing resources are purchased in terms of fairly abstract units of work, and where, when selecting the location of a major data-centre, the local price of a kilowatt hour is likely to be weighed just as heavily as the wholesale price of bandwidth, this is quite understandable.

Globally, data centres consumed between 1.1 and 1.5 percent of electricity in 2010 (Source: Koomey). The economic and ecological importance of minimizing that number is fairly obvious.

The broad trend towards increasing amounts of computing being performed within large data centres, with consolidated infrastructure, sold as a service rather than a product is undeniable. Of course, the term “cloud computing” is often applied to this phenomenon. That’s a term that I try to avoid, as it’s fairly ambiguous.

There has been considerable effort to reduce wake-ups when idle in software in general, including everything from web browsers to word processors, which is related to the increasing importance of mobile and embedded platforms. However, this effort is most pronounced among developers of software that is expected to be deployed in virtualised environment on many servers, as wake-ups prevent CPUs from entering various idle states that allow them to save electricity, and when these wakeups are multiplied by thousands of VM instances, they add up very quickly.

As part of 4CaaSt, a research project funded by the European Commission's Seventh Framework programme, that brings together members of industry and academia with the collective goal of producing an innovative platform-as-as-service offering, I spent time reducing the idle wake-ups per second in PostgreSQL. Postgres services firm 2ndQuadrant, where I work as a database architect, has had the development of several PostgreSQL features sponsored by 4CaaSt in furtherance of that goal, of which this is only one.

Historically, PostgreSQL has been weak in

[...]

Andrew Dunstan: Microsoft C extension building

Sat, 28 Jan 2012 14:18:57 GMT

Someone asked on IRC how to build an external module using Microsoft C. I have done a lot of building core Postgres with various versions of Microsoft C, and written or committed substantial parts of the build system we use for it, but sadly I don't know the answer, and couldn't find one anywhere in our docs or on the wiki. Maybe there isn't a simple answer. If so, it's long since time we should remedy that. Unix people tend to look down on Windows, but in fact some of the largest deployments of Postgres anywhere are on Windows. It should not be beyond our wits to support this better.

Planet PostgreSQL (short)

Mark Wong: PDXPUG: February meeting

Jim Smith: Announcing the Arizona Postgres Users Group

Bruce Momjian: Virtualizing Postgres

Chris Travers: Further Response to Robert Young

Robert Haas: My Patches Are Breeding

Selena Deckelmann: Catching up with pgsql-hackers: CRCs, checkpoint performance, SKIP LOCKED ROWS, dry-run pg_archivecleanup

Chris Travers: Robert Young is Wrong about Threads and PostgreSQL

Steve Singer: 10 years of PostgreSQL replication

Joe Abbate: Automated Database Augmentation

Andrew Dunstan: JSON for PG 9.2 ... and now for 9.1!

Leo Hsu and Regina Obe: Back from DDOS Attack

Chris Travers: LedgerSMB talk at FOSDEM

Hubert 'depesz' Lubaczewski: Waiting for 9.2 – pg_basebackup from slave

Jared Watkins: What I Do – Broadsoft CDR Files to Radius Accounting Records

Bruce Momjian: Let's See work_mem

Hubert 'depesz' Lubaczewski: Waiting for 9.2 – Trigger Depth

Jim Smith: Why aren't more people/companies using PostgreSQL?

Bruce Momjian: Postgres Memory Surprises

Hubert 'depesz' Lubaczewski: Waiting for 9.2 – ALTER IF EXISTS

Pavel Golub: Factorial using CTE in PostgreSQL

Damien Clochard: Write a Foreign Data Wrapper in 15 minutes (Part 1/2)

Greg Smith: Using the PostgreSQL System Columns

Hubert 'depesz' Lubaczewski: Change in anonmymization of plans on explain.depesz.com

Greg Sabino Mullane: Protecting and auditing your secure PostgreSQL data

Bruce Momjian: Revisiting Memory Reporting

David Wheeler: SQL Change Management Sans Duplication

Satoshi Nagayasu: PostgreSQL Conference 2012 on February 24 in Japan

Josh Berkus: pgCon CfP and other events

Peter Geoghegan: Power consumption in Postgres 9.2

Andrew Dunstan: Microsoft C extension building

David Wheeler:
SQL Change Management Sans Duplication