Planet PostgreSQL

Bruce Momjian: Template1 Me!

Wed, 16 May 2012 18:30:01 GMT

You might have heard of the template1 database before, or seen it in the output of pg_dumpall:

REVOKE ALL ON DATABASE template1 FROM PUBLIC;
…

Jeff Frost: Slony Upstart Script for Ubuntu 10.04 LTS

Tue, 15 May 2012 21:48:40 GMT

Since Ubuntu is moving towards using upstart instead of SysV init scripts, I thought I'd write a quick upstart script for starting slony compiled and installed in /usr/local/slony. Upstart 1.4+ supports setuid, but the version of upstart included with Ubuntu 10.04LTS is only 0.6.5 and even on Ubuntu 11.10, it's only 1.3, so instead I use start-stop-daemon's --chuid switch. The script should be installed as /etc/init/slony.conf. Here's the script:

---cut here---
# slony - slon daemon
#
# slon is the slony replication daemon

description "slony slon daemon"

start on net-device-up
stop on runlevel [06]

# can't use setuid until upstart 1.4
#setuid slony
# which is why we use start-stop-daemon with the --chuid switch

respawn

script
exec start-stop-daemon --start --chuid slony --exec /usr/local/slony/bin/slon -- -f /etc/slon.conf >> /var/log/slony/slony.log 2>&1
end script
---cut here---

This will have the slon daemon run as the slony user and log to /var/log/slony/slony.log. You'll also need a valid slony config file in /etc/slon.conf.

You can also find the script in the pgexperts upstart-scripts git repo:
https://github.com/pgexperts/upstart-scripts

Peter Eisentraut: My (anti-)take on database schema version management

Tue, 15 May 2012 20:46:51 GMT

There were a number of posts recently about managing schema versions and schema deployment in PostgreSQL. I have analyzed these with great interest, but I have concluded that they are all more or less significantly flawed. (Of course, most of these solutions do in fact work for someone, but they are not general enough to become canonical go-to solutions for this problem class, in my opinion.) I have developed a list of elimination criteria by which I can evaluate future developments in this area. So here are some of the things that I don't want in my schema version management system:

Using schemas for distinguishing multiple versions (like this, but that's actually more about API versioning). That simply won't work for deploying objects that are not in schemas, such as casts, languages, extensions, and, well, schemas.
Using extensions (like this). Well, this could work. But extensions by themselves do nothing about the core problem. They are just an SQL wrapper interface around upgrade scripts. You still need to write the upgrade scripts, order them, test them, package them. The extension mechanism might replace the, say, shell script that would otherwise run the upgrade files in a suitable order. Another issue is that extensions require access to the server file system. Changing this is being discussedas "inline extensions", but there is no consensus. This is a smaller problem, but it needs to be thought about. Also, I do need to support PostgreSQL 9.0 and earlier for little while more.
Requiring naming each change (patch names, like this). Naming things is hard. Numbering things is easy. And how many good names are you going to still be able to come up with after 100 or so changes?

Take a lesson from file version control systems: versions are numbers or, if it must be, hashes or the like (UUIDs have been suggested).
Using a version control tool for tracking upgrade paths (like this). Sqitch, unlike the initial draft of this concept, doesn't actually require a version control tool for deployment, which wouldn't have worked for me, because what we ship is a tarball or a deb/rpm-type package. But it still requires you to maintain some kind of sanity in your version control history so that the tool can make sense out of it. That sounds fragile and inconvenient. The other choice appears to be writing the plan files manually without any VCS involvement, but then this loses much of the apparent appeal of this tool, and it's really no better than the "naming each change" approach mentioned above.
Taking snapshots or the like of a production or staging or central development system. Production systems and staging systems should be off limits for this sort of thing. Central development systems don't exist, because with distributed version control, every developer has their own setups, branches, deployments, and world views.

You could set it up so that every developer gets their own test database, sets up the schema there, takes a dump, and checks that in. There are going to be problems with that, including that dumps produced by pg_dump are ugly and optimized for restoring, not for developing with, and they don't have a deterministic output order.
Storing the database source code in a different place or in a different manner than the rest of the source code. This includes using a version control system like mentioned above (meaning storing part of the information in the version control meta information rather than in the files that are checked into the version control system in the normal way), using a separate repository like Post Facto, or using something like the mentioned staging server.

The source is the source, and it must be possible to check out, edit, build, test, and deploy everything in a uniform and customary manner.
Allowing lattice-like dependencies between change sets (like most examples cited above). This sounds great on paper, especially if you want to support distributed development in branches. But then you can have conflicts, for example where two upgrades add a column to the same table. Depending on the upgrade path, you end up with different results. As your change graph grows, you will have an exploding number of possible upgrade paths that will need to be tested.

There needs to be an unambiguous, canonical state of the database schema for a given source checkout.
Requiring running through all the upgrade scripts for a fresh deployment (like this). There are two problems with this. First, it's probably going to be very slow. Fresh deployments need to be reasonably fast, because they will be needed for automated tests, including unit tests, where you don't want to wait for minutes to set up the basic schema. Second, it's inefficient. Over time, you might drop columns, add new columns, delete rows, migrate them to different tables, etc. If you run through all those upgrade scripts, then a supposedly fresh database will already contain a bunch of rubble, dropped attributes, dead rows, and the like.

Therefore, the current version needs to be deployable from a script that will not end up replaying history.
Using metalanguages or abstraction layers (like Pyrseas or Liquibase or any of the metaformats included in various web frameworks). It'd probably a good idea to check some of these out for simple applications. But my concern is whether using an abstraction layer would prevent me from using certain features. For example, look at the Pyrseas feature matrix. It's pretty impressive. But it doesn't support extensions, roles, or grants. So (going by that list), I can't use it. (It's being worked on.) And in a previous version, when I looked at it for a previous project, it didn't support foreign-data functionality, so I couldn't use it then either. And those are just the top-level things the author thought of. Actually, the Pyrseas author has gone through some effortto have almost complete coverage of PostgreSQL DDL features, so give this tool a try. But it won't be for everyone.

So, I'm going to keep looking.

Magnus Hagander: Call for Papers - PostgreSQL Conference Europe 2012

Tue, 15 May 2012 19:49:00 GMT

The call for papers for PostgreSQL Conference Europe 2012 in Prague, the Czech Republic has now been posted. As usual, we are looking for talks on all topics related to PostgreSQL. At this point, we are looking for submissions for regular conference sessions - we will post a separate call for papers for lightning talks at a later time.

We are also still looking for sponsors - please see our website for details about the sponsor benefits and the costs.

Follow the news feed on our site, or our Twitter feed, for news updates!

David Wheeler: Use of DBI in Sqitch

Tue, 15 May 2012 17:41:00 GMT

Sqitch uses the native database client applications (psql, sqlite3, mysql, etc.). So for tracking metadata about the state of deployments, I have been trying to stick to using them. I’m first targeting PostgreSQL, and as a result need to open a connection to psql, start a transaction, and be able to read and write stuff to it as migrations go along. The IPC is a huge PITA. Furthermore, getting things properly quoted is also pretty annoying — and it will be worse for SQLite and MySQL, I expect (psql’s --set support is pretty slick).

Martin Pitt: Debian/Ubuntu Packages for PostgreSQL 9.2 Beta 1

Tue, 15 May 2012 12:33:08 GMT

The first Beta of the upcoming PostgreSQL 9.2 was released yesterday (see announcement). Your humble maintainer has now created packages for you to test. Please give them a whirl, and report any problems/regressions that you may see to the PostgreSQL developers, so that we can have a rock solid 9.2 release.

Remember, with the postgresql-common infrastructure you can use pg_upgradecluster to create a 9.2 cluster from your existing 8.4/9.1 cluster and run them both in parallel without endangering your data.

For Debian the package is currently waiting in the NEW queue, I expect them to go into experimental in a day or two. For Ubuntu 12.04 LTS you can get packages from my usual PostgreSQL backports PPA. Note that you need at least postgresql-common version 0.130, which is available in Debian unstable and the PPA now.

I (or rather, the postgresql-common test suite) found one regression: Upgrades do not keep the current value of sequences, but reset them to their default value. I reported this upstream and will provide updated packages as soon as this is fixed.

Hans-Juergen Schoenig: PostgreSQL resource consumption limitations

Mon, 14 May 2012 08:30:52 GMT

PostgreSQL offers a great deal of flexibility when it comes to resource
allocation and efficient usage of hardware capacities. This allows fancy
monitoring of what PostgreSQL does in production. Sometimes, however, better
operating system integration is needed and helps to use hardware even more
efficiently.

On Linux one way to limit or allocate resources is to use the Linux kernel’s
cgroup interface. cgroups and PostgreSQL are a perfect team for …

more efficient monitoring
restriciting use of resources
resource accounting.

How do cgroups work?

On a Linux system cgroups (= control groups) are basically a way to group
various operating system processes into groups. A group like that can then be
assigned to a certain amount of RAM, a fraction of a CPU or to a certain number
of I/O operations.

Processes can be moved to a group manually or with the help of a daemon which
moves processes to the right group automatically given some user-defined rules.
It is easily possible to move all processes belonging to a user to some special
cgroup and apply constraints to it. Following this approach you can move an
entire PostgreSQL instance and all its processes to a certain cgroup.

A simple config could look like that (/etc/cgconfig.conf):

mount {
        cpu = /cgroup/main;
        cpuacct = /cgroup/main;
        memory = /cgroup/main;
        blkio = /cgroup/main;
}

group user_0004 {
    perm {
        admin {
            uid = root;
            gid = root;
        }
        task {
            uid = user_0004;
            gid = user_0004;
        }
    }
    memory {
        memory.limit_in_bytes = 256M;
    }
    cpu {
        cpu.shares = "75";
    }
}

group user_0005 {
    perm {
        admin {
            uid = root;
            gid = root;
        }
        task {
            uid = user_0005;
            gid = user_0005;
        }
    }
    memory {
        memory.limit_in_bytes = 512M;
    }
    cpu {
        cpu.shares = "125";
    }
}

We got two users which form separate groups. Each group will get certain
fractions of the system’s CPU as well as a certain amount of memory. In other
words: You can box a database instance nicely inside a group.

In the next step you can nicely put processes into a group. This is best done
using /etc/cgrules.conf:

user_0004        *       /user_0004
user_0005        *       /user_0005

In our case every user’s process is automatically moved to the right process on
startup.

More on cgroups will be available in further blog posts.

Michael Paquier: Postgres-XC 1.0beta2 is released

Sun, 13 May 2012 03:56:17 GMT

Postgres-XC, write-scalable multi-master symmetric cluster based on PostgreSQL, version 1.0beta2 has been released. This beta version is based on PostgreSQL 9.1.3. All the patches in PostgreSQL 9.1 stable branch have been merged up to commit 1c0e678 (4th of May 2012). You can download the tarball directly from here. This tarball contains all the HTML and [...]

Chris Travers: Thoughts on VoltDB and performance vs flexibility

Sun, 13 May 2012 02:27:30 GMT

VoltDB is the latest project of Mike Stonebreaker (original founder of Postgres) aimed at very high performance OLTP. It is fairly clearly a niche product however, and not really usable in the ERP space for reasons I will discuss below.

Stonebreaker's own presentations (one example, at 11:25) suggest he is aware of this issue given his triangle framework.

In the vertices of the triangle he places NoSQL, high performance OLTP (like VoltDB) and column-oriented databases, while in the second, as lower performance generalists, what he calls the legacy databases or elephants are in the middle. The argument is that you can get a significant performance boost by specializing your systems beyond what you could when you have every system doing everything. This is a fair argument to a point, but the implementation details show that it is true only in some areas.

Stonebreaker's analysis of processing time (see above presentation, at 14:36) in traditional databases places virtually all processing time in four areas, namely buffer management, locking, thread locks/semaphores, and recovery tasks, and he suggests that in order to get high degrees of performance one must eliminate these tasks. This requires, however, rethinking how we address concurrency (the locking issues) and durability (recovery and disk-based storage). The VoltDB approach is to get rid of concurrency entirely and see durability as a network, rather than a system, property. Concurrency elimination accepted because the db is fast enough that queries can be run one at a time, and intraquery parallelism can be used instead of interquery parallelism. However, this imposes significant limitations on the database because it means that every transaction is limited to a single query. You can't do round-tripping in your transactions because this would impose locking requirements on the database.

This approach works very well for certain kinds of processing, such as consuming rapid data feeds and then feeding that information into a data warehouse at specified intervals., However one thing that is generally missing from the discussion is that the more complex the application, the more general the database solution needs to be. One solution might be to separate data entry from reporting and use multiple tools, but who wants to pull their trial balance from a different system than they enter invoices on? Since ERP systems usually tightly integrate decision support and OLTP, there isn't really room here to use specialized databases like VoltDB.

As the database market has expanded, it has created niches for products like VoltDB. These niches may indeed grow with time. However, I think it is incorrect to treat generalists as legacy, outdated approaches.

In essence VoltDB typifies what I call a specialism paradox, namely that to perform well in a specialist niche one must give up generalist solutions. It is true that sometimes things improve and supplant older systems but at least as often they only do when the systems are not vastly more complex than the ones they are replacing. For example, Stonebreaker's comparison of high performance OLTP to the shift from cable-operated digging excavation equipment to hydrolics misses the fact that hydrolic systems are not that much more complex than cables and such. There is some additional complexity, but it isn't a huge gap. A better comparison might be comparing semi trucks to race cars and locomotives. These are all highly engineered systems, but the basic design considerations are very different, and specialization comes at the cost of flexibility. You will never see race cars hauling twenty-foot containers of furniture cross-country, and you will never see travelling along our nation's highways to reach out-of-the-way towns. The semi however will never pull the loads that the locomotive can, and will never win the Indie 500...... The semi truck, however, can go fast enough, and pull the loads it needs to pull.....

Andrew Dunstan: Upgradable JSON

Sat, 12 May 2012 15:51:00 GMT

If you're using my backport of the 9.2 JSON feature in your 9.1 Postgres, you have a problem if you want to use pg_upgrade when moving to release 9.2: the type has the same name and behaviour in 9.1 as in 9.2, but the OIDs are different and so the upgrade will fail. If you use pg_dump/pg_restore, you should just be able to exclude the restoration of the extension and everything will "just work".

But after a lot of trial and error I have come up with a different way to do things. In this scenario we don't install as an extension, and instead we borrow a bit of magic from pg_upgrade to make sure that the JSON type and its array type have the same OIDs as they will have in PostgreSQL 9.2. We also need to fool pg_upgrade when it runs, as it goes looking for a loadable module called json.so, and of course in 9.,2 there isn't one because JSON is built in. So we need to supply a dummy one. But, given that, this too "just works".

There might be some wrinkles I haven't foreseen, but I have done this quite a few times now without mishap. This is checked in on a separate branch of the json_91 repo callled "binup"

If you are going to use this backport, I also highly recommend using the backport of the fix for missing column names on row expressions. This requires patching Postgres, but it's worth doing anyway, even if you're not using JSON - notably it affects some uses of hstore as well. You can get this by pulling the port into a git repo set at REL9_1_STABLE by doing:

git pull https://bitbucket.org/adunstan/pgdevel.git rowexpr91

Enjoy.

Hitoshi Harada: PL/coffee Trial

Thu, 10 May 2012 17:58:16 GMT

We are getting to the final stage of the first stable release of PL/v8. This should be a good release with a lot of improvement such like

Subtransaction support
Better name space of built-in functions
OO style cursor, prepared plan
find_function
Start-up procedure with GUC
Separate context in user switch
A series of bug fixes.

And these days I'm getting increasing number of interests and feedback via email, SNS, plv8 project page. Upcoming PGCon will at least have two talks that may cover the usage of PL/v8.

In the mean time, there was a long standing feature request of PL/coffee, a procedural language in CoffeeScript, the dialect of JavaScript. CoffeeScript is only a source-source transformation, and the compiler is provided as a tiny JavaScript, so I tried to let the compiler to transform plv8 source if the language is "plcoffee".

=# CREATE OR REPLACE FUNCTION public.fibonacci(n integer)
 RETURNS integer LANGUAGE plcoffee IMMUTABLE STRICT
AS $function$
fibonacci = (x)->
  return 0 if x == 0
  return 1 if x == 1
  return fibonacci(x-1) + fibonacci(x-2)
return fibonacci n
$function$;
CREATE FUNCTION

=# select fibonacci(10);
 fibonacci 
-----------
        55
(1 row)

As I noted above, this is only a source transformation, so the engine is still v8, and the runtime environment and other stuff are shared with plv8. Of course it can reference plv8 function via find_function(). The plcoffee EXTENSION is separate and by default it's off. If you are interested in it, clone the latest source and say

make ENABLE_COFFEE=1 install

It works and it attracts more interest from the people around some area. However, it seems it is a little controversial to use CoffeeScript (of course not in the database usage context, but in the web browser and server context) that the source transformation is quite difficult to debug since the problems like run time error emitted by the JS engine is far from the actual script source, which is so annoying. And since I embed the compiler JS source file as a object file symbol, the size of plv8 shared object gets from around 200k byte to 600k byte. Thus, for now I leave it as an "experimental" feature and want to see if it gets popular or not.

Bruce Momjian: Postgres 9.2 Draft Release Note Published

Thu, 10 May 2012 04:45:01 GMT

I have completed the Postgres 9.2 release notes I started seven days ago. Writing the release notes is always a taxing experience. I have to basically dedicate an entire week to the process of digesting 1100 commit messages to produce 3000 lines of SGML text. Once I am done though, it is rewarding to see the finished product. It is like working on a painting all year, and spending a frustrating week framing it and hanging it on the all — once it is hung, you stand and admire it, and forget much of the sweat it took to produce.

Curious about how the 9.2 release item count compares to previous major releases? Here are the results:

Release	Items
9.2	245
9.1	213
9.0	252
8.4	330
8.3	237
8.2	230
8.1	187
8.0	238
7.4	280

Andrew Dunstan: Deploying file_text_array Foreign Data wrapper

Thu, 10 May 2012 03:45:43 GMT

Today, a moment planned 18 months ago arrived when I was able to deploy the file_text_array foreign data wrapper for the client for whom it was originally designed. It's taken this long for the client to move to the 9.1 release, and they suddenly emailed me today saying they needed the old ragged CSV feature I'd previously provided them. I explained to them that this was meant to replace it, and it took me all of about 5 minutes to deploy, and a further hour or two to adjust and test the code in the ETL tool that is meant to use it. It's all working flawlessly, I'm happy to say.

Leo Hsu and Regina Obe: File FDW Family: Part 2 file_textarray_fdw Foreign Data Wrapper

Thu, 10 May 2012 03:44:00 GMT

Last time we demonstrated how to query delimited text files using the fdw_file that comes pacakged with PostgreSQL 9.1+, this time we'll continue our journey into Flat file querying Foreign Data Wrapper using an experimental foreign data wrapper designed for also querying delimited data, but outputting it as a single column text array table. This one is called file_textarray_fdw and developed by Andrew Dunstan. It's useful if you are dealing with for example jagged files, where not all columns are not properly filled in for each record or there are just a ton of columns you don't want to bother itemizing before you bring in. The benefit is you can still query and decide how you want to break it apart. You can grab the source code from file_text_array_fdw source code. If you are on windows, we have compiled binaries in our Bag o' FDWs for both PostgreSQL 9.1 32-bit FDW for Windows bag and PostgreSQL 9.1 64-bit FDW for Windows bag that should work fine with the EDB installed windows binaries. For other systems, the compile is fairly easy if you have the postgresql development libraries installed.

Continue reading "File FDW Family: Part 2 file_textarray_fdw Foreign Data Wrapper"

Michael Paquier: Rules in Postgres-XC

Wed, 09 May 2012 00:16:48 GMT

One of the features that has been really improved the last couple of weeks is the stabilization of remote query planning for DML for Postgres-XC standard planner. And this has consequences on rules, because a rule is fired on Coordinators by design, and you need to provide a global way to plan queries correctly with [...]

Greg Sabino Mullane: Monitoring many Postgres files at once with tail_n_mail

Tue, 08 May 2012 17:22:54 GMT

This post discusses version 1.25.0 of tail_n_mail, which can be downloaded at http://bucardo.org/wiki/Tail_n_mail

One of our clients recently had one of their Postgres servers crash. In technical terms, it issued a PANIC because it tried to commit a transaction that had already been committed. We are using tail_n_mail for this client, and while we got notified six ways to Sunday about the server being down (from Nagios, tail_n_mail, and other systems), I was curious as to why the actual PANIC had not gotten picked up by tail_n_mail and mailed out to us.

The tail_n_mail program at its simplest is a Perl script that greps through log files, finds items of interest, and mails them out. It does quite a bit more than that, of course, including normalizing SQL, figuring out which log files to scan, and analyzing the data on the fly. This particular client of ours consolidates all of their logs to some central logging boxes via rsyslog. For the host in question that issued the PANIC, we had two tail_n_mail config files that looked like this:

## Config file for the tail_n_mail program
## This file is automatically updated
## Last updated: Fri Apr 27 18:00:01 2012
MAILSUBJECT: Groucho fatals: NUMBER

INHERIT: tail_n_mail.fatals.global.txt

FILE: /var/log/%Y/groucho/%m/%d/%H/pgsql-err.log
LASTFILE: /var/log/2012/groucho/04/27/18/pgsql-err.log
OFFSET: 10199

## Config file for the tail_n_mail program
## This file is automatically updated
## Last updated: Fri Apr 27 18:00:01 2012
MAILSUBJECT: Groucho fatals: NUMBER

INHERIT: tail_n_mail.fatals.global.txt

FILE: /var/log/%Y/groucho/%m/%d/%H/pgsql-warning.log
LASTFILE: /var/log/2012/groucho/04/27/18/pgsql-warning.log
OFFSET: 7145

The reason for two files was that rsyslog was splitting the incoming Postgres logs into multiple files. Which is normally a very handy thing, because the main file, pgsql-info.log, is quite large, and it's nice to have the mundane things filtered out for us already. Because rsyslog also splits things based on the timestamp, we don't give it an exact file name, but use a POSIX template instead, e.g. /var/log/apps/%Y/groucho/%m/%d/%H/pgsql-warning.log. By doing this, tail_n_mail knows where to find the latest file. It also uses the LASTFILE and OFFSET to know exactly where it stopped last time, and then walks through all files from LASTFILE until the current one.

So why did we miss the PANIC? Because it was in a heretofore unseen and untracked log file known as pgsql-crit.log. (Which goes to show how rarely Postgres crashes: this was the first time in well over 700,000 log files generated that a PANIC had occurred!) At this point, the solution was to either create yet another set of config files for each host to watch for and parse any pgsql-crit.log files, or to give tail_n_mail some more brains and allow it to handle multiple FILE entries in a single config file. Obviously, I chose the latter.

After some period of coding, testing, debugging, and caffeine consumption, a new tail_n_mail was ready. This one (version 1.25.0) allows multiple values of the FILE parameter inside of a single config. Thus, for the above, I was able to combine everything into a single tail_n_mail config file like so:

MAILSUBJECT: Groucho fatals: NUMBER

INHERIT: tail_n_mail.fatals.global.txt

FILE: /var/log/%Y/groucho/%m/%d/%H/pgsql-warning.log
FILE: /var/log/%Y/groucho/%m/%d/%H/pgsql-err.log
FILE: /var/log/%Y/groucho/%m/%d/%H/pgsql-crit.log

The INHERIT file is a way of keeping common config items in a single file: in this case, groucho and a bunch of other similar hosts all use it. It contains the rules on what tail_n_mail should care about, and looks similar to this:

## Global behavior for all "fatals" configs
EMAIL: acme-alerts@endpoint.com
FROM: postgres@endpoint.com
FIND_LINE_NUMBER: 0
STATEMENT_SIZE: 3000
INCLUDE: FATAL:
INCLUDE: PANIC:
INCLUDE: ERROR:

## Client specific exceptions:
EXCLUDE: ERROR:  Anvils cannot be delivered via USPS
EXCLUDE: ERROR:  Jetpack fuel quantity missing
EXCLUDE: ERROR:  Iron Carrots and Giant Magnets must go to different addresses
EXCLUDE: ERROR:  Rocket Powered Rollerskates no longer available

## Postgres excceptions:
EXCLUDE: ERROR:  aggregates not allowed in WHERE clause
EXCLUDE: ERROR:  negative substring length not allowed
EXCLUDE: ERROR:  there is no escaped character
EXCLUDE: ERROR:  operator is not unique
EXCLUDE: ERROR:  cannot insert multiple commands into a prepared statement
EXCLUDE: ERROR:  value "\d+" is out of range for type integer
EXCLUDE: ERROR:  could not serialize access due to concurrent update

Thus, we only have one file per host to worry about, in addition to a common shared file across all hosts. So now tail_n_mail can handle multiple files over a time dimension (by walking forward from LASTFILE to the present), as well as over a vertical dimension (by forcing together the files split by rsyslog). However, there is no reason we cannot handle multiple files over a horizontal dimension as well. In other words, putting multiple hosts into a single file. In this client's case, there were other hosts very similar to "groucho" that had files we wanted to monitor. Thus, the config file was changed to look like this:

MAILSUBJECT: Acme fatals: NUMBER

INHERIT: tail_n_mail.fatals.global.txt

FILE: /var/log/%Y/groucho/%m/%d/%H/pgsql-warning.log
FILE: /var/log/%Y/groucho/%m/%d/%H/pgsql-err.log
FILE: /var/log/%Y/groucho/%m/%d/%H/pgsql-crit.log

FILE: /var/log/%Y/dawson/%m/%d/%H/pgsql-warning.log
FILE: /var/log/%Y/dawson/%m/%d/%H/pgsql-err.log
FILE: /var/log/%Y/dawson/%m/%d/%H/pgsql-crit.log

FILE: /var/log/%Y/cosby/%m/%d/%H/pgsql-warning.log
FILE: /var/log/%Y/cosby/%m/%d/%H/pgsql-err.log
FILE: /var/log/%Y/cosby/%m/%d/%H/pgsql-crit.log

We've just whittled nine config files down to a single one. Of course, the config file cannot stay like that, as the LASTFILE and OFFSET entries need to be applied to specific files. Thus, when tail_n_mail does its first rewrite of the config file, it will assign numbers to each FILE, and the file will then look something like this:

FILE1: /var/log/%Y/groucho/%m/%d/%H/pgsql-warning.log
LASTFILE1: /var/log/2012/groucho/04/27/18/pgsql-warning.log
OFFSET1: 100


FILE2: /var/log/%Y/groucho/%m/%d/%H/pgsql-err.log
LASTFILE2: /var/log/2012/groucho/04/27/18/pgsql-err.log
OFFSET2: 2531

FILE3: /var/log/%Y/groucho/%m/%d/%H/pgsql-crit.log

FILE4: /var/log/%Y/dawson/%m/%d/%H/pgsql-warning.log
LASTFILE4: /var/log/2012/dawson/04/27/18/pgsql-warning.log
OFFSET4: 42

# etc.

By using this technique, we were able to reduce a slew of config files (the actual number was around 60), and their crontab entries, into a single config file and a single cron call. We also have a daily "error" report that mails a summary of all ERROR/FATAL calls in the last 24 hours. These were consolidated into a single email, rather than the half dozen that appeared before.

While tail_n_mail has a lot of built-in intelligence to handle Postgres logs, it is ultimately regex-based and can be used on any files which you want to track and receive alerts when certain items appear inside of them, so feel free to use it for more than just Postgres!

Andrew Dunstan: Mixing compilers

Tue, 08 May 2012 15:14:00 GMT

Regina has been reporting problems with running my Fixed Length Record Foreign Data Wrapper on Windows. I tried it yesterday on my MinGW 64 bit setup and it worked just fine. It turns out she's been trying to run a mingw compiled FDW DLL with a VC++ built Postgres. I'm not sure that we actually expect mixing compilers like that to work. In fact, I'm not sure how to build extension modules for VC++ builds at all. My approach would probably be to build them in tree, possibly modifying the build scripts slightly, but I haven't tried. It would be nice if this were documented better.

Josh Berkus: Amazonstravaganza tommorrow, on UStream

Tue, 08 May 2012 05:37:53 GMT

Tommorrow we're going to have two back-to-back SFPUG presentations about running PostgreSQL on AWS: Scalr presenting their stuff, and "Amazon With Reduced Tears" by Christophe Pettus. The event is sold out at 100 RSVPs, though, so I suggest you watch it on video!

Barring severe technical difficulties, it will be presented live on SFPUG Ustream. Festivities should start around 7:15 PM PDT.

John DeSoi: PostgreSQL support in Ace editor

Tue, 08 May 2012 01:55:00 GMT

Ace is a standalone code editor written in JavaScript. It now supports PostgreSQL specific syntax coloring, including support for pl/pgsql, pl/perl, and pl/python.

To try the live demo, just select "pgSQL" from the Document menu.

Hubert 'depesz' Lubaczewski: OmniPITR 0.7.0

Mon, 07 May 2012 20:34:27 GMT

Just released new version of OmniPITR. This version has one important new feature: when you’re calling omnipitr-backup-slave, it will make backups only of required xlog files, and not, as previously, of all in walarchive directory. This is important, especially in case you have multiple slaves, or you keep shared long-term walarchive. Previously – backups would [...]

Bruce Momjian: Postgres Drinking Game

Mon, 07 May 2012 13:15:01 GMT

My children have traveled to many conferences with me, and have heard many webcasts and training calls at home. I guess, after hearing about Postgres so often, they pick up certain phases of interest, and the big one for them is "shared buffers". Anytime someone uses those words, they start howling and make up some funny sentence using the words.

There are a variety of beer drinking games that trigger drinking when a word is said — perhaps "shared buffers" is enough on its own to make a drinking game.

Josh Berkus: Random Page Cost Revisited

Mon, 07 May 2012 05:16:35 GMT

Because the planner has limited information about the host system PostgreSQL is running on, we offer several planner configuration variables, which can be set either dynamically or in the postgresql.conf file. One of these, which has been a source of much contention in the past, is Random Page Cost (or random_page_cost, hereafter RPC). For the last couple of years RPC was not a variable you needed to worry about, but now, thanks to advances in storage technology, it's become one of the essential parameters again.

Random Page Cost is meant to represent the coefficient between the cost of looking up one row (out of many) via sequential reads, vs. the cost of looking up a single row individually using random access (disk seeks). This factor strongly influences the planner's decisions to use indexes vs. table scan, and composite vs. simple indexes, in executing queries.

For many generations of spinning hard drives, this ratio remained about the same, around 4.0.

Things have changed recently though. In fact, there are five now-common types of storage where you want to change RPC in order to match a different scan/seek ratio for different hardware.

1. High-End NAS/SAN: when working well, your large storage box combines a large volatile cache with many many disks, improving the number of concurrent seeks you can do. However, if storage is not direct-attach, you usually have limited throughput. This means that a random_page_cost of 2.5 or 3.0 is more appropriate.

2. Amazon EBS and Heroku: EBS, especially when "RAIDed", is an extreme example of the NAS/SAN case. Seeks are fast with virtually unlimited concurrency, while storage bandwitdh is erratic and slow, limiting full table scans. As such, we often use a random_page_cost of 2.0.

3. Bad SANs: the advent of iSCSI seems to have inspired a large variety of very poor-performing inexpensive storage units, such as the lower-end Dell Equallogics and various SATA-based Promise units. These boxes are cheaper for a reason, folks! Anyway, their quirks often give these boxes unique performance characteristics which strongly affect database access and prompt you to modify planner parameters. For example, we discovered through testing that one Promise box had decent throughput, but seek rates slower than a single drive due to a combination of architecture and driver issues. As such, we set RPC to 6.0. You'll have to trust the results of bonnie++ and fio in order to figure out where you should set things, though; each inferior unit is inferior in a different way.

4. SSDs: in general, SSDs have not lived up to their early hype for performance; random writes and sequential scans on most SSDs are only incrementally better than on HDDs. One area where SSDs shine, though, is random page access; a good array of SSDs can make even large index scans faster than sequential scans. Therefore, use a random_page_cost of 2.0 to 2.5.

5. NvRAM (or NAND): durable memory, such as FusionIO and Virident drives, do deliver on most of the performance promises originally made about SSDs. Everything is faster, sometimes orders of magnitude faster, than HDDs, most of all random access. While the concept of NvRAM might make you inclined to set RPC to 1, forcing almost 100% index access, that's going a little far because PostgreSQL still has a disk page structure. Therefore, I suggest 1.5.

Note that two other variables, effective_cache_size and effective_io_concurrency also affect seek/scan behavior. Possibly I will blog about them at a different time. Further, I have only been able to test with relatively few models and configurations of the above types of storage, so test, monitor and check your results before committing to a long-term configuration change.

Bruce Momjian: The Effectiveness of effective_cache_size

Fri, 04 May 2012 12:30:01 GMT

Having reported the methods for finding the size of the kernel cache on Linux, I wish to highlight the importance of the postgresql.conf setting effective_cache_size.

Unlike other memory settings that control how memory is allocated, effective_cache_size tells the optimizer how much cache is present in the kernel. This is important for determining how expensive large index scans will be. The optimizer knows the size of shared_buffers, but not the kernel cache size, which affects the probability of expensive disk access.

Josh Berkus: SELECT fire_proton_torpedoes();

Fri, 04 May 2012 05:41:05 GMT

At pgCon this year you will have the chance to reduce your fellow attendees to their component elements, and win prizes for it!

We will be having a Schemaverse tournament for the length of the conference. Schemaverse is an amazing game which demonstrates that PostgreSQL isn't just a database, it's an entire development environment. You build a fleet of starships, fight battles, take planets, trade, and conquer the galaxy ... all from the psql command line. Write PL/pgSQL functions to command your armies!

To make things even more interesting, if you can hack into the Postgres server, you can win the game.

The Schemaverse tournament will begin Wednesday night and end shortly before the closing session at pgCon. The winner will get some one-of-a-kind PostgreSQL and Schemaverse swag, and even runners up will get posters or tshirts.

If you've already registered for pgCon, look for email from Dan Langille about how to sign up for the tournament. In the meantime, get over to the Schemaverse website and practice your space battle skills to get ready for the big event.

gabrielle roth: PDXPUG: May meeting in two weeks

Fri, 04 May 2012 02:08:30 GMT

When: 7-9pm Thu May 17, 2012
Where: Iovation
Who: Ed Snajder
What: Replication Without Tears

“Database replication is a great way to tackle challenges with scaling, concurrency, availability and redundancy. It is common to all well-known database engines, and in Postgres, there are almost too many choices. In this demo-driven presentation, I will review a few of the options available for replication in PostgreSQL Systems. While the concept of replication is really quite straightforward, it can be unusually painful to configure and maintain. Furthermore, each particular method of replication includes its own significant pros and cons. My hope is that some of my discoveries will ease the pain and reduce the tears of configuring and managing your own replicated systems.”

Ed Snajder is the Database Administrator at Jive Software, and has been working with databases for over 10 years. His currently responsibility is predominantly PostgreSQL systems administration, though he still actively works through challenges with Oracle, SQL Server and MySql. A native New Yorker, Ed moved to Portland in 2006 and had the opportunity to operate as the DBA for the Systems running the NTIA Analog-to-digital Convertor Box Program. Outside of technology, Ed enjoys music and sports appreciation, and recently started a grass-roots effort to officially rename Beaverton to “The Tron.”

Our meeting will be held at Iovation, on the 32nd floor of the US Bancorp Tower at 111 SW 5th (5th & Oak). It’s right on the Green & Yellow Max lines. Underground bike parking is available in the parking garage; outdoors all around the block in the usual spots. No bikes in the office, sorry!

Building security will close access to the floor at 7:30.

After-meeting beer location TBD. See you there!

Leo Hsu and Regina Obe: File FDW Family: Part 1 file_fdw

Thu, 03 May 2012 19:04:00 GMT

Last time we demonstrated how to use the ODBC Foreign Data wrapper, this time we'll continue our journey into Foreign Data Wrapper land by demonstrating what I'll call the File FDW family of Foreign Data Wrappers. There is one that usually comes packaged with PostgreSQL 9.1 which is called fdw_file but there are two other experimental ones I find very useful which are developed by Andrew Dunstan both of which Andrew demoed in PostgreSQL Foreign Data Wrappers and talked about a little bit Text files from a remote source. As people who have to deal with text data files day in and out, especially ones from mainframes, these satisfy a certain itch.

file_fdw - for querying delimited text files.
file_fixed_length_fdw - this one deals with fixed length data. We discussed methods of importing fixed length data in Import Fixed width data. This is yet another approach but has the benefit that you can also use it to import just a subset of a file.
file_text_array_fdw - this one queries a delimited file as if each delimiete row was a text array. It is ideal for those less than perfect moments when someone gives you a file with a 1000 columns and you don't have patience to look at what the hell those columns mean just yet.

In this article, we'll just cover the file_fdw one, but will follow up in subsequent articles, demonstrating the array and fixed length record ones.

Continue reading "File FDW Family: Part 1 file_fdw"

Bruce Momjian: Starting on 9.2 Release Notes

Thu, 03 May 2012 14:00:01 GMT

As in previous years, I have started working on major release notes, this time for Postgres 9.2. I start with this command:

$ src/tools/git_changelog --since '2011-06-11 00:00:00 GMT' --master-only \
> --oldest-first --details-after

Bruce Momjian: Measuring Free Memory and Kernel Cache Size on Linux

Wed, 02 May 2012 14:00:01 GMT

Measuring Linux free memory and kernel cache size can be somewhat tricky. You might look at /proc/meminfo for the answer (commas added):

# cat /proc/meminfo
MemTotal:       24,736,604 kB
MemFree:         3,805,392 kB
Buffers:           743,016 kB
Cached:         18,188,208 kB
...

Frank Wiles: Three things you should never put in your database

Tue, 01 May 2012 21:24:03 GMT

As I've said in a few talks, the best way to improve your systems is by first not doing "dumb things". I don't mean you or your development staff is "dumb", it's easy to overlook the implications of these types of decisions and not realize how bad they are for maintainability let alone scaling. As a consultant I see this stuff all of the time and I have yet to ever see it work out well for anyone.

Images, files, and binary data

Your database supports BLOBs so it must be a good idea to shove your files in there right? No it isn't! Hell it isn't even very convenient to use with many DB language bindings.

There are a few of problems with storing files in your database:

read/write to a DB is always slower than a filesystem
your DB backups grow to be huge and more time consuming
access to the files now requires going through your app and DB layers

The last two are the real killers. Storing your thumbnail images in your database? Great now you can't use nginx or another lightweight web server to serve them up.

Do yourself a favor and store a simple relative path to your files on disk in the database or use something like S3 or any CDN instead.

Ephemeral data

Usage statistics, metrics, GPS locations, session data anything that is only useful to you for a short period of time or frequently changes. If you find yourself DELETEing an hour, day, or weeks worth of some table with a cron job, you're using the wrong tool for the job.

Use redis, statsd/graphite, Riak anything else that is better suited to that type of work load. The same advice goes for aggregations of ephemeral data that doesn't live for very long.

Sure it's possible to use a backhoe to plant some tomatoes in the garden, but it's far faster to grab the shovel in the garage than schedule time with a backhoe and have it arrive at your place and dig. Use the right tool(s) for the job at hand.

Logs

This one seems ok on the surface and the "I might need to use a complex query on them at some point in the future" argument seems to win people over. Storing your logs in a database isn't a HORRIBLE idea, but storing them in the same database as your other production data is.

Maybe you're conservative with your logging and only emit one log line per web request normally. That is still generating a log INSERT for every action on your site that is competing for resources that your users could be using. Turn up your logging to a verbose or debug level and watch your production database catch on fire!

Instead use something like Splunk, Loggly or plain old rotating flat files for your logs. The few times you need to inspect them in odd ways, even to the point of having to write a bit of code to find your answers, is easily outweighed by the constant resources it puts on your system.

But wait, you're a unique snowflake and your problem is SO different that it's ok for you to do one of these three. No you aren't and no it really isn't. Trust me.

David Wheeler: Sqitch Status: A Step at a Time

Tue, 01 May 2012 04:09:00 GMT

I've just released Sqitch v0.20-TRIAL, the third testing release of Sqitch. Since last week, I've implemented add-step. So let's have a look-see at what all it can do. First, let's initialize a Sqitch project.