Replication Deadlock Bug in Current Postgres Releases 14-16

The current minor releases of Postgres versions 14-16 (14.23, 15.18 and 16.14, released on May 14th) introduced a regression that can lead to a MultiXactOffsetSLRU deadlock during transaction log (WAL) replay in certain circumstances.

The bug was (to our knowledge) first reported in Bug#19490 on May 20th by Radim Marek from BoringSQL. Further reports were done on the pgsql-bugs and pgsql-admin mailing lists and we also got customer support requests through our Open-Source Support Center.

What is Currently Known About This Bug

The bug can be hit in two ways. First, during streaming replication where the standby eventually hangs due to the deadlock. The other possibility is hanging point-in-time-recovery (PITR). The following is currently known:

1. The bug is only live on Postgres versions 14-16. Version 17 and 18 (and earlier versions) are not affected.

2. The WAL needs to be generated by a leader running the Q4/2025 (November 13th) back-branch releases or earlier (14.20/15.15/16.11).

3. The standby or the instance running PITR needs to be updated to the latest minor release (14.23/15.18/16.14).

4. The startup process hangs with wait event LWLock/MultiXactOffsetSLRU in pg_stat_activity during WAL replay.

So to summarize, the leader needs to be at least a few minor versions behind and the standby needs to be updated to the latest minor version. Due to the recommended procedure of updating streaming standbys first before updating the leader, this bug is likely to be hit relatively often in the field, especially by organizations that only patch every few minor releases.

Circumventing the Issue

If one has not yet hit the issue, then not upgrading the standby or a PITR machine to the latest minor releases will avoid the problem. In the case that one is affected by the regression, there are three (well, currently two) ways to address the problem:

CloudNativePG 1.30 introduces the DatabaseRole CRD and built-in TLS client certificate issuance, letting application teams own their PostgreSQL credentials declaratively and connect without ever handling a password.

A practical guide to what .ready and .done mean, and why WAL sticks around

Introduction

It is 9:12 a.m. on a Monday. Someone on your team opens pg_wal/archive_status/ during a storage scare and sees a long list of files ending in .ready. They ask the question many of us have asked at least once: “Is replication broken?” Streaming replicas still look mostly fine, but .ready files keep piling up, disk usage keeps climbing, and nobody is fully sure what .ready and .done are actually telling you.

What is .ready and what (if any) action do I need to take? Let’s talk about that today.

Hint: It’s About WAL Delivery

Think of WAL delivery as three independent steps:

1. Generate WAL
2. Transport WAL
3. Replay or consume WAL

archive_command is one way to do transport. Streaming replication is another. Note, logical replication also has a transport channel, but what it transports is decoded logical change data rather than raw WAL segment files.

What archive_command Actually Does

When archive_mode=on, Postgres tries to copy each completed WAL segment to long-term storage by running archive_command.

Typical example:

archive_mode = on
archive_command = 'rsync -a %p backup@walbox:/archives/%f'

• %p is the local path to the WAL segment in pg_wal
• %f is just the filename

Postgres runs this command from the archiver process. If the command exits with status 0, Postgres treats it as success. Any non-zero exit code means failure, and it retries later.

Info: Archiving usually happens when a WAL segment is complete (typically 16 MB), not every transaction. So pure archive shipping can have more lag unless segment switches happen frequently.

What .ready and .done Are For

Inside pg_wal/archive_status/, Postgres tracks each WAL segment’s archiving state with tiny marker files.

For a segment named:

000000010000000A000000FE

you may see:

• 000000010000000A000000FE.ready
• 000000010000000A000000FE.done

.ready

Almost every conversation about pg_hardstorage's repository format ends up at the same question: "where's the incremental chain?"

Short answer: there isn't one. By design.

The chain footgun

In a chained-incremental format, pgBackRest's default, Barman's incremental mode, every incremental references the previous backup directly:

full A   ←   incr B   ←   incr C   ←   incr D   ←   diff E

This is fine when everything works. But every chain has a single property that bites you at the worst time: any single corrupted backup invalidates everything downstream.

Real failure modes we've all seen:

• An S3 lifecycle policy quietly deletes incr B; incr C..D and diff E become useless.
• A bit flip on the storage backend corrupts diff E; the chain restores up to incr D only.
• An operator runs backup expire with too-aggressive retention; full A goes; the entire chain is now floating.
• The base full's manifest schema bumped between releases and the migration script had a rounding bug.

Each of these is recoverable in isolation. The pattern that bites is: you don't find out until the restore. The 3am restore.

Content addressing kills the chain

pg_hardstorage's manifest references chunks by SHA-256, not by reference to a previous manifest:

{
  "id": "prod-2026-05-02-0334",
  "chunks": ["7e1f2a…ab", "9c4d3b…12", "2faabc…77", /* … */]
}

Each chunk's filename is its hash. Two backups that share data also share chunks — but at the storage level, not via a "parent" pointer. Delete the older backup; the newer one's chunks stay until garbage collection finds zero references.

This is what restic, kopia, and (importantly) borgbackup all do. Not novel — but worth saying out loud, because in the PostgreSQL ecosystem the chained-incremental model is so dominant it feels like the only way. See the how it works page for a full walkthrough of the manifest format.

"But what about storage cost?"

The wins of chained incrementals are:

1. Don't write the same byte twice when nothing

In HOT Updates in Postgres we covered page pruning clean up HOT chains, an elegant shortcut where PostgreSQL reclaims dead tuple space during ordinary reads. All that without waiting for any background process. But pruning is exactly that: a shortcut. It only works within a single page, and only for HOT-updated tuples. For everything else (cold updates that touch indexed columns, plain DELETEs, index entry cleanup, free space map registration, visibility map maintenance) we need VACUUM.

This article won't repeat what VACUUM does operationally. The DELETEs are difficult article covers autovacuum tuning, worker allocation, and the operational side of dead tuple cleanup. Here we are going to watch VACUUM work byte by byte. We'll snapshot a page before and after each phase, tracking exactly what changes in the page header, line pointers, tuple headers, free space map, and visibility map. Same tools as always: pageinspect, pg_visibility, and pg_freespacemap.

Setup

We need a table with enough rows to make the before-and-after comparison meaningful, plus indexes to demonstrate the full VACUUM cycle.

CREATE EXTENSION IF NOT EXISTS pageinspect;
CREATE EXTENSION IF NOT EXISTS pg_visibility;
CREATE EXTENSION IF NOT EXISTS pg_freespacemap;

CREATE TABLE vacuum_demo (
    id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    category text NOT NULL,
    payload text
);

INSERT INTO vacuum_demo (category, payload)
SELECT
    'cat_' || (i % 5),
    repeat('x', 100)
FROM generate_series(1, 50) AS i;

Fifty rows with a 100-byte payload each. The primary key gives us an index, which matters: VACUUM's behavior changes when indexes are involved. Run VACUUM once upfront so we start from a clean baseline:

VACUUM vacuum_demo;

Snapshot before any deletes

Record the baseline state of page 0. First the page header:

SELECT lower, upper, special, pagesize
FROM page_header(get_raw_page('vacuum_demo', 0));

 lower | upper | special | pagesize
-------+-------+---------+----------
   224 |  1392 |    8192 |     8192

The PostGIS Team is pleased to release PostGIS 3.7.0alpha1! Best Served with PostgreSQL 19 Beta1 and GEOS 3.15 which will be released soon.

This version requires PostgreSQL 14 - 19beta1, GEOS 3.10 or higher, and Proj 6.1+. To take advantage of all features, GEOS 3.15+ is needed. To take advantage of all SFCGAL features SFCGAL 2.3.0+ is needed.

• source download md5

• NEWS

• HTML Online en ja zh_Hans fr

• PDF docs: en ja, zh_Hans, fr

• Cheat Sheets:

  • postgis: en ja zh_Hans fr
  • postgis_raster: en ja zh_Hans fr
  • postgis_topology: en ja zh_Hans fr
  • postgis_sfcgal: en ja zh_Hans fr

This release is an alpha of a major release, it includes bug fixes since PostGIS 3.6.4 and new features.

It has been a month since PG DATA 2026, the first full-scale event organized by Prairie Postgres. Looking at the feedback we received from the seekers, sponsors, and participants (and regrets of those who were unable to come :)), I couldn’t be happier with how it went.

I know everyone says this, but let me repeat: this event wouldn’t be possible without everyone who contributed in so many different ways! One more time, I want to thank all organizational committee members, all CfP members. volunteers, speakers, and every single person who attended. You all helped us to build an open and inclusive event where everyone felt welcome.

The organization team and volunteers were so efficient that I was able to attend several talks (which is a huge improvement in comparison with all three PG Days I organized in previous years :)). My only regret is that I didn’t have time for longer conversations with speakers and attendees, especially those who visited Chicago for the first time, but I hope it wasn’t their last time in our city!

And guess what – we are already working on PG DATA 2027! The website is not up yet, but mark your calendar for June 11-12, 2027, and plan to join us in Chicago!

Things I hope will stay the same

• We will have a similarly amazing CfP committee, and will have a great program featuring both new and experienced speakers
• We will keep the ticket prices low, making the conference affordable for anyone
• We will have multiple community sponsors

Things I hope we will do more

• More people are using DEI – focused grants
• More training sessions
• More students participation
• We hope that at least some universities will participate in our Academic Partnership program

What will be better

• The conference will be held on Friday and Saturday, which we hope will allow more people to participate
• Better venue with more space and better floor plan (you won’t need to take the elevator to get from the Red Line to Green Line :))
• More flexible sponsori

Data checksums are one of those Postgres features that, when they are doing their job, are easily forgotten. They sit quietly in the header of every data page as a small integer fingerprint, forever waiting to thwart the threat of cosmic rays or errant hardware failures. Most clusters run from cradle to grave and never trip a single one.For years, that decision was etched in stone at the time of database initialization. It wasn't until version 12 that Postgres introduced the pg_checksums utility to change it. And even then, doing so is a fully offline affair, grinding through every page on disk and incurring a long outage window.That's a fairly painful ordeal for a basic safeguard that wasn't even enabled by default until version 18. So why go through all the trouble in the first place? Do we really need data checksums in our Postgres cluster? The short answer is "yes". The longer answer explains why Postgres 19 continues to improve the checksum system by adding an online conversion capability.

Bit Rot Never Sleeps

April 8 marked the start of feature freeze for PostgreSQL 19. For anyone unfamiliar with the PostgreSQL development cycle, that means that as of April 8 no new features are accepted for the upcoming major version. From April until the final release in the second part of the year, the community works on beta releases, bug fixes, and documentation. If a major feature isn't ready by the April deadline, it cannot be "snuck in." Conversely, stuff that is committed before the feature freeze isn't automatically making it into the new version as is. Patches might still get reversed, or stripped down. 

This blog is going to show you how to set up your own RAG Server on pgEdge Cloud. The Cloud UI makes this so easy it is almost insulting - a few clicks and you are done - so I am going to show you the harder and more interesting path instead: the Cloud API. Everything below is a real call you can adapt. Replace anything in with your own values, and keep your API keys out of your shell history.Click on "Services" under your database, click on "Add RAG Server" and enter your config. see? sooo easy. Soooo boring. Lets use the API.First things first: I need a dataset.  Because this is my blog, I get to choose the use-case, so I am inflicting my personal interests on you.  I am a huge Tabletop RPG nerd (yes, like Dungeons & Dragons), and my favourite system is GURPS 4th Edition (Generic Universal Roleplaying System) by Steve Jackson Games (No, you don’t have to care about this). Although mechanically simple, this is a huge sprawling game system that just grows and grows, because it is generic and universal. You can run a game in any setting or genre, so the amount of content that is available has become massive.  I have around 50 books… thankfully in electronic format.Finding the right rule at the table during play can sometimes be quite an exercise, let alone adding up all the modifiers, and applying the rule. So I did what any reasonable person with a distributed Postgres habit would do. I crammed all fifty books into my own personal RAG server, so I can ask it a plain-English question and get a cited answer back in seconds. Then when I and my degenerate friends gather every Sunday, away from the wives, clutching our beer, dice, pencils, and pizza, I can impose rapid smack-downs upon the Rules Lawyers who try to argue with me.Anyway… enough about my sample use-case, let's get into it.

Step 1: Get your content into Postgres

If you are of a certain age, the words 38911 BASIC BYTES FREE will do something to you that no amount of therapy can undo. You remember the blue screen. You remember typing in three pages of a listing from a magazine, getting ?SYNTAX ERROR IN 2340, and not knowing which of the three pages contained the typo. You remember that the disk drive was device 8, and that it was slower than continental drift.

I have some news. All of that now runs inside PostgreSQL.

PL/CBMBASIC is a procedural language extension that executes function bodies on Commodore 64 BASIC V2. Not a lookalike, not a tribute act: the actual Microsoft/Commodore interpreter from 1982, by way of Michael Steil's cbmbasic project, which statically recompiled the 6502 ROM into C. That C is compiled straight into the extension's shared library, so the interpreter lives inside your backend process. Every function call is an in-memory power cycle: zero the 64KB RAM array, reset the CPU registers, and re-enter the ROM at $E394. The whole ceremony costs about 15 to 20 microseconds, which is roughly a thousand times faster than the hardware ever managed, and quick enough to call per row over a large table without feeling guilty.

CREATE EXTENSION plcbmbasic;

CREATE FUNCTION hello(who text) RETURNS text AS $$
10 PRINT "HELLO, ";WHO$;"!"
$$ LANGUAGE plcbmbasic;

SELECT hello('WORLD');   -- HELLO, WORLD!

Yes, those are line numbers. Yes, they are mandatory. User code starts at line 10, like nature intended, because lines 0 to 9 are reserved: the extension injects your function arguments there as ordinary BASIC assignments before your code runs. A text parameter named who arrives as WHO$, a smallint named lives becomes a genuine 16-bit LIVES%, and everything numeric otherwise lands in a 40-bit CBM float, all nine glorious significant digits of it.

The validator has opinions, because BASIC V2 had opinions

Anyone who programmed a C64 for more than an hour discovered that you could not have a variable called TOTAL. The tokeniser crunched keyword

Yesterday, I had the pleasure of presenting at the Postgres User Group Estonia, and that was a delightful experience! Many thanks to Ervin Weber, who literally spent three years trying to make it happen. I was happy to give back to one of my favorite places in the world – the city of Tallinn.

I was a little bit hesitant when Ervin indicated his preference to listen to my pg_acm talk. I thought that this talk was often viewed as “too specialized”, “niche,” and not interesting enough to people who are “not very much into Postgres.” And I am so glad I ended up giving this talk to this particular group!

I have probably never heard such extensive and thoughtful feedback! Multiple people approached me during the break, saying they had run into all the problems I described, that they understand the challenges, and that they would love to give it a try! (and now I need to make sure all the bugs in the open-source version are fixed! – Watch for updates on this GitHub repo).

That was a slightly extended version of the talk I gave at PG DATA, and now that this talk has been accepted for PG.Conf EU, I need to extend it a little more, and I know what I will add and how I will incorporate the feedback I received yesterday! It always surprises me that application developers “get it” right away, unlike many DBAs, and understand the advantages of that approach. Each question I received yesterday was clear evidence that people had thought about the problems I was trying to solve and were happy to hear that a solution is available.

Thank you, Tallinn! We will do it again

For PostgreSQL administrators, DBAs, SREs, and platform teams, understanding how backup data moves through a system is just as important as knowing when a backup completed successfully. Questions about repository layout, WAL handling, metadata, integrity, and recovery usually surface when troubleshooting, validating a backup strategy, or preparing for recovery.

The Storage and Recovery Guide is written to answer those questions

Rather than focusing on commands or configuration, it follows the lifecycle of a backup through pg_hardstorage. Beginning with PostgreSQL, the guide walks through how base backups and WAL are captured, how data enters the repository, how chunks, manifests, and metadata are organized, and how those components come together to reconstruct a database during recovery.

Looking Behind the Repository

The guide also explains the engineering decisions that shape the repository itself. It explores topics such as content-addressed storage, chunking and deduplication, manifest design, metadata management, repository layout, integrity verification, corruption handling, crash safety, garbage collection, and the restore workflow, showing how these components work together rather than as isolated features.

If you like to explore the implementation alongside the architecture, the GitHub repository contains the project source code, documentation, and ongoing development of pg_hardstorage.

GitHub Repository: https://github.com/cybertec-postgresql/pg_hardstorage

Continue Exploring

Whether you are reviewing the repository design, evaluating the storage architecture, or simply interested in how pg_hardstorage approaches backup and recovery, the complete Storage and Recovery Guide provides a detailed walkthrough of the concepts, design decisions, and recovery flow behind the project.

Storage and Recovery Guide is accessible under resources section: https://www.cybertec-postgresql.com/en/products/pg-hardstorage

The post Following a Backup from PostgreSQL to Recovery usin

After more than 20 years working with PostgreSQL, I keep seeing the same problems surface at the worst possible times - bloat that sneaks up on you, replication slots quietly holding back WAL, transaction ID wraparound that nobody caught in time, backups that silently stopped working weeks ago. There are also data and catalog corruption issues like TOAST table corruption or a mismatch between heap state and VM state causing problems with vacuum operations. What I always wanted was a single tool I could point at any PostgreSQL instance and get a clear, actionable picture of its health. So I built one.pg-healthcheck is an open source utility written in Go that runs 180+ checks across 14 groups, querying live PostgreSQL system catalog views directly… no estimates, no simulated data. It works against single PostgreSQL instances as well as pgEdge multi-node Spock clusters, and gives you either coloured terminal output or structured JSON you can feed into a monitoring pipeline.

Getting Started

What Gets Checked

Hot off the press: pgcopydb v0.18 is out!

It’s the biggest release the project has had — 88 commits since v0.17, which shipped in August 2024. I took a break from my Open Source responsibilities for a while, because I was lacking employer support to make it happen.

What is pgcopydb

pgcopydb copies a PostgreSQL database to another PostgreSQL server, as fast as possible when physical file copy isn’t available. It parallelises the COPY across all tables simultaneously, builds indexes in parallel after data is loaded, and supports Change Data Capture via logical replication for minimal-downtime migrations. It is designed to be restartable: state is tracked in a local SQLite catalog so an interrupted run can resume where it left off.

Headline Features of pgcopydb v0.18

v0.18 brings compatibility with PostgreSQL 16, 17, and 18; a pgoutput-default CDC engine with significant reliability and performance improvements; regular-expression-based filtering; Citus-to-Citus migration support; and 24 bug fixes.

1. pgsql_tweaks is a bundle of functions and views for PostgreSQL
2. Changes In The pgsql_tweaks 1.0.5 Release

pgsql_tweaks is a bundle of functions and views for PostgreSQL

The source code is available on Codeberg.

The extension is also available on PGXN.

The extension is also availabe through the PostgreSQL rpm packages.

Changes In The pgsql_tweaks 1.0.5 Release

This minor update solves a problem in the deinstallation script.

Due to the changes in version 1.0 with installation of objects in a schema of its own, the generated uninstall script did not work anymore.

The deinstallation does now also work when the extension is installed in a different schema name.

PostgreSQL community images address a real gap in how a Kubernetes database operator earns your trust. Running a database operator on Kubernetes means trusting two things: the code, and the container images the operator pulls. The code is on GitHub, easy to inspect, easy to fork. The container images, the registry that hosts them, and the license that governs them all sit with the vendor, and any of those three can change without the source repository changing at all. Starting with Percona Operator for PostgreSQL 3.0.0, you can run the operator against community images you build yourself from the official PostgreSQL packages on download.postgresql.org, in a registry you control.

• Community Docker Images: tech preview in PGO 3.0.0, official in 3.1.0. Point the operator at upstream-built PostgreSQL images instead of the Percona Distribution images.
• Build them yourself from the official PostgreSQL source. The Dockerfiles pull packages from download.postgresql.org (the PGDG repositories), so the trust chain runs from PGDG to your registry with no vendor in the middle.
• There are limits. Anything Percona-specific (TDE in our distribution build, for example) does not exist in an upstream-built image. That trade is intentional.

In this post:

• How open source gets diluted in practice
• Why distributions exist anyway, honestly
• How Community Docker Images work
• Limits of the upstream path
• What to try, what to tell us

Open source has changed in the last few years, and not always for the better. Companies have learned that you can keep a project’s source code fully open and still capture most of the lock-in by quietly closing the parts that matter in production: the release artifacts, the container images, the supported OS list, the certified Kubernetes distributions, the marketplace listings.

You can have a fully community CNCF proj