pg_upgrade from 9.1.3 to 9.2 failed

[Rural Hunter]

I am trying to test the upgrade from my 9.1.3 db to 9.2 on ubuntu 10.10
server. I got error below when run the pg_upgrade command. What can I do
for this?

$ /opt/PostgreSQL/9.2/bin/pg_upgrade -b /opt/PostgreSQL/9.1/bin -B
/opt/PostgreSQL/9.2/bin -d /raid/pgsql -D /raid/pg92data
Performing Consistency Checks
-----------------------------
Checking current, bin, and data directories ok
Checking cluster versions ok
Checking database user is a superuser ok
Checking for prepared transactions ok
Checking for reg* system OID user data types ok
Checking for contrib/isn with bigint-passing mismatch ok
Creating catalog dump ok
Checking for presence of required libraries ok
Checking database user is a superuser ok
Checking for prepared transactions ok

If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.

Performing Upgrade
------------------
Analyzing all rows in the new cluster ok
Freezing all rows on the new cluster ok
Deleting files from new pg_clog ok
Copying old pg_clog to new server ok
Setting next transaction ID for new cluster ok
Resetting WAL archives ok
Setting frozenxid counters in new cluster ok
Creating databases in the new cluster ok
Adding support functions to new cluster ok
Restoring database schema to new cluster ok
Removing support functions from new cluster ok
Copying user relation files
/raid/pgsql/base/6087920/6088238
old and new databases "testdb" have a different number of relations
Failure, exiting

[Bruce Momjian]

Post by Rural Hunter
I am trying to test the upgrade from my 9.1.3 db to 9.2 on ubuntu
10.10 server. I got error below when run the pg_upgrade command.
What can I do for this?
$ /opt/PostgreSQL/9.2/bin/pg_upgrade -b /opt/PostgreSQL/9.1/bin -B
/opt/PostgreSQL/9.2/bin -d /raid/pgsql -D /raid/pg92data
Performing Consistency Checks
-----------------------------
Checking current, bin, and data directories ok
Checking cluster versions ok
Checking database user is a superuser ok
Checking for prepared transactions ok
Checking for reg* system OID user data types ok
Checking for contrib/isn with bigint-passing mismatch ok
Creating catalog dump ok
Checking for presence of required libraries ok
Checking database user is a superuser ok
Checking for prepared transactions ok
If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.
Performing Upgrade
------------------
Analyzing all rows in the new cluster ok
Freezing all rows on the new cluster ok
Deleting files from new pg_clog ok
Copying old pg_clog to new server ok
Setting next transaction ID for new cluster ok
Resetting WAL archives ok
Setting frozenxid counters in new cluster ok
Creating databases in the new cluster ok
Adding support functions to new cluster ok
Restoring database schema to new cluster ok
Removing support functions from new cluster ok
Copying user relation files
/raid/pgsql/base/6087920/6088238
old and new databases "testdb" have a different number of relations
Failure, exiting

That is an odd failure. That check was added in PG 9.1 and this is the
first time I am seeing this failure.

The check is to make sure that once we have created all the user schema
details in the new cluster, that there are the same number of objects in
the new and old databases.

Obviously there are a different number in your case here, but I don't
know why those would be different, and in fact, because we have never
hit this, there isn't even any debug output that shows the source of the
difference.

If I send you a patch can you compile it and send back the debug output
it produces?
--
Bruce Momjian <***@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

[Rural Hunter]

Post by Bruce Momjian

Post by Rural Hunter
I am trying to test the upgrade from my 9.1.3 db to 9.2 on ubuntu
10.10 server. I got error below when run the pg_upgrade command.
What can I do for this?
$ /opt/PostgreSQL/9.2/bin/pg_upgrade -b /opt/PostgreSQL/9.1/bin -B
/opt/PostgreSQL/9.2/bin -d /raid/pgsql -D /raid/pg92data
Performing Consistency Checks
-----------------------------
Checking current, bin, and data directories ok
Checking cluster versions ok
Checking database user is a superuser ok
Checking for prepared transactions ok
Checking for reg* system OID user data types ok
Checking for contrib/isn with bigint-passing mismatch ok
Creating catalog dump ok
Checking for presence of required libraries ok
Checking database user is a superuser ok
Checking for prepared transactions ok
If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.
Performing Upgrade
------------------
Analyzing all rows in the new cluster ok
Freezing all rows on the new cluster ok
Deleting files from new pg_clog ok
Copying old pg_clog to new server ok
Setting next transaction ID for new cluster ok
Resetting WAL archives ok
Setting frozenxid counters in new cluster ok
Creating databases in the new cluster ok
Adding support functions to new cluster ok
Restoring database schema to new cluster ok
Removing support functions from new cluster ok
Copying user relation files
/raid/pgsql/base/6087920/6088238
old and new databases "testdb" have a different number of relations
Failure, exiting

That is an odd failure. That check was added in PG 9.1 and this is the
first time I am seeing this failure.
The check is to make sure that once we have created all the user schema
details in the new cluster, that there are the same number of objects in
the new and old databases.
Obviously there are a different number in your case here, but I don't
know why those would be different, and in fact, because we have never
hit this, there isn't even any debug output that shows the source of the
difference.
If I send you a patch can you compile it and send back the debug output
it produces?

Yes sure, I will try to compile and retest with it.

[Bruce Momjian]

Post by Rural Hunter

Post by Bruce Momjian

Post by Rural Hunter
Performing Upgrade
------------------
Analyzing all rows in the new cluster ok
Freezing all rows on the new cluster ok
Deleting files from new pg_clog ok
Copying old pg_clog to new server ok
Setting next transaction ID for new cluster ok
Resetting WAL archives ok
Setting frozenxid counters in new cluster ok
Creating databases in the new cluster ok
Adding support functions to new cluster ok
Restoring database schema to new cluster ok
Removing support functions from new cluster ok
Copying user relation files
/raid/pgsql/base/6087920/6088238
old and new databases "testdb" have a different number of relations
Failure, exiting

That is an odd failure. That check was added in PG 9.1 and this is the
first time I am seeing this failure.
The check is to make sure that once we have created all the user schema
details in the new cluster, that there are the same number of objects in
the new and old databases.
Obviously there are a different number in your case here, but I don't
know why those would be different, and in fact, because we have never
hit this, there isn't even any debug output that shows the source of the
difference.
If I send you a patch can you compile it and send back the debug output
it produces?

Yes sure, I will try to compile and retest with it.

Actually, I have a simpler idea. At the point where it fails, you can
run pg_dump --schema-only on the testdb database in the old and new
cluster and then diff those output files and email the result to us; it
should show the mismatch. I am not sure if the dumps will output the
objects in the same order, it might.
--
Bruce Momjian <***@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

[Rural Hunter]

Post by Bruce Momjian

Post by Rural Hunter

Post by Bruce Momjian

Post by Rural Hunter
Performing Upgrade
------------------
Analyzing all rows in the new cluster ok
Freezing all rows on the new cluster ok
Deleting files from new pg_clog ok
Copying old pg_clog to new server ok
Setting next transaction ID for new cluster ok
Resetting WAL archives ok
Setting frozenxid counters in new cluster ok
Creating databases in the new cluster ok
Adding support functions to new cluster ok
Restoring database schema to new cluster ok
Removing support functions from new cluster ok
Copying user relation files
/raid/pgsql/base/6087920/6088238
old and new databases "testdb" have a different number of relations
Failure, exiting

That is an odd failure. That check was added in PG 9.1 and this is the
first time I am seeing this failure.
The check is to make sure that once we have created all the user schema
details in the new cluster, that there are the same number of objects in
the new and old databases.
Obviously there are a different number in your case here, but I don't
know why those would be different, and in fact, because we have never
hit this, there isn't even any debug output that shows the source of the
difference.
If I send you a patch can you compile it and send back the debug output
it produces?

Yes sure, I will try to compile and retest with it.

Actually, I have a simpler idea. At the point where it fails, you can
run pg_dump --schema-only on the testdb database in the old and new
cluster and then diff those output files and email the result to us; it
should show the mismatch. I am not sure if the dumps will output the
objects in the same order, it might.

diff attached.

[Bruce Momjian]

Post by Rural Hunter

Post by Bruce Momjian

Post by Rural Hunter

Post by Bruce Momjian
The check is to make sure that once we have created all the user schema
details in the new cluster, that there are the same number of objects in
the new and old databases.
Obviously there are a different number in your case here, but I don't
know why those would be different, and in fact, because we have never
hit this, there isn't even any debug output that shows the source of the
difference.
If I send you a patch can you compile it and send back the debug output
it produces?

Yes sure, I will try to compile and retest with it.

Actually, I have a simpler idea. At the point where it fails, you can
run pg_dump --schema-only on the testdb database in the old and new
cluster and then diff those output files and email the result to us; it
should show the mismatch. I am not sure if the dumps will output the
objects in the same order, it might.

diff attached.

OK, I see many new ALTER TABLE commands, but nothing that would cause a
difference in relation count.

Attached is a patch that will return the OID of the old/new mismatched
entries. Please research the pg_class objects on the old/new clusters
that have the mismatch and let me know. It might be something that
isn't in the old cluster, or not in the new cluster.

[Rural Hunter]

Post by Bruce Momjian

Post by Rural Hunter

Post by Bruce Momjian

Post by Rural Hunter

Post by Bruce Momjian
The check is to make sure that once we have created all the user schema
details in the new cluster, that there are the same number of objects in
the new and old databases.
Obviously there are a different number in your case here, but I don't
know why those would be different, and in fact, because we have never
hit this, there isn't even any debug output that shows the source of the
difference.
If I send you a patch can you compile it and send back the debug output
it produces?

Yes sure, I will try to compile and retest with it.

Actually, I have a simpler idea. At the point where it fails, you can
run pg_dump --schema-only on the testdb database in the old and new
cluster and then diff those output files and email the result to us; it
should show the mismatch. I am not sure if the dumps will output the
objects in the same order, it might.

diff attached.

OK, I see many new ALTER TABLE commands, but nothing that would cause a
difference in relation count.
Attached is a patch that will return the OID of the old/new mismatched
entries. Please research the pg_class objects on the old/new clusters
that have the mismatch and let me know. It might be something that
isn't in the old cluster, or not in the new cluster.

I ran the pg_upgrade with the patch and found the problematic object is
a toast object.
Copying user relation files
/raid/pgsql/base/6087920/6088238
Mismatch of relation OID in database "forummon": old OID 16439148, new
OID 16439322

In old cluster:
# select * from pg_class WHERE oid=16439148;
relname | relnamespace | reltype | reloftype | relowner | relam |
relfilenode | reltablespace | relpages | reltuples | reltoastrelid |
reltoastidxid | relhasindex | relisshared | relpersistence | relkind |
relnatts | relchecks | relhasoids | relhaspkey | relhasrules |
relhastriggers | relhassubclass | relfrozenxid | relacl | reloptions
-------------------+--------------+----------+-----------+----------+-------+-------------+---------------+----------+-----------+---------------+---------------+-------------+-------------+----------------+---------+----------+-----------+------------+------------+-------------+----------------+----------------+--------------+--------+------------
pg_toast_16439145 | 99 | 16439149 | 0 | 10 | 0 | 16439148 | 0 | 0 | 0 |
0 | 16439150 | t | f | p | t | 3 | 0 | f | t | f | f | f | 630449585 | |
(1 row)

But it doesn't exist in new cluster:
select * from pg_class WHERE oid=16439148;
relname | relnamespace | reltype | reloftype | relowner | relam |
relfilenode | reltablespace | relpages | reltuples | relallvisible |
reltoastrelid | reltoastidxid | relhasindex | relisshared |
relpersistence | relkind | relnatts | relchecks | relhasoids |
relhaspkey | relhasrules | relhastriggers | relhassubclass |
relfrozenxid | relacl | reloptions
---------+--------------+---------+-----------+----------+-------+-------------+---------------+----------+-----------+---------------+---------------+---------------+-------------+-------------+----------------+---------+----------+-----------+------------+------------+-------------+----------------+----------------+--------------+--------+------------
(0 rows)