Now that PostgreSQL is becoming the database of choice for independent software vendors, we're developing a new problem: software vendors do not apply updates. Within the last month, we've had the exact same conversation with four different ISV customers we have:
Customer: we have an instance of data corruption on one of our client's machines. detailed description follows
pgExperts: yes, that sounds like data corruption. What version of PostgreSQL are you running on that machine?
Customer: 8.4.1.
pgExperts: 8.4.1 is missing 2 years of patch updates, including fixes for several data corruption issues. You should have updated to 8.4.11.
Customer: so can you fix it?
pgExperts: you need to apply the update to the current PostgreSQL patch version first.
Customer: we can't do that. Can you fix it?
pgExperts: not for a reasonable cost, no.
It seems that many ISVs regularly deploy databases where they have neither mechanism nor regular practice of applying updates and patches. This could be from a practice of avoiding bad patches (like those from certain major database and OS vendors), poor QA and testing, lack of remote access, inability to schedule downtimes, or some other issue. The only strange thing is the level of resistance ISVs have to the idea of applying updates, as if they'd never heard of it before. Regardless, the result is the same: the user's data is lost/corrupt/hacked, and PostgreSQL will be blamed.
I doubt we're the only middleware software provider to encounter this. My question is, what can we do to educate vendors about the need to apply updates regularly, promptly, and throughout their customer base?
My experience is that when money is involved, money talks. That the best way to motivate vendors is by working backwards through their customers. The customers hold the purse strings. That being said there maybe a way to work the other way, through the vendors. Something along the lines of a vendor channel on the Postgres site. A place where the patches and accompanying information could be subscribed to via RSS or email. In other words push the information to the vendors to get their attention. There still remains the problem of how to find the vendors and then make them aware of the channel. I am going to have to think more on that.
ReplyDelete> A place where the patches and accompanying information could be subscribed to via RSS or email
DeleteYes. Someone on IRC suggested following the pgsql-announce mailing list, to be notified about updates. But there's so much noise on that mailing list (mixed with ads about proprietary products), making it useless for this purpose. In fact, I'll be surprised if anyone at all finds that mailing list useful.
I would hardly call 15-18 messages a month much noise. I find the list useful, particularly the PostgreSQL Weekly News. This would seem to be required reading for vendors as it is a succinct update of what has happened to Postgres in the past week and what is coming in the future.
DeleteThe News ticker for the front page has an RSS feed. If we really thought that vendors weren't subscribing or weren't reading it because of the volume of vendor updates, then that would be easy to fix. However, at least among pgExperts' clientele, that's not the issue. The ISVs know about the updates; they just have no plans (and no mechanism) to apply them.
DeleteMy first comment I deleted because when I published I saw two copies and so I deleted one(turns out both). I then created the second comment and again saw two copies. This time I left the second one alone and when I refreshed it went away. Just thought you should know.
ReplyDeleteMaintaining support for a variety of different versions is difficult, and incremental improvements are usually not clear to established customers. While the project ensures point releases are safe, major releases were basically impossible for datasets which had reasonable uptime expectations prior to the release of pg_upgrade, and because pg_upgrade discards stats, is still not really capable of performing upgrades in a reasonable time on actual production datasets.
ReplyDeleteTo be fair, the situation is better, but as one of those damn, dirty vendors, I have to say that if you want us to support more versions, keep focusing on making it easier.
Peter, I'm not talking about major version upgrades, I'm talking about patch (minor) releases.
DeleteMany ISVs probably don't upgrade anything at all, not only PostgreSQL. It's just very expensive (upfront) do arrange for that.
ReplyDeleteAlso, the minor releases of PostgreSQL haven't been all rock solid lately, so I couldn't blame anyone for playing it more conservatively.
Peter,
DeleteWhen's the last time we had a problem with a minor release? I don't remember anything since 8.3.1.
The problem is that few software projects follow the strict "bug-fix only" revision scheme that Postgres does. It is very common for software to add features, change behavior, etc. between releases, even if they claim not to. So, it's a tough sell to explain that Postgres is different. However, I was also going to make the same point as Peter: even we do not have a clean record in that regard lately. That's why I no longer blindly upgrade to the latest revision, but only if the release notes indicate it fixes a relevant bug.
ReplyDeleteP.S. Josh: this new blog doesn't seem to support viewing or posting comments via Opera.
Greg, It's blogger. I'll file a bug, but I doubt Google will care about Opera.
DeleteI agree with every single word Greg said. I've been dealing with ISVs in Brazil and they insist to not upgrade mainly for misinformation. They generally don't know how the upgrade process works. I mean, they think that 8.4.1 and 8.4.11 are different (in features) releases; so they are afraid to break the applications. I blogged some time ago [1] (in pt-br) to clarify these points. Of course, there are people that know how the upgrade process works and don't do it because it doesn't have the cycles to perform QA and testing for every minor releases (9.0 has a new minor release each ~ 2.5 month). In this case, the upgrade is forced only by a bug-fix that affects some application.
ReplyDeleteWhat could we improve to turn the upgrade process easier to understand?
(i) improve the search for bug fixes. If I want to know if a bug was fixed in 8.4.11 and I'm using 8.4.1, I have to click in all of the minor version release notes; that's too time-consuming (10 clicks);
(ii) a new ML for releases only. I don't know if it solves some problem but some people could be interested only in new (minor or major) releases;
(iii) new version string. Instead of "PostgreSQL 9.1.2" we could use "PostgreSQL 9.1 patchlevel 2". We could even indicate the release date "PostgreSQL 9.1 patchlevel 2 from 2012-02-27". Synonyms for "patchlevel" could be "fixpack", "service pack", or "maintenance level".
[1] http://eulerto.blogspot.com.br/2011/09/hora-de-considerar-versao-91.html
Your articles make complete sense out of each topic.
ReplyDeletec2logix.com
Your articles support me a lot in all mediums of subjects.
ReplyDeleteprice per head company
I want to know if a bug was fixed in 8.4.11 and I'm using 8.4.1, I have to click in all of the minor version release notes; that's too time-consuming (10 clicks);domain name
ReplyDelete