Version bump policies and beta packages in stable branch

[Frantz Dhin]

Hello all,

So we have this problem that I'd like to put up here for discussion. It has
been boggling my mind for quite a while.



The problem:

Best summarized using the Gentoo distribution as an example. In Gentoo you
basically have a choice of 2 distributions. A stable and an unstable branch.
Some would like you to believe that a choice between a wealth of package
versions exists in this "meta distribution", but in reality you are limited
to these two package sets.

If you type "emerge mysql" on the command line in Gentoo today you get
version 4.0.16 if you are running the stable branch, and 4.0.17 if you are
running unstable. Now question is: Where are v4.1 and v5.0 versions of MySQL
in the Portage tree? Answer: They are not there. So when will they be there
and when will stable or unstable versions suddenly prompt you to upgrade?
Good question that only the package maintainer can answer.

Now the problem in performing a MySQL v4.0 to v4.1 upgrade will be that some
upgrading will need to be done on the database tables themselves. In other
words a 4.0 database is most likely not compatible with a 4.1 database. For
a database of a reasonable size this is no trivial operation. It will
require planning, testing and a good chunk of human resource time. It is an
operation that needs to be carried out at a time of convenience.

For a while Gentoo was running with MySQL 3.23 in the stable branch and
4.0.x in the unstable branch. One day the package maintainer decided that
now it was time to "move v4.0.x to stable" and announced this on gentoo-dev
mailing list, and so it became.

I hope everyone is now beginning to see the problem. What happens when it is
time to move 4.1 into unstable or stable? How many oopses will we hear the
users say? How much annoyance is it going to cause when a package manager
constantly offers you a package upgrade that you don't have time for or may
not even want for years to come? And what happens when you in 2 years badly
need to restore your MySQL 4.0 system fast and smooth. - Is the ebuild for
the exact version you need still guaranteed to be there and accessible for
you?



A workaround could be to split it into 3 separate packages named such as
these. The naming is taken from mysql.com.



Mysql-production-release (Currently 4.0.x)

Mysql-alpha-release (Currently 4.1.x)

Mysql-development-tree (Currently 5.0.x)



But really that is just delaying and making a half solution to the problem,
because there will be a day where v4.1 will be recommended as the current
"production release".



I have here used Gentoo and MySQL as an example. Not intended as flaming,
but to illustrate a problem of a more general nature. Think of Apache 1.3.x
and 2.0.x which are both used in production and will be for a long time
still. Think of the Gimp (1.2.x) and the beta version of it (1.3.x) which
also get version incremented independently in both branches.

In Gentoo it was for a while as such that if you were running stable you got
Gimp 1.2 and if you were running unstable you got Gimp 1.3 handed to you.
IMHO you should have the choice of both no matter what OS branch you run,
but here obviously separating into 2 packages and naming them gimp and
gimp-beta would suffice.



To conclude this I suggest a general policy for the trees we develop to
provide as much choice as possible when it comes to running beta packages in
a so-called 'stable' release. We should, especially for desktop
applications, avoid forcing a whole set or branch of packages on a user who
in all innocence is curious about what the beta release of Gimp is like.

It would all-in-all be a more effective use of having a stable vs an
unstable branch if, for instance, gimp 1.2.4 and gimp-beta 1.3.22 were
placed in stable, while gimp 1.2.5 and gimp-beta 1.2.23 were in testing
stage. I suggest making this a general principle as long as the technical
implications do not become too much of a burden. I realize that running a
beta of an application could force an upgrade of a bunch of libs which again
could break a lot of things. I imagine a beta of Evolution for instance
would have this trait, so a policy like this of course has to be kept within
reason.

Furthermore, not neglecting the problem with a package such as MySQL, I am
still in search for solutions and mechanisms and policies to handle this
exact problem. Do not forget that we seek to create a data center worthy
distribution. The demands for such a distribution is that it can be
installed on large machinery and clusters and be able to run the next 15
years with 99.999% uptime. Take this into account, but I am eager to hear
comments and proposals for solutions.



Best Regards



Frantz Dhin

CEO, Zynot Foundation

[Low Zhen Lin]

I can offer the technical solution:

MySQL comes in three mutually incompatible branches: 4.0, 4.1 and 5.0.

Thus, the XBuild tree will have three MySQL projects: mysql-4.0,
mysql-4.1 and mysql-5.0.

'mysql' is an alias for all three projects. Thus, mysql is a virtual
project.

When a user issues 'xeta install mysql' on a system that has not had
mysql installed, it will be dealt with like any other virtual
dependency -- depending on user configuration, Xeta might prompt the
user to select one between the tree; or present the three options and
exit with an error; or select a tree-specified default; etc.

When a user issues 'xeta update mysql' on a system that has had mysql
installed, it will update to the newest in the version-branch installed.

Version constraints apply to virtual projects as well; if a package
specifies '[database server] mysql::lib ge 5.0', the version
constraint reduces the choices to only mysql-5.0.

[David Nielsen]

I agree with Zhen Lin, the far technically superior solution is to
compose a virtual package and default to sanity (stable branch) for all
such projects - gnome fx. runs and even/odd versioning scheme for stable
and development releases where this could be useful.

I noticed Debian's dpkg system has a user notification level for users
to set during install (we could have a setting in make.conf instead)
which sets the level of interaction the system will have with the user.
basically going from "gimme the sane defaults" to "I wanna do it all
myself" in something like 7 steps. A similar approach might be useful
for xeta in these cases and others like it that require tampering with
config files and/or recompilation of secondary package(s).

btw. I'm likely to be taking a 1 year off from school to adjust my
medical treatment, meaning I should have plenty of time to work on Zynot
related stuff should it be needed - I imagine the kernel might be
needing some love as well as the gnome desktop.

Regards
David 'Lovechild' Nielsen

[Frantz Dhin]

Post by David Nielsen
I noticed Debian's dpkg system has a user notification level for users
to set during install (we could have a setting in make.conf instead)
which sets the level of interaction the system will have with the user.
basically going from "gimme the sane defaults" to "I wanna do it all
myself" in something like 7 steps. A similar approach might be useful
for xeta in these cases and others like it that require tampering with
config files and/or recompilation of secondary package(s).

Yes we are going to need something similar in principle. A goal is to
provide opportunity for fully unattended installations, as well as
opportunity to configure any feature in the distribution before installation
takes place. May seem like I lost my marbles, but if you're faced with the
task of rolling out a couple of hundred nodes with linux on it then this
feature is something that adds real value.

Post by David Nielsen
btw. I'm likely to be taking a 1 year off from school to adjust my
medical treatment, meaning I should have plenty of time to work on Zynot
related stuff should it be needed - I imagine the kernel might be
needing some love as well as the gnome desktop.

I imagine that too :)

Frantz Dhin

Post by David Nielsen
Regards
David 'Lovechild' Nielsen
_______________________________________________
zynaut mailing list
http://lists.zynot.org/mailman/listinfo/zynaut

[Frantz Dhin]

I like it. You write that Xeta should "select a tree-specified default" for
virtual dependencies, and I think such possibility is a must-have with a
solution like this.
An example scenario that warrants it is if MySQL (or another virtual
dependency) is not the desired end result in itself. If a package such as qt
is requested to be compiled with MySQL support, then Xeta should default to
the 1 version that tree maintainers find most suitable for the overall
purpose of the tree. For a production tree that would almost certainly be
v4.0 of MySQL, but for an "I-like-to-live-life-dangerously" tree it could be
v5.0. This is of course assuming that qt will compile against all 3 versions
of MySQL, which I think is unlikely, but let's pretend for the moment that
it does. :) Of course the possibility to block deps known not to work
should still be there, and it is iirc.
I have a request, and that is that configuration of such tree-specified
defaults get separated from the other configuration options. It would be
best to provide a conf file that can be freely tampered with by user, and a
number of other files where the typical "If you edit this file you will void
your support" statement is. (If we were nasty we could even make Xeta do MD5
digest of these files and send them back to support server live for
user/customer to prove that he didn't mess with anything he shouldn't be
messing with :))
So anyway that would mean at least 1 conf file per tree that user syncs up
with and combines, so override policies and stuff like that would be in
demand. Of course this still leaves the quirk for people who never ever wish
to move from 4.0 to 4.1, but it somehow seems easier to reason with now. If
the tree conf file is versioned like the xbuilds are, then a user can stay
with a distribution version and never get untimely updates shoved in his
face.

Frantz Dhin

-----Original Message-----
Sent: 24. januar 2004 11:58
Subject: Re: Version bump policies and beta packages in stable branch
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
MySQL comes in three mutually incompatible branches: 4.0, 4.1 and 5.0.
Thus, the XBuild tree will have three MySQL projects: mysql-4.0,
mysql-4.1 and mysql-5.0.
'mysql' is an alias for all three projects. Thus, mysql is a virtual
project.
When a user issues 'xeta install mysql' on a system that has not had
mysql installed, it will be dealt with like any other virtual
dependency -- depending on user configuration, Xeta might prompt the
user to select one between the tree; or present the three options and
exit with an error; or select a tree-specified default; etc.
When a user issues 'xeta update mysql' on a system that has had mysql
installed, it will update to the newest in the version-branch installed.
Version constraints apply to virtual projects as well; if a package
specifies '[database server] mysql::lib ge 5.0', the version
constraint reduces the choices to only mysql-5.0.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iD8DBQFAEk+sv+6a/MPcjnERAgOZAJwI0CYR0HXFMsMUHpCeCfGhSru9CACfTrT/
UU6eHuqvfmrPYGRNFkFuetM=
=K2/j
-----END PGP SIGNATURE-----
_______________________________________________
zynaut mailing list
http://lists.zynot.org/mailman/listinfo/zynaut

[Chris Frey]

Post by Frantz Dhin
I have a request, and that is that configuration of such tree-specified
defaults get separated from the other configuration options. It would be
best to provide a conf file that can be freely tampered with by user, and a
number of other files where the typical "If you edit this file you will void
your support" statement is.

[...]

Post by Frantz Dhin
So anyway that would mean at least 1 conf file per tree that user syncs up
with and combines, so override policies and stuff like that would be in
demand. Of course this still leaves the quirk for people who never ever wish
to move from 4.0 to 4.1, but it somehow seems easier to reason with now. If
the tree conf file is versioned like the xbuilds are, then a user can stay
with a distribution version and never get untimely updates shoved in his
face.

Beautiful. Another way to state this, is to separate a package's version
selection from the XBuilds, and even separate that selection from the
"release". As a user, I want complete freedom to go backwards or forwards
in time, in relation to versions.

To do this, an XBuild file should never disappear, ever. I should always
be able to emerge a specific version that I had before, that worked.

For example:

XBuild respository
(holds every XBuild file ever released by Zynot)
|
.------------------+-----------------------.
Zynot Release 1.0 | Zynot Release 1.1
(conf file listing packages | (another conf file)
and versions for Release 1.0) |
|
.------------------+----------------------.
Stable Release Candidate Unstable
(conf file again) (conf file... again :)
this one would change regularly)


Now, as the user, I have a number of very handy options:
- I can archive the entire XBuild repository locally
- I can download only the XBuilds in my selected conf file to
save space locally
- I can pick the "release" conf file that I want to track
- I can mix and match from all versions with my own conf file
- I can go backwards in revision history if I find something
that doesn't work (especially if tracking something in
Unstable)
- Upgrades and security fixes are done via conf files, conf file
updates/merges, or versioned conf files. The updated
conf file would just refer to the fixed XBuild files
which can be found in the giant repositiory.
- If the software supports it, I should be able to install a
version from Release 1.0 and Unstable at the same time
(especially for libraries)

Ideally, the tools would be able to take a snapshot of a given system's
package versions, creating a conf file that could be used to
go back to an entire "known good" system at any time.

With control like this, people would eagerly track Unstable and get more
testing for the entire system.

- Chris

[Galik]

Hi,
There seem to be two aspects to this problem. One is how to manage it
from the perspective of the Package management System. The other is how
to manage it from the perspective of the Distribution.

1) Package Management System.

The best that the PMS can achieve is to implement a system that is
ultimately flexible and infinitely easy to use with respect to Package
Selection and Control. These two ideals (flexible and easy) are usually
mutually exclusive to one degree or another however if the underlaying
engine is flexible then the UI that gets bolted onto the engine can
implement an easy subset of all that is achievable.
This thread in the forums is useful I think for a nice flexible approach
to package selection.

http://forums.zynot.org/viewtopic.php?t=225

nall's proposed approach provides for an extremely flexible way to
prioritise packages (co)existing in multiple trees. Trees are
essentially distributions and are only called trees for historical
reasons. They do not need to be tree shaped any more although they may
be displayed in a hierarchy at the discretion of the UI.

2) Distribution.

This is allways the make or breake of a distribution as it determines:

a) How the user experiences the distro: are all the packages up to date
enough?
b) How easy the distro is to maintain: how many interdependent packages
are required to change to form a new stable?
c) Can the distro integrate with other distributed software not
affiliated with the distro? That is if I want to supply some audio apps
as a third party to anyone running a Zynot System using Xeta can people
use my mini-distro at the same time?

Solutions...(?)

a) Being source based (or at least source capable) allows for the most
fine grained upgrade progress achievable.
b) Maybe a more structured approach to the distro as opposed the the
more traditional monolithic model currently used by most. By this I mean
rather than having stable/unstable woody/metalic etc... Why not utilise
the power of the multiple trees enabled by the underlaying PMS and break
the distro down into components. For example:

RH style system
----------------

zynot-stable
zynot-testing
zynot-unstable

Componentised system
------------------------

zynot-base(-stable|-unstable)
zynot-audio(-stable|-unstable)
zynot-video(-stable|-unstable)
zynot-inet(-stable|-unstable)
zynot-gnome(-stable|-unstable)
zynot-kde(-stable|-unstable)
zynot-dev(-stable|-unstable)

Now I have to say I really don't know if breaking the distribution down
into sub-distributions like this will help or hinder the process of
building a stable platform but it's an idea that people might want to
think about.

c) A third party developer could be easily integrated into the above
scenario:

mikado-audio(-stable|-unstable) // Either an alternative to or a
suppliment of zynot-audio from mikado.
borland-dev // Borland's official development tools.

- Galik

Post by Frantz Dhin
Hello all,
So we have this problem that I’d like to put up here for discussion.
It has been boggling my mind for quite a while.
Best summarized using the Gentoo distribution as an example. In Gentoo
you basically have a choice of 2 distributions. A stable and an
unstable branch. Some would like you to believe that a choice between
a wealth of package versions exists in this “meta distribution”, but
in reality you are limited to these two package sets.
If you type “emerge mysql” on the command line in Gentoo today you get
version 4.0.16 if you are running the stable branch, and 4.0.17 if you
are running unstable. Now question is: Where are v4.1 and v5.0
versions of MySQL in the Portage tree? Answer: They are not there. So
when will they be there and when will stable or unstable versions
suddenly prompt you to upgrade? Good question that only the package
maintainer can answer.
Now the problem in performing a MySQL v4.0 to v4.1 upgrade will be
that some upgrading will need to be done on the database tables
themselves. In other words a 4.0 database is most likely not
compatible with a 4.1 database. For a database of a reasonable size
this is no trivial operation. It will require planning, testing and a
good chunk of human resource time. It is an operation that needs to be
carried out at a time of convenience.
For a while Gentoo was running with MySQL 3.23 in the stable branch
and 4.0.x in the unstable branch. One day the package maintainer
decided that now it was time to “move v4.0.x to stable” and announced
this on gentoo-dev mailing list, and so it became.
I hope everyone is now beginning to see the problem. What happens when
it is time to move 4.1 into unstable or stable? How many oopses will
we hear the users say? How much annoyance is it going to cause when a
package manager constantly offers you a package upgrade that you don’t
have time for or may not even want for years to come? And what happens
when you in 2 years badly need to restore your MySQL 4.0 system fast
and smooth. – Is the ebuild for the exact version you need still
guaranteed to be there and accessible for you?
A workaround could be to split it into 3 separate packages named such
as these. The naming is taken from mysql.com.
Mysql-production-release (Currently 4.0.x)
Mysql-alpha-release (Currently 4.1.x)
Mysql-development-tree (Currently 5.0.x)
But really that is just delaying and making a half solution to the
problem, because there will be a day where v4.1 will be recommended as
the current “production release”.
I have here used Gentoo and MySQL as an example. Not intended as
flaming, but to illustrate a problem of a more general nature. Think
of Apache 1.3.x and 2.0.x which are both used in production and will
be for a long time still. Think of the Gimp (1.2.x) and the beta
version of it (1.3.x) which also get version incremented independently
in both branches.
In Gentoo it was for a while as such that if you were running stable
you got Gimp 1.2 and if you were running unstable you got Gimp 1.3
handed to you. IMHO you should have the choice of both no matter what
OS branch you run, but here obviously separating into 2 packages and
naming them gimp and gimp-beta would suffice.
To conclude this I suggest a general policy for the trees we develop
to provide as much choice as possible when it comes to running beta
packages in a so-called ‘stable’ release. We should, especially for
desktop applications, avoid forcing a whole set or branch of packages
on a user who in all innocence is curious about what the beta release
of Gimp is like.
It would all-in-all be a more effective use of having a stable vs an
unstable branch if, for instance, gimp 1.2.4 and gimp-beta 1.3.22 were
placed in stable, while gimp 1.2.5 and gimp-beta 1.2.23 were in
testing stage. I suggest making this a general principle as long as
the technical implications do not become too much of a burden. I
realize that running a beta of an application could force an upgrade
of a bunch of libs which again could break a lot of things. I imagine
a beta of Evolution for instance would have this trait, so a policy
like this of course has to be kept within reason.
Furthermore, not neglecting the problem with a package such as MySQL,
I am still in search for solutions and mechanisms and policies to
handle this exact problem. Do not forget that we seek to create a data
center worthy distribution. The demands for such a distribution is
that it can be installed on large machinery and clusters and be able
to run the next 15 years with 99.999% uptime. Take this into account,
but I am eager to hear comments and proposals for solutions…
Best Regards
Frantz Dhin
CEO, Zynot Foundation
------------------------------------------------------------------------
_______________________________________________
zynaut mailing list
http://lists.zynot.org/mailman/listinfo/zynaut