Mirror layout proposal

[Graham Forest]

So, the time has come to take a first jab at how to layout mirrors.
There are, in my opinion, a few basic requirements which I'll get out of
the way first:

* Identical paths on all mirrors
Identical paths are necessary for roundrobin dns, and general
sanity. zynot.oregonstate.edu has our files at /pub/zynot, so it seems
logical to me to require that path point to our distribution trees on
all mirrors. /pub may be too ftp specific, in which case possibly /zynot
would be adequate.

* Organized distfiles section
It's my opinion that the distfiles should be kept in subcategories.
In my opinion there should be a folder per application, ie all
app-admin/foo distfiles be placed in an app-admin/foo/ subcategory. Of
course, this would require that some ebuilds that use the same distfiles
be tweaked to download such shared files into the most logical 'parent'
directory, but we're going to be thrashing the trees rather thoroughly
anyway, so that shouldn't be too much of a problem (also, it's not a
huge deal to mirror some files multiple times accidentally, and such can
be easily remedied later).

* Per-dev (or project) mirror space
For Zynot-hosted projects, and other various things devs may need to
distribute, there needs to be a section in the tree. This would be
synced either out of ~/something on jupiter (dev shell box), or part of
the SVN tree.

* Sane, standard gpg signature distribution
Mirrors should not have the ability to compromise our security by
changing the contents of tarballs. md5sums are not a practical method of
thwarting this, as more often than not it is assumed that the md5sum is
wrong, not the file. Users should be given the option to reject
distfiles that have not been gpg signed by X number of trusted devs, and
those signatures must exist somewhere in our distribution tree. An idea
brought up in #zynot-infra is to have the downloading of the gpg
signatures default to always coming from our main mirror, only getting
them off other mirrors after failing at that, and printing a nice
warning message. Of course, with proper key signing and distribution the
security benefits of this are reduced, but a nice side-effect is that we
would be able to track the approximate number of downloads without
heavily monitoring every mirror.

* Easy to navigate iso/stage file distribution
I think just about everyone has spent a few hours of their life
hunting around badly layed out ftp servers. Users should be able to
click to direcly what they want without having to search. In my mind,
this means not only having an intuitive layout, but also providing handy
things such as links to "current" versions (when possible, symlinks may
cause problems).


So, I throw the following out to be poked and prodded until somewhat
sane:

/pub/zynot/
distfiles/ # Organized distfiles dir
app-admin/
foo/
foo.tar.bz2
foo.tar.bz2.asc # Signatures (hopefully more than one
# per file)
...
moo/
...
...
app-benchmarks/
bar/
bar.tar.bz2
bar.tar.bz2.asc
...
...
...
people/
gforest/ # Synced from
/svn/zynot/people/gforest/something?
releases/
stable/ # Symlink?
unstable/ # Symlink?
0.1
x86/
iso/
zynot-moo-foo-ra-gu-0.1.iso.bz2
stages/
zynot-ug-ar-oof-oom-0.1.tar.bz2
ppc/
iso/
zynot-moo-foo-ra-gu-0.1.iso.bz2
stages/
zynot-ug-ar-oof-oom-0.1.tar.bz2
ppc64/
...
1.1_rc385 # Just kidding
x86/
ppc/
ppc64/
...
...
...

I appreciate all prompt constructive feedback.

Have fun but remain pleasant,
Graham

[Graham Forest]

It has been recommended (by our man at OSU) that, instead of trying to
require /zynot symlinks, we simply require HTTP, and use virtualhosting
so that our path is always /. I mean really, would you miss ftp? :)

Thanks,
Graham

[Mark Guertin]

Post by Graham Forest
It has been recommended (by our man at OSU) that, instead of trying to
require /zynot symlinks, we simply require HTTP, and use virtualhosting
so that our path is always /. I mean really, would you miss ftp? :)

I would miss ftp big time !! :)

Typically any (well setup) http server will throttle bandwidth to 20k or 40k
/sec. Ftp is generally not so throttled.

As for virtual hosting, that is not a great option for servers not in our
control, not many of the larger hosting mirrors will allow changes to their
setups WRT virtual hosting, etc. and I personally would not offer to host
files for a project that requires me reconfiguring my servers to that level.

Mark

[Jesse Nelson]

Post by Graham Forest
I mean really, would you miss ftp? :)

yes :P

Post by Graham Forest
Thanks,
Graham
______________________________________________________________________
_______________________________________________
Zynaut mailing list
http://lists.zynot.org/mailman/listinfo/zynaut

BOFH Excuse #283: Lawn mower blade in your fan need sharpening

[Mike Frysinger]

Post by Graham Forest
It has been recommended (by our man at OSU) that, instead of trying to
require /zynot symlinks, we simply require HTTP, and use virtualhosting
so that our path is always /. I mean really, would you miss ftp? :)
Thanks,
Graham

ftp is designed for file transfer, HTTP is not ...
if you want i can go into an analysis of the FTP protocol vs HTTP and why ftp
is much more suited ...
-mike

[Scott Kveton]

Post by Mike Frysinger
ftp is designed for file transfer, HTTP is not ...
if you want i can go into an analysis of the FTP protocol vs HTTP and why ftp
is much more suited ...

FTP is an awful protocol for many number of reasons. Getting past
firewalls, crappy server software and the nature of the beast (limiting
to a number of connections instead of amount of bandwidth) just to name
a few.

HTTP is not ideal, but its the easiest to deploy, Apache is _kind of_ a
good application ... 10.6Tb last month via HTTP and only 6Tb for ftp on
our end ... it also appears users prefer that method.

Scott :-)

[Mark Guertin]

Post by Graham Forest
* Organized distfiles section
It's my opinion that the distfiles should be kept in subcategories.
In my opinion there should be a folder per application, ie all
app-admin/foo distfiles be placed in an app-admin/foo/ subcategory. Of
course, this would require that some ebuilds that use the same distfiles
be tweaked to download such shared files into the most logical 'parent'
directory, but we're going to be thrashing the trees rather thoroughly
anyway, so that shouldn't be too much of a problem (also, it's not a
huge deal to mirror some files multiple times accidentally, and such can
be easily remedied later).

This is not something that is properly do-able at this time afaik. It would
mean a rewrite of large portions of porteus because it always defaults to a
single mirror location (i.e. Mirror:/distfiles/ ) and cannot be set within
builds to do it differently aside from removing that functionality from
porteus and updating ever ebuild by hand to point to the mirror locations
first).

As messy as it is I think we are more stuck with a single (huge) folder
holding the sources until we can build in a better mechanism.

Also we should avoid duplication of files on mirrors, that is just _asking_
for trouble. I.E. One tarball gets updated, the other doesn't and chaos
will happen. Vendors update tarballs without doing revision bumps all too
often which can lead to complications for all.

Mark

[Mark Guertin]

Post by Mark Guertin

Post by Graham Forest
* Organized distfiles section
It's my opinion that the distfiles should be kept in subcategories.
In my opinion there should be a folder per application, ie all
app-admin/foo distfiles be placed in an app-admin/foo/ subcategory. Of
course, this would require that some ebuilds that use the same distfiles
be tweaked to download such shared files into the most logical 'parent'
directory, but we're going to be thrashing the trees rather thoroughly
anyway, so that shouldn't be too much of a problem (also, it's not a
huge deal to mirror some files multiple times accidentally, and such can
be easily remedied later).

This is not something that is properly do-able at this time afaik. It would
mean a rewrite of large portions of porteus because it always defaults to a
single mirror location (i.e. Mirror:/distfiles/ ) and cannot be set within
builds to do it differently aside from removing that functionality from
porteus and updating ever ebuild by hand to point to the mirror locations
first).
As messy as it is I think we are more stuck with a single (huge) folder
holding the sources until we can build in a better mechanism.
Also we should avoid duplication of files on mirrors, that is just _asking_
for trouble. I.E. One tarball gets updated, the other doesn't and chaos
will happen. Vendors update tarballs without doing revision bumps all too
often which can lead to complications for all.

Also just had another thought on this. This would prevent us from having
multiple categories with the same packages without having funky symlinks or
duplicated code. This may be very desirable if not mandatory on some
arches.

Mark

[Graham Forest]

Ok, so in response to the responses:

* I still think files should be located at the same path on all mirrors,
though noone has commented on a better way to achieve such without just
requiring a standard path point to our directory to become a mirror.

* FTP may be designed for transferring files where HTTP is more for data
in general, but in my experience, users have less problems getting files
through firewalls and other such madness with HTTP. Doesn't it just erk
you when you hit an ftp site that doesn't tell your client how large a
file is while it's downloading it? Regardless, I don't really object to
staying around, or going the way of Richard Nixon. It was just an idea,
after all. Oh, and no Mike, no need to preach why FTP is the ultimate
something or other to me, the initial objections were enough. ^_^

* Organized distfiles is put off for now, but desired for the future,
when porteus version 42 is able to support it, and that whole multiple
builds using the same distfile problem is hacked around. I don't see it
as much of a problem for when porteus is cleanly re-written with a new
build format, and I don't feel it necessary enough to warrant wasting
the time on portage-porteus getting it working. So, in the meantime, all
distfile syncing scripts will be designed with such a layout in mind,
but ugly ugly monolithic distfiles dirs will be utilized.

* No comments on per-dev/project space - moo

* No comments on where gpg signatures should go - Security guys, we
actually need a whole new approach at this, not only for distfiles, but
also for build scripts.

* No comments on physical layout - I'd really like some comments on this
one, as I damn well know my layout isn't that good :).

* Until further notice, Gerk's idea flamethrower has been confiscated.
Actually, the previous statement was a joke. In reality I filled it with
cranberry juice (from concentrate) without telling him.

Keep hammering, and we'll forge something adequate out eventually.

Thanks,
Graham

[Jesse Nelson]

SNiP

Post by Graham Forest
* No comments on where gpg signatures should go - Security guys, we
actually need a whole new approach at this, not only for distfiles, but
also for build scripts.

I think that they should come with the files of the package .. mainly
for ease of packaging. distribute them with the package to the mirrors.
theres no reason to corral the sigs. If any mirror mucks with a sig its
invalidated.

The exception has got to be how we handle getting the "key ring" to the
user. I had proposed we have 2 signing keys just for the key ring and
that porteous or whatever it becomes will sync->install this
non-interactively. As well as check the sigs on that package against our
published pubkeys ( http/ldap/pgp keyserver).. This package can be
distributed to the mirrors, but it should never be installed without
first checking the sigs against an external source.

as for distfiles. I would like to see a mechanisim for shipping the
original sigs for these things.. tho i havent worked out how yet :P

i see distfiles sigs all being in one big directory as well
/distfiles/sigs/*.sig untill somthing better with distfiles is worked
out.

Post by Graham Forest
Thanks,
Graham
______________________________________________________________________
_______________________________________________
Zynaut mailing list
http://lists.zynot.org/mailman/listinfo/zynaut

I opened the drawer of my little desk and a single letter fell out, a
letter from my mother, written in pencil, one of her last, with
unfinished words and an implicit sense of her departure. It's so
curious: one can resist tears and "behave" very well in the hardest
hours of grief. But then someone makes you a friendly sign behind a
window... or one notices that a flower that was in bud only yesterday
has suddenly blossomed... or a letter slips from a drawer... and
everything collapses. -- Letters From Colette

[Amir Guindehi]

Hi,

Post by Graham Forest
* No comments on where gpg signatures should go - Security guys, we
actually need a whole new approach at this, not only for distfiles,
but also for build scripts.

I always imagined that those signatures are kept within the ebuild
directory. I would propose a sepparate file containing the signatures
and using a file type extention of the for gna-1.0.ebuild.gpg or
something along this line. This will allow us to use another files named
gna-1.0.ebuild.XXX for different sorts of signatures, eg: x509.

Naturally those signature files could also be placed to arbitrary
different places. I'm not sure we would want this. Shouldn't ebuild and
signatures share a common directory?

Post by Graham Forest
... that have not been gpg signed by X number of trusted devs ...

I would propose to implement this in an abstract way so that _different_
types of trust can be applied! I can imagine multiple form of trust as
for example:

GPG:
- Trust in a keyring of Zynot developers
- Trust in a top level signature of one or multiple Zynot GPG
Certificate Authority Certificates, created specially for this purpose.
This maps a hirarchical trust schema like the x509 schema to GPG. Trust
would essentially be provided by giving the End-User a keyring of
trusted GPG CA Certificates, and not individual developers. By defining
apropriate certificate verification depths these special signature
certificates could be checked.
- Trust in a simple web of trust given by a arbitrary keyring (user
supplied)

x509:
- Trust in the Zynot Certificate Authority's trust chain
- Trust in a foreign Certificate Authority's trust chain

Last but not lest:
- Trust in any combination of the above. This will allow the checking of
the GPG signatures by x509 signatures and vice versa. It will also allow
to use a web of trust and hirarchical trust chain in combination giving
the system a independency of arbitrary bugs in one or the other form of
signatures and trust schemas...

Please share your thoughts!
- Amir