I really like the idea of having a single URL and having the web server

    on pkgbox issue a redirect as appropriate. 
    We could even have pkgbox test the other mirrors periodically to

    determine whether they are up or not.
    I would even do it with a CGI so it could be made as smart as possible.

    So, e.g. we could build-in some load balancing which would allow us to

    have 'slow' mirrors in our rotation which we would only redirect a fixed

    percentage of the requests to.
					-Matt

					Matthew Dillon

					<dillon@backplane.com>

Or hosts periodically syncing get automatically checked if they serve

the data via http and get added to the list :)
cheers

  simon

(sorry, accidentally hit send on a previous version of the message.)
This is exactly why I said "use a CGI redirect" instead of DNS - I've

previously encountered this with other multi-host download systems like

Perl's CPAN - you manually pick a list of mirrors to set up, but if one of

the picks at the top of your list happens to be slow, you always end up

hitting that, and it's dumb to make every user out there need to configure

and reconfigure their list for speed when it can be done for everyone at

one spot.
FTP hosts could be used too, as only the original request has to be HTTP;

the redirect just has to be in the form of a URL, and

fetch/wget/firefox/whatever will all understand it.
Of course, now I've come up with the idea, so I suppose I have to put up

or shut up.

Okay, how about some rough design:
URL format:

http://host/DragonFly/1.10/packages/2007Q2  or

http://host/DragonFly/snapshots

------------^-----------------^
take the local part, have configs like this:
cgi-data/DragonFly/snapshots.hosts
of course, prevent any char ~! /[[:alnum:]_\/-]/ in the url to sanitize

the parameter.
The hosts file would then just contain

# hosts for pkgsrc 2007Q2

# hostname						prio

http://chlamydia....../2007Q2/DragonFly-1.10/All/	20
of course the prio field etc could be ignored now/added later.
The CGI just reads in the specified file and outputs a random host from

the list.  Later there could be a hostname lookaside data file which

contains data like priority, response time, etc.  all the stuff we don't

need anyways.
I guess that's under 50 lines of ruby code.
Mind you that this is to support < 10 hits per hour.  Maybe 200 per hour

when we are flashed on a release.
cheers

  simon

To follow up on the mirror idea...
Here's a quick draft for a CGI that would accept HTTP (haven't tried FTP

URLs yet) requests for pkgsrc packages and direct them to a random mirror,

pasted below.  In testing, pkg_add works with it, either as a directly

passed URL or if you set PKG_PATH.  BINPKG_SITES does not work for 'bmake

bin-install' targets, because it unhelpfully assumes it needs to add /All

onto the target URL.
#!/usr/pkg/bin/perl
@mirrors = (

"http://www.pkgsrc-box.org/packages/stable/DragonFly-1.10/All",

"http://chlamydia.fs.ei.tum.de/pub/DragonFly/packages/stable/DragonFly-1.10/All",
"http://www.theshell.com/pub/DragonFly/packages/stable/DragonFly-1.10/All",
"http://ftp.twaren.net/BSD/DragonFlyBSD/packages/stable/DragonFly-1.10/All",

);
my $target = $ENV{'REQUEST_URI'};

$target =~ s/^\/download//;
print "Location: " . $mirrors[int(rand(scalar(@mirrors)))] . "$target\n\n";