On 10/1/07, Jon Smirl <jonsmirl@gmail.com> wrote:
quoted text
> This is what full text is used for with code:
> http://lxr.mozilla.org/
>
> It makes grep instant.

I'd thought that keeping a full-text index of all my program files was
my dirty little secret that shows I'm not a "pro" programmer ;-)

quoted text
> For source code you can take the full text concept further and store
> parse trees.
[details snipped]

This sounds interesting in principle but is beyond what I'm thinking
in practice (particularly since I'm not in the "C is the only language
worth ever using" camp).

quoted text
> Full text indexing can also achieve high levels of compression as
> stated in the earlier threads. It is full scale dictionary
> compression. When it is being used for compression you want to apply
> it to all revisions.

Well, as I say I'm not convinced it makes sense to integrate this with
existing pack stuff precisely because I don't think it's universally
useful. So you seem to end up with all the usual tricks, eg, Golomb
coding inverted indexes, etc, _if_ you treat each blob as completely
independent. I was wondering if there was anything else you can do
given the special structure that might be both more useful and more
compact?

quoted text
> You would full text index the expanded source text for each revision,
> not the delta. There are forms of full text indexes that record the
> words position in the document. They let you search for "vision NEAR
> feedback"

Well, the kind of question I was thinking was "clearly you can use the
existing sort of full text indexing (eg, the stuff covered in Cleary,
Witten & Bell's covered Managing Gigabytes), but is that the most
useful way of doing things in the context of an evolving database?" If
you treat every blob as essentially a different document there are
indexing tools out there already you can use. What I was wondering was
if it's really that useful to a human user to report every revision of
a document containing those keywords even if the differences are in
other parts of the text far removed from the text containing the
keywords. I don't know the answer.

quoted text
> > (One question is "why do you want to build a table rather than
> > actively search the full git repo for each query (ie, combination of
> > words) as you make it?" My primary motivation is that I might in the
> > future like to do queries on some sort of low processor power
> > UMPC-type thing, having built the file containing a "full text index"
> > data structure for the index on a quite beefy desktop. The other point
> > is that searching natural language text based on a fallible memory
> > you're more likely to try different combinations of search terms
> > iteratively to try and hit the right one, so there might be some point
> > in trying to build an index.)
>
> I do admit that these indexes are used to make functions that can be
> done with brute force faster. As computers get faster the need for
> these decrease. Right now the size of the kernel repo is not growing
> faster than the progress of hardware. If you went back are tried to do
> these things on a 386 you'd be shouting for indexes tomorrow.

The other point is that direct searching is easier because you know
exactly what the query is at the point you have access to the full
text, whereas building an index you want to extract no more and no
less information to be able to answer all allowed queries. But I still
like the idea of getting a UMPC type thing if they become affordable.

-- 
cheers, dave tweed__________________________
david.tweed@gmail.com
Rm 124, School of Systems Engineering, University of Reading.
"we had no idea that when we added templates we were adding a Turing-
complete compile-time language." -- C++ standardisation committee
-
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html