[PATCH 06/16] vcs-svn: Improve support for reading large files

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
From: Jonathan Nieder
Date: Sunday, October 10, 2010 - 7:46 pm

Tweak the line_buffer API to permit seeking and cat-ing segments
longer than 4 GiB.  This would be particularly useful for applying
deltas that remove a large segment from the middle of a file.

Callers would still have to be updated to take advantage of this.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
Since off_t is a signed type, on systems with 32-bit file offsets,
this might make things worse.  Is that worth worrying about?

 vcs-svn/line_buffer.c |    8 ++++----
 vcs-svn/line_buffer.h |    4 ++--
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/vcs-svn/line_buffer.c b/vcs-svn/line_buffer.c
index 999368b..fd1d3c3 100644
--- a/vcs-svn/line_buffer.c
+++ b/vcs-svn/line_buffer.c
@@ -56,7 +56,7 @@ char *buffer_read_string(struct line_buffer *buf, uint32_t len)
 	return ferror(buf->infile) ? NULL : buf->blob_buffer.buf;
 }
 
-void buffer_copy_bytes(struct line_buffer *buf, uint32_t len)
+void buffer_copy_bytes(struct line_buffer *buf, off_t len)
 {
 	char byte_buffer[COPY_BUFFER_LEN];
 	uint32_t in;
@@ -72,12 +72,12 @@ void buffer_copy_bytes(struct line_buffer *buf, uint32_t len)
 	}
 }
 
-uint32_t buffer_skip_bytes(struct line_buffer *buf, uint32_t nbytes)
+off_t buffer_skip_bytes(struct line_buffer *buf, off_t nbytes)
 {
-	uint32_t done = 0;
+	off_t done = 0;
 	while (done < nbytes && !feof(buf->infile) && !ferror(buf->infile)) {
 		char byte_buffer[COPY_BUFFER_LEN];
-		uint32_t len = nbytes - done;
+		off_t len = nbytes - done;
 		uint32_t in = len < COPY_BUFFER_LEN ? len : COPY_BUFFER_LEN;
 		done += fread(byte_buffer, 1, in, buf->infile);
 	}
diff --git a/vcs-svn/line_buffer.h b/vcs-svn/line_buffer.h
index 2796ba7..2849faa 100644
--- a/vcs-svn/line_buffer.h
+++ b/vcs-svn/line_buffer.h
@@ -16,8 +16,8 @@ int buffer_init(struct line_buffer *buf, const char *filename);
 int buffer_deinit(struct line_buffer *buf);
 char *buffer_read_line(struct line_buffer *buf);
 char *buffer_read_string(struct line_buffer *buf, uint32_t len);
-void buffer_copy_bytes(struct line_buffer *buf, uint32_t len);
-uint32_t buffer_skip_bytes(struct line_buffer *buf, uint32_t len);
+void buffer_copy_bytes(struct line_buffer *buf, off_t len);
+off_t buffer_skip_bytes(struct line_buffer *buf, off_t len);
 void buffer_reset(struct line_buffer *buf);
 
 #endif
-- 
1.7.2.3

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

Messages in current thread:
[PATCH 0/8] Resurrect rr/svn-export, Ramkumar Ramachandra, (Thu Jul 15, 9:22 am)
[PATCH 1/8] Export parse_date_basic() to convert a date st ..., Ramkumar Ramachandra, (Thu Jul 15, 9:22 am)
[PATCH 2/8] Introduce vcs-svn lib, Ramkumar Ramachandra, (Thu Jul 15, 9:22 am)
[PATCH 3/8] Add memory pool library, Ramkumar Ramachandra, (Thu Jul 15, 9:22 am)
[PATCH 4/8] Add treap implementation, Ramkumar Ramachandra, (Thu Jul 15, 9:23 am)
[PATCH 5/8] Add string-specific memory pool, Ramkumar Ramachandra, (Thu Jul 15, 9:23 am)
[PATCH 6/8] Add stream helper library, Ramkumar Ramachandra, (Thu Jul 15, 9:23 am)
[PATCH 7/8] Add infrastructure to write revisions in fast- ..., Ramkumar Ramachandra, (Thu Jul 15, 9:23 am)
[PATCH 8/8] Add SVN dump parser, Ramkumar Ramachandra, (Thu Jul 15, 9:23 am)
Re: [PATCH 2/8] Introduce vcs-svn lib, Jonathan Nieder, (Thu Jul 15, 10:46 am)
Re: [PATCH 3/8] Add memory pool library, Jonathan Nieder, (Thu Jul 15, 11:57 am)
Re: [PATCH 4/8] Add treap implementation, Jonathan Nieder, (Thu Jul 15, 12:09 pm)
Re: [PATCH 3/8] Add memory pool library, Ramkumar Ramachandra, (Thu Jul 15, 12:12 pm)
Re: [PATCH 2/8] Introduce vcs-svn lib, Ramkumar Ramachandra, (Thu Jul 15, 12:15 pm)
Re: [PATCH 4/8] Add treap implementation, Ramkumar Ramachandra, (Thu Jul 15, 12:18 pm)
Re: [PATCH 6/8] Add stream helper library, Jonathan Nieder, (Thu Jul 15, 12:19 pm)
Re: [PATCH 8/8] Add SVN dump parser, Jonathan Nieder, (Thu Jul 15, 12:52 pm)
Re: [PATCH 8/8] Add SVN dump parser, Jonathan Nieder, (Thu Jul 15, 1:04 pm)
Re: [PATCH 0/8] Resurrect rr/svn-export, Jonathan Nieder, (Fri Jul 16, 3:13 am)
[PATCH 3/9] Add memory pool library, Jonathan Nieder, (Fri Jul 16, 3:16 am)
[PATCH 4/9] Add treap implementation, Jonathan Nieder, (Fri Jul 16, 3:23 am)
Re: [PATCH 4/9] Add treap implementation, Jonathan Nieder, (Fri Jul 16, 11:26 am)
[PATCH 0/10] rr/svn-export reroll, Jonathan Nieder, (Mon Aug 9, 2:57 pm)
[PATCH 02/10] Introduce vcs-svn lib, Jonathan Nieder, (Mon Aug 9, 3:04 pm)
[PATCH 03/10] Add memory pool library, Jonathan Nieder, (Mon Aug 9, 3:11 pm)
[PATCH 04/10] Add treap implementation, Jonathan Nieder, (Mon Aug 9, 3:17 pm)
[PATCH 05/10] Add string-specific memory pool, Jonathan Nieder, (Mon Aug 9, 3:34 pm)
[PATCH 06/10] Add stream helper library, Jonathan Nieder, (Mon Aug 9, 3:39 pm)
[PATCH 08/10] SVN dump parser, Jonathan Nieder, (Mon Aug 9, 3:55 pm)
PATCH 09/10] Update svn-fe manual, Jonathan Nieder, (Mon Aug 9, 3:55 pm)
Re: [PATCH 0/10] rr/svn-export reroll, Ramkumar Ramachandra, (Tue Aug 10, 5:53 am)
Re: [PATCH 0/10] rr/svn-export reroll, Jonathan Nieder, (Tue Aug 10, 6:53 pm)
Re: [PATCH 05/10] Add string-specific memory pool, Junio C Hamano, (Thu Aug 12, 10:22 am)
Re: [PATCH 04/10] Add treap implementation, Junio C Hamano, (Thu Aug 12, 10:22 am)
Re: [PATCH 08/10] SVN dump parser, Junio C Hamano, (Thu Aug 12, 10:22 am)
Re: [PATCH 05/10] Add string-specific memory pool, Jonathan Nieder, (Thu Aug 12, 2:30 pm)
Re: [PATCH 04/10] Add treap implementation, Jonathan Nieder, (Thu Aug 12, 3:02 pm)
Re: [PATCH 04/10] Add treap implementation, Jonathan Nieder, (Thu Aug 12, 3:11 pm)
Re: [PATCH 04/10] Add treap implementation, Junio C Hamano, (Thu Aug 12, 3:44 pm)
[PATCH/WIP 00/16] svn delta applier, Jonathan Nieder, (Sun Oct 10, 7:34 pm)
[PATCH 01/16] vcs-svn: Eliminate global byte_buffer[] array, Jonathan Nieder, (Sun Oct 10, 7:37 pm)
[PATCH 03/16] vcs-svn: Collect line_buffer data in a struct, Jonathan Nieder, (Sun Oct 10, 7:39 pm)
[PATCH 06/16] vcs-svn: Improve support for reading large files, Jonathan Nieder, (Sun Oct 10, 7:46 pm)
[PATCH 07/16] vcs-svn: Add binary-safe read() function, Jonathan Nieder, (Sun Oct 10, 7:47 pm)
[PATCH 10/16] vcs-svn: Allow character-oriented input, Jonathan Nieder, (Sun Oct 10, 7:52 pm)
[PATCH 13/16] vcs-svn: Learn to check for SVN\0 magic, Jonathan Nieder, (Sun Oct 10, 7:58 pm)
[PATCH 14/16] compat: helper for detecting unsigned overflow, Jonathan Nieder, (Sun Oct 10, 7:59 pm)
[PATCH/RFC 16'/16] vcs-svn: Add svn delta parser, Jonathan Nieder, (Sun Oct 10, 9:01 pm)
[PATCH/RFC 0/11] Building up the delta parser, Jonathan Nieder, (Wed Oct 13, 2:17 am)
[PATCH 02/11] vcs-svn: Skeleton of an svn delta parser, Jonathan Nieder, (Wed Oct 13, 2:21 am)
[PATCH 04/11] vcs-svn: Read inline data from deltas, Jonathan Nieder, (Wed Oct 13, 2:35 am)
[PATCH 05/11] vcs-svn: Read instructions from deltas, Jonathan Nieder, (Wed Oct 13, 2:38 am)
[PATCH 07/11] vcs-svn: Check declared number of output bytes, Jonathan Nieder, (Wed Oct 13, 2:41 am)
[PATCH 09/11] vcs-svn: Let deltas use data from postimage, Jonathan Nieder, (Wed Oct 13, 2:50 am)
[PATCH 11/11] vcs-svn: Allow deltas to copy from preimage, Jonathan Nieder, (Wed Oct 13, 2:58 am)
[PATCH 11/11] vcs-svn: Allow deltas to copy from preimage, Jonathan Nieder, (Wed Oct 13, 3:00 am)
Re: [PATCH/RFC 0/11] Building up the delta parser, Ramkumar Ramachandra, (Mon Oct 18, 10:00 am)
Re: [PATCH/RFC 0/11] Building up the delta parser, Jonathan Nieder, (Mon Oct 18, 10:03 am)