View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0002225 | GNUnet | file-sharing service | public | 2012-03-19 00:45 | 2012-06-02 19:15 |
Reporter | LRN | Assigned To | LRN | ||
Priority | normal | Severity | minor | Reproducibility | always |
Status | closed | Resolution | fixed | ||
Product Version | Git master | ||||
Target Version | 0.9.3 | Fixed in Version | 0.9.3 | ||
Summary | 0002225: W32: GNUnet fails to index a file that is larger than 2 gigabytes | ||||
Description | When selected and published in GNUnet-FS-GTK, such files are highlighted in "red" color in publishing tab, "Size" column shows 16777215 TiB instead of the real file size, and indexing is not progressing. This issue is most likely W32-only, and most likely related to the lack of LFS in W32 CRT. | ||||
Tags | No tags attached. | ||||
Attached Files | 0001-Have-stat64-on-W32-unconditionally.patch (974 bytes)
From 08921e613c5d55cf1aaf10f9028e1627b2c3ddb1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=A0=D1=83=D1=81=D0=BB=D0=B0=D0=BD=20=D0=98=D0=B6=D0=B1=D1?= =?UTF-8?q?=83=D0=BB=D0=B0=D1=82=D0=BE=D0=B2?= <lrn1986@gmail.com> Date: Mon, 19 Mar 2012 04:37:00 +0400 Subject: [PATCH] Have stat64 on W32 unconditionally In case it's unavailable (only on W9x, which we don't support anyway), plibc will provide a wrapper. --- configure.ac | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/configure.ac b/configure.ac index a558ffd..825d8c7 100644 --- a/configure.ac +++ b/configure.ac @@ -152,7 +152,7 @@ netbsd*) LDFLAGS="$LDFLAGS -Wl,-no-undefined -Wl,--export-all-symbols" LIBS="$LIBS -lws2_32 -lplibc -lgnurx -lole32" CFLAGS="-mms-bitfields $CFLAGS" - CPPFLAGS="-D_WIN32_WINNT=0x0501 $CPPFLAGS" + CPPFLAGS="-D_WIN32_WINNT=0x0501 -DHAVE_STAT64=1 $CPPFLAGS" build_target="mingw" AC_PROG_CXX LIBPREFIX=lib -- 1.7.4 0002-Calculate-file-size-for-single-files-when-needed.patch (13,279 bytes)
From 8fb4b6843e2d8b181962cc92681bf54005f5f470 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=A0=D1=83=D1=81=D0=BB=D0=B0=D0=BD=20=D0=98=D0=B6=D0=B1=D1?= =?UTF-8?q?=83=D0=BB=D0=B0=D1=82=D0=BE=D0=B2?= <lrn1986@gmail.com> Date: Mon, 19 Mar 2012 10:30:43 +0400 Subject: [PATCH 2/4] Calculate file size for single files when needed --- src/fs/fs_download.c | 4 ++-- src/fs/fs_unindex.c | 2 +- src/fs/gnunet-directory.c | 2 +- src/fs/test_fs_download.c | 2 +- src/fs/test_fs_download_indexed.c | 2 +- src/fs/test_fs_download_persistence.c | 2 +- src/hello/gnunet-hello.c | 2 +- src/include/gnunet_disk_lib.h | 10 ++++++---- src/testing/gnunet-testing.c | 2 +- src/testing/testing_group.c | 2 +- src/transport/transport-testing.c | 2 +- src/util/crypto_hash.c | 2 +- src/util/crypto_rsa.c | 2 +- src/util/disk.c | 17 +++++++++++++++-- src/util/pseudonym.c | 4 ++-- src/util/test_disk.c | 2 +- 16 files changed, 37 insertions(+), 22 deletions(-) diff --git a/src/fs/fs_download.c b/src/fs/fs_download.c index 0596d85..490beef 100644 --- a/src/fs/fs_download.c +++ b/src/fs/fs_download.c @@ -2068,7 +2068,7 @@ GNUNET_FS_download_start (struct GNUNET_FS_Handle *h, { dc->filename = GNUNET_strdup (filename); if (GNUNET_YES == GNUNET_DISK_file_test (filename)) - GNUNET_break (GNUNET_OK == GNUNET_DISK_file_size (filename, &dc->old_file_size, GNUNET_YES)); + GNUNET_break (GNUNET_OK == GNUNET_DISK_file_size (filename, &dc->old_file_size, GNUNET_YES, GNUNET_YES)); } if (GNUNET_FS_uri_test_loc (dc->uri)) GNUNET_assert (GNUNET_OK == @@ -2178,7 +2178,7 @@ GNUNET_FS_download_start_from_search (struct GNUNET_FS_Handle *h, { dc->filename = GNUNET_strdup (filename); if (GNUNET_YES == GNUNET_DISK_file_test (filename)) - GNUNET_break (GNUNET_OK == GNUNET_DISK_file_size (filename, &dc->old_file_size, GNUNET_YES)); + GNUNET_break (GNUNET_OK == GNUNET_DISK_file_size (filename, &dc->old_file_size, GNUNET_YES, GNUNET_YES)); } if (GNUNET_FS_uri_test_loc (dc->uri)) GNUNET_assert (GNUNET_OK == diff --git a/src/fs/fs_unindex.c b/src/fs/fs_unindex.c index 3c2ef8f..bf619b7 100644 --- a/src/fs/fs_unindex.c +++ b/src/fs/fs_unindex.c @@ -755,7 +755,7 @@ GNUNET_FS_unindex_start (struct GNUNET_FS_Handle *h, const char *filename, struct GNUNET_FS_ProgressInfo pi; uint64_t size; - if (GNUNET_OK != GNUNET_DISK_file_size (filename, &size, GNUNET_YES)) + if (GNUNET_OK != GNUNET_DISK_file_size (filename, &size, GNUNET_YES, GNUNET_YES)) return NULL; ret = GNUNET_malloc (sizeof (struct GNUNET_FS_UnindexContext)); ret->h = h; diff --git a/src/fs/gnunet-directory.c b/src/fs/gnunet-directory.c index 0721ea9..c722f57 100644 --- a/src/fs/gnunet-directory.c +++ b/src/fs/gnunet-directory.c @@ -136,7 +136,7 @@ run (void *cls, char *const *args, const char *cfgfile, i = 0; while (NULL != (filename = args[i++])) { - if ((GNUNET_OK != GNUNET_DISK_file_size (filename, &size, GNUNET_YES)) || + if ((GNUNET_OK != GNUNET_DISK_file_size (filename, &size, GNUNET_YES, GNUNET_YES)) || (NULL == (h = GNUNET_DISK_file_open (filename, GNUNET_DISK_OPEN_READ, diff --git a/src/fs/test_fs_download.c b/src/fs/test_fs_download.c index 570eab9..198a815 100644 --- a/src/fs/test_fs_download.c +++ b/src/fs/test_fs_download.c @@ -118,7 +118,7 @@ abort_download_task (void *cls, const struct GNUNET_SCHEDULER_TaskContext *tc) GNUNET_FS_download_stop (download, GNUNET_YES); download = NULL; } - GNUNET_assert (GNUNET_OK == GNUNET_DISK_file_size (fn, &size, GNUNET_YES)); + GNUNET_assert (GNUNET_OK == GNUNET_DISK_file_size (fn, &size, GNUNET_YES, GNUNET_NO)); GNUNET_assert (size == FILESIZE); GNUNET_DISK_directory_remove (fn); GNUNET_free (fn); diff --git a/src/fs/test_fs_download_indexed.c b/src/fs/test_fs_download_indexed.c index e8504f1..c00b9e7 100644 --- a/src/fs/test_fs_download_indexed.c +++ b/src/fs/test_fs_download_indexed.c @@ -119,7 +119,7 @@ abort_download_task (void *cls, const struct GNUNET_SCHEDULER_TaskContext *tc) GNUNET_FS_download_stop (download, GNUNET_YES); download = NULL; } - GNUNET_assert (GNUNET_OK == GNUNET_DISK_file_size (fn, &size, GNUNET_YES)); + GNUNET_assert (GNUNET_OK == GNUNET_DISK_file_size (fn, &size, GNUNET_YES, GNUNET_NO)); GNUNET_assert (size == FILESIZE); GNUNET_DISK_directory_remove (fn); GNUNET_free (fn); diff --git a/src/fs/test_fs_download_persistence.c b/src/fs/test_fs_download_persistence.c index bcb1c54..3a6d955 100644 --- a/src/fs/test_fs_download_persistence.c +++ b/src/fs/test_fs_download_persistence.c @@ -113,7 +113,7 @@ abort_download_task (void *cls, const struct GNUNET_SCHEDULER_TaskContext *tc) GNUNET_FS_download_stop (download, GNUNET_YES); download = NULL; } - GNUNET_assert (GNUNET_OK == GNUNET_DISK_file_size (fn, &size, GNUNET_YES)); + GNUNET_assert (GNUNET_OK == GNUNET_DISK_file_size (fn, &size, GNUNET_YES, GNUNET_NO)); GNUNET_assert (size == FILESIZE); GNUNET_DISK_directory_remove (fn); GNUNET_free (fn); diff --git a/src/hello/gnunet-hello.c b/src/hello/gnunet-hello.c index be7719e..9cf32f9 100644 --- a/src/hello/gnunet-hello.c +++ b/src/hello/gnunet-hello.c @@ -122,7 +122,7 @@ main (int argc, char *argv[]) _("Call with name of HELLO file to modify.\n")); return 1; } - if (GNUNET_OK != GNUNET_DISK_file_size (argv[1], &fsize, GNUNET_YES)) + if (GNUNET_OK != GNUNET_DISK_file_size (argv[1], &fsize, GNUNET_YES, GNUNET_YES)) { FPRINTF (stderr, _("Error accessing file `%s': %s\n"), diff --git a/src/include/gnunet_disk_lib.h b/src/include/gnunet_disk_lib.h index 18f5535..d6ec0fe 100644 --- a/src/include/gnunet_disk_lib.h +++ b/src/include/gnunet_disk_lib.h @@ -302,8 +302,8 @@ GNUNET_DISK_file_seek (const struct GNUNET_DISK_FileHandle *h, OFF_T offset, /** - * Get the size of the file (or directory) - * of the given file (in bytes). + * Get the size of the file (or directory) of the given file (in + * bytes). * * @param filename name of the file or directory * @param size set to the size of the file (or, @@ -311,11 +311,13 @@ GNUNET_DISK_file_seek (const struct GNUNET_DISK_FileHandle *h, OFF_T offset, * of all sizes of files in the directory) * @param includeSymLinks should symbolic links be * included? - * @return GNUNET_OK on success, GNUNET_SYSERR on error + * @param singleFileMode GNUNET_YES to only get size of one file + * and return GNUNET_SYSERR for directories. + * @return GNUNET_SYSERR on error, GNUNET_OK on success */ int GNUNET_DISK_file_size (const char *filename, uint64_t * size, - int includeSymLinks); + int includeSymLinks, int singleFileMode); /** diff --git a/src/testing/gnunet-testing.c b/src/testing/gnunet-testing.c index 0caa28e..3f6eb8a 100644 --- a/src/testing/gnunet-testing.c +++ b/src/testing/gnunet-testing.c @@ -170,7 +170,7 @@ create_hostkeys (const unsigned int no) return 1; } - if (GNUNET_YES != GNUNET_DISK_file_size (hostkey_src_file, &fs, GNUNET_YES)) + if (GNUNET_YES != GNUNET_DISK_file_size (hostkey_src_file, &fs, GNUNET_YES, GNUNET_YES)) fs = 0; if (0 != (fs % HOSTKEYFILESIZE)) diff --git a/src/testing/testing_group.c b/src/testing/testing_group.c index e8db98a..983a5ab 100644 --- a/src/testing/testing_group.c +++ b/src/testing/testing_group.c @@ -6078,7 +6078,7 @@ GNUNET_TESTING_daemons_start (const struct GNUNET_CONFIGURATION_Handle *cfg, return NULL; } - if (GNUNET_YES != GNUNET_DISK_file_size (hostkeys_file, &fs, GNUNET_YES)) + if (GNUNET_YES != GNUNET_DISK_file_size (hostkeys_file, &fs, GNUNET_YES, GNUNET_YES)) fs = 0; #if DEBUG_TESTING GNUNET_log (GNUNET_ERROR_TYPE_DEBUG, diff --git a/src/transport/transport-testing.c b/src/transport/transport-testing.c index 752106b..6a27d99 100644 --- a/src/transport/transport-testing.c +++ b/src/transport/transport-testing.c @@ -661,7 +661,7 @@ GNUNET_TRANSPORT_TESTING_init () return NULL; } - if (GNUNET_YES != GNUNET_DISK_file_size (hostkeys_file, &fs, GNUNET_YES)) + if (GNUNET_YES != GNUNET_DISK_file_size (hostkeys_file, &fs, GNUNET_YES, GNUNET_YES)) fs = 0; if (0 != (fs % HOSTKEYFILESIZE)) diff --git a/src/util/crypto_hash.c b/src/util/crypto_hash.c index f896390..a7f9c4c 100644 --- a/src/util/crypto_hash.c +++ b/src/util/crypto_hash.c @@ -204,7 +204,7 @@ GNUNET_CRYPTO_hash_file (enum GNUNET_SCHEDULER_Priority priority, return NULL; } fhc->bsize = blocksize; - if (GNUNET_OK != GNUNET_DISK_file_size (filename, &fhc->fsize, GNUNET_NO)) + if (GNUNET_OK != GNUNET_DISK_file_size (filename, &fhc->fsize, GNUNET_NO, GNUNET_YES)) { GNUNET_free (fhc->filename); GNUNET_free (fhc); diff --git a/src/util/crypto_rsa.c b/src/util/crypto_rsa.c index ee7e5e9..54d61b1 100644 --- a/src/util/crypto_rsa.c +++ b/src/util/crypto_rsa.c @@ -726,7 +726,7 @@ GNUNET_CRYPTO_rsa_key_create_from_file (const char *filename) return NULL; } - if (GNUNET_YES != GNUNET_DISK_file_size (filename, &fs, GNUNET_YES)) + if (GNUNET_YES != GNUNET_DISK_file_size (filename, &fs, GNUNET_YES, GNUNET_YES)) fs = 0; if (fs < sizeof (struct GNUNET_CRYPTO_RsaPrivateKeyBinaryEncoded)) { diff --git a/src/util/disk.c b/src/util/disk.c index b6b458f..ae3ac4c 100644 --- a/src/util/disk.c +++ b/src/util/disk.c @@ -112,6 +112,11 @@ struct GetFileSizeData * GNUNET_YES if symbolic links should be included. */ int include_sym_links; + + /** + * GNUNET_YES if mode is file-only (return total == -1 for directories). + */ + int single_file_mode; }; @@ -176,6 +181,11 @@ getSizeRec (void *cls, const char *fn) return GNUNET_SYSERR; } #endif + if ((S_ISDIR (buf.st_mode)) && (gfsd->single_file_mode == GNUNET_YES)) + { + errno = EISDIR; + return GNUNET_SYSERR; + } if ((!S_ISLNK (buf.st_mode)) || (gfsd->include_sym_links == GNUNET_YES)) gfsd->total += buf.st_size; if ((S_ISDIR (buf.st_mode)) && (0 == ACCESS (fn, X_OK)) && @@ -290,11 +300,13 @@ GNUNET_DISK_file_seek (const struct GNUNET_DISK_FileHandle * h, OFF_T offset, * of all sizes of files in the directory) * @param includeSymLinks should symbolic links be * included? + * @param singleFileMode GNUNET_YES to only get size of one file + * and return GNUNET_SYSERR for directories. * @return GNUNET_SYSERR on error, GNUNET_OK on success */ int GNUNET_DISK_file_size (const char *filename, uint64_t * size, - int includeSymLinks) + int includeSymLinks, int singleFileMode) { struct GetFileSizeData gfsd; int ret; @@ -302,6 +314,7 @@ GNUNET_DISK_file_size (const char *filename, uint64_t * size, GNUNET_assert (size != NULL); gfsd.total = 0; gfsd.include_sym_links = includeSymLinks; + gfsd.single_file_mode = singleFileMode; ret = getSizeRec (&gfsd, filename); *size = gfsd.total; return ret; @@ -1350,7 +1363,7 @@ GNUNET_DISK_file_copy (const char *src, const char *dst) struct GNUNET_DISK_FileHandle *in; struct GNUNET_DISK_FileHandle *out; - if (GNUNET_OK != GNUNET_DISK_file_size (src, &size, GNUNET_YES)) + if (GNUNET_OK != GNUNET_DISK_file_size (src, &size, GNUNET_YES, GNUNET_YES)) return GNUNET_SYSERR; pos = 0; in = GNUNET_DISK_file_open (src, GNUNET_DISK_OPEN_READ, diff --git a/src/util/pseudonym.c b/src/util/pseudonym.c index c2b9583..d2efdc9 100644 --- a/src/util/pseudonym.c +++ b/src/util/pseudonym.c @@ -320,7 +320,7 @@ GNUNET_PSEUDONYM_name_uniquify (const struct GNUNET_CONFIGURATION_Handle *cfg, len = 0; if (0 == STAT (fn, &sbuf)) - GNUNET_break (GNUNET_OK == GNUNET_DISK_file_size (fn, &len, GNUNET_YES)); + GNUNET_break (GNUNET_OK == GNUNET_DISK_file_size (fn, &len, GNUNET_YES, GNUNET_YES)); fh = GNUNET_DISK_file_open (fn, GNUNET_DISK_OPEN_CREATE | GNUNET_DISK_OPEN_READWRITE, @@ -475,7 +475,7 @@ GNUNET_PSEUDONYM_name_to_id (const struct GNUNET_CONFIGURATION_Handle *cfg, GNUNET_assert (fn != NULL); if ((GNUNET_OK != GNUNET_DISK_file_test (fn) || - (GNUNET_OK != GNUNET_DISK_file_size (fn, &len, GNUNET_YES))) || + (GNUNET_OK != GNUNET_DISK_file_size (fn, &len, GNUNET_YES, GNUNET_YES))) || ((idx + 1) * sizeof (GNUNET_HashCode) > len)) { GNUNET_free (fn); diff --git a/src/util/test_disk.c b/src/util/test_disk.c index 5462772..149cec0 100644 --- a/src/util/test_disk.c +++ b/src/util/test_disk.c @@ -97,7 +97,7 @@ testOpenClose () GNUNET_break (5 == GNUNET_DISK_file_write (fh, "Hello", 5)); GNUNET_DISK_file_close (fh); GNUNET_break (GNUNET_OK == - GNUNET_DISK_file_size (".testfile", &size, GNUNET_NO)); + GNUNET_DISK_file_size (".testfile", &size, GNUNET_NO, GNUNET_YES)); if (size != 5) return 1; GNUNET_break (0 == UNLINK (".testfile")); -- 1.7.4 0003-Add-a-FIXME.patch (1,037 bytes)
From d8f06c58032bd2381147793d1b75cdd2ae47d624 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=A0=D1=83=D1=81=D0=BB=D0=B0=D0=BD=20=D0=98=D0=B6=D0=B1=D1?= =?UTF-8?q?=83=D0=BB=D0=B0=D1=82=D0=BE=D0=B2?= <lrn1986@gmail.com> Date: Mon, 19 Mar 2012 10:33:47 +0400 Subject: [PATCH 3/4] Add a FIXME --- src/fs/fs_file_information.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/src/fs/fs_file_information.c b/src/fs/fs_file_information.c index 85a076f..7a2dcdb 100644 --- a/src/fs/fs_file_information.c +++ b/src/fs/fs_file_information.c @@ -145,6 +145,9 @@ GNUNET_FS_file_information_create_from_file (struct GNUNET_FS_Handle *h, #endif while (NULL != (ss = strstr (fn, DIR_SEPARATOR_STR))) fn = ss + 1; +/* FIXME: If we assume that on other platforms CRT is UTF-8-aware, then + * this should be changed to EXTRACTOR_METAFORMAT_UTF8 + */ #if !WINDOWS GNUNET_CONTAINER_meta_data_insert (ret->meta, "<gnunet>", EXTRACTOR_METATYPE_GNUNET_ORIGINAL_FILENAME, -- 1.7.4 0004-Use-GD_file_size-instead-of-STAT.patch (16,257 bytes)
From 62349122fef467a3971ac52261a0850e08238ae2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=A0=D1=83=D1=81=D0=BB=D0=B0=D0=BD=20=D0=98=D0=B6=D0=B1=D1?= =?UTF-8?q?=83=D0=BB=D0=B0=D1=82=D0=BE=D0=B2?= <lrn1986@gmail.com> Date: Mon, 19 Mar 2012 11:02:46 +0400 Subject: [PATCH 4/4] Use GD_file_size() instead of STAT() --- src/fs/fs_file_information.c | 7 +++-- src/fs/gnunet-helper-fs-publish.c | 10 +++++-- src/statistics/gnunet-service-statistics.c | 11 ++++--- src/testing/gnunet-testing.c | 2 +- src/testing/testing_group.c | 2 +- src/topology/gnunet-daemon-topology.c | 21 +++++++------- src/transport/gnunet-service-transport_blacklist.c | 29 ++++++++++--------- src/transport/plugin_transport_http_server.c | 11 ++++--- src/transport/transport-testing.c | 2 +- src/util/crypto_rsa.c | 2 +- 10 files changed, 53 insertions(+), 44 deletions(-) diff --git a/src/fs/fs_file_information.c b/src/fs/fs_file_information.c index 7a2dcdb..8065927 100644 --- a/src/fs/fs_file_information.c +++ b/src/fs/fs_file_information.c @@ -107,7 +107,7 @@ GNUNET_FS_file_information_create_from_file (struct GNUNET_FS_Handle *h, *bo) { struct FileInfo *fi; - struct stat sbuf; + uint64_t fsize; struct GNUNET_FS_FileInformation *ret; const char *fn; const char *ss; @@ -116,7 +116,8 @@ GNUNET_FS_file_information_create_from_file (struct GNUNET_FS_Handle *h, char fn_conv[MAX_PATH]; #endif - if (0 != STAT (filename, &sbuf)) + /* FIXME: should includeSymLinks be GNUNET_NO or GNUNET_YES here? */ + if (GNUNET_OK != GNUNET_DISK_file_size (filename, &fsize, GNUNET_NO, GNUNET_YES)) { GNUNET_log_strerror_file (GNUNET_ERROR_TYPE_WARNING, "stat", filename); return NULL; @@ -129,7 +130,7 @@ GNUNET_FS_file_information_create_from_file (struct GNUNET_FS_Handle *h, } ret = GNUNET_FS_file_information_create_from_reader (h, client_info, - sbuf.st_size, + fsize, &GNUNET_FS_data_reader_file_, fi, keywords, meta, do_index, bo); diff --git a/src/fs/gnunet-helper-fs-publish.c b/src/fs/gnunet-helper-fs-publish.c index 4f70464..4bb77fa 100644 --- a/src/fs/gnunet-helper-fs-publish.c +++ b/src/fs/gnunet-helper-fs-publish.c @@ -69,7 +69,8 @@ struct ScanTreeNode char *filename; /** - * Size of the file (if it is a file), in bytes + * Size of the file (if it is a file), in bytes. + * At the moment it is set to 0 for directories. */ uint64_t file_size; @@ -275,8 +276,11 @@ preprocess_file (const char *filename, { struct ScanTreeNode *item; struct stat sbuf; + uint64_t fsize = 0; - if (0 != STAT (filename, &sbuf)) + if ((0 != STAT (filename, &sbuf)) || + ((!S_ISDIR (sbuf.st_mode)) && (GNUNET_OK != GNUNET_DISK_file_size ( + filename, &fsize, GNUNET_NO, GNUNET_YES)))) { /* If the file doesn't exist (or is not stat-able for any other reason) skip it (but report it), but do continue. */ @@ -297,7 +301,7 @@ preprocess_file (const char *filename, item = GNUNET_malloc (sizeof (struct ScanTreeNode)); item->filename = GNUNET_strdup (filename); item->is_directory = (S_ISDIR (sbuf.st_mode)) ? GNUNET_YES : GNUNET_NO; - item->file_size = (uint64_t) sbuf.st_size; + item->file_size = fsize; if (item->is_directory == GNUNET_YES) { struct RecursionContext rc; diff --git a/src/statistics/gnunet-service-statistics.c b/src/statistics/gnunet-service-statistics.c index 58ef763..1ec07ad 100644 --- a/src/statistics/gnunet-service-statistics.c +++ b/src/statistics/gnunet-service-statistics.c @@ -224,6 +224,7 @@ load (struct GNUNET_SERVER_Handle *server) char *fn; struct GNUNET_BIO_ReadHandle *rh; struct stat sb; + uint64_t fsize; char *buf; struct GNUNET_SERVER_MessageStreamTokenizer *mst; char *emsg; @@ -232,12 +233,12 @@ load (struct GNUNET_SERVER_Handle *server) NULL); if (fn == NULL) return; - if ((0 != stat (fn, &sb)) || (sb.st_size == 0)) + if ((GNUNET_OK != GNUNET_DISK_file_size (fn, &fsize, GNUNET_NO, GNUNET_YES)) || (fsize == 0)) { GNUNET_free (fn); return; } - buf = GNUNET_malloc (sb.st_size); + buf = GNUNET_malloc (fsize); rh = GNUNET_BIO_read_open (fn); if (!rh) { @@ -245,7 +246,7 @@ load (struct GNUNET_SERVER_Handle *server) GNUNET_free (fn); return; } - if (GNUNET_OK != GNUNET_BIO_read (rh, fn, buf, sb.st_size)) + if (GNUNET_OK != GNUNET_BIO_read (rh, fn, buf, fsize)) { GNUNET_log_strerror_file (GNUNET_ERROR_TYPE_WARNING, "read", fn); GNUNET_break (GNUNET_OK == GNUNET_BIO_read_close (rh, &emsg)); @@ -256,10 +257,10 @@ load (struct GNUNET_SERVER_Handle *server) } GNUNET_log (GNUNET_ERROR_TYPE_INFO, _("Loading %llu bytes of statistics from `%s'\n"), - (unsigned long long) sb.st_size, fn); + fsize, fn); mst = GNUNET_SERVER_mst_create (&inject_message, server); GNUNET_break (GNUNET_OK == - GNUNET_SERVER_mst_receive (mst, NULL, buf, sb.st_size, + GNUNET_SERVER_mst_receive (mst, NULL, buf, fsize, GNUNET_YES, GNUNET_NO)); GNUNET_SERVER_mst_destroy (mst); GNUNET_free (buf); diff --git a/src/testing/gnunet-testing.c b/src/testing/gnunet-testing.c index 3f6eb8a..bdbb5e8 100644 --- a/src/testing/gnunet-testing.c +++ b/src/testing/gnunet-testing.c @@ -170,7 +170,7 @@ create_hostkeys (const unsigned int no) return 1; } - if (GNUNET_YES != GNUNET_DISK_file_size (hostkey_src_file, &fs, GNUNET_YES, GNUNET_YES)) + if (GNUNET_OK != GNUNET_DISK_file_size (hostkey_src_file, &fs, GNUNET_YES, GNUNET_YES)) fs = 0; if (0 != (fs % HOSTKEYFILESIZE)) diff --git a/src/testing/testing_group.c b/src/testing/testing_group.c index 983a5ab..49c0c1a 100644 --- a/src/testing/testing_group.c +++ b/src/testing/testing_group.c @@ -6078,7 +6078,7 @@ GNUNET_TESTING_daemons_start (const struct GNUNET_CONFIGURATION_Handle *cfg, return NULL; } - if (GNUNET_YES != GNUNET_DISK_file_size (hostkeys_file, &fs, GNUNET_YES, GNUNET_YES)) + if (GNUNET_OK != GNUNET_DISK_file_size (hostkeys_file, &fs, GNUNET_YES, GNUNET_YES)) fs = 0; #if DEBUG_TESTING GNUNET_log (GNUNET_ERROR_TYPE_DEBUG, diff --git a/src/topology/gnunet-daemon-topology.c b/src/topology/gnunet-daemon-topology.c index 38a648a..4e33249 100644 --- a/src/topology/gnunet-daemon-topology.c +++ b/src/topology/gnunet-daemon-topology.c @@ -970,7 +970,7 @@ read_friends_file (const struct GNUNET_CONFIGURATION_Handle *cfg) char *data; size_t pos; struct GNUNET_PeerIdentity pid; - struct stat frstat; + uint64_t fsize; struct GNUNET_CRYPTO_HashAsciiEncoded enc; unsigned int entries_found; struct Peer *fl; @@ -987,7 +987,8 @@ read_friends_file (const struct GNUNET_CONFIGURATION_Handle *cfg) GNUNET_DISK_fn_write (fn, NULL, 0, GNUNET_DISK_PERM_USER_READ | GNUNET_DISK_PERM_USER_WRITE); - if (0 != STAT (fn, &frstat)) + if (GNUNET_OK != GNUNET_DISK_file_size (fn, + &fsize, GNUNET_NO, GNUNET_YES)) { if ((friends_only) || (minimum_friend_count > 0)) GNUNET_log (GNUNET_ERROR_TYPE_ERROR, @@ -995,14 +996,14 @@ read_friends_file (const struct GNUNET_CONFIGURATION_Handle *cfg) GNUNET_free (fn); return; } - if (frstat.st_size == 0) + if (fsize == 0) { GNUNET_log (GNUNET_ERROR_TYPE_WARNING, _("Friends file `%s' is empty.\n"), fn); GNUNET_free (fn); return; } - data = GNUNET_malloc_large (frstat.st_size); + data = GNUNET_malloc_large (fsize); if (data == NULL) { GNUNET_log (GNUNET_ERROR_TYPE_ERROR, @@ -1011,7 +1012,7 @@ read_friends_file (const struct GNUNET_CONFIGURATION_Handle *cfg) GNUNET_free (fn); return; } - if (frstat.st_size != GNUNET_DISK_fn_read (fn, data, frstat.st_size)) + if (fsize != GNUNET_DISK_fn_read (fn, data, fsize)) { GNUNET_log (GNUNET_ERROR_TYPE_ERROR, _("Failed to read friends list from `%s'\n"), fn); @@ -1021,11 +1022,11 @@ read_friends_file (const struct GNUNET_CONFIGURATION_Handle *cfg) } entries_found = 0; pos = 0; - while ((pos < frstat.st_size) && isspace ((unsigned char) data[pos])) + while ((pos < fsize) && isspace ((unsigned char) data[pos])) pos++; - while ((frstat.st_size >= sizeof (struct GNUNET_CRYPTO_HashAsciiEncoded)) && + while ((fsize >= sizeof (struct GNUNET_CRYPTO_HashAsciiEncoded)) && (pos <= - frstat.st_size - sizeof (struct GNUNET_CRYPTO_HashAsciiEncoded))) + fsize - sizeof (struct GNUNET_CRYPTO_HashAsciiEncoded))) { memcpy (&enc, &data[pos], sizeof (struct GNUNET_CRYPTO_HashAsciiEncoded)); if (!isspace @@ -1037,7 +1038,7 @@ read_friends_file (const struct GNUNET_CONFIGURATION_Handle *cfg) ("Syntax error in topology specification at offset %llu, skipping bytes.\n"), (unsigned long long) pos); pos++; - while ((pos < frstat.st_size) && (!isspace ((unsigned char) data[pos]))) + while ((pos < fsize) && (!isspace ((unsigned char) data[pos]))) pos++; continue; } @@ -1068,7 +1069,7 @@ read_friends_file (const struct GNUNET_CONFIGURATION_Handle *cfg) } } pos = pos + sizeof (struct GNUNET_CRYPTO_HashAsciiEncoded); - while ((pos < frstat.st_size) && isspace ((unsigned char) data[pos])) + while ((pos < fsize) && isspace ((unsigned char) data[pos])) pos++; } GNUNET_free (data); diff --git a/src/transport/gnunet-service-transport_blacklist.c b/src/transport/gnunet-service-transport_blacklist.c index 2089949..1036268 100644 --- a/src/transport/gnunet-service-transport_blacklist.c +++ b/src/transport/gnunet-service-transport_blacklist.c @@ -221,7 +221,7 @@ read_blacklist_file () size_t colon_pos; int tsize; struct GNUNET_PeerIdentity pid; - struct stat frstat; + uint64_t fsize; struct GNUNET_CRYPTO_HashAsciiEncoded enc; unsigned int entries_found; char *transport_name; @@ -241,14 +241,15 @@ read_blacklist_file () GNUNET_DISK_fn_write (fn, NULL, 0, GNUNET_DISK_PERM_USER_READ | GNUNET_DISK_PERM_USER_WRITE); - if (0 != STAT (fn, &frstat)) + if (GNUNET_OK != GNUNET_DISK_file_size (fn, + &fsize, GNUNET_NO, GNUNET_YES)) { GNUNET_log (GNUNET_ERROR_TYPE_ERROR, _("Could not read blacklist file `%s'\n"), fn); GNUNET_free (fn); return; } - if (frstat.st_size == 0) + if (fsize == 0) { #if DEBUG_TRANSPORT GNUNET_log (GNUNET_ERROR_TYPE_DEBUG, _("Blacklist file `%s' is empty.\n"), @@ -258,9 +259,9 @@ read_blacklist_file () return; } /* FIXME: use mmap */ - data = GNUNET_malloc_large (frstat.st_size); + data = GNUNET_malloc_large (fsize); GNUNET_assert (data != NULL); - if (frstat.st_size != GNUNET_DISK_fn_read (fn, data, frstat.st_size)) + if (fsize != GNUNET_DISK_fn_read (fn, data, fsize)) { GNUNET_log (GNUNET_ERROR_TYPE_ERROR, _("Failed to read blacklist from `%s'\n"), fn); @@ -270,17 +271,17 @@ read_blacklist_file () } entries_found = 0; pos = 0; - while ((pos < frstat.st_size) && isspace ((unsigned char) data[pos])) + while ((pos < fsize) && isspace ((unsigned char) data[pos])) pos++; - while ((frstat.st_size >= sizeof (struct GNUNET_CRYPTO_HashAsciiEncoded)) && + while ((fsize >= sizeof (struct GNUNET_CRYPTO_HashAsciiEncoded)) && (pos <= - frstat.st_size - sizeof (struct GNUNET_CRYPTO_HashAsciiEncoded))) + fsize - sizeof (struct GNUNET_CRYPTO_HashAsciiEncoded))) { colon_pos = pos; - while ((colon_pos < frstat.st_size) && (data[colon_pos] != ':') && + while ((colon_pos < fsize) && (data[colon_pos] != ':') && (!isspace ((unsigned char) data[colon_pos]))) colon_pos++; - if (colon_pos >= frstat.st_size) + if (colon_pos >= fsize) { GNUNET_log (GNUNET_ERROR_TYPE_WARNING, _ @@ -298,12 +299,12 @@ read_blacklist_file () ("Syntax error in blacklist file at offset %llu, skipping bytes.\n"), (unsigned long long) colon_pos); pos = colon_pos; - while ((pos < frstat.st_size) && isspace ((unsigned char) data[pos])) + while ((pos < fsize) && isspace ((unsigned char) data[pos])) pos++; continue; } tsize = colon_pos - pos; - if ((pos >= frstat.st_size) || (pos + tsize >= frstat.st_size) || + if ((pos >= fsize) || (pos + tsize >= fsize) || (tsize == 0)) { GNUNET_log (GNUNET_ERROR_TYPE_WARNING, @@ -336,7 +337,7 @@ read_blacklist_file () ("Syntax error in blacklist file at offset %llu, skipping bytes.\n"), (unsigned long long) pos); pos++; - while ((pos < frstat.st_size) && (!isspace ((unsigned char) data[pos]))) + while ((pos < fsize) && (!isspace ((unsigned char) data[pos]))) pos++; GNUNET_free_non_null (transport_name); continue; @@ -367,7 +368,7 @@ read_blacklist_file () } pos = pos + sizeof (struct GNUNET_CRYPTO_HashAsciiEncoded); GNUNET_free_non_null (transport_name); - while ((pos < frstat.st_size) && isspace ((unsigned char) data[pos])) + while ((pos < fsize) && isspace ((unsigned char) data[pos])) pos++; } GNUNET_STATISTICS_update (GST_stats, "# Transport entries blacklisted", diff --git a/src/transport/plugin_transport_http_server.c b/src/transport/plugin_transport_http_server.c index 8ec5a5e..68b6a2a 100644 --- a/src/transport/plugin_transport_http_server.c +++ b/src/transport/plugin_transport_http_server.c @@ -95,12 +95,13 @@ static char * server_load_file (const char *file) { struct GNUNET_DISK_FileHandle *gn_file; - struct stat fstat; + uint64_t fsize; char *text = NULL; - if (0 != STAT (file, &fstat)) + if (GNUNET_OK != GNUNET_DISK_file_size (file, + &fsize, GNUNET_NO, GNUNET_YES)) return NULL; - text = GNUNET_malloc (fstat.st_size + 1); + text = GNUNET_malloc (fsize + 1); gn_file = GNUNET_DISK_file_open (file, GNUNET_DISK_OPEN_READ, GNUNET_DISK_PERM_USER_READ); @@ -109,13 +110,13 @@ server_load_file (const char *file) GNUNET_free (text); return NULL; } - if (GNUNET_SYSERR == GNUNET_DISK_file_read (gn_file, text, fstat.st_size)) + if (GNUNET_SYSERR == GNUNET_DISK_file_read (gn_file, text, fsize)) { GNUNET_free (text); GNUNET_DISK_file_close (gn_file); return NULL; } - text[fstat.st_size] = '\0'; + text[fsize] = '\0'; GNUNET_DISK_file_close (gn_file); return text; } diff --git a/src/transport/transport-testing.c b/src/transport/transport-testing.c index 6a27d99..fd5cc47 100644 --- a/src/transport/transport-testing.c +++ b/src/transport/transport-testing.c @@ -661,7 +661,7 @@ GNUNET_TRANSPORT_TESTING_init () return NULL; } - if (GNUNET_YES != GNUNET_DISK_file_size (hostkeys_file, &fs, GNUNET_YES, GNUNET_YES)) + if (GNUNET_OK != GNUNET_DISK_file_size (hostkeys_file, &fs, GNUNET_YES, GNUNET_YES)) fs = 0; if (0 != (fs % HOSTKEYFILESIZE)) diff --git a/src/util/crypto_rsa.c b/src/util/crypto_rsa.c index 54d61b1..0106f43 100644 --- a/src/util/crypto_rsa.c +++ b/src/util/crypto_rsa.c @@ -726,7 +726,7 @@ GNUNET_CRYPTO_rsa_key_create_from_file (const char *filename) return NULL; } - if (GNUNET_YES != GNUNET_DISK_file_size (filename, &fs, GNUNET_YES, GNUNET_YES)) + if (GNUNET_OK != GNUNET_DISK_file_size (filename, &fs, GNUNET_YES, GNUNET_YES)) fs = 0; if (fs < sizeof (struct GNUNET_CRYPTO_RsaPrivateKeyBinaryEncoded)) { -- 1.7.4 0001-Use-GNUNET_DISK_file_size-to-get-size.patch (1,769 bytes)
From 941dcc37d7520853f595bb2f370c0484a5e6d5f1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=A0=D1=83=D1=81=D0=BB=D0=B0=D0=BD=20=D0=98=D0=B6=D0=B1=D1?= =?UTF-8?q?=83=D0=BB=D0=B0=D1=82=D0=BE=D0=B2?= <lrn1986@gmail.com> Date: Tue, 20 Mar 2012 01:00:44 +0400 Subject: [PATCH] Use GNUNET_DISK_file_size() to get size --- src/fs/gnunet-fs-gtk_publish-dialog.c | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-) diff --git a/src/fs/gnunet-fs-gtk_publish-dialog.c b/src/fs/gnunet-fs-gtk_publish-dialog.c index 0460e65..b5cb74f 100644 --- a/src/fs/gnunet-fs-gtk_publish-dialog.c +++ b/src/fs/gnunet-fs-gtk_publish-dialog.c @@ -1012,6 +1012,7 @@ add_item (struct AddDirClientContext *adcc, GtkTreeRowReference *row_reference; GtkTreePath *path; struct stat sbuf; + uint64_t fsize; if (0 != stat (item->filename, &sbuf)) @@ -1019,6 +1020,10 @@ add_item (struct AddDirClientContext *adcc, GNUNET_log_strerror_file (GNUNET_ERROR_TYPE_ERROR, "stat", item->filename); return; } + + if ((item->is_directory != GNUNET_YES) && (GNUNET_OK != + GNUNET_DISK_file_size (item->filename, &fsize, GNUNET_YES, GNUNET_YES))) + return; gtk_tree_store_insert_after (ts, item_iter, parent, sibling); path = gtk_tree_model_get_path (GTK_TREE_MODEL (ts), item_iter); row_reference = gtk_tree_row_reference_new (GTK_TREE_MODEL (ts), path); @@ -1050,7 +1055,7 @@ add_item (struct AddDirClientContext *adcc, item->meta, adcc->directory_scan_do_index, &adcc->directory_scan_bo); - file_size_fancy = GNUNET_STRINGS_byte_size_fancy (sbuf.st_size); + file_size_fancy = GNUNET_STRINGS_byte_size_fancy (fsize); } gtk_tree_store_set (ts, item_iter, 0, file_size_fancy, -- 1.7.4 | ||||
|
Does it only impact indexing (or also inserting)? Is your datastore's quota high enough? |
|
Only tested it on indexing (i always index). My quota should be sufficient. I also get this in the log file: util-6292 WARNING `read' failed on file `$LARGEFILENAME' at crypto_hash.c:158 with error: No error |
|
The problem seems to be related to stat(). stat() on W32 is a mess, and plibc is not helping :( |
|
OK, there was a bug in plibc UTF8ization patch, where i forgot to add an extern declaration. I'll submit an after-patch to plibc utf8ization bug tracker. With that taken care of it is possible to use STAT64 on W32 (a patch to define HAVE_STAT64 is required, since otherwise HAVE_STAT64 remains undefined on W32 - it's from plibc, not from crt; i'll attach the patch shortly). And with THAT taken care of it is possible to use GNUNET_DISK_file_size() to get size of files. Right now the code is very POSIXly, and uses STAT() in many places to get st_size. GNUNET_DISK_file_size() does correct #ifdef magic to use STAT64(), if it is available. Therefore GNUNET_DISK_file_size() should be preferred when file size is required. Caveat: GNUNET_DISK_file_size() calculates total size for a directory (summing sizes of all files in it). This makes it unsuitable as a generic STAT()-to-get-st_size replacement. I'm requesting a permission to adjust GNUNET_DISK_file_size(), adding extra argument that will prevent it from calculating directory size (whether it should return size == 0 or size == -1 when argument is a directory, is open for discussion). Grepping the code has shown me that GNUNET_DISK_file_size() is often used with an assumption that its first argument actually IS a file instead of a directory. So, for example, after that it might open that file for reading... |
|
The issue of 'accidentally' processing a directory recursively when stat-ing a file was desired has occurred to me before. The real question is to what extend the plibc 'STAT' macro can or should be used -- I know it is used in some places instead, and my expectation was that the 'STAT' macro goes to 'STAT64' if it is available (just like 'stat' does on POSIX). Overall, we seem to be on route towards replacing all of the all-caps PLIBC-calls with even more high-level boxed code in gnunetutil, which is likely not a bad thing (especially given that our plibc maintainer is dormant...). Overall, I don't mind you adding an extra argument to specify recursive/non-recursive; however, the use of 'STAT' vs. 'stat' vs. GNUNET_DISK_file_size() in the overall code should likely still be investigated in this context. I'd like to have a good answer when which style should be used and I'd like to see that answer be actually used consistently. |
|
Uploaded 0002-Calculate-file-size-for-single-files-when-needed.patch - adds an argument to be able to use GNUNET_DISK_file_size() on single files only. Returns GNUNET_SYSERR for directories, if single-file mode is enabled. 0003-Add-a-FIXME.patch - Something i've noticed while looking into this issue 0004-Use-GD_file_size-instead-of-STAT.patch - Replaces some uses of STAT() with GNUNET_DISK_file_size(). In most places the context strongly implies that file in question is NOT a directory, so the use of GNUNET_DISK_file_size() in single-file mode adequately replaces the use of STAT(). |
|
Patches 1-4 committed. Does this solve this issue? |
|
With 0001-Use-GNUNET_DISK_file_size-to-get-size.patch for gnunet-fs-gtk it does solve this issue. |
|
OK, 0001-Use-GNUNET_DISK_file_size-to-get-size.patch is half-baked, hold it back for a while... |
|
Re-uploaded the right version of 0001-Use-GNUNET_DISK_file_size-to-get-size.patch . Now it should do the trick. |
|
I really fail to see the point of always first calling 'stat' (which gives us the filesize -- but I guess not for 64-bit files) and then again calling 'GNUNET_DISK_file_size'. Is this really the way to do this!? :-(. |
Date Modified | Username | Field | Change |
---|---|---|---|
2012-03-19 00:45 | LRN | New Issue | |
2012-03-19 00:53 | Christian Grothoff | Note Added: 0005625 | |
2012-03-19 00:53 | Christian Grothoff | Note Edited: 0005625 | |
2012-03-19 00:56 | LRN | Note Added: 0005626 | |
2012-03-19 02:57 | LRN | Note Added: 0005627 | |
2012-03-19 03:36 | LRN | Note Added: 0005628 | |
2012-03-19 03:36 | LRN | File Added: 0001-Have-stat64-on-W32-unconditionally.patch | |
2012-03-19 09:43 | Christian Grothoff | Note Added: 0005629 | |
2012-03-19 09:43 | LRN | File Added: 0002-Calculate-file-size-for-single-files-when-needed.patch | |
2012-03-19 09:44 | LRN | File Added: 0003-Add-a-FIXME.patch | |
2012-03-19 09:44 | LRN | File Added: 0004-Use-GD_file_size-instead-of-STAT.patch | |
2012-03-19 09:47 | LRN | Note Added: 0005630 | |
2012-03-19 11:30 | Christian Grothoff | Priority | low => normal |
2012-03-19 11:30 | Christian Grothoff | Target Version | => 0.9.3 |
2012-03-19 11:30 | Christian Grothoff | Assigned To | => LRN |
2012-03-19 11:30 | Christian Grothoff | Status | new => assigned |
2012-03-19 11:49 | Christian Grothoff | Note Added: 0005634 | |
2012-03-19 23:54 | LRN | File Added: 0001-Use-GNUNET_DISK_file_size-to-get-size.patch | |
2012-03-19 23:54 | LRN | Note Added: 0005636 | |
2012-03-20 00:38 | LRN | Note Added: 0005637 | |
2012-03-20 02:12 | LRN | File Deleted: 0001-Use-GNUNET_DISK_file_size-to-get-size.patch | |
2012-03-20 02:13 | LRN | File Added: 0001-Use-GNUNET_DISK_file_size-to-get-size.patch | |
2012-03-20 02:13 | LRN | Note Added: 0005638 | |
2012-03-20 09:26 | Christian Grothoff | Note Added: 0005644 | |
2012-03-20 09:28 | Christian Grothoff | Status | assigned => resolved |
2012-03-20 09:28 | Christian Grothoff | Fixed in Version | => 0.9.3 |
2012-03-20 09:28 | Christian Grothoff | Resolution | open => fixed |
2012-06-02 19:15 | Christian Grothoff | Status | resolved => closed |