View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0002300 | GNUnet | file-sharing service | public | 2012-05-03 21:52 | 2012-06-02 19:15 |
Reporter | gabe | Assigned To | Christian Grothoff | ||
Priority | normal | Severity | major | Reproducibility | random |
Status | closed | Resolution | unable to reproduce | ||
Platform | GNU/Linux | OS | Debian | OS Version | 6.0.4 |
Product Version | Git master | ||||
Target Version | 0.9.3 | Fixed in Version | 0.9.3 | ||
Summary | 0002300: gnunet-publish ERROR Assertion failed at fs_dirmetascan.c:344 | ||||
Description | This is a regression of bug 0002232 - I#m seeing this issue again in the current SVN HEAD. With certain files on my system, when I try to index them with 'gnunet-publish' I get these errors: May 03 20:46:45-693437 gnunet-publish-4193 ERROR Assertion failed at fs_dirmetascan.c:344. Internal error scanning directory. May 03 20:46:45-693483 gnunet-publish-4193 ERROR Assertion failed at fs_dirmetascan.c:385. Internal error scanning directory. The error also occurrs when trying to publish a directory that contains the file either directly or in a subdir somehwere. | ||||
Steps To Reproduce | This is 100% reproducible for those files on my system, but I couldn't find a commonality between the files or a recipe to generate files that provoke this behaviour. | ||||
Additional Information | I did try a few things to see what the cause may be and here's what I found out: - Making a symlink to the file in question on the same or another filesystem allows me to publish the file by giving gnunet-publish the symlink. - Making a hardlinked copy of the file in another location on the same filesystem allows me to publish the file by giving gnunet-publish the hardlink - Making a copy of the file on the same or another filesystem allows me to publish the copied file. - Moving the file to another location on the same or another filesystem allows me to publish it. - Moving it back (after moving it or deleting the original after having made a copy) still leads to the same error when publishing. This all seems to suggest a problem with the filename (although there are plenty of files in the same dir that work, with file names that only differ by one or two characters), so I tried renaming: I found that adding an underscore at the end of one particular file made the second error message go away, but adding an underscore in front or replacing all blanks with underscores did nothing. In other cases, though, the extra underscore at the end made the second error message appear where it hadn't shown up before. I renamed one of the problematic files to 'aaaaa', which allowed me to insert it. Then I made an empty file of the same name as the original file that caused the problem, but gnunet-publish didn't complain. Finally I filled the file with random bytes to the exact size of the original file and ran gnunet-publish on it, which didn't report any problems. Obviously, when I deleted the random file and restored the original to its former name, the issue reappeared. I also wrote a little shell script to generate regular hierarchies of directories with randomly generated names and fill the deepest level directories with randomly named files consisting of random blocks of data. I ran it several times to make hundreds of directories various levels of nesting and thousands of files with sizes ranging from 1M to 1G, but gnunet-publish never complained about any of them. It seems to be a specific combination of a file's contents, name and even location that causes this issue to appear. IMO this is a bizarre corner-case, but a very annoying one that might keep users from publishing interesting content. | ||||
Tags | No tags attached. | ||||
Attached Files | 2300-diagnostics.diff (1,667 bytes)
Index: src/fs/gnunet-helper-fs-publish.c =================================================================== --- src/fs/gnunet-helper-fs-publish.c (revision 21274) +++ src/fs/gnunet-helper-fs-publish.c (working copy) @@ -186,6 +186,9 @@ { struct GNUNET_MessageHeader hdr; + fprintf (stderr, "Helper sends %u-byte message of type %u\n", + (unsigned int) (sizeof (struct GNUNET_MessageHeader) + data_length), + (unsigned int) message_type); hdr.type = htons (message_type); hdr.size = htons (sizeof (struct GNUNET_MessageHeader) + data_length); if ( (GNUNET_OK != @@ -338,7 +341,7 @@ ssize_t size; size_t slen; - if (item->is_directory == GNUNET_YES) + if (GNUNET_YES == item->is_directory) { /* for directories, we simply only descent, no extraction, no progress reporting */ Index: src/fs/fs_dirmetascan.c =================================================================== --- src/fs/fs_dirmetascan.c (revision 21274) +++ src/fs/fs_dirmetascan.c (working copy) @@ -257,6 +257,9 @@ const char *filename; size_t left; + fprintf (stderr, "DMS parses %u-byte message of type %u\n", + (unsigned int) ntohs (msg->size), + (unsigned int) ntohs (msg->type)); left = ntohs (msg->size) - sizeof (struct GNUNET_MessageHeader); filename = (const char*) &msg[1]; switch (ntohs (msg->type)) @@ -325,7 +328,7 @@ NULL, GNUNET_SYSERR, GNUNET_FS_DIRSCANNER_ALL_COUNTED); ds->pos = ds->toplevel; - if (ds->pos->is_directory == GNUNET_YES) + if (GNUNET_YES == ds->pos->is_directory) ds->pos = advance (ds->pos); return; case GNUNET_MESSAGE_TYPE_FS_PUBLISH_HELPER_META_DATA: | ||||
|
Are your filenames all ASCII-characters, or do they contain special non-English characters from your native language? The problem is within the code that transfers the filename WITH metadata to the inserting process, so if you put random garbage it won't happen simply because no metadata would be found in such files. So yes, this seems likely to be an error related to the filename. Is it possible for me to get SSH access to your system to try to investigate further? |
|
I've looked over the code and cannot find anything. Assuming SSH access is not possible, I've attached a patch which might give me some useful diagnostics on 'stderr'. You should be able to apply the patch against your source code and then re-do the failing gnunet-publish operation. With some luck, this will point me towards the issue... |
|
I'm out of town with sporadic internet access and not much time on my hands. I'll have to think about the SSH access thing - not saying "no" right now, but you'll appreciate it's a sensitive question. I'll get back to this next week. |
|
I totally understand; I'd just really like to squash this one, and manual code inspection got me nowhere. Now, please remember the alternative I proposed: running with the additional instrumentation I attached. That would also help and certainly be less invasive. |
|
The issue seems to have been fixed. I updated sources from svn sometime last week, recompiled and tried to reproduce this issue, but couldn't. I ran gnunet-publish -s on very nearly all files in my collection and I got some other errors (maybe relevant for another bug report), but not this one. It must have gotten got fixed sometime between revisions r21256 (HEAD, when I submitted) and r21400 (that I'm currently still at). Good work! I would consider this one closed, but I will continue testing and if there's ever a regression I will re-open or post a new bug report. |
Date Modified | Username | Field | Change |
---|---|---|---|
2012-05-03 21:52 | gabe | New Issue | |
2012-05-04 09:45 | Christian Grothoff | Assigned To | => Christian Grothoff |
2012-05-04 09:45 | Christian Grothoff | Status | new => assigned |
2012-05-04 09:49 | Christian Grothoff | Note Added: 0005775 | |
2012-05-04 14:50 | Matthias Wachs | Target Version | => 0.9.3 |
2012-05-04 15:28 | Christian Grothoff | Priority | normal => urgent |
2012-05-05 09:12 | Christian Grothoff | File Added: 2300-diagnostics.diff | |
2012-05-05 09:13 | Christian Grothoff | Note Added: 0005785 | |
2012-05-05 09:14 | Christian Grothoff | Status | assigned => feedback |
2012-05-06 20:46 | Christian Grothoff | Target Version | 0.9.3 => |
2012-05-06 21:26 | Christian Grothoff | Priority | urgent => normal |
2012-05-07 09:53 | gabe | Note Added: 0005799 | |
2012-05-07 09:53 | gabe | Status | feedback => assigned |
2012-05-07 09:56 | Christian Grothoff | Note Added: 0005800 | |
2012-05-07 09:56 | Christian Grothoff | Status | assigned => feedback |
2012-05-17 16:25 | gabe | Note Added: 0005926 | |
2012-05-17 16:25 | gabe | Status | feedback => assigned |
2012-05-27 18:37 | Christian Grothoff | Status | assigned => resolved |
2012-05-27 18:37 | Christian Grothoff | Fixed in Version | => 0.9.3 |
2012-05-27 18:37 | Christian Grothoff | Resolution | open => unable to reproduce |
2012-05-28 15:06 | Christian Grothoff | Target Version | => 0.9.3 |
2012-06-02 19:15 | Christian Grothoff | Status | resolved => closed |