View Issue Details

IDProjectCategoryView StatusLast Update
0002300GNUnetfile-sharing servicepublic2012-06-02 19:15
Reportergabe Assigned ToChristian Grothoff  
PrioritynormalSeveritymajorReproducibilityrandom
Status closedResolutionunable to reproduce 
PlatformGNU/LinuxOSDebianOS Version6.0.4
Product VersionGit master 
Target Version0.9.3Fixed in Version0.9.3 
Summary0002300: gnunet-publish ERROR Assertion failed at fs_dirmetascan.c:344
DescriptionThis is a regression of bug 0002232 - I#m seeing this issue again in the current SVN HEAD.

With certain files on my system, when I try to index them with 'gnunet-publish'
I get these errors:

May 03 20:46:45-693437 gnunet-publish-4193 ERROR Assertion failed at fs_dirmetascan.c:344.
Internal error scanning directory.
May 03 20:46:45-693483 gnunet-publish-4193 ERROR Assertion failed at fs_dirmetascan.c:385.
Internal error scanning directory.

The error also occurrs when trying to publish a directory that contains the file either directly or in a subdir somehwere.
Steps To ReproduceThis is 100% reproducible for those files on my system, but I couldn't find a commonality between the files or a recipe to generate files that provoke this behaviour.
Additional InformationI did try a few things to see what the cause may be and here's what I found out:
- Making a symlink to the file in question on the same or another filesystem allows me to publish the file by giving gnunet-publish the symlink.
- Making a hardlinked copy of the file in another location on the same filesystem allows me to publish the file by giving gnunet-publish the hardlink
- Making a copy of the file on the same or another filesystem allows me to publish the copied file.
- Moving the file to another location on the same or another filesystem allows me to publish it.
- Moving it back (after moving it or deleting the original after having made a copy) still leads to the same error when publishing.

This all seems to suggest a problem with the filename (although there are plenty of files in the same dir that work, with file names that only differ by one or two characters), so I tried renaming:

I found that adding an underscore at the end of one particular file made the second error message go away, but adding an underscore in front or replacing all blanks with underscores did nothing. In other cases, though, the extra underscore at the end made the second error message appear where it hadn't shown up before.

I renamed one of the problematic files to 'aaaaa', which allowed me to insert it. Then I made an empty file of the same name as the original file that caused the problem, but gnunet-publish didn't complain. Finally I filled the file with random bytes to the exact size of the original file and ran gnunet-publish on it, which didn't report any problems. Obviously, when I deleted the random file and restored the original to its former name, the issue reappeared.

I also wrote a little shell script to generate regular hierarchies of directories with randomly generated names and fill the deepest level directories with randomly named files consisting of random blocks of data. I ran it several times to make hundreds of directories various levels of nesting and thousands of files with sizes ranging from 1M to 1G, but gnunet-publish never complained about any of them.

It seems to be a specific combination of a file's contents, name and even location that causes this issue to appear. IMO this is a bizarre corner-case, but a very annoying one that might keep users from publishing interesting content.
TagsNo tags attached.
Attached Files
2300-diagnostics.diff (1,667 bytes)   
Index: src/fs/gnunet-helper-fs-publish.c
===================================================================
--- src/fs/gnunet-helper-fs-publish.c	(revision 21274)
+++ src/fs/gnunet-helper-fs-publish.c	(working copy)
@@ -186,6 +186,9 @@
 {
   struct GNUNET_MessageHeader hdr;
 
+  fprintf (stderr, "Helper sends %u-byte message of type %u\n",
+	   (unsigned int) (sizeof (struct GNUNET_MessageHeader) + data_length),
+	   (unsigned int) message_type);
   hdr.type = htons (message_type);
   hdr.size = htons (sizeof (struct GNUNET_MessageHeader) + data_length);
   if ( (GNUNET_OK !=
@@ -338,7 +341,7 @@
   ssize_t size;
   size_t slen;
 
-  if (item->is_directory == GNUNET_YES)
+  if (GNUNET_YES == item->is_directory)
   {
     /* for directories, we simply only descent, no extraction, no
        progress reporting */
Index: src/fs/fs_dirmetascan.c
===================================================================
--- src/fs/fs_dirmetascan.c	(revision 21274)
+++ src/fs/fs_dirmetascan.c	(working copy)
@@ -257,6 +257,9 @@
   const char *filename;
   size_t left;
 
+  fprintf (stderr, "DMS parses %u-byte message of type %u\n",
+	   (unsigned int) ntohs (msg->size),
+	   (unsigned int) ntohs (msg->type));
   left = ntohs (msg->size) - sizeof (struct GNUNET_MessageHeader);
   filename = (const char*) &msg[1];
   switch (ntohs (msg->type))
@@ -325,7 +328,7 @@
 			   NULL, GNUNET_SYSERR,
 			   GNUNET_FS_DIRSCANNER_ALL_COUNTED);
     ds->pos = ds->toplevel;
-    if (ds->pos->is_directory == GNUNET_YES)
+    if (GNUNET_YES == ds->pos->is_directory)
       ds->pos = advance (ds->pos);
     return;
   case GNUNET_MESSAGE_TYPE_FS_PUBLISH_HELPER_META_DATA:
2300-diagnostics.diff (1,667 bytes)   

Activities

Christian Grothoff

2012-05-04 09:49

manager   ~0005775

Are your filenames all ASCII-characters, or do they contain special non-English characters from your native language? The problem is within the code that transfers the filename WITH metadata to the inserting process, so if you put random garbage it won't happen simply because no metadata would be found in such files. So yes, this seems likely to be an error related to the filename.

Is it possible for me to get SSH access to your system to try to investigate further?

Christian Grothoff

2012-05-05 09:13

manager   ~0005785

I've looked over the code and cannot find anything. Assuming SSH access is not possible, I've attached a patch which might give me some useful diagnostics on 'stderr'. You should be able to apply the patch against your source code and then re-do the failing gnunet-publish operation. With some luck, this will point me towards the issue...

gabe

2012-05-07 09:53

reporter   ~0005799

I'm out of town with sporadic internet access and not much time on my hands.
I'll have to think about the SSH access thing - not saying "no" right now, but you'll appreciate it's a sensitive question.
I'll get back to this next week.

Christian Grothoff

2012-05-07 09:56

manager   ~0005800

I totally understand; I'd just really like to squash this one, and manual code inspection got me nowhere. Now, please remember the alternative I proposed: running with the additional instrumentation I attached. That would also help and certainly be less invasive.

gabe

2012-05-17 16:25

reporter   ~0005926

The issue seems to have been fixed.

I updated sources from svn sometime last week, recompiled and tried to reproduce this issue, but couldn't. I ran gnunet-publish -s on very nearly all files in my collection and I got some other errors (maybe relevant for another bug report), but not this one.
It must have gotten got fixed sometime between revisions r21256 (HEAD, when I submitted) and r21400 (that I'm currently still at).
Good work!

I would consider this one closed, but I will continue testing and if there's ever a regression I will re-open or post a new bug report.

Issue History

Date Modified Username Field Change
2012-05-03 21:52 gabe New Issue
2012-05-04 09:45 Christian Grothoff Assigned To => Christian Grothoff
2012-05-04 09:45 Christian Grothoff Status new => assigned
2012-05-04 09:49 Christian Grothoff Note Added: 0005775
2012-05-04 14:50 Matthias Wachs Target Version => 0.9.3
2012-05-04 15:28 Christian Grothoff Priority normal => urgent
2012-05-05 09:12 Christian Grothoff File Added: 2300-diagnostics.diff
2012-05-05 09:13 Christian Grothoff Note Added: 0005785
2012-05-05 09:14 Christian Grothoff Status assigned => feedback
2012-05-06 20:46 Christian Grothoff Target Version 0.9.3 =>
2012-05-06 21:26 Christian Grothoff Priority urgent => normal
2012-05-07 09:53 gabe Note Added: 0005799
2012-05-07 09:53 gabe Status feedback => assigned
2012-05-07 09:56 Christian Grothoff Note Added: 0005800
2012-05-07 09:56 Christian Grothoff Status assigned => feedback
2012-05-17 16:25 gabe Note Added: 0005926
2012-05-17 16:25 gabe Status feedback => assigned
2012-05-27 18:37 Christian Grothoff Status assigned => resolved
2012-05-27 18:37 Christian Grothoff Fixed in Version => 0.9.3
2012-05-27 18:37 Christian Grothoff Resolution open => unable to reproduce
2012-05-28 15:06 Christian Grothoff Target Version => 0.9.3
2012-06-02 19:15 Christian Grothoff Status resolved => closed