View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0006241GNUnetfile-sharing servicepublic2020-05-17 21:582024-02-29 22:47
Reportercy1 Assigned To 
PrioritynormalSeverityminorReproducibilityalways
Status acknowledgedResolutionopen 
PlatformGentooOSLinux 
Product VersionGit master 
Summary0006241: Unindex does not unindex.
DescriptionI managed to index a bunch of junk files testing fs, and I can't unindex them. It unindexes them, and then they're still indexed.
Steps To Reproduce$ tar xzf testdata.tar.gz
$ gnunet-publish -R herp
...
$ gnunet-fs -i | grep filegix.html
.../herp/derpdximv/filegix.html
$ gnunet-unindex .../herp/derpdximv/filegix.html
Unindexing done.
$ gnunet-fs -i | grep filegix.html
.../herp/derpdximv/filegix.html

Additional InformationOther files that I have indexed, I do seem to be able to unindex. However, any files that I have indexed which get deleted can never be unindexed, since gnunet doesn't unindex them automatically, and gnunet-unindex just fails with an error. That's a different bug than this though. This bug is where the file exists, and the file remains indexed even after you unindex it. I'm not sure if it's my test data specifically, or something more general. If other people could confirm these files get stuck in their index, I'd appreciate it. I don't know how to get them out though, so be warned.
TagsNo tags attached.
Attached Files
testdata.tar.gz (29,627 bytes)

Activities

cy1

2020-05-20 01:03

reporter   ~0015932

I can't help getting the feeling that I figured out a fix for this a while ago. I don't remember what for the life of me though. Maybe if you... remove ~/.local/share/gnunet/fs/idxinfo.lst then all your indexed files will stop being indexed?

cy1

2020-05-21 22:33

reporter   ~0015935

I remembered the fix. It's not really a fix, but a hacky way to temporarily fix it is to stop gnunet, then remove ~/.local/share/gnunet/fs/idxinfo.lst
When you restart gnunet, all your indexed files will be unindexed, though their records might still be somewhere in the db of course. ALL your files of course, not just the ones you wanted to unindex.

I think the solution would have to rewrite unindexing so it unindexes files by path, then only gives a token effort to remove the content from the data store, if those paths happen to point to actual files. But even when the files exist, sometimes it won't unindex them, so I don't know what's going on there.

cy1

2020-05-22 06:12

reporter   ~0015941

Oh another problem with removing indexed files is you'll get endless error messages in your logs, trying and failing about once a second like:

May 22 04:01:51-376176 fs-28169 ERROR Assertion failed at gnunet-service-fs_indexing.c:303.
May 22 04:01:51-376211 fs-28169 ERROR Failed to find index M20F56KN

If you cannot recreate the file whose hash is M20F56KN, it just reports that error over and over again ad infinitum. For every file that is supposed to be indexed, but was deleted. I don't know if it ever gets tired of doing that. Removing idxinfo.lst doesn't seem to stop it from these endless attempts. Whatever calls GNUNET_FS_handle_on_demand_block does not even log what it's doing at the DEBUG level, so the above error messages are just back to back, without context, one after the other. I could attach gdb I suppose...

cy1

2020-05-22 06:20

reporter   ~0015942

This is the stack trace for the error:

#0 GNUNET_FS_handle_on_demand_block (key=key@entry=0x5602719c20c0, size=size@entry=72, data=data@entry=0x5602719c2100,
    type=type@entry=GNUNET_BLOCK_TYPE_FS_ONDEMAND, priority=priority@entry=365, anonymity=anonymity@entry=1, replication=0, expiration=...,
    uid=1809910, cont=0x56026f3b73f0 <process_local_reply>, cont_cls=0x5602717602a0) at gnunet-service-fs_indexing.c:304
#1 0x000056026f3b773e in process_local_reply (cls=cls@entry=0x5602717602a0, key=key@entry=0x5602719c20c0, size=72,
    data=data@entry=0x5602719c2100, type=GNUNET_BLOCK_TYPE_FS_ONDEMAND, priority=365, anonymity=1, replication=0, expiration=..., uid=1809910)
    at gnunet-service-fs_pr.c:1581
#2 0x00007f884089fa0a in handle_data (cls=<optimized out>, dm=0x5602719c2090) at /usr/include/bits/byteswap.h:52
#3 0x00007f88407d7ba3 in GNUNET_MQ_handle_message (handlers=<optimized out>, mh=0x5602719c2090) at mq.c:263
#4 0x00007f88407d7d60 in GNUNET_MQ_inject_message (mq=0x560270ac8750, mh=<optimized out>) at mq.c:207
#5 0x00007f88407a79ea in recv_message (cls=0x560270ac8620, msg=0x5602719c2090) at client.c:344
#6 0x00007f88407d6ce2 in GNUNET_MST_from_buffer (mst=0x560270ac83f0, buf=buf@entry=0x0, size=size@entry=0, purge=purge@entry=0,
    one_shot=one_shot@entry=0) at mst.c:231
#7 0x00007f88407d75b8 in GNUNET_MST_read (mst=<optimized out>, sock=<optimized out>, purge=purge@entry=0, one_shot=one_shot@entry=0)
    at mst.c:374
#8 0x00007f88407a8afd in receive_ready (cls=0x560270ac8620) at client.c:444
#9 0x00007f88407ee97a in GNUNET_SCHEDULER_do_work (sh=sh@entry=0x560270ae2280) at scheduler.c:2144
#10 0x00007f88407ef9a9 in select_loop (context=0x7ffdfbe48000, sh=0x560270ae2280) at scheduler.c:2444
#11 GNUNET_SCHEDULER_run (task=task@entry=0x7f88407f34d0 <service_main>, task_cls=task_cls@entry=0x7ffdfbe48100) at scheduler.c:738
#12 0x00007f88407f4cf2 in GNUNET_SERVICE_run_ (argc=<optimized out>, argv=<optimized out>, service_name=0x56026f3be08e "fs",
    options=<optimized out>, service_init_cb=0x56026f3acf30 <run>, connect_cb=0x56026f3ab310 <client_connect_cb>,
    disconnect_cb=0x56026f3ab040 <client_disconnect_cb>, cls=0x0, handlers=0x7ffdfbe48480) at service.c:2126
#13 0x000056026f3aac98 in main (argc=<optimized out>, argv=<optimized out>) at gnunet-service-fs.c:1395

I am not currently running gnunet-download, or anything at all, so I don't know what "local reply" it might be processing.

cy1

2020-05-22 19:23

reporter   ~0015947

Uh, I think I was writing that the "Endless" error stopped as of this morning, but the website erased my post again.

cy1

2020-05-22 19:37

reporter   ~0015948

According to the log the error began appearing at "May 22 03:57:20-704495" and ended at "May 22 15:34:40-750780" which was about 2 hours ago.

No other log messages from gnunet-fs anywhere

Issue History

Date Modified Username Field Change
2020-05-17 21:58 cy1 New Issue
2020-05-17 21:58 cy1 Tag Attached: file-sharing service
2020-05-17 21:58 cy1 File Added: testdata.tar.gz
2020-05-20 01:03 cy1 Note Added: 0015932
2020-05-21 22:33 cy1 Note Added: 0015935
2020-05-22 06:12 cy1 Note Added: 0015941
2020-05-22 06:20 cy1 Note Added: 0015942
2020-05-22 19:23 cy1 Note Added: 0015947
2020-05-22 19:37 cy1 Note Added: 0015948
2020-06-01 00:44 Adminknox Issue cloned: 0006271
2020-06-01 00:49 Adminknox Issue cloned: 0006274
2020-10-29 11:20 schanzen Assigned To => schanzen
2020-10-29 11:20 schanzen Status new => acknowledged
2022-02-22 23:10 schanzen Assigned To schanzen =>
2024-02-29 22:47 Christian Grothoff Tag Detached: file-sharing service