Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Solved Issue indexer queue not getting processed, 8000+ unfinished jobs

    Gitea
    3
    11
    443
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • yusf
      yusf last edited by yusf

      This instance is a couple of years old, so it's been through some updates, and one server migration.

      I tried deleting /app/data/appdata/indexers in order to try and rebuild the indexing but since found that workers do not process that queue anyway. The issue indexer work queue is building up without getting worked on. Adding workers manually doesn’t help.

      I found a note for a recommendation from the 1.15 release, stating:

      It is recommended to ensure that queues are empty before upgrading using gitea manager flush-queues or on the admin pages.

      Should I do that?

      1 Reply Last reply Reply Quote 0
      • yusf
        yusf last edited by

        I went ahead and tried to flush the queue:

        sudo -u git /home/git/gitea/gitea -c /run/gitea/app.ini  manager flush-queues --timeout 20m
        

        but even with that generous timeout value it timed out:

        flush timeout with non-empty queues: issue_indexer
        

        I've little to no idea of what I'm doing here btw.

        girish 1 Reply Last reply Reply Quote 0
        • yusf
          yusf last edited by

          The queue has still not being processed, but 13000 flush workes (and probably counting) have seemingly been added.

          I’d really appreciate some help before this instance blows up 😅

          fbartels 1 Reply Last reply Reply Quote 0
          • fbartels
            fbartels App Dev @yusf last edited by

            I'm using Gitea for quite some years as well and this got me curious. The current queue can be seen in the webinterface of Gitea at https://your-domain/admin/monitor. My queues however are at 0.

            If you have not done yet I would check the logs of gitea if there are any abnormalities in regards to queue handling. Error messages at startup?

            The same website also allows to see stacktraces for the running processes.

            In the end it may be a bug within Gitea that would need reporting and fixing upstream.

            yusf 1 Reply Last reply Reply Quote 2
            • girish
              girish Staff @yusf last edited by

              @yusf For me that command says "Flushed" instantly. Maybe something is stuck in the queue. Looks like you can pass a "--debug" flag to the command. Can you see what it's doing?

              yusf 1 Reply Last reply Reply Quote 1
              • yusf
                yusf @girish last edited by

                @girish said in Issue indexer queue not getting processed, 8000+ unfinished jobs:

                --debug

                With what command? At least not gitea manager flush-queues. 🤔

                girish 1 Reply Last reply Reply Quote 0
                • yusf
                  yusf @fbartels last edited by

                  @fbartels I have skimmed the logs but I haven't seen any abnormalities (though logs are quite verbose). I don't know what a stacktrace is.

                  1 Reply Last reply Reply Quote 0
                  • girish
                    girish Staff @yusf last edited by

                    @yusf said in Issue indexer queue not getting processed, 8000+ unfinished jobs:

                    With what command? At least not gitea manager flush-queues.

                    yes, --debug is an option for that command.

                    yusf 1 Reply Last reply Reply Quote 0
                    • yusf
                      yusf @girish last edited by

                      @girish Must've got it wrong somehow. Anyway, --debug only adds one line of little insight:

                      sudo -u git /home/git/gitea/gitea -c /run/gitea/app.ini manager flush-queues --debug
                      2023/01/16 09:45:29 ...dules/git/command.go:213:Run() [D] /usr/bin/git version
                      flush timeout with non-empty queues: issue_indexer
                      girish 1 Reply Last reply Reply Quote 0
                      • girish
                        girish Staff @yusf last edited by

                        @yusf I guess we have to ask upstream for ideas...

                        But I found https://github.com/go-gitea/gitea/issues/17563 . It suggests changing the indexer to db instead of the default bleve. Don't know if that will help your case.

                        I think another idea is to figure where these jobs are stored and clear that. There is a task table, probably there?

                        yusf 1 Reply Last reply Reply Quote 1
                        • yusf
                          yusf @girish last edited by yusf

                          @girish Setting

                          [indexer]
                          ISSUE_INDEXER_TYPE=db
                          

                          in app.ini solved the issue immediately. Thanks alot!

                          Now I'm curious to know what db means in this context (but the urgency is gone 😅). Should the Cloudron package have db as ISSUE_INDEXER_TYPE out of the box, for example?

                          Some Cloudron app config data got alot of legacy files from earlier versions, so sometimes it's hard to know what the true good defaults are.

                          1 Reply Last reply Reply Quote 0
                          • Topic has been marked as a question  yusf yusf 
                          • Topic has been marked as solved  yusf yusf 
                          • First post
                            Last post
                          Powered by NodeBB