Changelog format seems broken
-
As seen in https://forum.cloudron.io/post/112933 and compared to original changelog at https://github.com/syncthing/syncthing/releases/tag/v2.0.10 the changelog in Cloudron just adds useless bullet points and newlines.
Thanks!
-
S SansGuidon marked this topic as a question
-
Good catch @SansGuidon . I wonder why this happenned. What we do is to run the changelog through a summarizer which is based on Mistral . I guess it broke the lines for some reason. I will keep a look-out if this happens again.
-
G girish marked this topic as a regular topic
-
Thanks! LLMs will never be 100% reliable but maybe add some try catch mechanism/retry loop in case the ratio of bullet points in original content and new summary does not make sense like more bullet points in the final summary than the initial. I'm used to work with Mistral and end up always adding validation/ retries mechanisms for every work as LLMs are designed to hallucinate and also Mistral API is not designed to always answer something valuable nor always answer quick or at first try.
-
Thanks! LLMs will never be 100% reliable but maybe add some try catch mechanism/retry loop in case the ratio of bullet points in original content and new summary does not make sense like more bullet points in the final summary than the initial. I'm used to work with Mistral and end up always adding validation/ retries mechanisms for every work as LLMs are designed to hallucinate and also Mistral API is not designed to always answer something valuable nor always answer quick or at first try.
@SansGuidon that's my experience too. In fact, often, it makes up changelog items when it lacks information and no amount of prompt engineering seems to work. Because of all this, the app publishing pipeline is not fully automated, and we check the summary output manually. But clearly I missed this one!
-
Good to know, but it doesn't help to trust the notifications at all. It might be good to add a "generated by AI" at the bottom of those notifications /summaries as a matter of transparency for the users.
-
@SansGuidon that's my experience too. In fact, often, it makes up changelog items when it lacks information and no amount of prompt engineering seems to work. Because of all this, the app publishing pipeline is not fully automated, and we check the summary output manually. But clearly I missed this one!
@girish said in Changelog format seems broken:
@SansGuidon that's my experience too. In fact, often, it makes up changelog items when it lacks information and no amount of prompt engineering seems to work.
I don’t have good experience with Mistral tbh. Gemini 2.5 Pro is much more reliable for such tasks…
