I also have a lot of wear on the part of the idler block that guides the filament. I am wondering if the filament moving too far off the hobbed bolt could be causing some of the grinding issues.
I tried to address this by capturing the PTFE tube using a bracket above the extruder. I haven’t done much 2.85mm prints since the change so can’t tell if it fixed the issue.
Good to know that you are printing without the PTFE tube. I’ve wondered if that would work well. Did you move the spool above the printer?