GeezerVison is showing github.com — Create a script to remove dead, deprecated or invalid links from planet · Issue #233 · python/planet
Create a script to remove dead, deprecated or invalid links from planet #233Description
- pysoy hasn't been updated in about 5 years
- Cherrypy link is dead
- Turbo gears hasn't been updated for over a year and the project is marginal
- These could or should be replaced with links to the pyramid, flask, pandas, or jupyter feeds
Activitychanged the title [-]Remove dead or deprecated links for sidebar[/-] [+]Remove dead or deprecated links from sidebar[/+] on Jun 12, 2017rochacbruno commented on Jun 13, 2017 I suggested to create a validator script to run monthly #124 (comment)
That script would take each URL check its status and validate it in RSS validator. changed the title [-]Remove dead or deprecated links from sidebar[/-] [+]Create a script to remove dead, deprecated or invalid links from planet[/+] on Jun 13, 2017rochacbruno commented on Jun 13, 2017 @pydanny I changed the title of your issue to address the automation of this task. rochacbruno commented on Jun 13, 2017 Script should do
- Run weekly or monthly
- Iterate all urls on
config.ini and also python libraries and python planets list
- Call each URL and assert (following redirects) -
Check return code is 200
Check data of last update is newer than one year
Check feed is valid using feedvalidator/podcastvalidator/othervalidator
If not valid, store the URL in a simple database (text file, sqlite or something)
- ACTION: If link returns
bad status\invalid 3 times, remove from planet
- ACTION: If feed is outdated for more than one year, keep in planet but remove from sidebar
- ACTION: If feed is updated and not in sidebar, put it there again
Reacted by Mridu Bhatnagar pydanny commented on Jun 13, 2017 ContributorAuthor More actions Pysoy and the others I mentioned aren't in the general feeds. They are in the "python libraries" and "python planets" lists. Will these be covered by your script or is that just for RSS feeds?
As for your script, go ahead and remove my blog from your list already. I say that because it's going to fail your W3C validator check and I don't have the luxury of time to fix it. Do keep in mind that this aggregator is the ONLY one I know of that insists on W3C validation. rochacbruno commented on Jun 13, 2017 @pydanny yeah we must discuss if using w3c validator is a good choice or not, we only need to check if the feed doesn't break the planet as it happened in the past. Maybe we can write our own simple validator so we do not need to rely on w3c one. tjguk commented on Jun 13, 2017 For my info -- what's the issue with the W3C validator? Is it too strict? pydanny commented on Jun 13, 2017 ContributorAuthor More actions Yes, it's too strict. No other aggregator I'm on blocks my RSS feed. pybites commented on Jul 19, 2017 pybites commented on Apr 2, 2018 mridubhatnagar commented on Jul 2, 2018 Hi
Can I be assigned this issue to work on. I don't really know the complexity of the issue. But, I am curious to work on the same.
Based on the above discussion what I could understand is
- Parse through each url in config.ini
- Check the response status
- Validate the RSS feed (use existing service/Create a RSS feed validator)
Links which are no more working should be removed.
I am currently tied up with some other stuff. But, after a month shall start working on it.
Can I please be assigned this. It looks interesting.
Thanks mridubhatnagar commented on Jul 2, 2018 Also, are we validating a RSS feed based on the W3 validator?
Or we consider a RSS feed to be valid if the feedparser is giving us no error?
Or we are relying on none, And instead create one of our own and validate? rochacbruno commented on Jul 3, 2018 @mridubhatnagar the only problem with W3 is that it considers podcast feeds invalid, so we need to have a podcast feed validator. Reacted by Mridu Bhatnagar mridubhatnagar commented on Jul 3, 2018 @rochacbruno Fair enough. I would like to give it a shot.
Also, I think some RSS feeds are valid. But, still not showing up on planet. Like @pybites live feed.
I am not sure though if feedparser works for podcast feeds or not.
I will code podcast feed validator.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|