Page MenuHomePhabricator

Pywikibot: 'NoneType' object is not subscriptable in validate_tokens
Closed, ResolvedPublicBUG REPORT

Description

List of steps to reproduce (step by step, including full links if applicable):

  • Run ./pwb.py touch -page:Main\ Page

What happens?:

  File "/home/user/src/pywikibot/./pwb.py", line 414, in <module>
    if not main():
  File "/home/user/src/pywikibot/./pwb.py", line 406, in main
    run_python_file(filename,
  File "/home/user/src/pywikibot/./pwb.py", line 106, in run_python_file
    exec(compile(source, filename, 'exec', dont_inherit=True),
  File "./scripts/touch.py", line 117, in <module>
    main()
  File "./scripts/touch.py", line 111, in main
    bot_class(generator=gen, **options).run()
  File "/home/user/src/pywikibot/pywikibot/bot.py", line 1600, in run
    self.treat(page)
  File "./scripts/touch.py", line 55, in treat
    page.touch(botflag=self.opt.botflag)
  File "/home/user/src/pywikibot/pywikibot/page/__init__.py", line 1368, in touch
    self.save(summary=summary, watch='nochange',
  File "/home/user/src/pywikibot/pywikibot/tools/_deprecate.py", line 404, in wrapper
    return obj(*__args, **__kw)
  File "/home/user/src/pywikibot/pywikibot/tools/_deprecate.py", line 404, in wrapper
    return obj(*__args, **__kw)
  File "/home/user/src/pywikibot/pywikibot/page/__init__.py", line 1211, in save
    self._save(summary=summary, watch=watch, minor=minor, botflag=botflag,
  File "/home/user/src/pywikibot/pywikibot/page/_decorators.py", line 54, in wrapper
    handle(func, self, *args, **kwargs)
  File "/home/user/src/pywikibot/pywikibot/page/_decorators.py", line 33, in handle
    func(self, *args, **kwargs)
  File "/home/user/src/pywikibot/pywikibot/page/__init__.py", line 1223, in _save
    done = self.site.editpage(self, summary=summary, minor=minor,
  File "/home/user/src/pywikibot/pywikibot/site/_decorators.py", line 92, in callee
    return fn(self, *args, **kwargs)
  File "/home/user/src/pywikibot/pywikibot/site/_apisite.py", line 1608, in editpage
    token = self.tokens['edit']
  File "/home/user/src/pywikibot/pywikibot/site/_tokenwallet.py", line 58, in __getitem__
    key = self.site.validate_tokens([key])[0]
  File "/home/user/src/pywikibot/pywikibot/site/_apisite.py", line 1307, in validate_tokens
    types_wiki_old = self._paraminfo.parameter('query+info',
TypeError: 'NoneType' object is not subscriptable
CRITICAL: Exiting due to uncaught exception <class 'TypeError'>

This is after clearing the ~/.pywikibot/apcache-py3 cache, before that, it was

  File "/home/john/src/pywikibot/pywikibot/data/api.py", line 572, in parameter
    raise ValueError("paraminfo for '{}' not loaded".format(module))
ValueError: paraminfo for 'tokens' not loaded
CRITICAL: Exiting due to uncaught exception <class 'ValueError'>

Software version (if not a Wikimedia wiki), browser information, screenshots, other information, etc:

  • Pywikibot 6.6.0, locally and on PAWS
  • Site: enwikisource, wikidata (both group1)

Not on Commons or enwiki (group2) so it could be MW .23 deployment issue.

enwiki has the following, enwikisource does not:

{
    "index": 5,
    "name": "token",
    "type": [
        "block",
        "delete",
        "edit",
        "email",
        "import",
        "move",
        "protect",
        "unblock",
        "watch"
    ],
    "deprecated": "",
    "multi": "",
    "lowlimit": 50,
    "highlimit": 500,
    "limit": 50
},

Event Timeline

JJMC89 triaged this task as High priority.EditedSep 16 2021, 7:20 PM
JJMC89 added a subscriber: JJMC89.

Due to T280806: Remove old action api token parameters. We cannot check query+info for tokens from 1.37wmf23 on.

Changing this line in _apisite.py fixes it:

types_wiki_old = self._paraminfo.parameter('query+info', 'token')['type']

to

if mw_ver < '1.37wmf23':
    types_wiki_old = self._paraminfo.parameter('query+info', 'token')['type']
else:
    types_wiki_old = []

But I am not sure what the actual correct formulation would be in the context of that if/else statement.

Edit: Actually, this only fixes the "touch" script, it still ends up with

Requested token 'edit' is invalid on wikisource:en wiki.

Change 721539 had a related patch set uploaded (by Legoktm; author: Legoktm):

[mediawiki/core@master] Revert \"Drop action api token methods deprecated in 1.24\"

https://gerrit.wikimedia.org/r/721539

Change 721540 had a related patch set uploaded (by Legoktm; author: Legoktm):

[mediawiki/core@wmf/1.37.0-wmf.23] Revert \"Drop action api token methods deprecated in 1.24\"

https://gerrit.wikimedia.org/r/721540

I've proposed a temporary revert of the MediaWiki change for this week, but it'll still go out next week and needs a proper fix.

Change 721621 had a related patch set uploaded (by Jon Harald Søby; author: Jon Harald Søby):

[pywikibot/core@master] Fix for removed action API token parameters

https://gerrit.wikimedia.org/r/721621

Change 721540 merged by jenkins-bot:

[mediawiki/core@wmf/1.37.0-wmf.23] Revert \"Drop action api token methods deprecated in 1.24\"

https://gerrit.wikimedia.org/r/721540

@Xqt: Can we get this fixed in v6? I saw you already moved development to 7.0, but I don't think anything breaking was merged yet.

Note: I had to delete ~/.pywikibot/apicache-py3 to get this working again.

@Xqt: Can we get this fixed in v6? I saw you already moved development to 7.0, but I don't think anything breaking was merged yet.

I think we still can merge a patch to the 7.0 master but backport and publish a 6.6.1 stable release to solve this issue. I think Pywikibot 7 will come in 4-6 weeks from now as stable release and giving up Python 3.5.0-3.5.2. Maybe we can drop older MediaWiki Version 1.23 or 1.23-1.26 up to the next LTS.

I wasn't able to reproduce this issue:

C:\pwb\GIT\core>py -3.9 pwb.py touch -page:"Main Page" -site:wikisource:en
Retrieving 1 pages from wikisource:en.
WARNING: API error protectedpage: This page has been protected to prevent editing or other actions.
ERROR: Page [[Main Page]] is locked.

1 pages read
0 pages written
0 pages skipped
Execution time: 4 seconds
Read operation time: 4.0 seconds
Script terminated successfully.

C:\pwb\GIT\core>py -3.9 pwb.py version
Pywikibot: [ssh] pywikibot-core (a719db9, g15483, 2021/09/15, 10:49:42, stable)
Release version: 6.6.0
setuptools version: 57.0.0
mwparserfromhell version: 0.6.2
wikitextparser version: n/a
requests version: 2.20.1
  cacerts: C:\Python39\lib\site-packages\certifi\cacert.pem
    certificate test: ok
Python: 3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)]
PYWIKIBOT_DIR: Not set
PYWIKIBOT_DIR_PWB: C:\pwb\GIT\core
PYWIKIBOT_NO_USER_CONFIG: Not set
Config base dir: C:\pwb\GIT\core

@Xqt first, you probably need to try on a page your are allowed to edit (now I think of it, the main page is protected). You could try "Wikisource:Sandbox".

And also since Legoktm has rolled the change back temporarily, it should now be working again against production WMF wikis.

You can probably still provoke it against a Beta wiki, as they don't have the revert backported: https://en.wikisource.beta.wmflabs.org/wiki/Special:ApiSandbox#action=paraminfo&format=json&modules=query%2Binfo (not sure how to specify that with -site:)

APISite.validate_tokens already replaces a given token by csrf key. I removed the actions api calls for mw 1.38 with my proposed patch.

@Legoktm: iwill this be changed in 1.37wmf23+ or 1.38?

It will reach Wikimedia wikis with 1.38wmf1, but will be included in the MediaWiki 1.37.0 stable release. See also https://gerrit.wikimedia.org/r/c/pywikibot/core/+/721621/comment/227c7ef4_309049fa/

Just in case it helps, it also happens with Spanish Wikipedia and the script "replace". For instance with:

python3 pwb.py replace Spanish español -page:Femenino -simulate

Just in case it helps, it also happens with Spanish Wikipedia and the script "replace". For instance with:

python3 pwb.py replace Spanish español -page:Femenino -simulate

I think you have to clear the apichache-py3 folder. The MW change was reverted. The Pywikibot patch to support the new behaviour is coming soon and will be backported to stable release.

Change 721621 merged by jenkins-bot:

[pywikibot/core@master] [mw 1.37] Fix for removed action API token parameters

https://gerrit.wikimedia.org/r/721621

Change 722547 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [cleanup] Remove outdated intoken from TestRequest

https://gerrit.wikimedia.org/r/722547

Change 722547 abandoned by Xqt:

[pywikibot/core@master] [cleanup] Remove outdated intoken from TestRequest

Reason:

https://gerrit.wikimedia.org/r/722547

Change 722547 restored by Xqt:

[pywikibot/core@master] [cleanup] Remove outdated intoken from TestRequest

https://gerrit.wikimedia.org/r/722547

Change 722547 merged by jenkins-bot:

[pywikibot/core@master] [cleanup] Remove outdated intoken from TestRequest

https://gerrit.wikimedia.org/r/722547

Change 722933 had a related patch set uploaded (by Legoktm; author: Legoktm):

[mediawiki/core@wmf/1.38.0-wmf.1] Revert \"Drop action api token methods deprecated in 1.24\"

https://gerrit.wikimedia.org/r/722933

Change 722933 merged by jenkins-bot:

[mediawiki/core@wmf/1.38.0-wmf.1] Revert \"Drop action api token methods deprecated in 1.24\"

https://gerrit.wikimedia.org/r/722933

Change 724791 had a related patch set uploaded (by Ppchelko; author: Legoktm):

[mediawiki/core@wmf/1.38.0-wmf.2] Revert \"Drop action api token methods deprecated in 1.24\"

https://gerrit.wikimedia.org/r/724791

Change 724791 abandoned by Ppchelko:

[mediawiki/core@wmf/1.38.0-wmf.2] Revert \"Drop action api token methods deprecated in 1.24\"

Reason:

https://gerrit.wikimedia.org/r/724791

Change 724791 restored by Ppchelko:

[mediawiki/core@wmf/1.38.0-wmf.2] Revert \"Drop action api token methods deprecated in 1.24\"

https://gerrit.wikimedia.org/r/724791

Change 724791 merged by jenkins-bot:

[mediawiki/core@wmf/1.38.0-wmf.2] Revert \"Drop action api token methods deprecated in 1.24\"

https://gerrit.wikimedia.org/r/724791

Change 721539 abandoned by Ppchelko:

[mediawiki/core@master] Revert \"Drop action api token methods deprecated in 1.24\"

Reason:

This landed, no need for revert anymore

https://gerrit.wikimedia.org/r/721539