I got a very interesting crash when I launched Vim to edit a source code file in a python project. There are some clues to lead me to find the root cause, and this adventure is worth to write it down.
The last lines of message as below:
File "/usr/lib/python2.7/inspect.py", line 39, in import tokenize File "/usr/lib/python2.7/tokenize.py", line 38, in COMMENT = N_TOKENS NameError: name 'N_TOKENS' is not defined
First, these is stack trace for python exception. But actually I just launched the Vim, most of plugins should be written as pure vim language, but this time I installed Kirill Klenov’s Python Mode , it will call/import python code.
So, this should be a python related error. But seems it is crashed in a standard library file. Why ? There shouldn’t have a bug in a released standard library. Must something wrong!
Then I check the source code of tokenize.py, N_TOKENS belongs to token.py, and I find a statement “from token import *”. Bad smell, right ? !
Yes! Normally, we shouldn’t write the code to import everything from a package, sometimes that will hide some bugs. In this case, I have a folder named “token” in current working directory I launched the Vim. So when Vim launched the plugin Python-mode, an embedded pylint was launched. And unfortunately, pylint will add the current working directory into PYTHONPATH sys.path (the root paths of searching packages). Once sys.path is modified, the standard library token.py is name conflict with my folder ‘token’, of cause ‘token’ is a subpackage.
Python world is built on packages same as Java, so we can’t have two identical packages inside. As Java suggest, we organize our private package with organization names or package hierarchy. Yup, I had already done that, package them in a root folder, but the naughty guy Pylint change the rule :(
The solutions can be:
- change the name for sub package to avoid the conflict.
- don’t launch the vim inside the package, and make sure you have a unique package name.
I selected the Solution 2. You can’t check every folder name, but you can design a unique package hierarchy. For running pylint, we need the PYTHONPATH can point to the root directory of project, so we should launch the editor in root directory only. How to open the files in sub-packages ? That’s not a deal, we can depend on another plugin NERDTree to find the file, in this way, we can get correct PYTHONPATH for pylint, and avoid any name conflict. I have to change the bad habit of jumping forward and back between lots of tabs.