A case study about name conflict in python

I got a very interesting crash when I launched Vim to edit a source code file in a python project. There are some clues to lead me to find the root cause, and this adventure is worth to write it down.

The last lines of message as below:

File "/usr/lib/python2.7/inspect.py", line 39, in 
  import tokenize
File "/usr/lib/python2.7/tokenize.py", line 38, in 
NameError: name 'N_TOKENS' is not defined

First, these is stack trace for python exception. But actually I just launched the Vim, most of plugins should be written as pure vim language, but this time I installed Kirill Klenov’s Python Mode , it will call/import python code.

So, this should be a python related error. But seems it is crashed in a standard library file. Why ? There shouldn’t have a bug in a released standard library. Must something wrong!

Then I check the source code of tokenize.py, N_TOKENS belongs to token.py, and I find a statement “from token import *”. Bad smell, right ? !

Yes! Normally, we shouldn’t write the code to import everything from a package, sometimes that will hide some bugs. In this case, I have a folder named “token” in current working directory I launched the Vim. So when Vim launched the plugin Python-mode, an embedded pylint was launched. And unfortunately, pylint will add the current working directory into PYTHONPATH sys.path (the root paths of searching packages). Once sys.path is modified, the standard library token.py is name conflict with my folder ‘token’, of cause ‘token’ is a subpackage.

Python world is built on packages same as Java, so we can’t have two identical packages inside. As Java suggest, we organize our private package with organization names or package hierarchy. Yup, I had already done that, package them in a root folder, but the naughty guy Pylint change the rule :(

The solutions can be:

  1. change the name for sub package to avoid the conflict.
  2. don’t launch the vim inside the package, and make sure you have a unique package name.

I selected the Solution 2. You can’t check every folder name, but you can design a unique package hierarchy. For running pylint, we need the PYTHONPATH can point to the root directory of project, so we should launch the editor in root directory only. How to open the files in sub-packages ? That’s not a deal, we can depend on another plugin NERDTree to find the file, in this way, we can get correct PYTHONPATH for pylint, and avoid any name conflict. I have to change the bad habit of jumping forward and back between lots of tabs.

4 thoughts on “A case study about name conflict in python

    • Thank you for reading my post. I think it’ better that pylint can use PYTHONPATH first if exists, elase adding the current directory ‘.’ into sys.path for a single python file not in a project.

  1. Hey There. I found your blog the use of msn. This is an extremely
    neatly written article. I’ll make sure to bookmark it and come back to learn extra of your useful info. Thank you for the post. I will certainly return.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s