python中requirements.txt的编码问题

TL;DR requirements.txt中不要使用非ASCII编码的字符,否则会造成字符集错误,无法解析内容

实验设计

  1. 两份requirements.txt文件,包含相同的依赖,但不同的地方是:一份是ASCII编码(或者说只包含英文字符);另一份包含非ASCII字符(比如中文注释之类的)
  2. 分别安装两份requirements.txt文件,观察现象

实验材料

ASCII编码的requirements.txt

命名为:all_ascii_requirements.txt

具体内容如下:

1
dummy

非ASCII编码的requirements.txt

命名为:non_ascii_requirements.txt

具体内容如下:

1
2
# 这个是中文注释
dummy

实验过程

安装ASCII编码的requirements.txt

执行命令 pip install -r all_ascii_requirements.txt

命令工作正常,输入内容如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Collecting dummy (from -r all_ascii_requirements.txt (line 1))
Downloading dummy-0.1.0.zip
Requirement already up-to-date: jinja2>=2.0.0 in /usr/local/lib/python2.7/dist-packages (from dummy->-r all_ascii_requirements.txt (line 1))
Collecting numpy>=1.0.0 (from dummy->-r all_ascii_requirements.txt (line 1))
Downloading numpy-1.13.1-cp27-cp27mu-manylinux1_x86_64.whl (16.6MB)
100% |################################| 16.6MB 66kB/s
Requirement already up-to-date: mock>=1.0.0 in /usr/local/lib/python2.7/dist-packages (from dummy->-r all_ascii_requirements.txt (line 1))
Requirement already up-to-date: MarkupSafe>=0.23 in /usr/local/lib/python2.7/dist-packages (from jinja2>=2.0.0->dummy->-r all_ascii_requirements.txt (line 1))
Requirement already up-to-date: six>=1.9 in /usr/local/lib/python2.7/dist-packages (from mock>=1.0.0->dummy->-r all_ascii_requirements.txt (line 1))
Requirement already up-to-date: funcsigs>=1; python_version < "3.3" in /usr/local/lib/python2.7/dist-packages (from mock>=1.0.0->dummy->-r all_ascii_requirements.txt (line 1))
Requirement already up-to-date: pbr>=0.11 in /usr/local/lib/python2.7/dist-packages (from mock>=1.0.0->dummy->-r all_ascii_requirements.txt (line 1))
Building wheels for collected packages: dummy
Running setup.py bdist_wheel for dummy ... done
Stored in directory: /root/.cache/pip/wheels/fb/72/de/c12e171be0c7bff52d4bcebf680bd3b012203c68b8372b02a5
Successfully built dummy
Installing collected packages: numpy, dummy
Found existing installation: numpy 1.11.0
Uninstalling numpy-1.11.0:
Successfully uninstalled numpy-1.11.0
Successfully installed dummy-0.1.0 numpy-1.13.1

安装非ASCII编码的requirements.txt

执行命令 pip install -r non_ascii_requirements.txt

命令工作出现异常,输出内容如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Exception:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 312, in run
wheel_cache
File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 295, in populate_requirement_set
wheel_cache=wheel_cache):
File "/usr/local/lib/python2.7/dist-packages/pip/req/req_file.py", line 84, in parse_requirements
filename, comes_from=comes_from, session=session
File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 422, in get_file_content
content = auto_decode(f.read())
File "/usr/local/lib/python2.7/dist-packages/pip/utils/encoding.py", line 31, in auto_decode
return data.decode(locale.getpreferredencoding(False))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe8 in position 2: ordinal not in range(128)

实验结论

pip在解析requirements.txt,只能处理ASCII编码的文件,否则会出现Unicode错误。在编写requirements.txt时,切记使用ASCII编码,不要夹杂中文等非ASCII字符