[Tutor] pip install lxml fails

[Alex Kleider]

In the process of trying to learn about web scraping
(http://docs.python-guide.org/en/latest/scenarios/scrape/)
I tried to pip install lxml.

Can anyone suggest a remedy?

The output goes on for many pages and the resulting .pip/pip.log
file is almost 900 lines long
but the main meat of the failure seems to be in the following:

(env)***@t61p:~/P3env/declination.d$ pip install lxml
Downloading/unpacking lxml
Downloading lxml-3.4.2.tar.gz (3.5MB): 3.5MB downloaded
Running setup.py (path:/home/alex/P3env/env/build/lxml/setup.py)
egg_info for package lxml
Building lxml version 3.4.2.
Building without Cython.
Using build configuration of libxslt 1.1.28
/usr/lib/python3.4/distutils/dist.py:260: UserWarning: Unknown
distribution option: 'bugtrack_url'
warnings.warn(msg)

warning: no previously-included files found matching '*.py'
Installing collected packages: lxml
Running setup.py install for lxml
Building lxml version 3.4.2.
Building without Cython.
Using build configuration of libxslt 1.1.28
building 'lxml.etree' extension
i686-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -g -fstack-protector --param=ssp-buffer-size=4
-Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC
-I/usr/include/libxml2
-I/home/alex/P3env/env/build/lxml/src/lxml/includes
-I/usr/include/python3.4m -I/home/alex/P3env/env/include/python3.4m -c
src/lxml/lxml.etree.c -o build/temp.linux-i686-3.4/src/lxml/lxml.etree.o
-w
src/lxml/lxml.etree.c:8:22: fatal error: pyconfig.h: No such file or
directory
#include "pyconfig.h"
^
compilation terminated.
/usr/lib/python3.4/distutils/dist.py:260: UserWarning: Unknown
distribution option: 'bugtrack_url'
warnings.warn(msg)
error: command 'i686-linux-gnu-gcc' failed with exit status 1
Complete output from command /home/alex/P3env/env/bin/python3 -c
"import setuptools,
tokenize;__file__='/home/alex/P3env/env/build/lxml/setup.py';exec(compile(getattr(tokenize,
'open', open)(__file__).read().replace('\r\n', '\n'), __file__,
'exec'))" install --record /tmp/pip-sknxo73w-record/install-record.txt
--single-version-externally-managed --compile --install-headers
/home/alex/P3env/env/include/site/python3.4:
......
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Alex Kleider]

I'm a python newbie, but it looks to me like your compiler cannot
find your header files, and in particular pyconfig.h.
I tried searching my system and found a file with that name at
/home/bambams/src/pyenv/versions/2.7.9/include/python2.7/pyconfig.h
/home/bambams/src/pyenv/versions/3.4.2/include/python3.4m/pyconfig.h
/usr/include/python2.6/pyconfig.h
/usr/include/python2.7/pyconfig.h
/usr/include/python3.2mu/pyconfig.h
Based on that I am assuming that you should have a pyconfig.h in
either /usr/include/python3.4m or
/home/alex/P3env/env/include/python3.4m. I would probably start
there to verify that I have that header file in a location where
it's expected (-I flags in the above command, or system include
directories). If not then I would wonder why...
Regards,

Thank you very much for looking into this.

I have corresponding files in /usr/include for 2.7 but not for 3*:
***@t61p:~/P3env/env$ ls -ld /usr/include/python*
drwxr-xr-x 2 root root 4096 Apr 8 19:05 /usr/include/python2.7

***@t61p:~/P3env/env$ ls
bin lib
Nothing in the env file (generated by virtualenv.)

My Ubuntu 14.04 system comes with Python3 by default so it does exist:
***@t61p:~/P3env/env$ which python3
/usr/bin/python3
It's a mystery why it doesn't come with the corresponding include
directory.

I'm guessing this is a system level problem that can probably only be
solved by someone at Ubuntu or one of the lxml maintainers.

Thanks again,
Alex

_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Stefan Behnel]

Post by Alex Kleider

I'm a python newbie, but it looks to me like your compiler cannot
find your header files, and in particular pyconfig.h.
I tried searching my system and found a file with that name at
/home/bambams/src/pyenv/versions/2.7.9/include/python2.7/pyconfig.h
/home/bambams/src/pyenv/versions/3.4.2/include/python3.4m/pyconfig.h
/usr/include/python2.6/pyconfig.h
/usr/include/python2.7/pyconfig.h
/usr/include/python3.2mu/pyconfig.h
Based on that I am assuming that you should have a pyconfig.h in
either /usr/include/python3.4m or
/home/alex/P3env/env/include/python3.4m. I would probably start
there to verify that I have that header file in a location where
it's expected (-I flags in the above command, or system include
directories). If not then I would wonder why...

drwxr-xr-x 2 root root 4096 Apr 8 19:05 /usr/include/python2.7
bin lib
Nothing in the env file (generated by virtualenv.)
/usr/bin/python3
It's a mystery why it doesn't come with the corresponding include directory.
I'm guessing this is a system level problem that can probably only be
solved by someone at Ubuntu or one of the lxml maintainers.

It's solved already. :)

All you need to do is install the "-dev" package that goes with your Python
installation, e.g. "python3-dev" should match Python 3.4 in current Ubuntu
releases.

The reason why it's in a separate package is that many people actually
don't need this, e.g. when they only install plain Python packages or use
the Ubuntu provided binary packages that they can install via apt (e.g.
"sudo apt-get install python-lxml").

Stefan


_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Alex Kleider]

Post by Stefan Behnel
It's solved already. :)
All you need to do is install the "-dev" package that goes with your Python
installation, e.g. "python3-dev" should match Python 3.4 in current Ubuntu
releases.
The reason why it's in a separate package is that many people actually
don't need this, e.g. when they only install plain Python packages or use
the Ubuntu provided binary packages that they can install via apt (e.g.
"sudo apt-get install python-lxml").

Thanks Brandon and Sefan. It proved correct that I did not have
python3-dev installed
.. but after intalling it, the pip install lxml still fails!

There's a huge amount of output but what follows might provide clues as
to the current problem:

................

/usr/lib/python3.4/distutils/dist.py:260: UserWarning: Unknown
distribution option: 'bugtrack_url'
warnings.warn(msg)

warning: no previously-included files found matching '*.py'
.......

Cleaning up...
Command /home/alex/P3env/env/bin/python3 -c "import setuptools,
tokenize;__file__='/home/alex/P3env/env/build/lxml/setup.py';exec(compile(getattr(tokenize,
'open', open)(__file__).read().replace('\r\n', '\n'), __file__,
'exec'))" install --record /tmp/pip-56yiw_ye-record/install-record.txt
--single-version-externally-managed --compile --install-headers
/home/alex/P3env/env/include/site/python3.4 failed with error code 1 in
/home/alex/P3env/env/build/lxml
Storing debug log for failure in /home/alex/.pip/pip.log


Looking at ~/.pip/pip.log
there seem to be hundreds of Skipping link ... unknown archive format:
.egg, .exe, lines typified by the following two:
Skipping link
https://pypi.python.org/packages/2.7/l/lxml/lxml-2.3beta1-py2.7-win32.egg#md5=4c0da2b308dd2cd0095de21c26647842
(from https://pypi.python.org/simple/lxml/); unknown archive format:
.egg
Skipping link
https://pypi.python.org/packages/2.7/l/lxml/lxml-2.3beta1.win32-py2.7.exe#md5=ea0224feed6f649cdb1ccb656ab91d81
(from https://pypi.python.org/simple/lxml/); unknown archive format:
.exe

Then it found some tar.gz links (103 of them to be exact)
Then "Ignoring link .." for bout 27 lines and finally
Some possibly useful information:

Using version 3.4.2 (newest of versions: 3.4.2, 3.4.1, 3.4.0, 3.3.6,
3.3.5, 3.3.4, 3.3.3, 3.3.2, 3.3.1, 3.3.0, 3.2.5, 3.2.4, 3.2.3, 3.2.2,
3.2.1, 3.2.0, 3.1.2, 3.1.1, 3.1.0, 3.0.2, 3.0.1, 3.0, 2.3.6, 2.3.5,
2.3.4, 2.3.3, 2.3.2, 2.3.1, 2.3, 2.2.8, 2.2.7, 2.2.6, 2.2.5, 2.2.4,
2.2.3, 2.2.2, 2.2.1, 2.2, 2.1.5, 2.1.4, 2.1.3, 2.1.2, 2.1.1, 2.1,
2.0.11, 2.0.10, 2.0.9, 2.0.8, 2.0.7, 2.0.6, 2.0.5, 2.0.4, 2.0.3, 2.0.2,
2.0.1, 2.0, 1.3.6, 1.3.5, 1.3.4, 1.3.3, 1.3.2, 1.3, 1.2.1, 1.2, 1.1.2,
1.1.1, 1.1, 1.0.4, 1.0.3, 1.0.2, 1.0.1, 1.0, 0.9.2, 0.9.1, 0.9)
Downloading from URL
https://pypi.python.org/packages/source/l/lxml/lxml-3.4.2.tar.gz#md5=429e5e771c4be0798923c04cb9739b4e
(from https://pypi.python.org/simple/lxml/)
Running setup.py (path:/home/alex/P3env/env/build/lxml/setup.py)
egg_info for package lxml
Building lxml version 3.4.2.
Building without Cython.
Using build configuration of libxslt 1.1.28
running egg_info
creating pip-egg-info/lxml.egg-info
writing pip-egg-info/lxml.egg-info/PKG-INFO
writing dependency_links to
pip-egg-info/lxml.egg-info/dependency_links.txt
writing top-level names to pip-egg-info/lxml.egg-info/top_level.txt
writing requirements to pip-egg-info/lxml.egg-info/requires.txt
writing manifest file 'pip-egg-info/lxml.egg-info/SOURCES.txt'
/usr/lib/python3.4/distutils/dist.py:260: UserWarning: Unknown
distribution option: 'bugtrack_url'
warnings.warn(msg)
warning: manifest_maker: standard file '-c' not found

..........

i686-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -g -fstack-protector --param=ssp-buffer-size=4
-Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC
-I/usr/include/libxml2
-I/home/alex/P3env/env/build/lxml/src/lxml/includes
-I/usr/include/python3.4m -I/home/alex/P3env/env/include/python3.4m -c
src/lxml/lxml.etree.c -o build/temp.linux-i686-3.4/src/lxml/lxml.etree.o
-w
i686-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions
-Wl,-z,relro -g -fstack-protector --param=ssp-buffer-size=4 -Wformat
-Werror=format-security -D_FORTIFY_SOURCE=2
build/temp.linux-i686-3.4/src/lxml/lxml.etree.o -lxslt -lexslt -lxml2
-lz -lm -o build/lib.linux-i686-3.4/lxml/etree.cpython-34m.so
/usr/bin/ld: cannot find -lz
collect2: error: ld returned 1 exit status
/usr/lib/python3.4/distutils/dist.py:260: UserWarning: Unknown
distribution option: 'bugtrack_url'
warnings.warn(msg)
error: command 'i686-linux-gnu-gcc' failed with exit status 1
Complete output from command /home/alex/P3env/env/bin/python3 -c
"import setuptools,
tokenize;__file__='/home/alex/P3env/env/build/lxml/setup.py';exec(compile(getattr(tokenize,
'open', open)(__file__).read().replace('\r\n', '\n'), __file__,
'exec'))" install --record /tmp/pip-56yiw_ye-record/install-record.txt
--single-version-externally-managed --compile --install-headers
/home/alex/P3env/env/include/site/python3.4:
Building lxml version 3.4.2.

Building without Cython.

Using build configuration of libxslt 1.1.28

running install

running build

running build_py

creating build

..........


i686-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -g -fstack-protector --param=ssp-buffer-size=4
-Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC
-I/usr/include/libxml2
-I/home/alex/P3env/env/build/lxml/src/lxml/includes
-I/usr/include/python3.4m -I/home/alex/P3env/env/include/python3.4m -c
src/lxml/lxml.etree.c -o build/temp.linux-i686-3.4/src/lxml/lxml.etree.o
-w

i686-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions
-Wl,-z,relro -g -fstack-protector --param=ssp-buffer-size=4 -Wformat
-Werror=format-security -D_FORTIFY_SOURCE=2
build/temp.linux-i686-3.4/src/lxml/lxml.etree.o -lxslt -lexslt -lxml2
-lz -lm -o build/lib.linux-i686-3.4/lxml/etree.cpython-34m.so

/usr/bin/ld: cannot find -lz

collect2: error: ld returned 1 exit status

/usr/lib/python3.4/distutils/dist.py:260: UserWarning: Unknown
distribution option: 'bugtrack_url'

warnings.warn(msg)

error: command 'i686-linux-gnu-gcc' failed with exit status 1

----------------------------------------
Cleaning up...
Removing temporary dir /home/alex/P3env/env/build...
Command /home/alex/P3env/env/bin/python3 -c "import setuptools,
tokenize;__file__='/home/alex/P3env/env/build/lxml/setup.py';exec(compile(getattr(tokenize,
'open', open)(__file__).read().replace('\r\n', '\n'), __file__,
'exec'))" install --record /tmp/pip-56yiw_ye-record/install-record.txt
--single-version-externally-managed --compile --install-headers
/home/alex/P3env/env/include/site/python3.4 failed with error code 1 in
/home/alex/P3env/env/build/lxml
Exception information:
Traceback (most recent call last):
File
"/home/alex/P3env/env/lib/python3.4/site-packages/pip/basecommand.py",
line 122, in main
status = self.run(options, args)
File
"/home/alex/P3env/env/lib/python3.4/site-packages/pip/commands/install.py",
line 283, in run
requirement_set.install(install_options, global_options,
root=options.root_path)
File "/home/alex/P3env/env/lib/python3.4/site-packages/pip/req.py",
line 1435, in install
requirement.install(install_options, global_options, *args,
**kwargs)
File "/home/alex/P3env/env/lib/python3.4/site-packages/pip/req.py",
line 706, in install
cwd=self.source_dir, filter_stdout=self._filter_install,
show_stdout=False)
File "/home/alex/P3env/env/lib/python3.4/site-packages/pip/util.py",
line 697, in call_subprocess
% (command_desc, proc.returncode, cwd))
pip.exceptions.InstallationError: Command
/home/alex/P3env/env/bin/python3 -c "import setuptools,
tokenize;__file__='/home/alex/P3env/env/build/lxml/setup.py';exec(compile(getattr(tokenize,
'open', open)(__file__).read().replace('\r\n', '\n'), __file__,
'exec'))" install --record /tmp/pip-56yiw_ye-record/install-record.txt
--single-version-externally-managed --compile --install-headers
/home/alex/P3env/env/include/site/python3.4 failed with error code 1 in
/home/alex/P3env/env/build/lxml


_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Stefan Behnel]

Post by Alex Kleider

Post by Stefan Behnel
All you need to do is install the "-dev" package that goes with your Python
installation, e.g. "python3-dev" should match Python 3.4 in current Ubuntu
releases.
The reason why it's in a separate package is that many people actually
don't need this, e.g. when they only install plain Python packages or use
the Ubuntu provided binary packages that they can install via apt (e.g.
"sudo apt-get install python-lxml").

Thanks Brandon and Sefan. It proved correct that I did not have
python3-dev installed
.. but after intalling it, the pip install lxml still fails!
There's a huge amount of output but what follows might provide clues as to
[...]
/usr/bin/ld: cannot find -lz

Same thing, you need to make zlib available to the build. Ubuntu calls the
package "zlib1g-dev". And while you're at it, make sure you also have
"libxml2-dev" and "libxslt-dev".

The official installation instructions can be found here, BTW:

http://lxml.de/installation.html

Stefan


_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Peter Otten]

Post by Stefan Behnel

Post by Alex Kleider

Post by Stefan Behnel
All you need to do is install the "-dev" package that goes with your
Python installation, e.g. "python3-dev" should match Python 3.4 in
current Ubuntu releases.
The reason why it's in a separate package is that many people actually
don't need this, e.g. when they only install plain Python packages or
use the Ubuntu provided binary packages that they can install via apt
(e.g. "sudo apt-get install python-lxml").

Thanks Brandon and Sefan. It proved correct that I did not have
python3-dev installed
.. but after intalling it, the pip install lxml still fails!
There's a huge amount of output but what follows might provide clues as
[...]
/usr/bin/ld: cannot find -lz

Same thing, you need to make zlib available to the build. Ubuntu calls the
package "zlib1g-dev". And while you're at it, make sure you also have
"libxml2-dev" and "libxslt-dev".
http://lxml.de/installation.html

I'd try

$ sudo apt-get build-dep python3-lxml

This should install all build dependencies of the python3-lxml package which
are likely the same as those of a manual installation of lxml.

_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Alex Kleider]

Post by Peter Otten
I'd try
$ sudo apt-get build-dep python3-lxml

build-dep is not an apt-get command I've seen before but it did the
trick!

Post by Peter Otten
This should install all build dependencies of the python3-lxml package which
are likely the same as those of a manual installation of lxml.

I can now
import lxml

Thank you very much, indeed!

Alex

_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Alex Kleider]

I tried something different and although not successful, there seems to
be some progress:

----------------------------------------
(env)***@t61p:~/P3env$ CFLAGS="-O0" STATIC_DEPS=true pip install lxml

.....
Exception: Command "make -j3" returned code 512

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in
/home/alex/P3env/env/build/lxml
Storing debug log for failure in /home/alex/.pip/pip.log

The part of .pip/pip.log that might prove useful is as follows:

make[1]: Leaving directory
`/home/alex/P3env/env/build/lxml/build/tmp/libiconv-1.14/srclib'
make: *** [all] Error 2
Building lxml version 3.4.2.
Latest version of libiconv is 1.14
Downloading libiconv into libs/libiconv-1.14.tar.gz
Unpacking libiconv-1.14.tar.gz into build/tmp
Latest version of libxml2 is 2.9.2
Downloading libxml2 into libs/libxml2-2.9.2.tar.gz
Unpacking libxml2-2.9.2.tar.gz into build/tmp
Latest version of libxslt is 1.1.28
Downloading libxslt into libs/libxslt-1.1.28.tar.gz
Unpacking libxslt-1.1.28.tar.gz into build/tmp
Starting build in build/tmp/libiconv-1.14
Traceback (most recent call last):
File "<string>", line 17, in <module>
File "/home/alex/P3env/env/build/lxml/setup.py", line 230, in
<module>
**setup_extra_options()
File "/home/alex/P3env/env/build/lxml/setup.py", line 144, in
setup_extra_options
STATIC_CFLAGS, STATIC_BINARIES)
File "/home/alex/P3env/env/build/lxml/setupinfo.py", line 57, in
ext_modules
multicore=OPTION_MULTICORE)
File "/home/alex/P3env/env/build/lxml/buildlibxml.py", line 348,
in build_libxml2xslt
cmmi(configure_cmd, libiconv_dir, multicore, **call_setup)
File "/home/alex/P3env/env/build/lxml/buildlibxml.py", line 285,
in cmmi
cwd=build_dir, **call_setup)
File "/home/alex/P3env/env/build/lxml/buildlibxml.py", line 268,
in call_subprocess
raise Exception('Command "%s" returned code %s' % (cmd_desc,
returncode))
Exception: Command "make -j3" returned code 512
Complete output from command python setup.py egg_info:
checking for a BSD-compatible install... /usr/bin/install -c

checking whether build environment is sane... yes

....

_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor