Results 1 to 4 of 4
Enjoy an ad free experience by logging in. Not a member yet? Register.
- Join Date
- May 2009
Scipy and numpy install on Slackware 13
I have been installing Scipy and Numpy on my slackware 13 distro. As there don't seem to be pre-built packages friendly to Slackware, I am compiling from source.
This first post is going to be a description of what I have learned to help others get an overview of the task -- the next post will be about where I finally came to a problem I couldn't solve easily on my own....
I have done this before on a python 2.6 interpreter, but because I had trouble I tried to upgrade to a python 2.7 interpreter, which is now fully functional (see thread in this same forum on Slackware 12/13 and Python 2.7); but it doesn't really help. The problem appears to be in the distutils of scipy and numpy, and with respect to shared libraries. So, I am studying the issue closer, and wanted to share results & frustration.
The instructions I can find online, generally, stink.
Numpy can be installed without any support packages, but Scipy expects a linear algebra library and a fast Forier transform. The standard ones are FFTW for Forier and LAPACK (Linear Algebra PAckage) for matricies, and for speed reasons ATLAS. (Auto Tuned Linear Algebra Subroutines). These have both fortran and C interfaces, but for full use require fortran. The altals and lapack libraries need to be built before numpy and scipy -- which causes a chicken and egg fortran compiler problem.
Slackware has GNU fortran tools -- but perhaps too many. There are two fortran compilers -- gfortran and g77 (different from f77 alias of gfortran) and apparently they can conflict and make worthless binaries (and waste you time). The numpy installer, however, likes to find and use them both... and the select a compiler methods explained online don't work.... so, because I found that the compiler for both scipy and numpy are selected by the distutils found under the *numpy* package; part of the solution to get the name of the compiler that running ./setup.py build finds and then change to the subdirectory -- numpy-1.6.0/numpy/distutils/fcompiler and there are separate scripts for the various manufacturers/systems. Delete all scripts that locate the compiler you DON'T want. Even though it might be a bit slower, I chose to use gfortran instead of g77 -- because it was easier....
The same compiler has to be used for ATLAS and LAPACK *and* numpy. The linear algebra pack is supposed to make four libraries that are used by numpy and then scipy. Note: even though online documentation suggests that ATLAS or BLAS is *required* to use LAPACK, that isn't true. You can compile LAPACK by itself, and it will provide a very poor -- but operational subroutine library for itself. OTOH, if ATLAS doesn't optimize right -- you're going to be slow anyway -- so you might want to do it in two steps. If you are after a fast compile, just use LAPACK -- the extra steps for ATLAS are a pain.
I compiled ATLAS and LAPACK by doing a dry run configure of ATLAS:
../configure -b32 Si not77 0
This creates a file Make.inc in the build directory. It will list whatever compiler is "best" for your system in it. After noting the compiler and flags in that file, the dummy configuration may be wiped out: rm *
Then one gets and unpacks lapack-3.3.1 and changes into that directory. There is a demo "make.inc" file in it which may be copied to make.inc and then edited to match the one in ATLAS. In mine the changes were:
FORTRAN = gfortran
OPTS = -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 -fpic -m32
DRVOPTS = $(OPTS)
NOOPT = -O0
LOADER = gfortran
LOADOPTS = $(OPTS)
... and the TIMER is set/uncommented to INT_ETIME instead of the default one which gets commented out because I am using gfortran. Notice the -fpic on the OPTS line -- that causes dynamic library compatible code to be generated; it will reduce performance... but, iI wanted the dynamic libs if possible for my first try.
I am running on a 2.4xx GHZ SSE2 instruction set machine.
After that I did a 'make'; which built the lapack libraries and placed them in archived format in the same directory (eg: lapack_LINUX.a and tmglib_LINUX.a). It is important to note, that at this point the libraries are linked against a BasicLinearAlgebraSubroutine that is inferior to ATLAS; and one acutally has to un-archive the linked file and delete/copy over the BLAS objects and replace them with ATLAS objects for full speed.
Supposedly, ATLAS will do that for you automatically by using --with-netlib, but I am not sure it did. However, returning to the ATLAS package and the LinuxBuild directory (which is empty now), I do:
../configure -b 32 -Si nof77 0 --with-netlib-lapack=/home/mydir/lapack-3.3.1/lapack_LINUX.a --prefix=/home/mydir/local
The prefix was worthless, as it doesn't install correctly. Manual copying is better, but that's how I tried it. After it is configured, and I checked to see that gfortran was still in this make.inc (it was) -- I issued a 'make' command which built the atlas objects and appeared to combine them with the LAPACK (HOORAY) -- but they too are still in archive format. (.a). and are found in the ./lib subdirectory. Changing to that directory -- I did a 'make shared' to generate all the shared libraries 'c' based ones and fortran based ones.
There are now four archive style libraries, and four dynamic link libraries:
libatlas.so libcblas.so libf77blas.so liblapack.so
Which I copied to my local ~/local/lib.
I then compiled the fast fourier transform: fftw-3.2.2
In the top directory I did:
./configure --enable-shared --enable-sse2 --enable-float --prefix /home/mydir/local
And it refused to build the float (less precision) library. So I issued it again without --enable-float, and it worked. So, then I did a 'make; make install' and walah... no problems.
Now, to where I got stuck....(next post)
- Join Date
- May 2009
The problem with numpy and shared user libraries....
Once I had the linear algebra and FFTW packages installed, I went into numpy -- deleted the 'g77' compiler detections as previously mentioned, and then did a './setup.py build' which worked just fine. I then did a './setup.py install home=~/local' which also worked fine. So then changing to super user, I did a ldconfig -- (forgetting that my local libs are not in the ldso.config file....); and switching back to normal user (mydir), I tried to run python and test numpy.
It didn't work -- couldn't find the shared libraries...
Checking the libraries themselves with ldd, they all found their dependencies correctly as they were in the system library directories. However, PYTHON couldn't import them with numpy but will give the error:
'ImportError: liblapack.so: cannot open shared object file: No such file or directory'
So, I know the problem is that python doesn't know where to look for them and the dynamic linker 'ldconfig' shouldn't be looking for them because they are user libraries and not system ones.
So, the question I have is -- where do I put these .so files so that python will find them correctly? in the lib/python subtdirectory -- or in the numpy directory or ??? I don't want to install them system wide and I am not sure how to get numpy to link with the archive (.a) files I made or the (.so) files. Either way would be adequate...
Anybody have an idea? (I'll post it if I figure it out.)
- Join Date
- May 2009
Problem solved, I knew it was simple -- I just couldn't remember how....
To get python to find local dynamic libraries, add in your home directory ~/.bashrc
export PYTHONPATH=~/local/lib/python export PYTHONSTARTUP=~/.pythonrc export PATH=$PATH:~/local/bin export PATH export LD_LIBRARY_PATH=~/local/lib
Next, be sure to add the following file (appropriately edited) for local library installation of lapack to your home directory before compiling LAPACK, ATLAS, numpy, scipy.
[DEFAULT] library_dirs = /home/andrew3/.local/lib inclide_dirs = /home/andrew3/.local/include [atlas] library_dirs = /home/my_home_name/local/lib atlas_libs = lapack, f77blas, cblas, atlas [fftw3] library_dirs = /home/my_home_name/local/lib include_dirs = /home/my_home_name/local/include fftw3_libs = fftw3_libs = fftw3, fftw3f
- Join Date
- May 2009
Just a follow up note,
The rest of the install for both Scipy and Numpy went flawlessly. They work quite well together, and I am enjoying them very much. It was worth the struggle.
And rereading my notes, (confusing...eh?) note that numpy, scipy, and python 2.7 additional packages (GTK) are installed in a directory I created .local/lib which I use for all my non-system wide install tools. Python is installed globally, but the Scipy and Numpy packages are installed in my home directory without needing superuser access -- since not even ldso needs to be run if the path's are added as shown above. My instructions above are based on that, so adjust them according to how you install... of course!