PyCharm remote interpreter and Tensorflow -> can not import Cudart.so

I'm using PyCharm for remote debugging, client Win10, server Ubuntu 16. On the Ubuntu machine I have Cuda 8.0 correctly installed, along with python 2.7 and tensorflow 1.0.0. Tensorflow works perfectly from the server (ssh + invoking python, import tensorflow etc..).

In PyCharm I set up a remote debugger to my server machine, but now importing tensorflow results in an error:

Traceback (most recent call last):
File "/home/*****/*****/********/*******.py", line 24, in <module>
import tensorflow as tf
File "/usr/local/lib/python2.7/dist-packages/tensorflow/__init__.py", line 24, in <module>
from tensorflow.python import *
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/__init__.py", line 60, in <module>
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/__init__.py", line 49, in <module>
from tensorflow.python import pywrap_tensorflow
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 28, in <module>
_pywrap_tensorflow = swig_import_helper()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow', fp, pathname, description)
ImportError: libcudart.so.8.0: cannot open shared object file: No such file or directory


Error importing tensorflow. Unless you are using bazel,
you should not try to import tensorflow from its source directory;
please exit the tensorflow source tree, and relaunch your python interpreter
from there.


I tried:

adding the correct paths to the sys.path variable

adding LD_LIBRARY_PATH to os.environ and/or to the PyCharm environment variables

No success!

I recall time ago I was able to do debugging without any problem, but I can't figure out what's wrong. Any suggestion?

UPDATE 1
After rebooting, the error changed to this

ssh://****@192.168.***.***:22/usr/bin/python -u /home/*****/.pycharm_helpers/pydev/pydevd.py --multiproc --qt-support --client '0.0.0.0' --port 35523 --file /home/*/*/*/*/test.py
warning: Debugger speedups using cython not found. Run '"/usr/bin/python" "/home/*/.pycharm_helpers/pydev/setup_cython.py" build_ext --inplace' to build.
pydev debugger: process 1913 is connecting

Connected to pydev debugger (build 163.10154.50)
Traceback (most recent call last):
File "/home/*/.pycharm_helpers/pydev/pydevd.py", line 1596, in <module>
globals = debugger.run(setup['file'], None, None, is_module)
File "/home/*/.pycharm_helpers/pydev/pydevd.py", line 974, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home/*/*/*/*/test.py", line 1, in <module>
import tensorflow
File "/usr/local/lib/python2.7/dist-packages/tensorflow/__init__.py", line 24, in <module>
from tensorflow.python import *
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/__init__.py", line 72, in <module>
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/__init__.py", line 61, in <module>
from tensorflow.python import pywrap_tensorflow
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 28, in <module>
_pywrap_tensorflow = swig_import_helper()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow', fp, pathname, description)
ImportError: libcudart.so.8.0: cannot open shared object file: No such file or directory


Failed to load the native TensorFlow runtime.

See https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/get_started/os_setup.md#import_error

for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.

Process finished with exit code 1


UPDATE 2 / SOLVED
As I tried before, the solution was to add on pycharm the LD_LIBRARY_PATH to the environment variables on the configuration panel. I think before it didn't work for a blank space after the name (-> 'LD_LIBRARY_PATH '). Here the solution:
solution

3 Answers
3

Solution is to add the correct path to the Pycharm environment panel:

LD_LIBRARY_PATH='/usr/local/cuda/lib64'


or the one that suits your configuration.

The error message you get is "please exit the tensorflow source tree, and relaunch your python interpreter from there.".

Normally you get this error message if you download the tensorflow source, compile it manually etc. install it, and then start python and import tensorflow while still in the source folder.

Things you can try to stop this error:
- Check if the tensorflow source is in the folder you are running your remote application in (did you follow something like this to get it working? http://www.pinchofintelligence.com/faster-writing-and-testing-machine-learning-applications/ )
- Try to find and remove the tensorflow source folder
- Restart your pc (this clears the /tmp folder you are probably running your stuff in)

Hopefully one of these things work, let me know!

Remote nvidia-docker case

I spended a couple of hours to solve similar problem. I actually read stackoverflow several times and add all this env variables without any effort.

So if you have remote nvidia-docker with oppened ssh then read this thread:
https://github.com/bethgelab/docker/issues/5

nvidia-docker starts incorrectly terminal when you connect with ssh.

Pycharm actually usses ssh to connect for remote debugging, so it also can't start tensorflow-gpu

Required, but never shown

Required, but never shown

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.