Some notes about building Caffe RC3 with Mac OS X 10.11.3, Anaconda, CUDA 7.5, cuDNN 4, Intel MKL and MATLAB R2015b

Environment: Mac OS X 10.11.3, Xcode 7.2, Anaconda Python 2.7.11, CUDA 7.5, cuDNN 4, Intel parallel_studio_xe_2016.1.043 mkl, homebrew boost 1.6.0, homebrew OpenCV 2.4.12, MATLAB R2015b.

Note 1: If your run into this problem afterwards "ld: warning: directory not found for option '-L/opt/intel/mkl/lib/intel64'", you can solve this problem as:

cd /opt/intel/mkl/lib/

sudo ln -s . /opt/intel/mkl/lib/intel64

Suppose the environment is setup as above, first follow the official Caffe OS X Installation guide step by step.

Then modify your .bash_profile file as:

export PATH=/usr/local/bin:/Applications/MATLAB_R2015b.app/bin:/Developer/NVIDIA/CUDA-7.5/bin:$PATH

export PATH="/Users/ylzhao/anaconda/bin:$PATH"

export DYLD_LIBRARY_PATH=/Developer/NVIDIA/CUDA-7.5/lib:$DYLD_LIBRARY_PATH

export DYLD_FALLBACK_LIBRARY_PATH=/usr/local/cuda/lib::$HOME/anaconda/lib:/usr/local/lib:/usr/lib:$DYLD_FALLBACK_LIBRARY_PATH

export PYTHONPATH=/usr/local/lib/python2.7/site-packages:$PYTHONPATH

Note 2: If you run into this problem when invoking "make runtest"afterwards:

dyld: Library not loaded: @rpath/libcudart.7.0.dylib

Referenced from: .../caffe/.build_release/tools/caffe

Reason: image not found

make: *** [runtest] Trace/BPT trap: 5

That is because on Mac OS X El Capitan, DYLD_FALLBACK_LIBRARY_PATH is cleared by the new System Integrity Protection feature of El Capitan. See the discussions 2320 and 3628. One possible way to solve this problem is disabled System Integrity Protection as:

1. Boot to Recovery OS by restarting your machine and holding down the Command and R keys at startup.
2. Launch Terminal from the Utilities menu.
3. Enter the following command: csrutil disable
4. restart your computer

 Then follow the official Caffe compilation with Make guide, my 'Makefile.config' file is:

## Refer to http://caffe.berkeleyvision.org/installation.html

# Contributions simplifying and improving our build system are welcome!

# cuDNN acceleration switch (uncomment to build with cuDNN).

USE_CUDNN := 1

# CUDA directory contains bin/ and lib/ directories that we need.

CUDA_DIR := /usr/local/cuda

# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := mkl

# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
MATLAB_DIR := /Applications/MATLAB_R2015b.app

# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
# PYTHON_INCLUDE := /usr/include/python2.7 \
# /usr/lib/python2.7/dist-packages/numpy/core/include
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
ANACONDA_HOME := $(HOME)/anaconda
PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
$(ANACONDA_HOME)/include/python2.7 \
$(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \

# We need to be able to find libpythonX.X.so or .dylib.
# PYTHON_LIB := /usr/lib
PYTHON_LIB := $(ANACONDA_HOME)/lib

# Uncomment to support layers written in Python (will link against Python libs)
# WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib

My modification parts are labelled as bold.

Then you can start the compilation in Terminal:

make all

make test

make runtest

Note 3: If your "LIBRARY_DIRS" like these:

LIBRARY_DIRS := $(PYTHON_LIB) $(MATLAB_DIR)/bin/maci64 usr/local/lib  /usr/lib

you may run into this protobuf problem afterwards:

Undefined symbols for architecture x86_64:

 "google::protobuf::io::CodedOutputStream::WriteStringWithSizeToArray(std::__1::basic_string, std::__1::allocator > const&, unsigned char*)", referenced from:

caffe::Datum::SerializeWithCachedSizesToArray(unsigned char*) const in caffe.pb.o

This is because Caffe tries to link with MATLAB's protobuf instead of homebrew protobuf, and this will also happen with the boost library and CUDA. (More detail: Caffe was seeing Matlab's internal libraries earlier in the library path than the local homebrew libraries in /usr/local/lib. Caffe might be trying to link Matlab's version of the boost library, which, needless to say, isn't compatible with Caffe.) See discussion 915.

So before compile the Caffe Matlab interface, your "LIBRARY_DIRS" should like:

LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib

After the commands "make all", "make test", "make runtest" successfully, you should modify the "LIBRARY_DIRS" like:

LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib $(MATLAB_DIR)/bin/maci64 /usr/lib

Thus we can avoid the conflict between homebrew's protobuf/boost/cuda libraries and MATLAB's protobuf/boost/cuda libraries.

Then you can build the Caffe Matlab interface as:

make matcaffe

make mattest 

Note 4: If you forget to add "(MATLAB_DIR)/bin/maci64" in "LIBRARY_DIRS" line, you may run into this problem:

Undefined symbols for architecture x86_64:

  "_mxArrayToString", referenced from:

      get_solver(int, mxArray_tag**, int, mxArray_tag const**) in caffe_.o

      solver_restore(int, mxArray_tag**, int, mxArray_tag const**) in caffe_.o

      get_net(int, mxArray_tag**, int, mxArray_tag const**) in caffe_.o

      net_copy_from(int, mxArray_tag**, int, mxArray_tag const**) in caffe_.o

      net_save(int, mxArray_tag**, int, mxArray_tag const**) in caffe_.o

      read_mean(int, mxArray_tag**, int, mxArray_tag const**) in caffe_.o

      write_mean(int, mxArray_tag**, int, mxArray_tag const**) in caffe_.o

      ...

That why we need to change the LIBRARY_DIRS := ... line in your Makefile.config so that the /usr/local/lib directory is before your Matlab library directory as above described.

Note 5: when you compile Caffe Python interface like:

make pycaffe

make pytest 

you may run into this problem:

======================================================================

ERROR: test_forward_backward (test_net.TestNet)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/Users/ylzhao/Project/caffe/python/caffe/test/test_net.py", line 45, in setUp

    size=self.net.blobs['label'].data.shape)

  File "caffe/pycaffe.py", line 28, in _Net_blobs

    return OrderedDict(zip(self._blob_names, self._blobs))

TypeError: No to_python (by-value) converter found for C++ type: boost::shared_ptr >

It seems that this problem is related to Boost 1.60, not 1.59. You can find the discussion at 3494 and 3575, and the fix is at 3575. The commit you want to apply is at 3575 commit.

Unfortunately, this commit has not been merged into Caffe master branch, so you should do the merge by hand.

That's all, have fun with Caffe building!