TL;DR using relative path as metadata_path to projector will cause TensorBoard cannot find metadata. The correct way is use FQPN (fully-qualified path name, aka absolute path)

How it happened

Follow code copied from TensorFlow official website https://www.tensorflow.org/versions/r1.3/programmers_guide/embedding, which is also the official approach to use TensorFlow’s embedding projector. This is also most people actually do in their code.

    from tensorflow.contrib.tensorboard.plugins import projector

    # Create randomly initialized embedding weights which will be trained.
    vocabulary_size = 10000
    embedding_size = 200
    embedding_var = tf.get_variable('word_embedding', [vocabulary_size, embedding_size])

    # Format: tensorflow/tensorboard/plugins/projector/projector_config.proto
    config = projector.ProjectorConfig()

    # You can add multiple embeddings. Here we add only one.
    embedding = config.embeddings.add()
    embedding.tensor_name = embedding_var.name
    # Link this tensor to its metadata file (e.g. labels).
    embedding.metadata_path = os.path.join(LOG_DIR, 'metadata.tsv')

    # Use the same LOG_DIR where you stored your checkpoint.
    summary_writer = tf.summary.FileWriter(LOG_DIR)

    # The next line writes a projector_config.pbtxt in the LOG_DIR. TensorBoard will
    # read this file during startup.
    projector.visualize_embeddings(summary_writer, config)

Above code binding tensor embedding_var to it’s metadata file location by metadata_path, then the config information will write to disk by summary_writer.

The problem is the path of metadata file assign to embedding.metadata_path is relative path. If current path is log directory (i.e: LOG_DIR in source code), program can find the metadata file. But people usually don’t start TensorBoard in log directory, most people like me usually use TensorBoard --logdir ./log to start TensorBoard, in such condition, TensorBoard read the config and get the location of metadata file, but TensorBoard can not find the file by this location info, this is because location is correct only if your current work path is log directory, which most likely not true. Then TensorBoard cannot find the metadata file, loading process will never be done.

PS Someone also notice this issue and give same solution