Tableau Server in a Container - Troubleshooting

Introduction

Tableau Server in a Container is Tableau's first container-based server offering. Tableau Server in a Container is an all-in-one Tableau Server instance running inside of a Linux Docker container. In other words, a Tableau Server in a Container image is a docker image that runs an entire self-contained Tableau Server application. Tableau Server in a Container is our first of many steps to support Tableau Server running in container-based environments. The easiest way to understand the concept of Tableau Server in a Container is to think of it like a VM with Tableau Server pre-installed. The image is based on a UBI 8 image (CentOS 7.x for version 2022.1 and earlier) and runs supervisord (instead of systemd) inside the container. When the container starts supervisord, it will immediately attempt to initialize and start Tableau Server. Much of the documentation here aims to describe how to provide configuration and leverage automation so you can run Tableau Server in Docker environments.

The Tableau Server in a Container Image Setup Tool helps you create and customize container images to include custom packages and artifacts. One of the primary functions of the tool is to build the container image and install custom data connectors.

To test out the Tableau Server in a Container Image quickly in a prrof-of-concept scenarios, see Tableau Server in a Container - Quick Start.

Limitations

  • Tableau Server in a Container only supports license activation using Server ATR, which requires the container to have internet access. Therefore, offline activation in an air-gapped environment is not possible.
  • Tableau Server in a Container does not currently support the Resource Monitoring Tool (RMT) agent.
  • Kerberos is not supported in Tableau Server in a Container

Troubleshooting

If you encounter issues when running Tableau Server, there are a number of avenues one can pursue to find a solution. This section covers general Tableau Server troubleshooting advice, such as where to find logs and what they mean. It also covers some specific known scenarios and mitigation paths.

If you are working with Tableau Support to debug an issue, it can be helpful if you provide the following:

  • Tableau Server logs (gathering these logs is explained below).
  • Docker container stdout logs.
  • Dockerfile of Tableau Server (if any customizations have been made).
  • Deployment configuration including:

    • Kubeconfig (or any equivalent deployment config).
    • Static configuration files that configure the Tableau Server Container.

Installation and initialization failures

If you are initializing Tableau Server for the first time, or you have done a fresh install inside a container, the server will not recover by simply restarting the container. Every installation attempt should use a clean data directory. This may mean deleting persistent volume data from previous container runs. If you do this make sure to save logs and information that could be useful for debugging.

Debugging failing installation

Tableau Server containers are designed to exit when an installation failure is encountered. This pattern makes it easy to automate and identify when an install failure has occurred. However, it can make debugging challenging because the container will exit and not leave any examinable runtime state. If you would like to have a debug session inside a running container that is failing during initialization, follow these steps:

  1. Prepare a fresh Tableau Server in a Container deployment.
  2. Configure the container to run with the TSM_ONLY=1 environment variable. The TSM_ONLY=1 environment variable tells Tableau Server to only initialize TSM. This is the equivalent of just running the initialize-tsm script in a standard, non-container installation.
  3. Run the Tableau Server container.
  4. Open a shell inside the container.
  5. You can now run TSM commands, even though Tableau Server has not been initialized. To resume the automation that takes place normally during initialization, execute the tsm-commands script: "${DOCKER_CONFIG}"/config/tsm-commands

Tableau Support and Kubernetes

Tableau Server in a Container can be run using Kubernetes, but it is not a requirement to do so. Our expectation is that most customers will use Kubernetes or one of its associated managed cloud environments (EKS, AKS or GKS) to run and manage Tableau Server in a Container.

Kubernetes can be a complex environment to run and debug and often includes dependencies on individual companies infrastructure and setup. Because of this, Tableau Support cannot help customers resolve Kubernetes (or infrastructure deployment) issues associated with running Tableau Server in a Container. However, Tableau does support running Tableau Server in a Docker container. Therefore, if you are having issues with running Tableau Server in a Container using Kubernetes, Tableau Support can only go as far as validating that the Docker container works properly by itself.

For more information on how to run Tableau Server in a Container using Kubernetes, see this Github site: https://github.com/tableau/tableau-server-in-kubernetes(Link opens in a new window).

Logs

Logs are an essential resource for finding, understanding, and solving problems in Tableau Server. They are useful for helping our support teams find the root the cause of issues you encounter. Logs may also be useful for your own debugging and troubleshooting.

Extracting All Logs

If you need to extract all logs for further debugging or to send to our support teams, there are a couple methods for retrieving this information.

Ziplogs

TSM can create a compressed archive containing all relevant server logs. You can trigger this by running the tsm maintenance ziplogs command. When the command completes it will report the filepath of the log archive. You will need to copy the archive using whatever file transfer method works best for your situation. For details on ziplogs, see tsm maintenance ziplogs.

Example command executed inside container:

tsm maintenance ziplogs
Manual Tar Command

If you cannot run the ziplogs command, for example, if the server does not manage to reach a consistent state, you can still retrieve the logs by executing a tar command inside the container. You will need to copy out the archive using whatever file transfer method works best for your situation.

Example command executed inside container (writes the tar to a temp directory in the container's data directory):

tar -zcvf /var/opt/tableau/tableau_server/temp/<archive_name>.tar.gz \
/var/opt/tableau/tableau_server/data/tabsvc/logs/. \
/var/opt/tableau/tableau_server/supervisord/ \
/var/opt/tableau/tableau_server/data/tabsvc/config/ \
/docker/.metadata.conf \
--exclude='*keystores' --exclude='*.jks' --exclude='*.tks' \
--exclude='*asset_keys.yml' --exclude='*.ks' --exclude='*.ts' \
--exclude='*.crt' --exclude='*cacerts' --exclude='*.key'
Navigating Logs and Debugging Tips

There are common steps to diagnosing most problems in Tableau Server. If you are thinking of taking a look at your server logs, it can be helpful to break down what information to look for depending on where in the server life cycle the error occurred.

Starting The Container (initial / install)

If the container is crashing immediately or failing to install or initialize, check the following resources:

Container stdout

Examine stdout for the docker container. This is most often accessible by looking at the container output collected by your container orchestration system (for example, Kubernetes). Because Tableau Server is a multi-process system running inside a container, stdout is often not useful and will not report the root cause of the problem, unless there are catastrophic failures on startup. It is recommended you check stdout of the failing container before digging further into Tableau Server logs.

Example:

docker logs <container-name>

Tableau Server Container Startup Log

The Tableau Server Container startup log captures output from automation that is initializing, configuring, and starting Tableau Server. If you find your container is encountering issues while booting or running for the first time, this is the first log to check:

/var/opt/tableau/tableau_server/supervisord/run-tableau-server.log

Check the bottom of the log and see if there is a reported failure. Sometimes the error will be reported and be immediately obvious from the log. If the error is not clear from the log, it's possible the root cause is only visible in a stage-specific or service-specific log file. The logs listed below cover these possibilities.

Tableau Server Install Log

If the startup log indicates there was an issue with the automation handling the initialize TSM stage, check this log:

/var/opt/tableau/tableau_server/logs/app-install.log

Tableau Server Controller Log

If the startup log indicates there was an issue with the initializing and starting Server stage (CLI only), check the tabadmincontroller service log:

/var/opt/tableau/tableau_server/data/tabsvc/logs/tabadmincontroller/tabadmincontroller_node1-0.log

This log file is for a specific service called tabadmincontroller. Tabadmincontroller is responsible for orchestrating the initialization and startup functionality in Server. This log can be complex and verbose. Errors in this log file may still not point to the root cause. Sometimes the errors are caused by services that tabadmincontroller is relying on to complete a certain task. Check the Server runtime section below for more details.

Service Logs - Server Runtime

If Tableau Server encounters issues during normal runtime or issues with services failing to complete tasks or are down, you can check service logs for more information. Every service running as part of Tableau Server has a service log file. If you know which service you would like to examine, you can find that service's logs under this general directory:

/var/opt/tableau/tableau_server/data/tabsvc/logs/<service_name>

Provide the name of the service in the <service_name> arg of the file path. Any service can write multiple kinds of logs files. Also, if you have more than one of the same service running (more than one instance), all service logs will be written into the same service directory.

General Service-Specific Log File Classifications

This table covers the most common Service Log file names, types, and descriptions for Tableau Server services. The "Failure types" column indicates which log files are likely to be useful in a given failure scenario.

Name Filename format Description Failure types Example
Control-App control_<service_name>_<node_id>-<instance_id>.log Contains information of the control-app process which is responsible for installing and configuring a service. This is often the first log written related to a service. For service install and configure failures, look here first. Install, Configure, Status control_backgrounder_node1-0.log
Service log <service_name>_<node_id>-<instance_id>.log Primary log for a running service. Most often this log contains output from the spring/java application layer. Start, Runtime, Status backgrounder_node1-1.log
Stdout log stdout_<service_name>_<instance_id>.log Contains stdout output for the service. Most services do not output much content to stdout and instead write to the primary log. Sometimes this log can contain useful information when a service exits. Start, Stop stdout_backgrounder_0.log
NativeAPI log nativeapi_<service_name>_<instance_id>.txt Some services run a native code layer. This log captures that portion of the application's runtime. Licensing, Start, Runtime, Status nativeapi_backgrounder_1-1_2021_05_10_00_00_00.txt
Tomcat log tomcat_<service_name>_<node_id>-<instance_id>.log This is only for services that run inside a tomcat container and contains tomcat logs. It rarely provides information regarding service failure. It can be useful to debug some network issues. Network, Start tomcat_backgrounder_node1-0.2021-05-10.log
Stopped Container

If the container is stopped or is otherwise difficult to execute commands in, you can still look at the logs if the data directory of the server is externalized to a mounted volume. Otherwise, only the stdout of the container will be examinable in the container's orchestration system, which is often not going to contain the root cause.

Failure to set authentication properties

There appears to be an issue with setting authentication properties in Tableau Server without the identity store getting set first. To work around this issue just set the identity store in the pre-initialization hook.

  1. Create a file called ./customer-files/pre_init_command in the Tableau Server Image Build Tool customer-files directory and edit it to contain:

    #!/bin/bash
    tsm configuration set -k wgserver.authenticate -v local --force-keys
  2. Set the script to be executable.

    chmod +x ./customer-files/pre_init_command
  3. Build and run the image.

Failure during fresh startup (e.g. why isn't Tableau Server starting?)

  • If you are encountering issues with Tableau Server initializing or starting up there are a number of troubleshooting options that may help uncover the issue.
  • If the container cannot start at all, you'll want to check the stdout from the PID 1 process using docker logs <container-name> command.
  • If the container is running but Tableau Server does not seem to be initializing or running properly, the second place to check for errors is this file:
${DATA_DIR}/supervisord/run-tableau-server.log

Example:

docker exec -it <container-name> bash -c 'cat $DATA_DIR/supervisord/run-tableau-server.log'

This log file contains all events orchestrated by the tableau container initializing service that is handling the startup of tableau server as well as executing any setup scripts or custom configuration that you may have provided in the container. Most startup errors will report issues here. Sometimes if the error is related to a TSM or Tableau Server process it will suggest another log file to look at for more detailed information.

Failure during restart or starting a container with existing data

Server Won't Start PostGRES (or other processes)

When data is persisted outside the container and you are starting another Tableau Server in a Container image instance using that old data, it is important to note that the internal hostname of the new container must match the hostname of the container that initialized the persisted data. Tableau Server does not handle dynamic hostname changes well and starting up a new container with a different internal hostname is effectively causing that scenario.

The remedy is to simply make sure the container's hostname is set to the same value as the container that was previously running with that data. This is not to be confused with Multi-node, workers can (and probably should) have different hostnames from each other. What matters is when a given container is restarted or killed the subsequent container must have the same hostname as its predecessor.

Deployment Configuration Examples

Docker

Tableau Server in a Container Basic Usage
docker run \
-e LICENSE_KEY=<key>
-p 8080:8080
--hostname=<static (internal) name of host machine> \
-d <Tableau Server in a Container image ID or tag>
Tableau Server in a Container Basic Usage with Automated Initial Admin User
docker run \
-e LICENSE_KEY=<key> \
-e TABLEAU_USERNAME=<myadmin> \
-e TABLEAU_PASSWORD_FILE=/etc/tableau-admin-secret \
-v <full-path-to-pw-file>:/etc/tableau-admin-secret \
--hostname=<static (internal) name of host machine> \
-p 8080:8080 -d <Tableau Server in a Container image ID or tag>
TSM only mode
docker run \
-e TSM_ONLY=1 \
-p 8080:8080 -d <Tableau Server in a Container image ID or tag>
Multi-Node Basic Usage
Initial Node

Option 1: Use this if the server configuration (CONFIG_FILE) specifies a multi-node topology:

docker run \
-v <network-shared-directory>:/docker/config/bootstrap \
-v <full-path-to-config-file>:/docker/config/config.json:ro \
-e LICENSE_KEY=<key> \
-p 8080:8080 -p 8800-9000:8800-9000 -p 27000-27010:27000-27010 \
--hostname=<name-of-host-machine> \
-d <Tableau Server in a Container image ID or tag>

Option 2: Use this if you want a multi-node deployment even if server configuration does not specify multi-node topology:

docker run \
-v <network-shared-directory>:/docker/config/bootstrap \
-e LICENSE_KEY=<key> -e ALWAYS_WRITE_BOOTSTRAP_FILE=1 \
-p 8080:8080 -p 8800-9000:8800-9000 -p 27000-27010:27000-27010 \
--hostname=<name-of-host-machine> \
-d <Tableau Server in a Container image ID or tag>
Additional node
docker run \
-v <network-shared-directory>:/docker/config/bootstrap \
-e BOOTSTRAP_INSTALL=1 \
-p 8080:8080 -p 8800-9000:8800-9000 \
--hostname=<static (internal) name of host machine> \
-d <Tableau Server in a Container image ID or tag>
Externalize Data Usage
docker run \
-v <empty-data-dir>:/var/opt/tableau \
-e LICENSE_KEY=<key> \
---hostname=<static (internal) name of host machine> \
-p 8080:8080 -d <Tableau Server in a Container image ID or tag>
Init-Container Basic Usage

Init Container

docker run \
-v <empty-data-dir>:/var/opt/tableau \
-e LICENSE_KEY=<key> \
-e INIT_CONTAINER=1 \
--hostname=<static (internal) name of host machine> \
-p 8080:8080 -d <Tableau Server in a Container image ID or tag>

Run Container

docker run \
-v <empty-data-dir>:/var/opt/tableau \
--hostname=<static (internal) name of host machine> \
-p 8080:8080 -d <Tableau Server in a Container image ID or tag>
Basic Restore from Backup Single-Node
docker run \
-v <full-path-to-backup-file>:/docker/config/backup/backup-file.tsbak \
-v <full-path-to-config-only-file>:/docker/config/config.json:ro \
-e LICENSE_KEY=<key> \
-p 8080:8080 -d <Tableau Server in a Container image ID or tag>

Docker-Compose

version: '3.2'
services:
    tableau-server:
         hostname: localhost
         volumes:
              - <your-tsm-command-file>:/docker/config/tsm-commands:ro
              - <your-config-file >:/docker/config/config.json:ro
         ports:
              - "8080:8080"
         image: ${IMAGE_NAME}
         environment:
              - LICENSE_KEY=<license-key>

 

 

Thanks for your feedback!Your feedback has been successfully submitted. Thank you!