================ On Premise Setup ================ .. contents:: Table of Contents :local: Hardware requirements ===================== Hardware requirements are highly dependent on the number of active users simultaneously working on documents inside the single SMASHDOCs installation. Under the active user we understand the user who is typing text in the Editor, using copy-paste, moving sections around, writing comments and so on. **Minimal hardware requirements are:** One server/VM with latest version of Ubuntu, CentOS, Redhat, Debian or any other Linux 64-bit distribution, 2 CPU cores, 10GB RAM, 160GB storage (SSD is preferred) Our performance tests with version 2.23.4 are showing following results regarding required hardware in term of active users count: +------------------------+------------------+----------+------------------------------------+ | Number of active users | CPU | RAM | Storage | +========================+==================+==========+====================================+ | 20 | 2 cores | 10GB | 160GB storage, SSD is preferred | +------------------------+------------------+----------+------------------------------------+ | 50 | 4 cores | 12GB | 160GB storage, SSD is preferred | +------------------------+------------------+----------+------------------------------------+ | 100 | 8 cores | 16GB | 160GB storage, SSD is mandatory | +------------------------+------------------+----------+------------------------------------+ | 300 | 16 cores | 32GB | 160GB storage, SSD is mandatory | +------------------------+------------------+----------+------------------------------------+ | >300 | Please contact support@smashdocs.net | +------------------------+------------------------------------------------------------------+ Software Requirements ===================== To setup a SMASHDOCs On-premise installation you will need: * Access to the private SMASHDOCs docker hub repository (Please contact support@smashdocs.net) * `Docker engine`_ (Version > 17.05.0-ce) * `Docker compose`_ (Version > 1.21.2) * SSL certificates for your Domain * Your business logos in the correct sizes: ======================== ================================================== default_logo .. image:: _static/default_logo.jpg :alt: default_logo.jpg :class: smashdocs-logo brand_logo .. image:: _static/brand_logo.png :alt: brand_logo.png :class: smashdocs-logo email_logo .. image:: _static/email_logo.png :alt: email_logo.png :class: smashdocs-logo favicon .. image:: _static/favicon.png :alt: favicon.png :class: smashdocs-logo ======================== ================================================== `Docker engine`_ is required to complete this installation. We deliver our services as docker containers, so this is a prerequisite for running SMASHDOCs. If you do not already have docker installed, obtain `docker engine`_ on your operating system of choice. .. _`docker engine`: https://docs.docker.com/engine/installation/ `Docker compose`_ is a required tool to compose and orchestrate the SMASHDOCs environment. If you do not already have docker compose installed, obtain `docker compose`_ on your operating system of choice. .. _`docker compose`: https://docs.docker.com/compose/install/ Docker Compose set-up ===================== System Architecture ------------------- .. image:: _static/smashdocs_architecture.png :alt: smashdocs_architecture.png .. toctree:: :glob: A basic SMASHDOCs installation consists of the following few docker images: 1. mongo:4.4 2. redis:5 3. registry.smashdocs.net/smashdocs/nginx:2.23.4 A standard Nginx Webserver Version 1.17.4 with an nginx-upload-module and the SMASHDOCs nginx config file 4. registry.smashdocs.net/smashdocs/backend:2.23.4 The SMASHDOCs Backend which will be used in 4 Docker Containers (Beat, Worker, Backend) 5. registry.smashdocs.net/smashdocs/frontend:2.23.4 The SMASHDOCs Frontend 6. registry.smashdocs.net/smashdocs/socketserver:2.23.4 The SMASHDOCs Socket Server 7. registry.smashdocs.net/smashdocs/minio:latest Asset management To ensure an easy setup of a SMASHDOCs installation, we are providing a docker compose file: ``. This docker compose file will spawn all docker containers which are needed for one SMASHDOCs Instance. Certificates ------------ SMASHDOCs can be served via SSL (by default) or as a plain HTTP. For the SSL setup (it can be a wildcard certificate or any other) you must ensure that your SSL certificates are placed in the the ``/opt/smashdocs/certs/`` folder. In our example below we use ``wildcard.crt`` and ``wildcard.key`` names, but it can be any other name. Just be sure the correct name is used in compose environment variable (nginx-proxy container). For plain HTTP Setup (for example in a case of external Load Balancer) please set following environment variable in nginx-proxy container: .. code-block:: yaml services: nginx-proxy: ... environment: - "SSL_DISABLE=true" If you don't have signed SSL certificates at hand you can generate a self-signed certificate as follows: .. hint:: If you dont have SSL certs and quickly need to create some selfsigned ones try: $(Host) cd /tmp $(Host) openssl req -x509 -newkey rsa:2048 -keyout wildcard.key -out wildcard.crt -days XXX \ -nodes -subj "/C=DE/ST=Bayern/L=Muenchen/O=Smashdocs Example/OU=Org/CN=*.smashdocs.local" .. code-block:: shell $(Host) mkdir -p /opt/smashdocs/certs/ $(Host) cp -R /tmp/* /opt/smashdocs/certs/ Traffic Routing and required web servers --------------------------------------- SMASHDOCs is using Nginx to route and to server all the traffic to an application. **1: Nginx:** The "Nginx-Proxy" docker container accepts incoming TCP (Port 4434) and HTTP (Ports 80, 443) traffic and is responsible for the SSL termination (or not in a case of SSL protocol is disabled). SMASHDOCs ships the NGINX-Proxy with a default nginx.config for easy installation and usage. Some of the properties in the Nginx default config will be overwritten by defining the environment variables by the docker entryfile. **2: Frontend** The "frontend" docker container serves the frontend HTML, CSS and JavaScript code served by Nginx **3: Backend** The "backend" docker container serves the backend. The backend traffic is routed by Nginx **4: Socket Server** The "socketserver" docker container serves the socket server. The socket server traffic is routed by Nginx Docker container setup ---------------------- The docker-compose.customer.yml file is an example docker compose file which sets up a SMASHDOCs Installation. If you setup you own SMASHDOCs instance, make sure you change the configuration accordingly to your desired domain. Download: :download:`docker-compose.customer.yml <_static/on_premise/docker-compose.customer.yml>` .. code-block:: yaml version: "2.4" networks: customer: volumes: mongo-data: minio-data: services: nginx-proxy: image: "registry.smashdocs.net/smashdocs/nginx:2.23.4" mem_limit: 256m user: root restart: always volumes: - "/opt/docker/ssl/certs:/etc/nginx/certs:ro" environment: - "SSL_CERT=wildcard.crt" - "SSL_KEY=wildcard.key" networks: - customer ports: - 80 - 443 - 4434 frontend: image: "registry.smashdocs.net/smashdocs/frontend:2.23.4" mem_limit: 256m user: root restart: always networks: - customer environment: - "SERVICE_NAME=customer-frontend" - "API_URL=https://customer.smashdocs.net/api" - "API_KEY=" - "ERRORREPORTING_ENABLED=true" - "ERRORREPORTING_SENTRY_DSN=" - "ERRORREPORTING_SENTRY_SERVER_NAME=customer-frontend" - MODE=${MODE_customer:-normal} - "SMARTDOCUMENTS_BRANDING=legal" adminui: image: "registry.smashdocs.net/smashdocs/admin-ui:2.23.4" mem_limit: 256m user: root restart: always networks: - customer environment: - "API_URL=https://customer.smashdocs.net/api" backend: image: "registry.smashdocs.net/smashdocs/backend:2.23.4" mem_limit: 5g user: root restart: always networks: - customer environment: - "DATABASE_DATABASE=customer" - "DATABASE_ADDRESS=mongodb://mongodb:27017/customer" - "API_URL_API_URL=https://customer.smashdocs.net/api" - "HTTP_SERVER_ADDRESS=http://customer.smashdocs.net" - "HTTP_SERVER_SSL_ADDRESS=https://customer.smashdocs.net" - "ERRORREPORTING_REPORTING_ENABLED=true" - "ERRORREPORTING_SENTRY_DSN=" - "ERRORREPORTING_SENTRY_SERVER_NAME=customer-backend" - "PROVISIONING_ENABLED=true" - "PROVISIONING_KEY=" - "EXAMPLE_DOC_ENABLED=false" - "EXAMPLE_DOC_USER_EMAIL=admin@smashdocs.net" - "EMAIL_SMTP_SERVER_ADDRESS=" - "EMAIL_SMTP_SERVER_PORT=" - "EMAIL_SMTP_USERNAME=" - "EMAIL_SMTP_PASSWORD=" - "LOCAL_ENABLED=true" - "CELERY_ENABLED=true" - "SYSTEM_ROLE=production" - "LOG_LEVEL=DEBUG" - "PROMETHEUS_METRICS_ENABLED=true" - "TRANSFORM_SERVICE_ADDRESS=transform-service:50051" - "SMART_DOCUMENTS_BRIDGE_ADDRESS=smart-documents-bridge:50052" - "SDOX2HTML_TRANSFORM_SERVICE_ADDRESS=sdox2html-transform-service:50051" - "SIGNATURIT_TOKEN=" - "CLOUDCONVERT_TOKEN=" - 'SIGNATURIT_TEMPLATE_UUID=' - "SPELL_CHECKER_LANGUAGE=en-US" depends_on: - minio - nginx-proxy - mongodb - redis socketserver: image: "registry.smashdocs.net/smashdocs/socketserver:2.23.4" mem_limit: 1g user: nobody restart: always networks: - customer environment: - "REDIS_URL=redis://redis:6379" - "SM_SPC_SIO_REDIS_BUS=redis://redis:6379/0" - "SM_SIO_BE_AUTH_URL=http://backend:8080/users/document-authenticate" depends_on: - redis - backend beat: image: "registry.smashdocs.net/smashdocs/backend:2.23.4" mem_limit: 512m user: nobody restart: always networks: - customer command: "beat" environment: - "SERVICE_NAME=beat-customer" - "SERVICE_TAGS=smashdocs,rest" - "CELERY_ENABLED=true" - "REDIS_URL=redis://redis:6379" - "CELERY_BEAT_SCHEDULE_PATH=/usr/local/data/celery/smashdocs_beat_schedule" depends_on: - redis worker: image: "registry.smashdocs.net/smashdocs/backend:2.23.4" mem_limit: 512m user: nobody restart: always networks: - customer command: "worker" environment: - "DATABASE_DATABASE=customer" - "DATABASE_ADDRESS=mongodb://mongodb/customer" - "DATABASE_PORT=27017" - "API_URL_API_URL=https://customer.smashdocs.net/api" - "HTTP_SERVER_ADDRESS=http://customer.smashdocs.net" - "HTTP_SERVER_SSL_ADDRESS=https://customer.smashdocs.net" - "CELERY_ENABLED=true" - "CELERY_BROKER=redis://redis:6379/0" - "CELERY_BACKEND=redis://redis:6379/0" - "CELERY_BEAT_SCHEDULE_PATH=/usr/local/data/celery/smashdocs_beat_schedule" - "MINIO_ACCESS_KEY=" - "MINIO_SECRET_KEY=" - "EMAIL_SMTP_SERVER_ADDRESS=" - "EMAIL_SMTP_SERVER_PORT=" - "EMAIL_SMTP_USERNAME=" - "EMAIL_SMTP_PASSWORD=" depends_on: - mongodb - redis spellcheck: image: "registry.smashdocs.net/smashdocs/spellcheck:develop" mem_limit: 1g user: nobody restart: always networks: - customer environment: - "SERVICE_NAME=customer-spellcheck" - "REDIS_URL=redis://redis:6379" - "SM_SPC_SIO_REDIS_BUS=redis://redis:6379/0" - "SM_SPC_SIO_EVENT_NAME=search_results" - "SM_SPC_LANGUAGETOOL_URL=http://languagetool:8010/v2/check" - "SM_SPC_CONCURRENCY=5" - "SM_SPC_THREAD_POOL=5" - "SM_SPC_CACHE_TTL=86400" languagetool: image: "registry.smashdocs.net/smashdocs/languagetool" mem_limit: 3g cpuset: 0-3 user: root restart: always networks: - customer environment: - "SERVICE_NAME=customer-languagetool" - "JAVA_OPTS=-Xms1024m -Xmx2 redis: image: "redis:3.2" mem_limit: 512m user: redis restart: always networks: - customer mongodb: image: "mongo:4.4" mem_limit: 5g user: root restart: always networks: - customer ports: - 27017 volumes: - "mongo-data:/data/db:rw" command: "--storageEngine wiredTiger" minio: image: "registry.smashdocs.net/smashdocs/minio:latest" command: server /export restart: always volumes: - "minio-data:/export:rw" networks: - customer environment: - "MINIO_ACCESS_KEY=" - "MINIO_SECRET_KEY=" transform-service: image: "registry.smashdocs.net/smashdocs/transform-service:2.23.4" mem_limit: 4g networks: - customer environment: - "JAVA_OPTS=-Xms256m -Xmx2048m" sdox2html-transform-service: image: "registry.smashdocs.net/smashdocs/sdox2html-transform-service:2.23.4" user: root restart: always networks: - customer External Database Setup ~~~~~~~~~~~~~~~~~~~~~~~ SMASHDOCs can be used with an existing MongoDB Database Server. In this case the dependencies in the example docker compose file above need to be removed. The following environment variables can be used in: * Backend * Worker * Socketserver =============================== ========= ====================== Name Type Description =============================== ========= ====================== DATABASE_ADDRESS String Database Address (default: 127.0.0.1) DATABASE_PORT String Database Port (default: 27017) DATABASE_USER String Database User (empty string for no authentication) DATABASE_PASSWORD String Database Password (empty string for no authentication) DATABASE_DATABASE String Database Name (required) =============================== ========= ====================== E-Mail Server Setup ~~~~~~~~~~~~~~~~~~~ For your own SMASHDOCs installation you can use your own SMTP server to send emails to your customers. The following environment variables can be configured in: * Worker =============================== ========= ====================== Name Type Description =============================== ========= ====================== EMAIL_ENABLED Boolean `True/False` value to enable or disable sending E-mail out of SMASHDOCs completely. `True` by default. EMAIL_WHITELIST String E-mail whitelist pattern (e.g. `=.+smashdocs[.]net`) to whitelist all outgoing E-mail traffic. No while-list by default. EMAIL_SMTP_SERVER_ADDRESS String E-Mail SMTP Address (default SMASHDOCs Mail Server) EMAIL_SMTP_SERVER_ADDRESS String E-Mail SMTP Address (default SMASHDOCs Mail Server) EMAIL_SMTP_SERVER_PORT String E-Mail SMTP Port (default 587) EMAIL_SMTP_USERNAME String SMTP Username (empty string for no authentication) EMAIL_SMTP_PASSWORD String SMTP Password (empty string for no authentication) EMAIL_STANDARD_EMAIL String Sender E-Mail Address (default: no-reply@smashdocs.net) EMAIL_STANDARD_FROM String Sender E-Mail Display Text (default: SMASHDOCs Team) =============================== ========= ====================== Spell-checker Setup ~~~~~~~~~~~~~~~~~~~ To set the default document language, in the `docker-compose.yml` file in the `backend` container, you need to add the variable `SPELL_CHECKER_LANGUAGE=en-US`, where any language can be substituted for `en-US` (`de-DE`, `fr-FR`, ...) Step 1: Prepare the configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Copy the Docker compose example setup file from above and place it to your hosts file system as `/opt/docker-compose/docker-compose.smashdocs.yml`. The example configuration contains the domains smashdocs.example.com These should be changed to reflect your environments needs. .. code-block:: shell $(Host) sed -i -- "s/customer.smashdocs.net/smashdocs.yourdomain.net/g" /opt/docker-compose/docker-compose.smashdocs.yml Step 2: Replace the Provisioning key ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ SMASHDOCs has a Provisioning API (``__) which can be used to configure a SMASHDOCs installation. The provisioning key is a random key a partner can choose by himself and enable/disable to his needs. For security reasons we advice to enable the Provisioning API only if need. In this example setup the provisioning key is generated using a python expression piped by sha256sum and written to the ``PROVISIONING_KEY` env variable .. code-block:: shell $(Host) export PROVISIONING_KEY=`python -c "import random; print random.randint(5,1000000)" | sha256sum | awk '{print $1}'` $(Host) sed -i -- "s/REPLACE_PROVISIONING_KEY/$PROVISIONING_KEY/g" /opt/docker-compose/docker-compose.smashdocs.yml Step 3: Replace the Minio Credentials ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ SMASHDOCs uses Minio Asset Management for storing files (e.g. images, templates or inline files). Minio credentials are random pair of strings. In this example setup the Minio credentials are generated using a bash expression and written to the ``MINIO_ACCESS_KEY`` and ``MINIO_SECRET_KEY`` env variables .. code-block:: shell $(Host) MINIO_ACCESS_KEY=$(head /dev/urandom | LC_ALL=C tr -dc A-Za-z0-9 | head -c 16) $(Host) MINIO_SECRET_KEY=$(head /dev/urandom | LC_ALL=C tr -dc A-Za-z0-9 | head -c 32) $(Host) sed -i -- "s||$MINIO_ACCESS_KEY|g" /opt/docker-compose/docker-compose.smashdocs.yml $(Host) sed -i -- "s||$MINIO_SECRET_KEY|g" /opt/docker-compose/docker-compose.smashdocs.yml Step 4: Select the Frontend MODE ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ SMASHDOCs can be run in 2 different modes: Standalone and Partner mode. 1. In Standalone mode (config variable ``"MODE=normal"``) a user can create an account and login. The user will see a document list and can create and open documents 2. In Partner mode (config variable ``"MODE=partner"``) the system can only be accessed via the Partner API `` .. code-block:: shell frontend: ... environment: ... - "MODE=normal" - "MODE=partner" Step 5: Authenticate with Dockerhub ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This step requires a contract with SMASHDOCs. The containers which are required to run a SMASHDOCs environment are located in a protected private registry. Please contact SMASHDOCs if you require authentication. .. code-block:: shell $(Host) docker login -u -p registry.smashdocs.net Step 6: Run the docker compose file ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ On the host system run the following: .. code-block:: shell $(Host) /usr/local/bin/docker-compose -f /opt/docker-compose/docker-compose.smashdocs.yml -p smashdocs up -d Wait until all docker containers are spawned. See running docker containers using: .. code-block:: shell $(Host) /usr/local/bin/docker-compose -f /opt/docker-compose/docker-compose.smashdocs.yml -p smashdocs ps Any changes to the compose file will be re-run on the specific parts of the configuration on consecutive executions. Docker compose will restart the changed service and dependant services. Step 7: Configure the DNS ~~~~~~~~~~~~~~~~~~~~~~~~~ The domain names chosen above are required to be resolvable from hosts using the SMASHDOCs system. For different customers this step will be quite different. SMASHDOCs consultation can be acquired. Kubernetes setup for Azure (AKS) ================================ This guide describes Kubernetes cluster provisioning using Azure managed kubernetes-as-a-service (AKS), and SMASHDOCs app install on Kubernetes (k8s) cluster. The main architectural considerations are: - all SMASHDOCs app components: frontend, adminui, backend, socketserver, beat, worker, nginx, redis, mongodb are running in k8s cluster - one k8s pod running one SMASHDOCs app component - one k8s service is created for each SMASHDOCs app component - nginx ingress controller is used for http/https termination - kube-lego is used for SSL certificate generation - persistent data stored at Azure storage, using Disk Storage and Azure Files services At the end of this guide you will have working instance of SMASHDOCs app in Azure cloud with public HTTPS access. Install CLI utilities --------------------- The following prerequisites are required for installing AKS - ``az`` - azure CLI utility - ``kubectl`` - Kubernetes CLI utility - ``helm`` - Kubernetes package manager client - ``jq`` - JSON parsing utility Step to install on MacOS: :: $ brew install azure-cli $ brew install kubernetes-cli $ brew install kubernetes-helm $ brew install jq in Linux use appropriate package manager, apt/yum Prerequisite steps ------------------ Before creating k8s AKS cluster, we need to make changes in our Azure account: - sign in using ``az`` CLI utility - enable write permissions for Azure account - register Azure providers in namespaces: Compute, Network, Storage Those steps are done once, before cluster creation. Sign in using ``az`` CLI utility. This step is required for further Azure resource provisioning from CLI. :: $ az login To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code B4JGSFDGSV to authenticate. Enable write permissions as described here - https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal Make sure that Azure providers available from your account, have proper permissions for cloud resource provisioning. This command will list available Azure providers with respective status - Registered/NotRegistered :: $ az provider list --query "[].{Provider:namespace, Status:registrationState}" --out table The providers need to be registered in following namespaces: Compute, Network, Storage to be able to create resources. :: # register providers in namespaces $ az provider register --namespace Microsoft.Compute $ az provider register --namespace Microsoft.Network $ az provider register --namespace Microsoft.Storage Create K8S in AKS ----------------- Azure Kubernetes Service is supported on following locations: - East US - West Europe - Central US - Canada Central - Canada East Check https://docs.microsoft.com/en-us/azure/aks/container-service-quotas for updates You need to choose Azure location, in which SMASHDOCs app in AKS will be deployed. List short names for all locations, and pick short name for desired one. :: # list short name for all Azure locations $ az account list-locations | jq .[].name We will use ``canadacentral`` Pick desired resource group name and create it in Azure location, we will use ``k8s-cc-gr`` as group name and create resources group in Canada Central Azure location. :: # list resource groups you already have $ az group list # create resource group at desired location for AKS deployment $ az group create --name k8s-cc-gr --location canadacentral # make sure it's created $ az group list # show newly created group details $ az group show --name k8s-cc-gr Choose node flavor at Azure website - https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes-general We will use ``Standard_B2ms`` node flavor - 2 vCPU, 8G RAM, 16G SSD Pick a name for k8s cluster, we will use ``k8s-cc-test`` Following command will create it with specified name, node count, and node flavor. It may take some time to finish provisioning. :: # create k8s cluster $ az aks create --resource-group k8s-cc-gr --name k8s-cc-test --node-count 3 --node-vm-size Standard_B2ms --generate-ssh-keys # show details about newly created cluster $ az aks show --resource-group k8s-cc-gr --name k8s-cc-test [JSON output with k8s Azure location, name, provisioningState, resourceGroup] Look for JSON keys ``"provisioningState": "Succeeded"`` in output, it means cluster is ready to use. Get k8s cluster credentials --------------------------- To use k8s cluster we need to get it’s credentials (certificates), this command will put it in ``kubectl`` config located at ``$HOME/.kube/config`` :: # get k8s credentials and merge it to kubectl config $ az aks get-credentials -g k8s-cc-gr -n k8s-cc-test k8s cli tool ``kubectl`` allows managing multiple k8s cluster, local/dev/staging/prod for example, in this step we will view available k8s cluster contexts, select context for future use, and check cluster connectivity :: # view available contexts $ kubectl config get-contexts * k8s-cc-test k8s-cc-test clusterUser_k8s-cc-gr_k8s-cc-test # use specified context $ kubectl config use-context k8s-cc-test Switched to context "k8s-cc-test". # view cluster nodes $ kubectl get nodes NAME STATUS ROLES AGE VERSION aks-nodepool1-24768417-0 Ready agent 20h v1.9.6 aks-nodepool1-24768417-1 Ready agent 20h v1.9.6 aks-nodepool1-24768417-2 Ready agent 20h v1.9.6 # view cluster pods (containers) $ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system azureproxy-79c5db744-2xp79 1/1 Running 2 20h kube-system heapster-55f855b47-q9qxn 2/2 Running 0 20h kube-system kube-dns-v20-7c556f89c5-7pwxh 3/3 Running 0 20h kube-system kube-dns-v20-7c556f89c5-g7v6g 3/3 Running 0 20h kube-system kube-proxy-96qbk 1/1 Running 0 20h kube-system kube-proxy-x9vms 1/1 Running 0 20h kube-system kube-svc-redirect-84wq8 1/1 Running 0 20h kube-system kube-svc-redirect-cnn77 1/1 Running 0 20h kube-system kubernetes-dashboard-546f987686-99xd7 1/1 Running 4 20h kube-system tunnelfront-646d798b4-dkrxc 1/1 Running 0 20h # view cluster info $ kubectl cluster-info Kubernetes master is running at https://k8s-cc-tes-k8s-cc-gr-2acabf-255c18b1.hcp.canadacentral.azmk8s.io:443 Heapster is running at https://k8s-cc-tes-k8s-cc-gr-2acabf-255c18b1.hcp.canadacentral.azmk8s.io:443/api/v1/namespaces/kube-system/services/heapster/proxy KubeDNS is running at https://k8s-cc-tes-k8s-cc-gr-2acabf-255c18b1.hcp.canadacentral.azmk8s.io:443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy kubernetes-dashboard is running at https://k8s-cc-tes-k8s-cc-gr-2acabf-255c18b1.hcp.canadacentral.azmk8s.io:443/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. Status ``Ready`` is OK, now nodes can take workloads. Create Azure cloud storage account ---------------------------------- Using AKS we can store cluster persistent data in Azure, but first we need to create Azure storage account. Previous step will create resource group list with prefix ``MC`` - in our case it will be ``MC_k8s-cc-gr_k8s-cc-test_canadacentral`` :: $ az group list --output table k8s-cc-gr canadacentral Succeeded MC_k8s-cc-gr_k8s-cc-test_canadacentral canadacentral Succeeded Look for group matching our cluster name with ``MC`` prefix, and create storage account with this ``MC``-prefixed resource group, account name must be in lowercase, and don’t have ‘-’ or ’_’ in account name. We choose ``sdk8sccstorageaccount`` as storage account name. :: # create Azure storage account $ az storage account create --resource-group MC_k8s-cc-gr_k8s-cc-test_canadacentral --name sdk8sccstorageaccount --location canadacentral --sku Standard_LRS # show Azure storage account details $ az storage account show --name sdk8sccstorageaccount --resource-group MC_k8s-cc-gr_k8s-cc-test_canadacentral Kubernetes storage class provisioning ------------------------------------- Kubernetes abstractions called ``StorageClass`` with special modes support needs to be provisioned in cluster before installing SMASHDOCs app: - ReadWriteOnce mode - ``azure-disk`` Storage Class for ``mongodb`` and ``minio`` volume mode is used with mongodb and minio volume Download: :download:`azure-disk-sc.yaml <_static/kubernetes/azure-disk-sc.yaml>` :: kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: azure-disk provisioner: kubernetes.io/azure-disk parameters: skuName: Standard_LRS kind: Managed cachingmode: None Create storage classes in k8s :: # create storage class "azure-disk" $ kubectl create -f azure-disk-sc.yaml # display storage clasess $ kubectl get sc NAME PROVISIONER AGE azure-disk kubernetes.io/azure-disk 1h Initialize k8s package manager - helm ------------------------------------- Initialize kubernetes package manager ``helm`` server-side part, which is called ``tiller``, empty output is OK, it means no releases have been installed by ``helm`` in k8s yet. :: # initialize server-side pod, tiller # wait a few moments, before server-side initialization is finished # upgrade tiller to latest version $ helm init --upgrade # show helm-deployed releases $ helm list -A # empty output excepted, we didn't deploy anything by helm yet $ Create k8s ingress ------------------ The standard way to receive incoming traffic in k8s is through special component, provisioned by creating ``Ingress`` object Install nginx k8s ingress controller to ``ingress-nginx`` namespace, without RBAC enabled :: # install nginx helm chart $ helm install stable/nginx-ingress --namespace ingress-nginx --set rbac.create=false --set rbac.createRole=false --set rbac.createClusterRole=false Ingress deployment will create Azure LoadBalancer, pointing to newly created k8s cluster, we need to find out it’s public IP, and create DNS A record - ``azure.smashdocs.net`` in our case. Execute following command and look for ``EXTERNAL-IP`` address - it will be first, but later will be like ``40.85.221.174`` - this is Azure load balancer public IP address, this operation may take a few minutes to complete. :: $ kubectl get service -n ingress-nginx -l app=nginx-ingress NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE irreverant-ladybird-nginx-ingress-controller LoadBalancer 10.0.162.217 40.85.221.174 80:31933/TCP,443:30924/TCP 13d Let’s assume, we want to deploy SMASHDOCs app to https://azure.smashdocs.net Create DNS A-record pointing to EXTERNAL-IP, in our case we’ve created ``azure.smashdocs.net`` with value ``40.85.221.174`` Verify that DNS provisioned correctly and k8s is working - try to open your URL in browser or by CLI utility like ``curl``, the following output is expected :: default backend - 404 Install all cert-manager components ----------------------------------- Link: https://cert-manager.io/docs/installation/kubectl/ :: kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.10.1/cert-manager.yaml You can view some of the resources that have been installed as follows: :: kubectl -n cert-manager get all Issuer Configuration -------------------- Link: https://cert-manager.io/docs/configuration/ https://cert-manager.io/docs/tutorials/getting-started-with-cert-manager-on-google-kubernetes-engine-using-lets-encrypt-for-ingress-ssl/#7-create-an-issuer-for-lets-encrypt-staging The first thing you'll need to configure after you've installed cert-manager is an Issuer or a ClusterIssuer. Save the following content to a file called issuer-lets-encrypt.yaml, change the email field to use your email address and apply it: :: # issuer-lets-encrypt.yaml apiVersion: v1 kind: List items: - apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-prod spec: acme: email: email@mail.net ❗ Replace this with your email address privateKeySecretRef: name: letsencrypt-prod server: https://acme-v02.api.letsencrypt.org/directory solvers: - http01: ingress: class: nginx :: kubectl apply -f issuer-lets-encrypt.yaml Install SMASHDOCs app --------------------- SMASHDOCs app can be deployed using helm chart - the templates collecton, which describes k8s components like deployments, services, volumes, configmaps, ingress, etc. One of possible ways to organize resources (deployments, pods, volumes) in kubernetes by using namespaces. One of recommendations is to have one namespace per application. Let’s choose namespace name - we will use ``azure`` and k8s release name, we will use ``azure`` Release name is optional, will be assigned to random word1-word2 value, if not specified, can be looked up later by running ``helm list`` SMASHDOCs helm chart can be downloaded from our public repo - https://bitbucket.org/smashdocs/helm-chart/src/master/ Actual chart content (deployment templates)located in chart archive ``smashdocs-x.x.x.tgz``, download, extract files, then we need to change install-specific settings, change current directory :: # untar archive $ tar zxvf smashdocs-x.x.x.tgz # change directory $ cd smashdocs There is ``smashdocs-azure-values.yaml`` file there, which describes SMASHDOCs app setings, like version, keys, domain name settings, etc Download: :download:`smashdocs-azure-values.yaml <_static/kubernetes/smashdocs-azure-values.yaml>` :: customer: dnsname: customer shortname: customer domain: smashdocs.net tags: mongodb-subchart: false minio-subchart: false replicaCount: 1 frontend: name: frontend mode: normal image: repository: registry.smashdocs.net/smashdocs/frontend tag: 2.25.4 adminui: enabled: true name: adminui image: repository: registry.smashdocs.net/smashdocs/admin-ui tag: 2.25.4 backend: name: backend command: backend provisioningKey: 12345 image: repository: registry.smashdocs.net/smashdocs/backend tag: 2.25.4 pullPolicy: Always systemRole: development email: smtpServerAddress: email-smtp.eu-west-1.amazonaws.com smtpServerPort: 587 smtpUsername: smtpUsername smtpPassword: smtpPassword email: no-reply@smashdocs.net from: "SMASHDOCs Team" socketserver: name: socketserver image: repository: registry.smashdocs.net/smashdocs/socketserver tag: 2.25.4 worker: name: worker command: worker beat: name: beat command: beat nginx: name: nginx image: repository: registry.smashdocs.net/smashdocs/nginx tag: 2.25.4 pullPolicy: Always spellcheck: enabled: true name: spellcheck image: repository: registry.smashdocs.net/smashdocs/spellcheck tag: 2.25.4 languagetool: name: languagetool image: repository: registry.smashdocs.net/smashdocs/languagetool tag: 5.9 redis: name: redis image: repository: redis tag: 7-alpine mongodb: enabled: true name: mongodb image: repository: mongo tag: 6.0 pullPolicy: IfNotPresent persistentVolume: size: 10Gi storageClass: azure-disk minio: enabled: true name: minio accessKey: 12345 secretKey: 12345678 image: repository: "registry.smashdocs.net/smashdocs/minio:latest" tag: latest pullPolicy: IfNotPresent persistentVolume: size: 10Gi storageClass: azure-disk resources: {} # We usually recommend not to specify default resources and to leave this as a conscious # choice for the user. This also increases chances charts run on environments with little # resources, such as Minikube. If you do want to specify resources, uncomment the following # lines, adjust them as necessary, and remove the curly braces after 'resources:'. # limits: # cpu: 100m # memory: 128Mi # requests: # cpu: 100m # memory: 128Mi nodeSelector: {} tolerations: [] affinity: {} You will need to review and edit this file, replacing following values: - provisioningKey - mandatory value, used internally for frontend-backend authentication, can be generated by python one-liner, requires ``python`` installed in system :: $ python -c "import random; print(random.randint(5,1000000))" | shasum -a 256 | awk '{print $1}' 2ae777842419e4ab1691655b3db1625412b816e8af70573378b32c81882cc13c place it in your ``smashdocs-values.yaml`` file, replacing ``provisioningKey`` default value :: backend: name: backend provisioningKey: 2ae777842419e4ab1691655b3db1625412b816e8af70573378b32c81882cc13c image: repository: registry.smashdocs.net/smashdocs/backend tag: 2.12.1 - dnsname - first word in fully qualified domain name, ``azure`` in our case - shortname - used in chart internally, equals to dnsname without dashes in name, ``azure`` in our case - domain name - customer domain, ``smashdocs.net`` :: customer: dnsname: azure shortname: azure domain: smashdocs.net - email settings - server name, user name, password - outgoing email, user confirmation during registration default values - SMASHDOCs AWS SES credentials Optional values: SMASHDOCs frontend and backend components can report runtime errors to Sentry, the following chart values need to be enabled in ``sentry`` section: :: sentry: enabled: true sentyURL: https://54a7890c3aa9457688c6560eb77bb28b:fffa5d77bd59403791ea038247d9cd36@sentry.io/278077 serverName: azure-smashdocs-backend - enabled- ``true`` or ``false`` - sentryURL - Sentry project DSN URL, like ``https://54a7890c3aa9457688c6560eb77bb28b@sentry.io/278077`` - serverName - component name for Sentry reporting, default is ``shortname-frontend`` or ``shortname-backend`` Now, we need to create kubernetes secret with docker login credentials, we use hub.docker.com, for private docker image registry, you can request your existing docker login to be linked, as our partner, for image pull. :: # create kubernetes docker login secret in given namespace with given credentials kubectl -n azure create secret docker-registry smashdocs-dockerlogin --docker-username=YOURLOGIN --docker-password=YOURPASSWORD --docker-email=your@email.com Now we are ready to install SMASHDOCs app into your k8s cluster. Following command will check configuration and deployment files syntax. It will output k8s deployment yaml files, but no objects will be created. :: # check install, dry run $ helm install --dry-run --name azure --namespace azure -f azure-values.yaml . Now, let’s install SMASHDOCs app from helm chart, to namespace ``azure`` with app settings (values) defined in ``azure-values.yaml`` :: # install SMASHDOCs app $ helm install --name azure --namespace azure -f azure-values.yaml . # list installed releases $ helm list NAME REVISION UPDATED STATUS CHART NAMESPACE azure 6 Fri Jun 1 08:55:05 2018 DEPLOYED smashdocs-0.1.0 azure bumptious-sheep 1 Fri May 11 13:32:28 2018 DEPLOYED kube-lego-0.4.0 default zeroed-markhor 1 Fri May 25 15:03:40 2018 DEPLOYED nginx-ingress-0.19.2 kube-system # get running pods in namespace = 'azure', it's freshly installed SMASHDOCs app $ kubectl -n azure get pods NAME READY STATUS RESTARTS AGE azure-adminui-5f59b78468-p987n 1/1 Running 0 azure-backend-7b6769474b-xzcpt 0/1 ContainerCreating 0 azure-beat-756549df5b-tmqrw 0/1 ContainerCreating 0 azure-frontend-6cdb5df48d-mcm6v 1/1 Running 0 azure-mongodb-85b76bdd85-fxfrn 0/1 Pending 0 azure-nginx-6cd6c7c784-cjj5x 1/1 Running 0 azure-redis-f45f6d49c-nhwhz 0/1 PodInitializing 0 azure-socketserver-559f4ffcbb-w7948 0/1 ContainerCreating 0 azure-worker-7cccccf465-l542h 0/1 ContainerCreating 0 # wait a few minutes, while Azure will create volumes and attach them to containers $ kubectl get pods -n azure NAME READY STATUS RESTARTS AGE azure-adminui-5f59b78468-p987n 1/1 Running 0 3m azure-backend-7b6769474b-xzcpt 0/1 Running 0 3m azure-beat-756549df5b-tmqrw 1/1 Running 0 3m azure-frontend-6cdb5df48d-mcm6v 1/1 Running 0 3m azure-mongodb-85b76bdd85-fxfrn 1/1 Running 0 3m azure-nginx-6cd6c7c784-cjj5x 1/1 Running 0 3m azure-redis-f45f6d49c-nhwhz 1/1 Running 0 3m azure-socketserver-559f4ffcbb-w7948 1/1 Running 0 3m azure-worker-7cccccf465-l542h 1/1 Running 0 3m App component statuses: * READY 0/1 - app component is up, but readinessProbe is not OK yet READY * READY 1/1 - app component is up and running, readinessProbe is OK First initialization for backend component may take some time, because Azure network attached storages with ReadWriteMany mode are slow. Open ``azure.smashdocs.net`` in browser Congratulations, you have SMASHDOCs running in Azure cloud! Debug option: you can add ``--debug`` to helm command while installing/upgrading release Upgrade SMASHDOCs app --------------------- Review CHANGELOG in helm chart directory, look for steps you need to complete before pgrade :: # change current directory to one, that contains smashdocs chart $ cd smashdocs_chart_directory # list releases, look for smashdocs-x.x.x in CHART column $ helm list # upgrade SMASHDOCs release $ helm upgrade -f azure-values.yaml RELEASE_NAME . Troubleshooting --------------- Get namespaces list :: $ kubectl get namespaces NAME STATUS AGE azure Active 31d default Active 36d ingress-nginx Active 36d kube-public Active 36d kube-system Active 36d Display pods in given namespace :: $ kubectl get pods -n azure Describe pod lifecycle, look for pod events :: $ kubectl -n azure describe pod azure-mongodb-85b76bdd85-fxfrn ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 4m (x7 over 5m) default-scheduler PersistentVolumeClaim is not bound: "azure-mongodb" (repeated 2 times) Normal Scheduled 4m default-scheduler Successfully assigned azure-mongodb-85b76bdd85-fxfrn to aks-nodepool1-24768417-1 Normal SuccessfulMountVolume 4m kubelet, aks-nodepool1-24768417-1 MountVolume.SetUp succeeded for volume "default-token-d7zdh" Warning FailedMount 1m kubelet, aks-nodepool1-24768417-1 Unable to mount volumes for pod "azure-mongodb-85b76bdd85-fxfrn_azure(ee0a6920-68ea-11e8-92cf-0a58ac1f0520)": timeout expired waiting for volumes to attach/mount for pod "azure"/"azure-mongodb-85b76bdd85-fxfrn". list of unattached/unmounted volumes=[mongodb-data] Normal SuccessfulMountVolume 1m kubelet, aks-nodepool1-24768417-1 MountVolume.SetUp succeeded for volume "pvc-edc6720e-68ea-11e8-92cf-0a58ac1f0520" Normal Pulling 1m kubelet, aks-nodepool1-24768417-1 pulling image "mongo:4.4" Normal Pulled 48s kubelet, aks-nodepool1-24768417-1 Successfully pulled image "mongo:4.4" Normal Created 48s kubelet, aks-nodepool1-24768417-1 Created container Normal Started 48s kubelet, aks-nodepool1-24768417-1 Started container Get pod logs from stdout/stder (nginx, frontend, backend, socketserver) :: $ kubectl -n azure log azure-backend-7b6769474b-xzcpt ... [backend logs here] Check storage classes :: $ kubectl get sc NAME PROVISIONER AGE azure-disk kubernetes.io/azure-disk 6h $ kubectl describe sc azure-disk Name: azure-disk IsDefaultClass: No Annotations: Provisioner: kubernetes.io/azure-disk Parameters: cachingmode=None,kind=Managed,skuName=Standard_LRS AllowVolumeExpansion: MountOptions: dir_mode=0777 file_mode=0777 ReclaimPolicy: Retain VolumeBindingMode: Immediate Events: Check persistent volume claims, and persistent volume status, Status Bound is OK :: $ kubectl -n azure get pv,pvc NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pvc-2003bef6-ec30-11e8-87ec-0a58ac1f027c 10Gi RWO Retain Bound azure/azure-smashdocs-minio azure-disk 2y56d persistentvolume/pvc-e0436dba-ec2e-11e8-87ec-0a58ac1f027c 10Gi RWO Retain Bound azure/azure-smashdocs-mongodb azure-disk 2y56d NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/azure-smashdocs-minio Bound pvc-2003bef6-ec30-11e8-87ec-0a58ac1f027c 10Gi RWO azure-disk 2y52d persistentvolumeclaim/azure-smashdocs-mongodb Bound pvc-e0436dba-ec2e-11e8-87ec-0a58ac1f027c 10Gi RWO azure-disk 2y52d Delete stuck in ``Terminating`` status pods :: kubectl get pods --all-namespaces | awk -v ns="YOUR_NAMESPACE" '$3=="Terminating" {print "kubectl delete -n "ns" pod " $2 " --grace-period=0 --force"}' | xargs -0 bash -c Access pods within k8s cluster ------------------------------ Let's assume that our release is named ``azure-smashdocs``, it's running in ``azure`` namespace, and we need to get access to mongo database, this can be done by ``kubectl port-forwarding`` Please change ``azure`` to ``YOUR_NAMESPACE`` when executing command from example given below: :: # list installed releases, to find out proper namespace $ helm list NAME REVISION UPDATED STATUS CHART NAMESPACE azure-smashdocs 166 Tue Jul 10 17:46:02 2018 DEPLOYED smashdocs-0.2.0 azure # OR $ kubectl get namespaces NAME STATUS AGE azure Active 28d default Active 33d ingress-nginx Active 33d kube-public Active 33d kube-system Active 33d # so, our components are running in 'azure' namespace, $ kubectl -n azure get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE azure-adminui ClusterIP 10.0.172.143 80/TCP 1h azure-backend ClusterIP 10.0.187.210 8080/TCP 3h azure-beat ClusterIP 10.0.169.224 8080/TCP 3h azure-frontend ClusterIP 10.0.108.36 80/TCP 3h azure-mongodb ClusterIP 10.0.23.47 27017/TCP 3h azure-nginx ClusterIP 10.0.109.157 80/TCP 3h azure-redis ClusterIP 10.0.159.33 6379/TCP 3h azure-socketserver ClusterIP 10.0.233.101 8080/TCP 3h azure-worker ClusterIP 10.0.193.207 8080/TCP 3h # forward local 37017 to remote service azure-mongodb port 27017 $ kubectl -n azure port-forward svc/azure-mongodb 37017:27017 Forwarding from 127.0.0.1:37017 -> 27017 Forwarding from [::1]:37017 -> 27017 Now you can connect with MongoHUB or Studio3T or mongodb cli to localhost:37017, and that connect will be forwarded to mongodb pod inside cluster. Delete SMASHDOCs release ---------------------------- WARNING - DESTRUCTIVE actions delete SMASHDOCs k8s release :: helm delete RELEASE_NAME delete k8s cluster from AKS :: az aks delete -n k8s-cc-test -g k8s-cc-gr Kubernetes setup for AWS (kops) ================================ This guide describes Kubernetes cluster provisioning using `KOPS `_ and following SMASHDOCs app install on this cluster (k8s) cluster. The main architectural considerations are: - all SMASHDOCs app components: frontend, adminui, backend, socketserver, beat, worker, nginx, redis, mongodb are running in k8s cluster - one k8s pod is running one SMASHDOCs app component - one k8s service is created for each SMASHDOCs app component - nginx ingress controller is used for http/https termination - kube-lego is used for SSL certificate generation - persistent data stored at AWS EBS storage At the end of this guide you will have working instance of SMASHDOCs app in AWS cloud with public HTTPS access. Install CLI utilities --------------------- The following prerequisites are required for installing KOPS - ``aws`` - AWS CLI utility - ``kubectl`` - Kubernetes CLI utility - ``kops`` - KOPS CLI utility - ``helm`` - Kubernetes package manager client - ``jq`` - JSON parsing utility Step to install on MacOS: :: $ brew install aws-cli $ brew install kubernetes-cli $ brew install kops $ brew install kubernetes-helm $ brew install jq in Linux use appropriate package manager, apt/yum Prerequisite steps ------------------ If your AWS account is not empty, and you don't want to mix existing AWS resources with resources, which will be created by kops, create new sub-account in your organization. Login to your AWS account, go to `Organizations `_ menu, select add account, enter email, add new organizational unit, assign permissions for newly created account, it must have full access to organizational unit. Switch role from your root account to new one. Choose domain name for your k8s cluster, it will be used later in AWS/k8s resources creation, we will use ``aws.smashdocs.net`` in this guide as domain name Go to IAM, create `kops` group with following permissions :: AmazonEC2FullAccess AmazonRoute53FullAccess AmazonS3FullAccess IAMFullAccess AmazonVPCFullAccess Create `kops` user, add it to `kops` group, generate `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` and configure aws profile `kops` using those credentials. :: aws configure --profile kops AWS Access Key ID [None]: YOUR_AWS_ACCESS_KEY_ID AWS Secret Access Key [None]: YOUR_AWS_SECRET_ACCESS_KEY Default region name [None]: YOUR_AWS_REGION Default output format [None]: Create S3 bucket where kops state will be stored, we will use bucket name `aws-smashdocs-net-kops-state-store` in this guide :: aws s3api create-bucket \ --bucket aws-eu-smashdocs-io-kops-state-store --region eu-central-1 \ --create-bucket-configuration LocationConstraint=eu-central-1 Enable versioning for this bucket :: aws s3api put-bucket-versioning --bucket aws-eu-smashdocs-io-kops-state-store \ --versioning-configuration Status=Enabled Create K8S in AWS ----------------- Set cluster name :: export NAME=aws.smashdocs.net export KOPS_STATE_STORE=s3://aws-smashdocs-net-kops-state-store List and set availibility zones for cluster :: aws ec2 describe-availability-zones --region eu-central-1 export ZONES="eu-central-1a" export ZONES="eu-central-1a eu-central-1b eu-central-1c" Choose node size for masters and workers :: export NODE_SIZE=t2.large export MASTER_SIZE=m3.medium Generate cluster manifests :: kops create cluster $NAME \ --node-count 1 \ --zones $ZONES \ --node-size $NODE_SIZE \ --master-size $MASTER_SIZE \ --master-zones $ZONES \ --networking calico \ --authorization RBAC --dry-run --output yaml | tee $NAME.k8s.yaml Review generated manifests, create cluster, create secret for access, validate cluster :: kops create -f $NAME.k8s.yaml kops create secret --name $NAME sshpublickey admin -i ~/.ssh/id_rsa.pub kops update cluster --yes kops validate cluster Get k8s cluster credentials --------------------------- To use k8s cluster we need to get it’s credentials (certificates), this command will put it in ``kubectl`` config located at ``$HOME/.kube/config`` With kops credentials will be put in your kubeconfig on cluster creation :: # view available contexts $ kubectl config get-contexts * aws.smashdocs.net aws.smashdocs.net aws.smashdocs.net # use specified context $ kubectl config use-context aws.smashdocs.net Switched to context "aws.smashdocs.net". # view cluster nodes $ kubectl get nodes NAME STATUS ROLES AGE VERSION ip-172-20-124-62.eu-central-1.compute.internal Ready node 1h v1.11.6 ip-172-20-49-115.eu-central-1.compute.internal Ready master 1h v1.11.6 # view cluster pods (containers) $ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE [output skipped] # view cluster info $ kubectl cluster-info [output skipped] To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. Status ``Ready`` is OK, now nodes can take workloads. Kubernetes storage class provisioning ------------------------------------- SMASHDOCs app requires persistent storage, so StorageClass must be provisioned in k8s cluster, kops is using ``gp2 (default)`` with ``kubernetes.io/aws-ebs provisioner`` by default. :: kubectl get sc NAME PROVISIONER AGE default kubernetes.io/aws-ebs 7d gp2 (default) kubernetes.io/aws-ebs 7d standard kubernetes.io/aws-ebs 7d Initialize k8s package manager - helm ------------------------------------- Initialize kubernetes package manager ``helm`` server-side part, which is called ``tiller``, empty output is OK, it means no releases have been installed by ``helm`` in k8s yet. :: # initialize server-side pod, tiller # wait a few moments, before server-side initialization is finished # upgrade tiller to latest version # show helm-deployed releases $ helm list -A # empty output excepted, we didn't deploy anything by helm yet $ Create k8s ingress ------------------ The standard way to receive incoming traffic in k8s is through special component, provisioned by creating ``Ingress`` object Install nginx k8s ingress controller to ``ingress-nginx`` namespace, with RBAC enabled :: # install nginx helm chart $ helm install stable/nginx-ingress --namespace ingress-nginx Ingress deployment will create AWS LoadBalancer, pointing to newly created k8s cluster, we need to find out it’s public IP, and create DNS A record - ``aws.smashdocs.net`` in our case. Execute following command and look for ``EXTERNAL-IP`` address - it will be first, but later will be like ``52.58.70.14`` - this is AWS load balancer public IP address, this operation may take a few minutes to complete. :: $ kubectl get service -n ingress-nginx -l app=nginx-ingress NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE irreverant-ladybird-nginx-ingress-controller LoadBalancer 10.0.162.217 52.58.70.14 80:31933/TCP,443:30924/TCP 13d Let’s assume, we want to deploy SMASHDOCs app to https://aws.smashdocs.net Create DNS A-record pointing to EXTERNAL-IP, in our case we’ve created ``aws.smashdocs.net`` with value ``52.58.70.14`` Verify that DNS provisioned correctly and k8s is working - try to open your URL in browser or by CLI utility like ``curl``, the following output is expected :: default backend - 404 Install all cert-manager components ----------------------------------- Link: https://cert-manager.io/docs/installation/kubectl/ :: kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.10.1/cert-manager.yaml You can view some of the resources that have been installed as follows: :: kubectl -n cert-manager get all Issuer Configuration -------------------- Link: https://cert-manager.io/docs/configuration/ https://cert-manager.io/docs/tutorials/getting-started-with-cert-manager-on-google-kubernetes-engine-using-lets-encrypt-for-ingress-ssl/#7-create-an-issuer-for-lets-encrypt-staging The first thing you'll need to configure after you've installed cert-manager is an Issuer or a ClusterIssuer. Save the following content to a file called issuer-lets-encrypt.yaml, change the email field to use your email address and apply it: :: # issuer-lets-encrypt.yaml apiVersion: v1 kind: List items: - apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-prod spec: acme: email: email@mail.net ❗ Replace this with your email address privateKeySecretRef: name: letsencrypt-prod server: https://acme-v02.api.letsencrypt.org/directory solvers: - http01: ingress: class: nginx :: kubectl apply -f issuer-lets-encrypt.yaml Install SMASHDOCs app --------------------- SMASHDOCs app can be deployed using helm chart - the templates collecton, which describes k8s components like deployments, services, volumes, configmaps, ingress, etc. One of possible ways to organize resources (deployments, pods, volumes) in kubernetes by using namespaces. One of recommendations is to have one namespace per application. Let’s choose namespace name - we will use ``aws`` and k8s release name, we will use ``aws`` Release name is optional, will be assigned to random word1-word2 value, if not specified, can be looked up later by running ``helm list`` SMASHDOCs helm chart can be downloaded from our public repo - https://bitbucket.org/smashdocs/helm-chart/src/master/ Actual chart content (deployment templates)located in chart archive ``smashdocs-x.x.x.tgz``, download, extract files, then we need to change install-specific settings, change current directory :: # untar archive $ tar zxvf smashdocs-x.x.x.tgz # change directory $ cd smashdocs There is ``smashdocs-aws-values.yaml`` file there, which describes SMASHDOCs app setings, like version, keys, domain name settings, etc Download: :download:`smashdocs-aws-values.yaml <_static/kubernetes/smashdocs-aws-values.yaml>` :: customer: dnsname: customer shortname: customer domain: smashdocs.net tags: mongodb-subchart: false minio-subchart: false replicaCount: 1 frontend: name: frontend mode: normal image: repository: registry.smashdocs.net/smashdocs/frontend tag: 2.25.4 adminui: enabled: true name: adminui image: repository: registry.smashdocs.net/smashdocs/admin-ui tag: 2.25.4 backend: name: backend command: backend provisioningKey: 12345 image: repository: registry.smashdocs.net/smashdocs/backend tag: 2.25.4 pullPolicy: Always systemRole: development email: smtpServerAddress: email-smtp.eu-west-1.amazonaws.com smtpServerPort: 587 smtpUsername: smtpUsername smtpPassword: smtpPassword email: no-reply@smashdocs.net from: "SMASHDOCs Team" socketserver: name: socketserver image: repository: registry.smashdocs.net/smashdocs/socketserver tag: 2.25.4 worker: name: worker command: worker beat: name: beat command: beat nginx: name: nginx image: repository: registry.smashdocs.net/smashdocs/nginx tag: 2.25.4 pullPolicy: Always spellcheck: enabled: true name: spellcheck image: repository: registry.smashdocs.net/smashdocs/spellcheck tag: 2.25.4 languagetool: name: languagetool image: repository: registry.smashdocs.net/smashdocs/languagetool tag: 5.9 redis: name: redis image: repository: redis tag: 7-alpine mongodb: enabled: true name: mongodb image: repository: mongo tag: 6.0 pullPolicy: IfNotPresent persistentVolume: size: 10Gi storageClass: azure-disk minio: enabled: true name: minio accessKey: 12345 secretKey: 12345678 image: repository: "registry.smashdocs.net/smashdocs/minio:latest" tag: latest pullPolicy: IfNotPresent persistentVolume: size: 10Gi storageClass: default resources: {} # We usually recommend not to specify default resources and to leave this as a conscious # choice for the user. This also increases chances charts run on environments with little # resources, such as Minikube. If you do want to specify resources, uncomment the following # lines, adjust them as necessary, and remove the curly braces after 'resources:'. # limits: # cpu: 100m # memory: 128Mi # requests: # cpu: 100m # memory: 128Mi nodeSelector: {} tolerations: [] affinity: {} You will need to review and edit this file, replacing following values: - provisioningKey - mandatory value, used internally for frontend-backend authentication, can be generated by python one-liner, requires ``python`` installed in system :: $ python -c "import random; print(random.randint(5,1000000))" | shasum -a 256 | awk '{print $1}' 2ae777842419e4ab1691655b3db1625412b816e8af70573378b32c81882cc13c place it in your ``smashdocs-aws-values.yaml`` file, replacing ``provisioningKey`` default value :: backend: name: backend provisioningKey: 2ae777842419e4ab1691655b3db1625412b816e8af70573378b32c81882cc13c image: repository: registry.smashdocs.net/smashdocs/backend tag: 2.25.4 - dnsname - first word in fully qualified domain name, ``azure`` in our case - shortname - used in chart internally, equals to dnsname without dashes in name, ``aws`` in our case - domain name - customer domain, ``smashdocs.net`` :: customer: dnsname: aws shortname: aws domain: smashdocs.net - email settings - server name, user name, password - outgoing email, user confirmation during registration default values - SMASHDOCs AWS SES credentials Optional values: SMASHDOCs frontend and backend components can report runtime errors to Sentry, the following chart values need to be enabled in ``sentry`` section: :: sentry: enabled: true sentyURL: https://54a7890c3aa9457688c6560eb77bb28b:fffa5d77bd59403791ea038247d9cd36@sentry.io/278077 serverName: smashdocs-backend - enabled- ``true`` or ``false`` - sentryURL - Sentry project DSN URL, like ``https://54a7890c3aa9457688c6560eb77bb28b@sentry.io/278077`` - serverName - component name for Sentry reporting, default is ``shortname-frontend`` or ``shortname-backend`` Now, we need to create kubernetes secret with docker login credentials, we use hub.docker.com, for private docker image registry, you can request your existing docker login to be linked, as our partner, for image pull. :: # create kubernetes docker login secret in given namespace with given credentials kubectl -n aws create secret docker-registry smashdocs-dockerlogin --docker-username=YOURLOGIN --docker-password=YOURPASSWORD --docker-email=your@email.com Now we are ready to install SMASHDOCs app into your k8s cluster. Following command will check configuration and deployment files syntax. It will output k8s deployment yaml files, but no objects will be created. :: # check install, dry run $ helm install --dry-run --name aws --namespace aws -f smashdocs-aws-values.yaml . Now, let’s install SMASHDOCs app from helm chart, to namespace ``aws`` with app settings (values) defined in ``smashdocs-aws-values.yaml`` :: # install SMASHDOCs app $ helm install --name aws --namespace aws -f smashdocs-aws-values.yaml . # list installed releases $ helm list NAME REVISION UPDATED STATUS CHART NAMESPACE aws 6 Fri Jun 1 08:55:05 2018 DEPLOYED smashdocs-0.4.3 aws bumptious-sheep 1 Fri May 11 13:32:28 2018 DEPLOYED kube-lego-0.4.0 default zeroed-markhor 1 Fri May 25 15:03:40 2018 DEPLOYED nginx-ingress-0.19.2 kube-system # get running pods in namespace = 'aws', it's freshly installed SMASHDOCs app $ kubectl -n aws get pods NAME READY STATUS RESTARTS AGE aws-adminui-5f59b78468-p987n 1/1 Running 0 aws-backend-7b6769474b-xzcpt 0/1 ContainerCreating 0 aws-beat-756549df5b-tmqrw 0/1 ContainerCreating 0 aws-frontend-6cdb5df48d-mcm6v 1/1 Running 0 aws-mongodb-85b76bdd85-fxfrn 0/1 Pending 0 aws-nginx-6cd6c7c784-cjj5x 1/1 Running 0 aws-redis-f45f6d49c-nhwhz 0/1 PodInitializing 0 aws-socketserver-559f4ffcbb-w7948 0/1 ContainerCreating 0 aws-worker-7cccccf465-l542h 0/1 ContainerCreating 0 # wait a few minutes, while aws will create volumes and attach them to containers $ kubectl get pods -n aws NAME READY STATUS RESTARTS AGE aws-adminui-5f59b78468-p987n 1/1 Running 0 3m aws-backend-7b6769474b-xzcpt 0/1 Running 0 3m aws-beat-756549df5b-tmqrw 1/1 Running 0 3m aws-frontend-6cdb5df48d-mcm6v 1/1 Running 0 3m aws-mongodb-85b76bdd85-fxfrn 1/1 Running 0 3m aws-nginx-6cd6c7c784-cjj5x 1/1 Running 0 3m aws-redis-f45f6d49c-nhwhz 1/1 Running 0 3m aws-socketserver-559f4ffcbb-w7948 1/1 Running 0 3m aws-worker-7cccccf465-l542h 1/1 Running 0 3m App component statuses: * READY 0/1 - app component is up, but readinessProbe is not OK yet READY * READY 1/1 - app component is up and running, readinessProbe is OK Open ``aws.smashdocs.net`` in browser Congratulations, you have SMASHDOCs running in AWS cloud! Debug option: you can add ``--debug`` to helm command while installing/upgrading release Upgrade SMASHDOCs app --------------------- Review CHANGELOG in helm chart directory, look for steps you need to complete before pgrade :: # change current directory to one, that contains smashdocs chart $ cd smashdocs_chart_directory # list releases, look for smashdocs-x.x.x in CHART column $ helm list # upgrade SMASHDOCs release $ helm upgrade -f smashdocs-aws-values.yaml RELEASE_NAME . Troubleshooting --------------- Get namespaces list :: $ kubectl get namespaces NAME STATUS AGE azure Active 31d default Active 36d ingress-nginx Active 36d kube-public Active 36d kube-system Active 36d Display pods in given namespace :: $ kubectl get pods -n azure Describe pod lifecycle, look for pod events :: $ kubectl -n azure describe pod azure-mongodb-85b76bdd85-fxfrn ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 4m (x7 over 5m) default-scheduler PersistentVolumeClaim is not bound: "azure-mongodb" (repeated 2 times) Normal Scheduled 4m default-scheduler Successfully assigned azure-mongodb-85b76bdd85-fxfrn to aks-nodepool1-24768417-1 Normal SuccessfulMountVolume 4m kubelet, aks-nodepool1-24768417-1 MountVolume.SetUp succeeded for volume "default-token-d7zdh" Warning FailedMount 1m kubelet, aks-nodepool1-24768417-1 Unable to mount volumes for pod "azure-mongodb-85b76bdd85-fxfrn_azure(ee0a6920-68ea-11e8-92cf-0a58ac1f0520)": timeout expired waiting for volumes to attach/mount for pod "azure"/"azure-mongodb-85b76bdd85-fxfrn". list of unattached/unmounted volumes=[mongodb-data] Normal SuccessfulMountVolume 1m kubelet, aks-nodepool1-24768417-1 MountVolume.SetUp succeeded for volume "pvc-edc6720e-68ea-11e8-92cf-0a58ac1f0520" Normal Pulling 1m kubelet, aks-nodepool1-24768417-1 pulling image "mongo:4.4" Normal Pulled 48s kubelet, aks-nodepool1-24768417-1 Successfully pulled image "mongo:4.4" Normal Created 48s kubelet, aks-nodepool1-24768417-1 Created container Normal Started 48s kubelet, aks-nodepool1-24768417-1 Started container Get pod logs from stdout/stder (nginx, frontend, backend, socketserver) :: $ kubectl -n azure log azure-backend-7b6769474b-xzcpt ... [backend logs here] Check storage classes :: $ kubectl get sc NAME PROVISIONER AGE azure-disk kubernetes.io/azure-disk 6h $ kubectl describe sc azure-disk Name: azure-disk IsDefaultClass: No Annotations: Provisioner: kubernetes.io/azure-disk Parameters: cachingmode=None,kind=Managed,skuName=Standard_LRS AllowVolumeExpansion: MountOptions: dir_mode=0777 file_mode=0777 ReclaimPolicy: Retain VolumeBindingMode: Immediate Events: Check persistent volume claims, and persistent volume status, Status Bound is OK :: $ kubectl -n azure get pv,pvc persistentvolume/pvc-2003bef6-ec30-11e8-87ec-0a58ac1f027c 10Gi RWO Retain Bound azure/azure-smashdocs-minio azure-disk 2y56d persistentvolume/pvc-e0436dba-ec2e-11e8-87ec-0a58ac1f027c 10Gi RWO Retain Bound azure/azure-smashdocs-mongodb azure-disk 2y56d NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/azure-smashdocs-minio Bound pvc-2003bef6-ec30-11e8-87ec-0a58ac1f027c 10Gi RWO azure-disk 2y52d persistentvolumeclaim/azure-smashdocs-mongodb Bound pvc-e0436dba-ec2e-11e8-87ec-0a58ac1f027c 10Gi RWO azure-disk 2y52d Delete stuck in ``Terminating`` status pods :: kubectl get pods --all-namespaces | awk -v ns="YOUR_NAMESPACE" '$3=="Terminating" {print "kubectl delete -n "ns" pod " $2 " --grace-period=0 --force"}' | xargs -0 bash -c Access pods within k8s cluster ------------------------------ Let's assume that our release is named ``azure-smashdocs``, it's running in ``azure`` namespace, and we need to get access to mongo database, this can be done by ``kubectl port-forwarding`` Please change ``azure`` to ``YOUR_NAMESPACE`` when executing command from example given below: :: # list installed releases, to find out proper namespace $ helm list NAME REVISION UPDATED STATUS CHART NAMESPACE azure-smashdocs 166 Tue Jul 10 17:46:02 2018 DEPLOYED smashdocs-0.2.0 azure # OR $ kubectl get namespaces NAME STATUS AGE azure Active 28d default Active 33d ingress-nginx Active 33d kube-public Active 33d kube-system Active 33d # so, our components are running in 'azure' namespace, $ kubectl -n azure get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE azure-adminui ClusterIP 10.0.172.143 80/TCP 1h azure-backend ClusterIP 10.0.187.210 8080/TCP 3h azure-beat ClusterIP 10.0.169.224 8080/TCP 3h azure-frontend ClusterIP 10.0.108.36 80/TCP 3h azure-mongodb ClusterIP 10.0.23.47 27017/TCP 3h azure-nginx ClusterIP 10.0.109.157 80/TCP 3h azure-redis ClusterIP 10.0.159.33 6379/TCP 3h azure-socketserver ClusterIP 10.0.233.101 8080/TCP 3h azure-worker ClusterIP 10.0.193.207 8080/TCP 3h # forward local 37017 to remote service azure-mongodb port 27017 $ kubectl -n azure port-forward svc/azure-mongodb 37017:27017 Forwarding from 127.0.0.1:37017 -> 27017 Forwarding from [::1]:37017 -> 27017 Now you can connect with MongoHUB or Studio3T or mongodb cli to localhost:37017, and that connect will be forwarded to mongodb pod inside cluster. Delete SMASHDOCs release ---------------------------- WARNING - DESTRUCTIVE actions delete SMASHDOCs k8s release :: helm delete RELEASE_NAME delete k8s cluster from AKS :: az aks delete -n k8s-cc-test -g k8s-cc-gr Multitenancy ============ SMASHDOCs supports multiple tenants on one installation. Each tenant in SMASHDOCs is called Organization and can be created and updated by the Provisioning API: ``__ Redundant Setup =============== please contact us