1. Architecture
The application and all its dependencies are compiled into a Docker image.
As soon as the container is deployed to a GCP instance, it pairs with a Salesforce org that has the Unify UI package (Installable directly through the AppExchange).
This document describes a possible configuration to manage access to the deployed instance, for example through a LoadBalancer.)
Flow Diagram
Here is the diagram describing all the inbound/outbound flows as well as the IPs and Ports requested by the application.
Security Measures
We cannot provide an example of how to implement the security layer that would fit perfectly into your architecture. However, we can provide some recommendations that are relevant to the use of the app itself.
- Protocols and Ports : All outgoing/incoming flows from the application go through the HTTPS protocol (port 443). There is no need to open another port on your VM. The DQE IPs that must be accessible are described in the preceding diagram.
- IP Filtering: The application server itself only needs to be accessible from Salesforce's servers. It is therefore recommended to set up a filtering of incoming IPs.
There is an exception to this, however, being the CustomUI web server included in the docker-compose. It must be accessible by a web browser. One solution could therefore be to allow a specific VPN to access the server. See here the list of IPs published by Salesforce. (Don't forget the Hyperforce IPs)
- Authentication protocol : Salesforce natively offers several protocols to secure requests between its servers and the DQE One server. When configuring the "DQE One Customer" Connected App in your organization, you will be able to select the "Basic Auth" authentication method and configure your application firewall accordingly.
Recommendation
In this section, we describe the list of components required to install the Unify Server DQE instance on GCP.
Note: Estimation are made for a base of 1 million records. Depending on the volume of the databases processed, it may be necessary to increase the memory capacity of container instances to optimize processing times.
Deployment with GCP VM INSTANCES
- Server type: Unix, e.g. Ubuntu.
- Load Balancer: Not essential but can be used if already existing.
For more information, see the following resources:
- VM Configuration:
https://cloud.google.com/compute/docs/instances?hl=fr
Composition and services
The docker image is a web application that exposes the services and APIs that are requested from Salesforce through the Lightning DQE Unify UI app.
For each service, we outline the attributes and their functions below.
WEB APPLICATION
A backend container that contains a microservices application. This application exposes all the APIs called to launch processes, manage processing queues, instantiate processing workers. It is dependent on the following services to function properly. This worker must be exposed on the web as described in the nginx configuration presented later in this document.
Docker Compose settings
- image: The name of the image used by the web application. It is necessary to rename the image according to the names of the registries used. For images hosted in an Azure container registry, the name is constructed as follows:
<Name of the registry container>.azurecr.io/<name of the image>
- container_name: Container Name
- depends_on: Name of the services that precede this service
- domainname: The name of the domain for the FQDN. Azure Rated Result:
<domain name>
- environment: Service environment variables
- command: Command executed when the container is launched
Worker
A backend container that contains a microservices application. This application exposes all the APIs called to launch processes, manage processing queues, instantiate processing workers. It is dependent on the following services to function properly. This worker must be exposed on the web as described in the nginx configuration presented later in this document.
Parameters to set in Docker Compose
- image: The name of this image is the same as the web service
- container_name: The name of the container that appears in Azure
- depends_on: Name of the services that precede this service
- environment: Service environment variables
- command: Command executed when the container is launched
CustomUI
This service is a web application exposing a front used to configure deduplication rules. Like the first service, it must therefore also be exposed on the web. This service also consumes some APIs from the backend, for example to retrieve metadata from the CRM.
Registry Name: *
Image name: *
Available versions: *
Redis
This service is a key/value database used by the various departments to store internal operating keys such as Salesforce org configurations, sessions, etc.
It is possible to use official images published by Redis such as "redis:alpine", but as a security measure some cloud providers block the retrieval of public images and only allow images from private registries. To overcome this restriction, we also publish on the private registry "dqeone" a version of redis that is perfectly compatible with our applications.
Parameters
- image: Official image of redis => redis:alpine
- container_name: The name of the container that appears in Azure
- volumes: The path of the volume to be used to back up Redis data
<name of the volume defined in this docker compose>:/data
RabbitMQ
This service is a powerful queue manager allowing the reception and scheduling of processing requests. Until a process has been assigned to a worker and completed, it will be parked in a queue. This also allows for resiliency to failures, if the server were to be restarted, it would resume the last processing in the queue where it left off.
As with the Redis service image, you have the option to use a public version (rabbitmq:3-management-alpine) or the one in the private dqeone registry.
Parameters
- image : Official image of rabbitmq => rabbitmq:3-management-alpine
- container_name : Name of the container that appears in aws
2. Installation
2 scenarios are possible :
-
Scenario 1: The customer does not yet have a Load Balancer configured on their VM
Follow all the steps - Scenario 2: The GCP client-user already has a VM with Load Balancer configured
2.1. Prequisites
Launch the dedicated server instance and connect with one of the connection options:
For more details on accessing instances through the GCP console, see the official documentation on GCP website.
NGINX installation
$ sudo apt update
$ sudo apt install nginx
Configuration :
- If the VM is only accessible by the load balancer (private vnet for example)
server {
listen 80;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://127.0.0.1:8001;
}
Location /api/ {
rewrite /api/(.*) /$1 break;
proxy_set_header Host $host;
proxy_pass http://127.0.0.1:8000;
}
- If the VM is directly opened in https:
server {
listen 443 ssl;
server_name <e.g. exemple.com>;
ssl_certificate < location of the .crt certificate>
ssl_certificate_key < location of the private key.key>
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers HIGH: !aNULL: ! MD5;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://127.0.0.1:8001;
}
Location /api/ {
rewrite /api/(.*) /$1 break;
proxy_set_header Host $host;
proxy_pass http://127.0.0.1:8000;
}
}
Docker installation
- Update the existing package
$ sudo apt-get update - Installing prerequisites for using packages over HTTPS
$ sudo apt install apt-transport-https ca-certificates curl software-properties-common
- Adding the GPG key from the official Docker repository
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
- Adding the Docker repository to APT sources
$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"
- Updating packages
$ sudo apt-get update
- Verification installation from Docker repository
$ apt-cache policy docker-ce
- Docker installation
$ sudo apt install docker-ce
Docker-compose installation
- Download the desired version (Version number to change in the url)
$ sudo curl -L "https://github.com/docker/compose/releases/download/<NUMEROS_DE_VERSION>/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
- Setting permissions
$ sudo chmod +x /usr/local/bin/docker-compose
Azure-cli installation
$ sudo curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
2.2. GCP Unify Server
version: '3.8'
services:
rabbitmq:
image: rabbitmq:3-management-alpine
container_name: rabbitmq
hostname: rabbitmq
logging:
driver: none
depends_on:
- redis
environment:
RABBITMQ_DEFAULT_PASS: guest
RABBITMQ_DEFAULT_USER: guest
RABBITMQ_DEFAULT_VHOST: admin
volumes:
- rabbitmq_data:/var/lib/rabbitmq/
- rabbitmq_log:/var/log/rabbitmq/
ports:
- "15672:15672"
redis:
container_name: redis
image: "redis:alpine"
hostname: redis
logging:
driver: none
ports:
- "6379:6379"
unify:
container_name: one-server
image: <imageURL>
expose:
- "8000"
ports:
- "8000:8000"
depends_on:
- redis
- rabbitmq
environment:
- SFAPIVERSION=v57.0
- WAIT_HOSTS= redis:6379
- WAIT_HOSTS_TIMEOUT=300
- WAIT_SLEEP_INTERVAL=5
- WAIT_HOST_CONNECT_TIMEOUT=30
- REDIS_URL=redis://redis
- CLOUDAMQP_URL=amqp://guest:guest@rabbitmq/admin
command:
- "bash"
- "./entrypoint.sh"
custom:
container_name: custom
image: <imageURL>
command: npm start
environment:
- PORT=8001
expose:
- "8001"
ports:
- "8001:8001"
worker:
container_name: queue-worker
image: <imageURL>
restart: always
depends_on:
- unify
- redis
- rabbitmq
environment:
- SFAPIVERSION=v57.0
- WORKDIRPATH=/app/unify
- REDIS_URL=redis://redis
- CLOUDAMQP_URL=amqp://guest:guest@rabbitmq/admin
command: python -u ./unify/queue_worker.py
volumes:
rabbitmq_data:
rabbitmq_log:
redis_data:
nginx.conf:
3. Launcher
3.1. Connecting DQE One Image to Azure Hub
Connect the docker to the ONE Server DQE image on Azure:
$ sudo docker login **************
Username: <Login provided by DQE>
Password: <Pwd provided by DQE>
3.2. Launch of docker-compose.yml
Go to the folder containing the docker-compose.yml and launch the.
$ sudo docker-compose up
Related to