1. Architecture
The application itself and all its dependencies are compiled in a docker-compose file that is provided by DQE-Software.
As soon as the container is deployed in an AWS instance (EC2, Elastic BeanStalk, ...), it pairs with a Salesforce organization with the Unify UI package (installation from the AppExchange).
This document describes a possible configuration for managing access to the deployed instance, such as through a LoadBalancer.
Flow Diagram
Here is the diagram describing all the inbound/outbound flows as well as the IPs and Ports requested by the application.
Security Measures
We cannot provide an example of how to implement the security layer that would fit perfectly into your architecture. However, we can provide some recommendations that are relevant to the use of the app itself.
- Protocols and Ports : All outgoing/incoming flows from the application go through the HTTPS protocol (port 443). There is no need to open another port on your VM. The DQE IPs that must be accessible are described in the preceding diagram.
- IP Filtering: The application server itself only needs to be accessible from Salesforce's servers. It is therefore recommended to set up a filtering of incoming IPs.
There is an exception to this, however, being the CustomUI web server included in the docker-compose. It must be accessible by a web browser. One solution could therefore be to allow a specific VPN to access the server. See here the list of IPs published by Salesforce. (Don't forget the Hyperforce IPs)
- Authentication protocol : Salesforce natively offers several protocols to secure requests between its servers and the DQE One server. When configuring the "DQE One Customer" Connected App in your organization, you will be able to select the "Basic Auth" authentication method and configure your application firewall accordingly.
Recommendation
Here of all the components needed for an instance of DQE One to be deployed on AWS.
Note: Those estimations are made on a base of 1 million records. Depending on the size of your databases, you might need to increase the capacities of your container instances to optimize the processing time.
Deployment with AWS EC2
- Server type: Unix, ex. Ubuntu Server 20.04 LTS (HVM),EBS General Purpose (SSD) Volume Type.
- Load Balancer: Need to be deployed in order for the vm to communicate with the exterior.
For more information, see the following resources:
- Amazon EC2 Setup:
https://docs.aws.amazon.com/fr_fr/AWSEC2/latest/UserGuide/get-set-up-for-amazon-ec2.html
https://docs.aws.amazon.com/fr_fr/AWSEC2/latest/UserGuide/ec2-best-practices.html
- EC2 instances:
https://docs.aws.amazon.com/fr_fr/AWSEC2/latest/UserGuide/Instances.html
https://aws.amazon.com/fr/ec2/instance-types/
Composition and services
The docker image is a web application that exposes the services and APIs that are requested from Salesforce through the Lightning DQE Unify UI app.
For each service, we outline the attributes and their functions below.
WEB APPLICATION
A backend container that contains a microservices application. This application exposes all the APIs called to launch processes, manage processing queues, instantiate processing workers. It is dependent on the following services to function properly. This worker must be exposed on the web as described in the nginx configuration presented later in this document.
Docker Compose settings
- image: The name of the image used by the web application. It is necessary to rename the image according to the names of the registries used. For images hosted in an Azure container registry, the name is constructed as follows:
<Name of the registry container>.azurecr.io/<name of the image>
- container_name: Container Name
- depends_on: Name of the services that precede this service
- domainname: The name of the domain for the FQDN. Azure Rated Result:
<domain name>
- environment: Service environment variables
- command: Command executed when the container is launched
Worker
A backend container that contains a microservices application. This application exposes all the APIs called to launch processes, manage processing queues, instantiate processing workers. It is dependent on the following services to function properly. This worker must be exposed on the web as described in the nginx configuration presented later in this document.
Parameters to set in Docker Compose
- image: The name of this image is the same as the web service
- container_name: The name of the container that appears in Azure
- depends_on: Name of the services that precede this service
- environment: Service environment variables
- command: Command executed when the container is launched
CustomUI
This service is a web application exposing a front used to configure deduplication rules. Like the first service, it must therefore also be exposed on the web. This service also consumes some APIs from the backend, for example to retrieve metadata from the CRM.
Registry Name: *
Image name: *
Available versions: *
Redis
This service is a key/value database used by the various departments to store internal operating keys such as Salesforce org configurations, sessions, etc.
It is possible to use official images published by Redis such as "redis:alpine", but as a security measure some cloud providers block the retrieval of public images and only allow images from private registries. To overcome this restriction, we also publish on the private registry "dqeone" a version of redis that is perfectly compatible with our applications.
Parameters
- image: Official image of redis => redis:alpine
- container_name: The name of the container that appears in Azure
- volumes: The path of the volume to be used to back up Redis data
<name of the volume defined in this docker compose>:/data
RabbitMQ
This service is a powerful queue manager allowing the reception and scheduling of processing requests. Until a process has been assigned to a worker and completed, it will be parked in a queue. This also allows for resiliency to failures, if the server were to be restarted, it would resume the last processing in the queue where it left off.
As with the Redis service image, you have the option to use a public version (rabbitmq:3-management-alpine) or the one in the private dqeone registry.
Parameters
- image : Official image of rabbitmq => rabbitmq:3-management-alpine
- container_name : Name of the container that appears in aws
2. Installation
2.1. Prerequisite
Launch the dedicated server instance and connect with one of the connection options offered:
For details on accessing instances through the AWS console, see the official documentation here.
NGINX installation
$ sudo apt update
$ sudo apt install nginx
In order to redirect port 80 to our API:
It will be necessary to add the key "location /" inside the key "server" in the configuration file of the Nginx located in "/etc/nginx/sites-enabled/default"
server {
listen 80;
location / {
proxy_pass http://127.0.0.1:8001;
}
location /api/ {
rewrite /api/(.*) /$1 break;
proxy_set_header Host $host;
proxy_pass http://127.0.0.1:8000;
}
}
Docker installation
- Update the existing package
$ sudo apt-get update - Installing prerequisites for using packages over HTTPS
$ sudo apt install apt-transport-https ca-certificates curl software-properties-common
- Adding the GPG key from the official Docker repository
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
- Adding the Docker repository to APT sources
$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"
- Updating packages
$ sudo apt-get update
- Verification installation from Docker repository
$ apt-cache policy docker-ce
- Docker installation
$ sudo apt install docker-ce
Docker-compose installation
- Download the desired version (Version number to change in the url)
$ sudo curl -L "https://github.com/docker/compose/releases/download/<NUMEROS_DE_VERSION>/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
- Setting permissions
$ sudo chmod +x /usr/local/bin/docker-compose
Azure-cli installation
$ sudo curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
CertBot installation
- Updating snapd
$ sudo snap install core; sudo snap refresh core
- Installing CERTBOT
$ sudo snap install --classic certbot
- Preparation of CERTBOT
$ sudo ln -s /snap/bin/certbot /usr/bin/certbot
- Creating the certificate
$ sudo certbot --nginx
2.2. AWS One Server
version: '3.8'
services:
rabbitmq:
image: rabbitmq:3-management-alpine
container_name: rabbitmq
hostname: rabbitmq
logging:
driver: none
depends_on:
- redis
environment:
RABBITMQ_DEFAULT_PASS: guest
RABBITMQ_DEFAULT_USER: guest
RABBITMQ_DEFAULT_VHOST: admin
volumes:
- rabbitmq_data:/var/lib/rabbitmq/
- rabbitmq_log:/var/log/rabbitmq/
ports:
- "15672:15672"
redis:
container_name: redis
image: "redis:alpine"
hostname: redis
logging:
driver: none
ports:
- "6379:6379"
unify:
container_name: one-server
image: <imageURL>
expose:
- "8000"
ports:
- "8000:8000"
depends_on:
- redis
- rabbitmq
environment:
- SFAPIVERSION=v57.0
- WAIT_HOSTS= redis:6379
- WAIT_HOSTS_TIMEOUT=300
- WAIT_SLEEP_INTERVAL=5
- WAIT_HOST_CONNECT_TIMEOUT=30
- REDIS_URL=redis://redis
- CLOUDAMQP_URL=amqp://guest:guest@rabbitmq/admin
command:
- "bash"
- "./entrypoint.sh"
custom:
container_name: custom
image: <imageURL>
command: npm start
environment:
- PORT=8001
expose:
- "8001"
ports:
- "8001:8001"
worker:
container_name: queue-worker
image: <imageURL>
restart: always
depends_on:
- unify
- redis
- rabbitmq
environment:
- SFAPIVERSION=v57.0
- WORKDIRPATH=/app/unify
- REDIS_URL=redis://redis
- CLOUDAMQP_URL=amqp://guest:guest@rabbitmq/admin
command: python -u ./unify/queue_worker.py
volumes:
rabbitmq_data:
rabbitmq_log:
redis_data:
nginx.conf:
3. Launcher
3.1. Connecting DQE One Image to Azure Hub
Connect the docker to the ONE Server DQE image on Azure:
$ sudo docker login **************
Username: <Login provided by DQE>
Password: <Pwd provided by DQE>
3.2. Launch of docker-compose.yml
Go to the folder containing the docker-compose.yml and launch the.
$ sudo docker-compose up
Related to