Nota: All the elements described below constitute a recommendation from DQE related to the experience of deployments for our various customers.

The customer is responsible for the integration in its own architecture.

An architect familiar with the context and internal infrastructures will be in charge of making the link between DQE's recommendation and client infrastructure.

As part of the installation, we have "dockerized" the approach and all the components are deployed in this docker.

• Redis for data storage and processing
• RabbitMQ – Scheduler of actions performed by DQE engine
• The Redis database is deleted after each treatment

1. Deployment Architecture

The exchanges between the front and the back are detailed below.

To deploy an instance of the Unify Server Application on your Azure organization, DQE-Software provides a dedicated access to the container registry. There are different ways to deploy this container on your side, but we recommend to use an Azure Container Instance.

The procedure will be explained in the installation section.

This involves the configuration of an Azure Gateway, or any other load balancer to expose this application with a public IP address and a DNS. Otherwise, the Salesforce organization where you installed the user interface package will not be able to access the application.

To deploy this application you need to create a YAML configuration file that will be detailed in this document.

2. Flow Matrix

Here is a diagram describing all the flows coming in and out of the azure App gateway in a standard Unify data process.

Note that all the IP addresses and ports described in this diagram need to be opened in your gateway / firewall for the process to complete. All the incoming connections are made through an HTTPS channel.

3. Salesforce Connexions

Authentication

When you install the Unify-UI package on your Salesforce organization you will have to go through some configuration steps. During those steps the Salesforce org will register to the Unify Server. At this step the server will create a unique password and a key (credentials) that are stored in your org and used to authenticate to your Unify Server instance.

The security protocol used by Salesforce to authenticate the users and allow the various operations performed by the package (bulk import/export) is JSON Web Token (JWT).

This JWT can only be validated by Salesforce for the users you enabled in the connected app installed with the package Unify-UI.

You can find more information about it on the official Salesforce documentation here.

Process

Below you will find the description of the flows between Salesforce and the ACI Unify Server and between the ACI Unify Server and the DQE servers.

Step 1 – A Salesforce user defines a new process.

Description of the process

• • Object : Person Account
  • Type of treatment: Email validation
  • Processed field: Email
  • Filters: Cf Business rules

Step 2 - A Salesforce user starts the process. This user needs to be part of the authorized logins in the connected App setup.

Step 3 - The processing request sent to the ACI Unify Server

Step 4 - ACI Unify Server authenticates with Salesforce through the connected app using a JWT token

Step 5 - Salesforce allows the server to perform a data export

Step 6 - The ACI Unify Server exports the data concerned (example: the Id and Email fields of the Person Account object) and saves them in the Azure file in csv format.

Step 7

If Data Quality processing: The ACI Unify Server makes unit API calls (Anonymized) on each email in the extracted file. (Only for the qualification of emails).

During the whole process this is the only step where some data may be sent by REST API via a secure connexion to the DQE-Software production server.

If Deduplication processing: it calculates groups of duplicates and reconciles records according to the Matching rules defined during the workshops.

Step 8

The ACI Unify Server aggregates all processing responses into a final file in csv format.

If the processing is deduplication, the ACI Unify Server calculates the result of the data merge within the identified duplicate groups. This result will be stored in the "DQE_Fusion_Json_c" field created for this purpose in the Lead object while waiting to be potentially used if the merge process is triggered (automatically or not) at the end of the processing.

Step 9 - The ACI Unify Server authenticates to Salesforce.

Step 10 - Salesforce allows the server to perform a data IMPORT/UPDATE.

Step 11 - The ACI Unify Server sends the processing results in the relevant records (Here Person Account) through the Bulk APIs exposed by Salesforce. Person Account are enriched with several fields created specifically for this purpose such as the number of the duplicate group to which it has been attached, if applicable, as well as the field keeping the result of the merge of the data of this group.

Step 12 - The ACI Unify Server deletes the csv file initially received during export (step 6) and the generate csv file containing the results (step 8).

Step 13 - The ACI Unify Server sends a purely statistical processing report (number of records processed, processing time, ...) to Salesforce visible in the Unify application in the Runs object.

4. Installation

In this section, we describe the installation protocol to create an instance of DQE Unify Server as an Azure Container Instance.

We also provide an example of how to create an Azure Application Gateway.

However, this part will highly depend on your own organization. Technical team of the customer is responsible of the different recommendations.

If you need any further information on how to configure your Azure App Gateway, you may find some interesting resources here.

4.1. Create an application gateway

4.1.1. Basic TAB

The first tab should present a form as follows. You should fill in the different parts as shown below.

4.1.1.1. WAF Policy

The WAF (Web Application Firewall) is used to define the IPs range that will be authorized to call the Application Gateway (Salesforce’s Ips range).

Creating a new WAF from the “WAF policy” in the 4.1.1. Basic Tab.

4.1.1.1. VNET

The VNET (Virtual Network) is used to connect the Application Gateway and the ACI (Azure Container Instance). All the flows will pass through that private network.

Creating a new VNET from the “Virtual network” in the 4.1.1. Basic Tab.

4.1.2. Frontends

The first tab should present a form as follows. You should fill in the different parts as shown below.

4.1.2.1. Public IP Address

The public IP is the IP that will be exposed by the Application Gateway. It is used to route traffic to the ACI.

Creating a new Public IP from the “Public IP address” in the 4.1.2 Frontends

4.1.3. Backends

The first tab should present a form as follows. You should fill in the different parts as shown below.

4.1.3.1. Back Backend pool

The Backend pool will allow the Application Gateway to route the flows through the VNET to the ACI. It will automatically be filled in the “Network” of the ACI creation part.

Creating a new Backend pool from the “Add a backend pool” in the 4.1.2. Backends

4.1.4. Configuration

The first tab should present a form as follows. You should fill in the different parts as shown below.

4.1.4.1. Routing Rules - Listener

This configuration allows call to the Application Gateway through Https Protocol.

4.1.4.2. Routing Rules - Backend target

The Backend Target used by the Application Gateway to know where to route the incoming traffic. It will use the Backend pool previously created.

4.1.4.2. Routing Rules - Backend setting

The Backend setting is used by the Application Gateway to know which protocol to use when routing the traffic to the ACI. This traffic will be sent through the private VNET.

Then click on Review + create

4.2. Create a container instance from a YAML

4.2.1. Add a VNET Subnet for the ACI

Create a new Subnet delegate to “Microsoft.ContainerInstance/containerGroups” from the previously created in the 4.1.1.1. VNET

4.2.2. Azure CLI

Azure CLI is required on your computer.

4.2.2.1. UNIX

$ sudo curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

4.2.2.2. Windows

Please find the information here.

4.2.3. Create account storage

4.2.3.1. Basics tab

4.2.3.2. Advanced

Nothing to change.

4.2.3.3. Networking

4.2.3.4. Data protection

Nothing to change.

4.2.3.5. Encryption

Nothing to change.

4.2.3.6. Tags

Nothing to change.

4.2.3.7. Review + Create

Create the account storage

4.2.4. Create file share

The Container will need 3 file shares: rabbitvol, nginxconf and redisvol.

To use https on the ACI, Nginx will be provisioned. To ensure that it has all the information required, you need to add several files in the nginxconf file share:

• The SSL certificate .pem or .crt file
• The SSL Key file .key
• If the .pem has a password, add it in a text file and add that text file on the file share
• The default.conf file that is defined in the 4.2.8. Configure the nginx

4.2.5. Create log analytic worker

The Log Analytics Worker will be used to store (30 days by default) all the logs from the ACI.

4.2.5.1. Basics Tab

4.2.5.2. Review + Create

Create the Log Analytics Worker

4.2.5.2. Get the workspace Id/Key

4.2.6. ACI YAML configuration file

On your local machine create a YAML file. Find the YAML configuration file below.

You need to complete some keys:

• name: < Container Group Name >
• location: < Location >
• imageRegistryCredentials:
  • username: < Username provide by DQE >
  • password: < Password provide by DQE >
• diagnostic: loganalytics:
  • < Log Analytics Workspace Id > can be found 4.2.5.2. Get the workspace Id/Key
  • < Log Analytics Workspace Key > can be found 4.2.5.2. Get the workspace Id/Key
• All the <file share name>, <storage account name>, <storage account key>.

Example with the previously created storage account and file share:

<file share name>= redisvol

<storage account name>= myaccountstoragename

<storage account key>= (find it on the Account storage > Access keys > Show keys)

• Id: subscriptions/{subscriptionsId}/resourceGroups/{resourceGroupsName}/providers/Microsoft.Network/virtualNetworks/{VNETName}/subnets/{VNETSubnetsName}

Example: 

name: <Container Group Name> # Name of the container group
apiVersion: '2021-10-01'
location: <Location>
tags: {"docker-compose-application": "docker-compose-application"}
properties: # Properties of container group
containers: # Array of container instances in the group
# Redis Image configuration
- name: redis
properties: # Properties of an instance
image: ***** # Container image used to create the instance
resources: # Resource requirements of the instance
requests:
memoryInGB: 1
cpu: 0.5
volumeMounts: # Array of volume mounts for the instance
- name: redisvol
mountPath: /data

# RabbitMQ Image configuration
- name: rabbitmq # Name of an instance
properties: # Properties of an instance
image: ****** # Container image used to create the instance
ports: # External-facing ports exposed on the instance, must also be set in group ipAddress property
- protocol: TCP
port: 5672
environmentVariables:
- name: RABBITMQ_DEFAULT_PASS
value: guest
- name: RABBITMQ_DEFAULT_USER
value: guest
- name: RABBITMQ_DEFAULT_VHOST
value: admin

resources: # Resource requirements of the instance
requests:
memoryInGB: 1
cpu: 1
volumeMounts: # Array of volume mounts for the instance
- name: "rabbitvol"
mountPath: /bitnami
readOnly: false

# Nginx Image configuration
- name: nginx
properties: # Properties of an instance
image: ********* # Container image used to create the instance
ports: # External-facing ports exposed on the instance, must also be set in group ipAddress property
- protocol: TCP
port: 80
resources: # Resource requirements of the instance
requests:
memoryInGB: 1
cpu: 0.25
volumeMounts: # Array of volume mounts for the instance
- name: nginxconf
mountPath: /etc/nginx/conf.d

# Unify UI web server Image configuration
- name: unify-ui
properties: # Properties of an instance
image: ******* # Container image used to create the instance
command:
- "npm"
- "start"
ports: # External-facing ports exposed on the instance, must also be set in group ipAddress property
- protocol: TCP
port: 8001
environmentVariables:
- name: SESSION_SECRET
value: myveryimportantSecret
- name: PORT
value: 8001
resources: # Resource requirements of the instance
requests:
memoryInGB: 1
cpu: 0.25

# Unify web server Image configuration
- name: one-server
properties: # Properties of an instance
image: ******* # Container image used to create the instance
command:
- "bash"
- "./entrypoint.sh"
ports: # External-facing ports exposed on the instance, must also be set in group ipAddress property
- protocol: TCP
port: 8000
environmentVariables:
- name: SFAPIVERSION 
value: v59.0
- name: REDIS_URL
value: redis://127.0.0.1:6379
- name: CLOUDAMQP_URL
value: amqp://guest:guest@127.0.0.1:5672/admin
- name: CUSTOMUI
value: http://127.0.0.1:8001
- name: UNIFYSERVERURL
value: http://127.0.0.1:8000
- name: PORT
value: 8000
resources: # Resource requirements of the instance
requests:
memoryInGB: 1
cpu: 0.25

# Unify Worker Image configuration
- name: queue-worker
properties: # Properties of an instance
image: ****** # Container image used to create the instance
command:
- "python"
- "-u"
- "./unify/queue_worker.pyc"
environmentVariables:
- name: SFAPIVERSION 
value: v59.0
- name: WORKDIRPATH 
value: /app/unify
- name: REDIS_URL
value: redis://127.0.0.1:6379
- name: CLOUDAMQP_URL
value: amqp://guest:guest@127.0.0.1:5672/admin
resources: # Resource requirements of the instance
requests:
memoryInGB: 5
cpu: 1

# Credential to pull the Unify image from private container
imageRegistryCredentials:
- server: **************
username: <Username provide by DQE>
password: <Password provide by DQE>

diagnostics:
logAnalytics:
workspaceId: <Log Analytics Workspace Id>
workspaceKey: <Log Analytics Workspace Key>

restartPolicy: Always
ipAddress: # IP address configuration of container group
ports:
- protocol: TCP
port: 80
type: Private
osType: Linux

# Volumes parametre (azure fileshared)
volumes: # Array of volumes available to the instances
- name: rabbitvol
azureFile:
shareName: <file share name>
readOnly: false
storageAccountName: <storage account name>
storageAccountKey: <storage account key>

- name: nginxconf
azureFile:
shareName: <file share name>
readOnly: false
storageAccountName: <storage account name>
storageAccountKey: <storage account key>

- name: redisvol
azureFile:
shareName: <file share name>
readOnly: false
storageAccountName: <storage account name>
storageAccountKey: <storage account key>

subnetIds: # Subnet to deploy the container group into
- id: subscriptions/{subscriptions Id}/resourceGroups/{resourceGroupsName}/providers/Microsoft.Network/virtualNetworks/{VNETName}/subnets/{VNETSubnetsName}

4.2.7. Create the ACI

On your local machine, open a command prompt, and login to your Azure with Azure CLI:

$ az login

Create the ACI from the previously YAML file:

$ az container create -g < your Ressource Group Name > -f < .yaml file path >

After the operation is finished, you should get those informations:

4.2.8. Configure the nginx

Create a default.conf file that contain the following content: (This configuration is for http connection between the gateway and the ACI through the private subnet)

server {
listen 80;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://127.0.0.1:8001;
}

location /unify/ {
rewrite /unify/(.*) /$1 break;
proxy_set_header Host $host;
proxy_pass http://127.0.0.1:8000;
}
}

Upload this file in the nginxconf file share previously created.

/!\ Restart the ACI in order for the Nginx to update with the last change.

4.3. SETTING UP THE APPLICATION GATEWAY BACKEND POOL

After the ACI is created, go to the previously created Application Gateway > Backend Pool > {Your Backend Pool} > Edit the backend pool.

Add the Private IP of your created ACI

4.4. SETTING UP THE HEALTH PROBE OF THE APPLICATION GATEWAY

The health probe is used by the Application Gateway to frequently check if the API on the ACI is up. Set the health probe then click on Test. If the Status is Healthy, the ACI is properly configured and started.

Adding a health probe from the Application Gateway:

Click on Test > Add

4.4.1. Setup the DNS 

You must contact your administrators and ask them (or anybody having access to the DNS zone) to add a DNS name pointing to your Application Gateway public IP.

4.4.1. Waf configuration

The WAF is used to authorized or block specific IP that try to call the Application Gateway. For the Salesforce processes to work, the WAF needs to authorize the Salesforce’s IP range.

In the WAF click on “switch to prevention mode”. That will allow the WAF to use the below custom rules.

In the Custom Rules, add:

• Salesforce IP address range. (basic and hyperforce)
• DQE Check license server: ask the support
• SF Automation 1 : ask the support
• SF Automation 2 : ask the support

You can find the exhaustive list of Salesforce IP addresses here.

Once you finished those steps you can check your deployment 4.4. SETTING UP THE HEALTH PROBE OF THE APPLICATION GATEWAY