Azure Container Instance (ACI) — Salesforce installation

Note: All elements described below are recommendations from DQE, based on deployment experience with different customers.

The customer is responsible for integrating the solution into their own architecture.

An architect familiar with the customer context and internal infrastructure should align DQE's recommendations with the client infrastructure.

As part of the installation, the approach has been dockerized and all components are deployed in Docker:

• Redis: used for data storage and processing.
• RabbitMQ: schedules actions performed by the DQE engine.
• The Redis database is deleted after each process.

1. Deployment Architecture

The exchanges between the front end and the back end are detailed below.

To deploy an instance of the Unify Server application on your Azure organization, DQE-Software provides dedicated access to the container registry. There are several ways to deploy this container, but DQE recommends using an Azure Container Instance.

The procedure is explained in the installation section.

This requires the configuration of an Azure Gateway, or another load balancer, to expose this application with a public IP address and DNS. Otherwise, the Salesforce organization where the user interface package is installed will not be able to access the application.

To deploy this application, you need to create a YAML configuration file, which is detailed in this document.

2. Flow Matrix

The diagram below describes all flows coming in and out of the Azure Application Gateway in a standard Unify data process.

All IP addresses and ports described in this diagram must be opened in your gateway or firewall for the process to complete. All incoming connections use HTTPS.

3. Salesforce Connections

Authentication

When you install the Unify-UI package on your Salesforce organization, you must complete several configuration steps. During these steps, the Salesforce org registers with the Unify Server. At this stage, the server creates a unique password and key credentials, which are stored in your org and used to authenticate to your Unify Server instance.

The security protocol used by Salesforce to authenticate users and allow the operations performed by the package, such as bulk import and export, is JSON Web Token (JWT).

This JWT can only be validated by Salesforce for users enabled in the connected app installed with the Unify-UI package.

More information is available in the official Salesforce documentation here.

Process

The following section describes the flows between Salesforce and the ACI Unify Server, and between the ACI Unify Server and DQE servers.

Step 1 - Define a New Process

A Salesforce user defines a new process.

Example process description:

• Object: Person Account
• Type of processing: Email validation
• Processed field: Email
• Filters: see business rules.

Step 2 - Start the Process

A Salesforce user starts the process. This user must be part of the authorized logins in the connected app setup.

Step 3 - Send the Processing Request

The processing request is sent to the ACI Unify Server.

Step 4 - Authenticate with Salesforce

The ACI Unify Server authenticates with Salesforce through the connected app using a JWT token.

Step 5 - Export Data

Salesforce allows the server to perform a data export.

Step 6 - Save Exported Data

The ACI Unify Server exports the relevant data, such as the Id and Email fields of the Person Account object, and saves it in Azure File Storage in CSV format.

Step 7 - Process the Data

If the process is Data Quality processing: the ACI Unify Server makes anonymized unit API calls for each email in the extracted file. This applies only to email qualification.

During the whole process, this is the only step where some data may be sent by REST API through a secure connection to the DQE-Software production server.

If the process is Deduplication processing: the ACI Unify Server calculates duplicate groups and reconciles records according to the matching rules defined during the workshops.

Step 8 - Generate the Result File

The ACI Unify Server aggregates all processing responses into a final CSV file.

If the process is deduplication, the ACI Unify Server calculates the data merge result within the identified duplicate groups. This result is stored in the DQE_Fusion_Json_c field, created for this purpose in the Lead object, while waiting to be used if the merge process is triggered automatically or manually at the end of processing.

Step 9 - Authenticate to Salesforce

The ACI Unify Server authenticates to Salesforce.

Step 10 - Allow Import or Update

Salesforce allows the server to perform a data import or update.

Step 11 - Send Processing Results

The ACI Unify Server sends the processing results to the relevant records, for example Person Account, through the Bulk APIs exposed by Salesforce. Person Accounts are enriched with fields created for this purpose, such as the number of the duplicate group to which the record has been attached and, where applicable, the field storing the merge result for that group.

Step 12 - Delete CSV Files

The ACI Unify Server deletes the CSV file initially received during export in step 6 and the generated CSV file containing the results from step 8.

Step 13 - Send the Processing Report

The ACI Unify Server sends a statistical processing report to Salesforce, including the number of records processed and processing time. This report is visible in the Unify application in the Runs object.

4. Installation

This section describes the installation protocol used to create an instance of DQE Unify Server as an Azure Container Instance.

It also provides an example of how to create an Azure Application Gateway.

This part depends heavily on the customer's own organization. The customer's technical team is responsible for applying the different recommendations.

For more information about configuring an Azure Application Gateway, refer to the Microsoft documentation here.

4.1. Create an Application Gateway

4.1.1. Basic Tab

The first tab displays a form similar to the one below. Fill in the different parts as shown.

4.1.1.1. WAF Policy

The WAF (Web Application Firewall) is used to define the IP ranges authorized to call the Application Gateway, such as Salesforce IP ranges.

Create a new WAF from the WAF policy field in the 4.1.1. Basic Tab.

4.1.1.2. VNET

The VNET (Virtual Network) is used to connect the Application Gateway and the ACI (Azure Container Instance). All flows pass through this private network.

Create a new VNET from the Virtual network field in the 4.1.1. Basic Tab.

4.1.2. Frontends

The Frontends tab displays a form similar to the one below. Fill in the different parts as shown.

4.1.2.1. Public IP Address

The public IP address is exposed by the Application Gateway and is used to route traffic to the ACI.

Create a new public IP from the Public IP address field in 4.1.2. Frontends.

4.1.3. Backends

The Backends tab displays a form similar to the one below. Fill in the different parts as shown.

4.1.3.1. Backend Pool

The backend pool allows the Application Gateway to route flows through the VNET to the ACI. It is automatically filled in during the Network step of ACI creation.

Create a new backend pool from Add a backend pool in 4.1.3. Backends.

4.1.4. Configuration

The Configuration tab displays a form similar to the one below. Fill in the different parts as shown.

4.1.4.1. Routing Rules - Listener

This configuration allows calls to the Application Gateway through HTTPS.

4.1.4.2. Routing Rules - Backend Target

The backend target is used by the Application Gateway to determine where to route incoming traffic. It uses the backend pool created previously.

4.1.4.3. Routing Rules - Backend Setting

The backend setting is used by the Application Gateway to determine which protocol to use when routing traffic to the ACI. This traffic is sent through the private VNET.

Then click Review + create.

4.2. Create a Container Instance from a YAML File

4.2.1. Add a VNET Subnet for the ACI

Create a new subnet delegated to Microsoft.ContainerInstance/containerGroups from the VNET created in 4.1.1.2. VNET.

4.2.2. Azure CLI

Azure CLI is required on your computer.

4.2.2.1. UNIX

sudo curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

4.2.2.2. Windows

Refer to the Microsoft documentation here.

4.2.3. Create a Storage Account

4.2.3.1. Basics Tab

4.2.3.2. Advanced

Nothing to change.

4.2.3.3. Networking

4.2.3.4. Data Protection

Nothing to change.

4.2.3.5. Encryption

Nothing to change.

4.2.3.6. Tags

Nothing to change.

4.2.3.7. Review + Create

Create the storage account.

4.2.4. Create File Shares

The container requires three file shares:

• rabbitvol
• nginxconf
• redisvol

To use HTTPS on the ACI, Nginx is provisioned. Add the following files to the nginxconf file share:

• The SSL certificate .pem or .crt file. It must contain the server certificate first, followed by the intermediate certificates. A reversed order prevents Nginx from starting (key values mismatch error).
• The SSL key .key file, which must match the first certificate in the file.
• If the key is password-protected, a text file containing the password.
• The default.conf file defined in 4.2.8.

Before uploading, check that the certificate and the key match (both results must be identical):

openssl x509 -in <certificate> -noout -pubkey | openssl md5
openssl pkey -in <key> -pubout | openssl md5

Important: Salesforce checks the certificate presented by the server: recognized certificate authority, complete chain and validity date. Renew it before it expires and restart the ACI after each replacement.

4.2.5. Create a Log Analytics Workspace

The Log Analytics Workspace is used to store all logs from the ACI. By default, logs are stored for 30 days.

4.2.5.1. Basics Tab

4.2.5.2. Review + Create

Create the Log Analytics Workspace.

4.2.5.3. Get the Workspace ID and Key

4.2.6. ACI YAML Configuration File

On your local machine, create a YAML file. The YAML configuration file is provided below.

You must complete the following keys:

• name: <Container Group Name>
• location: <Location>
• imageRegistryCredentials.username: <Username provided by DQE>
• imageRegistryCredentials.password: <Password provided by DQE>
• diagnostics.logAnalytics.workspaceId: <Log Analytics Workspace ID>, available from 4.2.5.3. Get the Workspace ID and Key.
• diagnostics.logAnalytics.workspaceKey: <Log Analytics Workspace Key>, available from 4.2.5.3. Get the Workspace ID and Key.
• All <file share name>, <storage account name>, and <storage account key> values.

Example with the previously created storage account and file share:

• <file share name> = redisvol
• <storage account name> = myaccountstoragename
• <storage account key> = available from Storage account > Access keys > Show keys.
• id: subscriptions/{subscriptionsId}/resourceGroups/{resourceGroupsName}/providers/Microsoft.Network/virtualNetworks/{VNETName}/subnets/{VNETSubnetsName}

Example:

# =============================================================================
#  Unify sur Azure Container Instances (groupe privé dans un VNet, Nginx en frontal)
#  Déploiement : az container create -g <rg> --file aci-unify.yaml
#
#  Rappel ACI : tous les conteneurs du groupe partagent le même réseau -> 127.0.0.1
# =============================================================================
name: <Container Group Name>
apiVersion: '2023-05-01'
location: <Location>
properties:
  osType: Linux
  restartPolicy: Always

  containers:
    # ------------------------------------------------------------------ Redis
    - name: redis
      properties:
        image: <ImageURL>
        resources:
          requests:
            memoryInGB: 1
            cpu: 0.5
        volumeMounts:
          - name: redisvol
            mountPath: /data

    # --------------------------------------------------------------- RabbitMQ
    # Image officielle (comme dans le docker-compose) : variables RABBITMQ_DEFAULT_*.
    # Si l'image est Bitnami : variables RABBITMQ_USERNAME / RABBITMQ_PASSWORD /
    # RABBITMQ_VHOST et montage sur /bitnami/rabbitmq/mnesia.
    - name: rabbitmq
      properties:
        image: <ImageURL>
        environmentVariables:
          - name: RABBITMQ_DEFAULT_USER
            value: guest
          - name: RABBITMQ_DEFAULT_PASS
            secureValue: <RabbitMQ password>
          - name: RABBITMQ_DEFAULT_VHOST
            value: admin
          - name: RABBITMQ_NODENAME          # nom de nœud stable => données retrouvées après recréation
            value: rabbit@localhost
        resources:
          requests:
            memoryInGB: 1
            cpu: 1
        volumeMounts:
          # Uniquement mnesia : le .erlang.cookie (chmod 400) ne peut pas vivre sur un partage SMB
          - name: rabbitvol
            mountPath: /var/lib/rabbitmq/mnesia

    # ------------------------------------------------------------------ Nginx
    # La conf (conf.d) doit viser 127.0.0.1:8000 et 127.0.0.1:8001
    - name: nginx
      properties:
        image: <ImageURL>
        ports:
          - protocol: TCP
            port: 80
        resources:
          requests:
            memoryInGB: 1
            cpu: 0.25
        volumeMounts:
          - name: nginxconf
            mountPath: /etc/nginx/conf.d
            readOnly: true

    # --------------------------------------------------------------- Unify UI
    - name: unify-ui
      properties:
        image: <ImageURL>
        command: ["npm", "start"]
        environmentVariables:
          - name: SESSION_SECRET
            secureValue: <Session secret>
          - name: PORT
            value: '8001'
        resources:
          requests:
            memoryInGB: 1
            cpu: 0.25

    # ------------------------------------------------------------- One server
    - name: one-server
      properties:
        image: <ImageURL>
        # ACI n'a pas de depends_on : on attend Redis et RabbitMQ avant de lancer l'entrypoint
        command:
          - bash
          - -c
          - "until (echo > /dev/tcp/127.0.0.1/6379) 2>/dev/null && (echo > /dev/tcp/127.0.0.1/5672) 2>/dev/null; do echo waiting-for-redis-rabbitmq; sleep 3; done; exec bash ./entrypoint.sh"
        environmentVariables:
          - name: SFAPIVERSION
            value: v59.0                     # v65.0 dans le docker-compose : à aligner
          - name: PORT
            value: '8000'
          - name: REDIS_URL
            value: redis://127.0.0.1:6379
          - name: CLOUDAMQP_URL
            secureValue: amqp://guest:<RabbitMQ password>@127.0.0.1:5672/admin
          - name: CUSTOMUI
            value: http://127.0.0.1:8001
          - name: UNIFYSERVERURL
            value: http://127.0.0.1:8000     # URL Nginx si utilisée par un navigateur / callback
        resources:
          requests:
            memoryInGB: 2
            cpu: 1

    # ----------------------------------------------------------- Queue worker
    - name: queue-worker
      properties:
        image: <ImageURL>
        command:
          - bash
          - -c
          - "until (echo > /dev/tcp/127.0.0.1/5672) 2>/dev/null && (echo > /dev/tcp/127.0.0.1/8000) 2>/dev/null; do echo waiting-for-rabbitmq-web; sleep 3; done; exec python -u ./unify/queue_worker.pyc"
        environmentVariables:
          - name: SFAPIVERSION
            value: v59.0
          - name: WORKDIRPATH
            value: /app/unify
          - name: REDIS_URL
            value: redis://127.0.0.1:6379
          - name: CLOUDAMQP_URL
            secureValue: amqp://guest:<RabbitMQ password>@127.0.0.1:5672/admin
        resources:
          requests:
            memoryInGB: 5
            cpu: 1

  imageRegistryCredentials:
    - server: <Registry server>   # nom d'hôte seul (ex. xxx.azurecr.io), pas l'URL complète de l'image
      username: <Username provided by DQE>
      password: <Password provided by DQE>

  diagnostics:
    logAnalytics:
      workspaceId: <Log Analytics Workspace Id>
      workspaceKey: <Log Analytics Workspace Key>
      logType: ContainerInstanceLogs

  ipAddress:
    type: Private
    ports:
      - protocol: TCP
        port: 80

  subnetIds:
    # Le chemin doit commencer par "/" ; sous-réseau délégué à Microsoft.ContainerInstance/containerGroups
    - id: /subscriptions/<subscriptionId>/resourceGroups/<rg>/providers/Microsoft.Network/virtualNetworks/<vnet>/subnets/<subnet>

  # Un partage Azure Files DISTINCT par volume
  volumes:
    - name: redisvol
      azureFile:
        shareName: <redis-share>
        storageAccountName: <storage account name>
        storageAccountKey: <storage account key>
    - name: rabbitvol
      azureFile:
        shareName: <rabbitmq-share>
        storageAccountName: <storage account name>
        storageAccountKey: <storage account key>
    - name: nginxconf
      azureFile:
        shareName: <nginx-conf-share>
        readOnly: true
        storageAccountName: <storage account name>
        storageAccountKey: <storage account key>

4.2.7. Create the ACI

On your local machine, open a command prompt and log in to Azure with Azure CLI:

az login

Create the ACI from the YAML file:

az container create -g <your Resource Group Name> -f <.yaml file path>

After the operation is complete, you should see the following information:

4.2.8. Configure Nginx

Create a default.conf file containing the following content. This configuration is used for the HTTP connection between the gateway and the ACI through the private subnet.

server {
listen 80;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://127.0.0.1:8001;
}

location /unify/ {
rewrite /unify/(.*) /$1 break;
proxy_set_header Host $host;
proxy_pass http://127.0.0.1:8000;
}
}

Upload this file to the nginxconf file share previously created.

Warning: restart the ACI so that Nginx is updated with the latest change.

4.3. Setting Up the Application Gateway Backend Pool

After the ACI is created, go to Application Gateway > Backend Pool > your backend pool > Edit backend pool.

Add the private IP address of your created ACI.

4.4. Setting Up the Application Gateway Health Probe

The health probe is used by the Application Gateway to frequently check whether the API on the ACI is up. Set up the health probe, then click Test. If the status is Healthy, the ACI is properly configured and started.

Add a health probe from the Application Gateway:

Click Test > Add.

4.4.1. Set Up the DNS

Contact your administrators, or anyone with access to the DNS zone, and ask them to add a DNS name pointing to your Application Gateway public IP.

4.4.2. WAF Configuration

The WAF is used to authorize or block specific IP addresses trying to call the Application Gateway. For Salesforce processes to work, the WAF must authorize Salesforce IP ranges.

In the WAF, click Switch to prevention mode. This allows the WAF to use the custom rules below.

In the custom rules, add:

• Salesforce IP address ranges, including basic and Hyperforce ranges.
• DQE check license server: ask support.
• SF Automation 1: ask support.
• SF Automation 2: ask support.

You can find the exhaustive list of Salesforce IP addresses here.

After completing these steps, check your deployment as described in 4.4. Setting Up the Application Gateway Health Probe.