Mustafa Bootwala

How to approach integraon with two systems: How do you get System “A” to talk with System “B”
Lets look at the overall components.
Middleware - It is the glue that holds two systems together
Many major companies like MuleSoft
Interoperability - Interoperability is the property that facilitates unrestricted sharing and use of data or resources between disparate systems via local area networks (LANs) or wide area networks (WANs) efficient automated data sharing between applications, databases, and other computer systems is a crucial component throughout networked computerized system
Protocols -
Data Transfer protocols
CSV import / export
XML Export / import
Webservices - APIs, using APIs is very common method of connecting different systems, RestAPis using JSON objects is a very common method
Webhooks - webhooks or HTTP callbacks are an alternative to APIs.
ISC and orchestrations

How to build a Robust Integration solution
Today in many companies we can see examples of data silos, in this age of digitalization we are seeing that more and more companies are integrating their systems (ERP, CRM, Accounts, Finance, etc) and levaraging the advantages of connected systems and data.
To make this happen we require a robust integration system

So what makes a robust integration system
Firstly, Its scalable - while desiging the integration system we need to make sure that is our system scalable?, are we taking into considerations the current and future growth of systems, new data types, increased number of data transfers, integrating more objects etc ?
All the future scenarios should be thought of and considered as
rebuilding later would cost a lot of time, efforts and money wasted.
Secondly, Easy to upgrade - once the new upgrades are available or necessary the system should be able to upgrade without affecting the current solution

Handle Data security
When we integrate two systems and data transfers from one to another there is high probability that data is exposed and company is in vulnerable position. So to avoid data security breach we need to ensure our data transfers securely via SSL encrypted transfers. Data should be read and write directly to and from the databases via the APIs and avoid data staging (i.e FTP server).

How to handle errors in integrations
When data transfers from one system to another we need to make sure that while designing the integration interface we have considered below
a) While importing data how to handle data error
b) How to handle connection error
Often times we are so focused on building interface that we forget to take care of data and connection issues

Firstly we need to start by finding the source of error
In most cases, the error is in
1) bad or poor designed connections, these connectors are not able to handle
all data / data types and crash
2) bad data or missing data, the destination system requires data in
certain format, certain required fields, but the data from the source is
not satisfying all the conditions and errors pop up

Secondly, the platform that we choose
If we are going with a integration platform like MuleSoft or Dellbhomi, then these platforms provide in built error handling tools. These should help us in solving most of the errors

Thirdly, if you are running periodic batches, then maintain a log of past weeks history, this way it is easy to track the data that was processed by the system

How to handle concurrency
In any system with multiple users and large systems there will be concurrent transations, these often lead to

Dirty read
A dirty read happens when a transaction accesses data that has been modified by another transaction but the change has not been committed or rolled back yet.

Dirty write
When two transactions access the same record and both update the same record. One transaction will overwrite the other transactions values, Thus it results in dirty write.
There are various other issues like Phantom reads, fuzzy reads

To solve these issues we need to make sure
1) That our transactions maintian their ACID properties
2) We use SQL servers default locks to avoid concurrency problems
3) Another option is to use row versions to support concurrency