One of our biggest challenges is to manage customer and partner expectations when it comes to available (hosting) capacity for integrations. After all, there are a lot of variables to take into consideration when trying to estimate the required server capacity in advance.
We’ll name a few;
- what is the behaviour of the (external) endpoints
- how much data (simultaneously) comes in
- how many components are used
- how many records are processed simultaneously
- how often does a transformation take place
- what is the run frequency of the entire integration flow
- is processing done in memory or via a queue
- and more…
“Anything is possible”
The rise of cloud applications has contributed to the generic expectation that “anything is possible”. With that expectation, it seems that there is no cost component in hosting which feeds into the concept that anything should be possible.
In reality there is a cost component to it. We often compare it to taking a car trip from the Netherlands to the South of Spain. With a Fiat 500 at 140 km/h in one go to Malaga it is questionable whether you will make it to Malaga. With a Mercedes 220D, chances are that you will have no problems and you will even arrive relatively rested.
The route you choose and the speed you drive partly determines the risk and the total costs of the trip. A matter of making choices!
Expanded health check
Long story short, predicting the exactly required capacity is impossible. So the need to collect more numbers and statistics from technology is growing, not only to monitor hosting capacity, but also to find opportunities for optimisation.
In version 4.13.1 our health check API has been further expanded. Now we can get detailed information from the used instance(s) and docker containers. This gives us insight into memory configuration and usage and CPU usage per Docker container. In practice, we mainly see a need for detailed information about the peak use of memory in the Test and Production container.
Information from the health check API can be requested and saved every minute. This provides insight into the peak load and helps us determine when it is sensible to expand the underlying capacity. Timely expansion prevents downtime!
