Recently, my ex-teammates had been engaged in a POC (Proof of Concept) test for the biggest Mobile operator in China . One of the test cases is an exception test. It requirements kill the process to simulate system crash and then restart the process. At last check if there are any data lost during the exception test. My teammates met some trouble in this test case; about twenty records of data lost, the total data count is about several hundred thousand. They sent the BW process to me and try to find out the reason that cause the data lost. The BW process is like this.
This BusinessWorks (BW) process receive message from a Queue in EMS server, process the message, then sent the result to another Queue in EMS server. The Acknowledge Mode of the JMS Queue Receiver is Auto. From this process, it’s very easy to find the problem, when the process had been killed during the process running, the EMS message had been received and acknowledged by JMS Queue Receiver, but it had not been processed completely, so this message lost. And in this test, we increase the thread count to 64 (default is 8) to get better performance, this means when the process had been killed, maybe less than 64 process threads are running, so we might get less than 64 records of data lost (we got about 20).
To resolve this problem, the messages should not be acknowledged on the server before the process finished running. If the message don’t acknowledge, process can re-receive the message again after it restart. So we modify the BW process like this:
To resolve this problem, the messages should not be acknowledged on the server before the process finished running. If the message don’t acknowledge, process can re-receive the message again after it restart. So we modify the BW process like this:
In this BW process, JMS Queue Receiver’s Acknowledge Mode is Client, and a Confirm activity had been added as the last activity of the process. The EMS message will only be acknowledged after the Confirm executed. If the process crashed, the EMS won’t be lost and it can be processed again after the process restart.
But there is problem in this process, too. There is a JMS Queue Sender in this process. It sends the processed message to another Queue. If use the above process, JMS Queue Sender maybe invoke repeated. For example, the process crash after JMS Queue Sender and before Confirm, the message won’t be acknowledge in the server, when the process restart, this message will be reprocessed. So in the test record, maybe some repeated records will be found. The BW process should be modified more.
But there is problem in this process, too. There is a JMS Queue Sender in this process. It sends the processed message to another Queue. If use the above process, JMS Queue Sender maybe invoke repeated. For example, the process crash after JMS Queue Sender and before Confirm, the message won’t be acknowledge in the server, when the process restart, this message will be reprocessed. So in the test record, maybe some repeated records will be found. The BW process should be modified more.
In this process, Confirm is added before JMS Queue Sender and a CheckPoint is added between Confirm and JMS Queue Sender. A checkpoint saves the current process data and state so that it can be recovered at a later time in the event of a failure. If a process engine fails, all process instances can be recovered and resume execution at the location of their last checkpoint in the process definition. If a process instance fails due to an unhandled exception or manual termination, it can optionally be recovered at a later time, if the process engine is configured to save checkpoint data for failed processes. So let’s check this process, if the process crash before Confirm, then it will reprocess the message after restart. If it crash after Checkpoint, then when the BW process archive restart, the process will continue execute from Checkpoint. Those can ensure the transaction integrality. But this process is not prefect too. If the process crash between Confirm and Checkpoint, then the data will lost. But the probability is tiny, because the Elapsed Time of Confirm is very shorter than JMS Queue Sender, it can’t take many time to execute, so the probability of the process crash at confirm is very low. Maybe lost one or two records in one million records; it’s an acceptable result in the POC. And the performance of this process is very good.
If the scenario is the customers’ product environment, it requires no data lost. How to configure the process? Use Transaction is a good idea. BW process is like this:
If the scenario is the customers’ product environment, it requires no data lost. How to configure the process? Use Transaction is a good idea. BW process is like this:
Add the Confirm and JMS Queue Sender into a Transaction. Only the two activities all success, the process will be finished. So wherever the process crash, the data won’t lost or duplicate. But the performance will a little lower when use the Transaction. We can use it according the customers’ actual product environment.
Here is only a very simple sample, the actual business environment will be more complex, those techniques should be used flexibility to keep the business system more reliability.
For training on TIBCO BusinessWorks mail us at [email protected]
Here is only a very simple sample, the actual business environment will be more complex, those techniques should be used flexibility to keep the business system more reliability.
For training on TIBCO BusinessWorks mail us at [email protected]
RSS Feed
