Way in the @future - Problem of using Salesforce as Real-time Relay System

Problem

For one of the project I was on, we had following as one of the requirements.  In this requirement, the System 1 would send a record to Salesforce, which would store the record, transform it based on the data stored in Salesforce and transmit the transformed record to System 2.  The whole process from System 1 to System 2 need to take less than 5 minute to complete.

Originally, we had implemented a trigger on the creation of the original record - Record A - that would transform it and send the transformation - Record A" - to System 2 by calling a web service through a future method.

This has been working as expected for quite sometime.  However, the production org was on NA14, and when the NA14 performance degradation occurred last year, we were witnessing that this request was taking more than half hours, and sometimes more than several hours.

Our investigation revealed that the problem was that the request to the future method gets stuck in the Apex Job Queue due to the degradation issue.  We had investigated both the batch apex and outbound-message workflow, but the problem was that both batch Apex and outbound-message jobs are queued the same way as the future method was.


Workaround

The way we addressed this problem was by implementing a custom web service.  This web service stored the Record A into Salesforce as well as transform the record the same way the trigger was doing.  We had proposed two solutions on how to transmit the transformed record to system 2.  One was for the custom web service make a callout; the second was to return the transformed record to System 1 as the response to the web service call, and System 1 would transmit it to System 2.  Our test revealed that both methods eliminated the queueing issue; however, the client went with the second option given the importance of this process.

Takeaway

The takeaway for me was that Salesforce really isn't a ideal tool for implementing a real-time processing of data, especially if you are using one of asynchronous methods (@future method, batch apex, etc.) as Salesforce do not provide any SLA on the timeliness of these processes.