Porter v5 – Asynchronous, scalable and durable data import abstraction library for publishing and consuming APIs

Porter v5 now released, after two years in development and testing, with a new headline feature: high concurrency asynchronous imports.

This new version boasts large scale (hundreds of thousands) of CPU-bound data imports with async, built on top of Amp. Now we can import multiple data streams at once and not have to wait for each request to resolve before proceeding to the next. The new API has been refined and proven stable after running daily for the last two years in production.

Libraries such as Guzzle already offer async HTTP, but Porter is not bound to any specific protocol, and offers structure for data services. Rather than creating a class and putting all the service method calls into it, Porter provides opinionated interfaces (Provider and ProviderResource) to structure each API method call as separate classes. Following this structure provides better separation, easier testing, and access to additional Porter features, such durability and caching. Durability protects against intermittent network failures by default and caching avoids the need to request the same resource more than once. The interfaces also promote working with iterators instead of arrays, to process each record of a results set one at a time instead of attempting to load and process large data sets in memory at once.

Full details in the readme.