Stream support for arrays of objects

[martin-moser]

Hi,

Maybe I don't get it, and you can help me. With OpenAPI I can describe an API that returns an array of objects, say Pets. Pets are simply structured objects an in swagger-samples\java\java-resteasy-appclasses (branch 2.0).
This is fine as long as I know how many Pets there are and their number is reasonably small. When it comes to huge numbers of objects where you sometimes do not know ahead of time how many there will be, I am stuck. What I think is needed is a "stream" type that leaves the size of the response open, and provides the response as "chunked". This allows the sender to send piece by piece and the receiver to process the incoming pieces one by one. The advantages of this kind of streaming are obvious, I assume. (Would also nicely interface with technologies like Java streams ...)
Any support for this kind of streams planned?

Best, Martin

[darrelmiller]

If you had to the response headers Transfer-encoding : chunked then you can indicate to the the consumer that the response is going to be returned in chunked using standard HTTP chunk encoding. The problem is that there really are not any JSON parsers that support stream parsing. So, even though you will get the payload back in chunks, the parser won't allow you to start processing the JSON response until it has been completely read.

There is an alternative mediate type called JSON Sequence https://tools.ietf.org/html/rfc7464 that supports returning many smaller JSON documents within a single payload. Your client could read that stream and then hand off each JSON document to your application.

I don't believe there is anything that OpenAPI needs to do to support this scenario.

[martin-moser]

Thanks for your quick response. I experimented a little with Jackson and came to the impression that the Jackson streaming API lets you read a chunked response in increments, parse it on the fly, and emit each parsed object into a Java stream as soon as it has been parsed. This is the behaviors that is needed in cloud implementations: you want to process streams of data rather than large blocks.
Integrating JSON Text Sequences might further improve an implementation as it can help to keep sync in case there is an error in the data stream.

Concerning what OpenAPI could do: I think it would be great if there was the type "stream" of objects, not only an array. Streaming is something fundamentally different from using an array: An array has a clearly defined size, while a stream does not. An array suggests "Read me in one block" while a stream suggests "Read me bit by bit."

A syntax could be:
type: stream
items:
$ref: '#/components/schemas/Pet'

[handrews]

@martin-moser the array and object type values come from JSON Schema, and there is no assumption that the array has a fixed length. So there would not be any difference with a stream type, and there are in fact streaming implementations of JSON Schema (no clue if any are integrated with OpenAPI). Arrays only have fixed length if the maxItems and minItems keywords are used.

If there are tools that don't handle indefinite-length arrays correctly, that issue should be raised with the tools rather than the spec.

I'm closing this since it's half a year since the original answer, and I don't see the type change as likely, but feel free to comment/re-open if needed.

[bobwebwen]

Hello,
I am running into the same situation as @martin-moser here, I can easily use JAVA spring @RestController to implement this API with returned type:
ResponseEntity<StreamingResponseBody>
but I can't define this using Open API, would want to know any further discussion on this?

my API defined using spring (example):

public ResponseEntity<StreamingResponseBody> getTradeStream(){
    return new ResponseEntity<>( x->tradeService.writeAllToStream(x), HttpStatus.OK);
}

All our other APIs are defined in Open API 3 yaml, this is now the only exceptional case, we can't find a way to define it with open API 3.

Thanks,
Bob

[handrews]

@martin-moser while it won't do anything special for streaming application/json, PR #4554 will add support for media types such as JSON Lines that are designed to be parsed in a streaming way, and can therefore be validated without needing all of JSON Schema to function in a streaming manner.