I am trying to get hug to receive a multipart/form-data POST request and stream body in chunks straight to disk. I was able to successfully upload stream a large binary file using application/octet-stream POST method. Here is my hug method:

And here is my curl snippet:

The above works and I'm able to stream upload the file like this because in the upload_file function, body is a gunicorn.http.body.Body instance which I am able to stream straight to disk in chunks.

However I need to be able to upload files from browser, which sends a multipart/form-data POST request. To emulate this with curl, I do:

This time, in hug, the body is a dictionary, and body['file'] is a Bytes instance. However I don't know how to stream this to disk without loading the whole thing in memory first.

Gta 5 fatal error fix. Is there a way I could obtain the body as a file object that I could stream straight to disk?

Any help much appreciated and thank you for the fantastic work on Hug!

I did some digging and the issue seems to stem from hug.input_formats.multipart, which invokes cgi.parse_multipart:

Here, body is a gunicorn.http.body.Body instance in my case, which is a file-like object.

cgi.parse_multipart reads the whole bytestream into memory before returning, which results in the behavior that I described in my original post. The docstring for cgi.parse_multipart indeed suggests that this is not suitable for large files, and that cgi.FieldStorage should be used instead:

I tried to go ahead and replace the call to parse_multipart with working with a FieldStorage instance, however I was unable to get any data through it:

Output from print(form):

@timothycrosley If you have any suggestion on how I could move forward with implementing this with FieldStorage I would be happy to work on this feature as I need it.

Update: Using this kind of invocation:

I was able to get an instance where form.fp is a file-like object I can stream, however the multipart is not parsed, i.e. other form items as well as the boundary are part of the stream.

My gut-feeling is that cgi.FieldStorage can be used to get the buffered file object without other form items, but I am not invoking it correctly. Any input much appreciated.

@timothycrosley I had success working this out by incorporating the multipart parser into hug.input_format.multipart:

What multipart.MultipartParser does is it writes form items to disk if their size exceeds mem_limit (default 2**20).

The form returned by multipart is still a dict, where the dict values are (filename, <io.BytesIO>) tuple if key is a file (e.g. file upload), and string otherwise. For example, for a request:

Hug example code:

The resulting form is: