Asked  7 Months ago    Answers:  5   Viewed   34 times

I'm performing a simple task of uploading a file using Python requests library. I searched Stack Overflow and no one seemed to have the same problem, namely, that the file is not received by the server:

import requests
url='http://nesssi.cacr.caltech.edu/cgi-bin/getmulticonedb_release2.cgi/post'
files={'files': open('file.txt','rb')}
values={'upload_file' : 'file.txt' , 'DB':'photcat' , 'OUT':'csv' , 'SHORT':'short'}
r=requests.post(url,files=files,data=values)

I'm filling the value of 'upload_file' keyword with my filename, because if I leave it blank, it says

Error - You must select a file to upload!

And now I get

File  file.txt  of size    bytes is  uploaded successfully!
Query service results:  There were 0 lines.

Which comes up only if the file is empty. So I'm stuck as to how to send my file successfully. I know that the file works because if I go to this website and manually fill in the form it returns a nice list of matched objects, which is what I'm after. I'd really appreciate all hints.

Some other threads related (but not answering my problem):

  • Send file using POST from a Python script
  • http://docs.python-requests.org/en/latest/user/quickstart/#response-content
  • Uploading files using requests and send extra data
  • http://docs.python-requests.org/en/latest/user/advanced/#body-content-workflow

 Answers

26

If upload_file is meant to be the file, use:

files = {'upload_file': open('file.txt','rb')}
values = {'DB': 'photcat', 'OUT': 'csv', 'SHORT': 'short'}

r = requests.post(url, files=files, data=values)

and requests will send a multi-part form POST body with the upload_file field set to the contents of the file.txt file.

The filename will be included in the mime header for the specific field:

>>> import requests
>>> open('file.txt', 'wb')  # create an empty demo file
<_io.BufferedWriter name='file.txt'>
>>> files = {'upload_file': open('file.txt', 'rb')}
>>> print(requests.Request('POST', 'http://example.com', files=files).prepare().body.decode('ascii'))
--c226ce13d09842658ffbd31e0563c6bd
Content-Disposition: form-data; name="upload_file"; filename="file.txt"


--c226ce13d09842658ffbd31e0563c6bd--

Note the filename="file.txt" parameter.

You can use a tuple for the files mapping value, with between 2 and 4 elements, if you need more control. The first element is the filename, followed by the contents, and an optional content-type header value and an optional mapping of additional headers:

files = {'upload_file': ('foobar.txt', open('file.txt','rb'), 'text/x-spam')}

This sets an alternative filename and content type, leaving out the optional headers.

If you are meaning the whole POST body to be taken from a file (with no other fields specified), then don't use the files parameter, just post the file directly as data. You then may want to set a Content-Type header too, as none will be set otherwise. See Python requests - POST data from a file.

Tuesday, June 1, 2021
 
dmp
answered 7 Months ago
dmp
44
  • Make sure the file exists: use os.listdir() to see the list of files in the current working directory
  • Make sure you're in the directory you think you're in with os.getcwd() (if you launch your code from an IDE, you may well be in a different directory)
  • You can then either:
    • Call os.chdir(dir), dir being the folder where the file is located, then open the file with just its name like you were doing.
    • Specify an absolute path to the file in your open call.
  • Remember to use a raw string if your path uses backslashes, like so: dir = r'C:Python32'
    • If you don't use raw-string, you have to escape every backslash: 'C:\User\Bob\...'
    • Forward-slashes also work on Windows 'C:/Python32' and do not need to be escaped.

Let me clarify how Python finds files:

  • An absolute path is a path that starts with your computer's root directory, for example 'C:Pythonscripts..' if you're on Windows.
  • A relative path is a path that does not start with your computer's root directory, and is instead relative to something called the working directory. You can view Python's current working directory by calling os.getcwd().

If you try to do open('sortedLists.yaml'), Python will see that you are passing it a relative path, so it will search for the file inside the current working directory. Calling os.chdir will change the current working directory.

Example: Let's say file.txt is found in C:Folder.

To open it, you can do:

os.chdir(r'C:Folder')
open('file.txt') #relative path, looks inside the current working directory

or

open(r'C:Folderfile.txt') #full path
Tuesday, June 1, 2021
 
Guesser
answered 7 Months ago
62

PythonAnywhere dev here. This is a good question about Flask and web development in general rather than specific to our system, so I'll try to give a generic answer without anything specific to us :-)

There are a few things that I'd need to know to give a definitive answer to your question, so I'll start by listing the assumptions I'm making -- leave me a comment if I'm wrong with any of them and I'll update the answer appropriately.

  • I'm assuming that the files you're uploading aren't huge and can fit into a reasonable amount of memory -- let's say, smaller than a megabyte.
  • I'm assuming that the program that you've already written to generate the CSV from the text file is in Python, and that it has (or, perhaps more likely, could be easily changed to have) a function that takes a string containing the contents of the text file, and returns the contents that need to be written into the CSV.

If both of those are the case, then the best way to structure your Flask app would be to handle everything inside Flask. A code sample is worth a thousand words, so here's a simple one I put together that allows the user to upload a text file, runs it through a function called transform (which is where the function from your conversion program would slot in -- mine just replaces = with , throughout the file), and sends the results back to the browser. There's a live version of this app on PythonAnywhere here.

from flask import Flask, make_response, request

app = Flask(__name__)

def transform(text_file_contents):
    return text_file_contents.replace("=", ",")


@app.route('/')
def form():
    return """
        <html>
            <body>
                <h1>Transform a file demo</h1>

                <form action="/transform" method="post" enctype="multipart/form-data">
                    <input type="file" name="data_file" />
                    <input type="submit" />
                </form>
            </body>
        </html>
    """

@app.route('/transform', methods=["POST"])
def transform_view():
    request_file = request.files['data_file']
    if not request_file:
        return "No file"

    file_contents = request_file.stream.read().decode("utf-8")

    result = transform(file_contents)

    response = make_response(result)
    response.headers["Content-Disposition"] = "attachment; filename=result.csv"
    return response

Regarding your other questions:

  • Templates: I didn't use a template for this example, because I wanted it all to fit into a single piece of code. If I were doing it properly then I'd put the stuff that's generated by the form view into a template, but that's all.
  • Can you do it by writing to files -- yes you can, and the uploaded file can be saved by using the save(filename) method on the file object that I'm using the stream property of. But if your files are pretty small (as per my assumption above) then it probably makes more sense to process them in-memory like the code above does.

I hope that all helps, and if you have any questions then just leave a comment.

Sunday, August 8, 2021
 
Mawg says reinstate Monica
answered 4 Months ago
76

I would use argparse to create an option parser that accepts a file path and opens it.

import argparse

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('infile', type='open')
    args = parser.parse_args()

    for line in args.infile:
        print line

if __name__ == '__main__':
    main()

If type='open' does not provide enough control, it can be replaced with argparse.FileType('o') which accepts bufsize and mode args (see http://docs.python.org/dev/library/argparse.html#type)

EDIT: My mistake. This will not support your use case. This will allow you to provide a filepath, but not pipe the file contents into the process. I'll leave this answer here as it might be useful as an alternative.

Monday, August 16, 2021
 
NIKHIL
answered 4 Months ago
33

OP here, solution is as follows:

class SomeModelForm(forms.Form):
    csv_file = forms.FileField(required=False, label="please select a file")


class SomeModel(admin.ModelAdmin):
    change_list_template = 'admin/my_app/somemodel/change_list.html'

    def get_urls(self):
        urls = super().get_urls()
        my_urls = patterns("",
                           url(r"^upload_csv/$", self.upload_csv, name='upload_csv')
                       )
        return my_urls + urls

    urls = property(get_urls)

    def changelist_view(self, *args, **kwargs):
        view = super().changelist_view(*args, **kwargs)
        view.context_data['submit_csv_form'] = SomeModelForm
        return view

    def upload_csv(self, request):
        if request.method == 'POST':
            form = MineDifficultyResourceForm(request.POST, request.FILES)
            if form.is_valid():
                # process form

with the template overridden as so:

{% extends "admin/change_list.html" %}
{% load i18n admin_urls admin_static admin_list %}

{% block object-tools %}
    {% if has_add_permission %}
        <div>
            <ul class="object-tools">
                {% block object-tools-items %}
                    <form id="upload-csv-form" action="{% url 'admin:upload_csv' %}" method="post" enctype="multipart/form-data">
                    {% csrf_token %}
                        <p>{{ form.non_field_errors }}</p>
                        <p>{{ submit_csv_form.as_p }}</p>
                        <p>{{ submit_csv_form.csv_file.errors }}</p>
                        <p><input type="submit" value="Upload" />
                            <input type="reset" value="Reset"></p>
                    </form>
                {% endblock %}
            </ul>
        </div>
     {% endif %}
{% endblock %}

The form needs some custom validation but otherwise this solves the difficult part of customizing the admin page.

To elaborate what is going on here:

  1. get_urls is overridden so that an additional endpoint can be added to the admin page, this can point to any view, in this case it points upload_csv

  2. changelist_view is overridden to append the form info to the view

  3. the change_list.html template block "object-tools" is overridden with the form fields

Hopefully someone else finds this helpful as well.

Saturday, November 20, 2021
 
Mike Kwan
answered 2 Weeks ago
Only authorized users can answer the question. Please sign in first, or register a free account.
Not the answer you're looking for? Browse other questions tagged :  
Share