What is the proper way to read a text file and display in HTML via Flask/Python?
Question: Specifically, let's talk about large text/log files that we want to display in
HTML. I've seen various methods, but for scalability purposes,

What I would like to do, is take a log file with space separated data and then
put this into a table displayed in HTML.

Example from the text file:

    
    
    2016-05-10 21:13:56.030616 Button_Pressed
    2016-05-10 21:14:44.534093 Door_Opened
    2016-05-10 21:14:46.850801 Button_Pressed
    2016-05-10 21:15:04.383880 Door_Closed
    

I'd like to take each line and convert this to a table with headings "Date,
Time, Event".

The easy way would be to use some kind of list, where we read each line into
the list, then pass the list to the HTML on a return.

Example Python code excerpt from a Flask web app (app.py):

    
    
    @app.route('/history')
    def history():
    
        with open('events.log') as event_log:
        event_lines = event_log.readlines()
    
        door_history = []
        events = len(event_lines)
    
        for x in range(events):
            door_history.append(event_lines[x].split(" "))
    
        return render_template('door-log.html', events=events, door_history=door_history)
    

Example HTML/Jinja code excerpt from the Flask web app (door-log.html)

    
    
    <table>
        <tr>
            <th>Date</th>
            <th>Time</th>
            <th>Event</th>
        </tr>
    
        {% for row in range(events) %}
            <tr>
                {% for column in range(3) %}
                <td> {{ door_history[row][column] }} </td>
                {% endfor %}
            </tr>
        {% endfor %}
    </table>
    

However, for this method, would this not be the best use of memory? Would it
scale if you had many people accessing the same page? Is this the proper way
to do this kind of thing?


Answer: I would not say that your way is exactly the proper way of doing this. However
for a small application it will work and be faster than most other solutions.

If your log file grows over time you would not want to show it in it's
entirety. Instead you might want to implement only showing the latest 10 or so
events, offering a second page with more events (pagination). In this case you
could probably still work with an adapted version of the above solution.

If however you either have lots of page views or want to have advanced
features such as: showing the last week of events, just showing door opening
events or showing all door opening events that happend in the first calendar
week. Then you would want to make your data more accessible by importing it
into some kind of database.

You might use SQLite as a database though if you really have a lot of users
something like PostgreSQL would be the better option. You would not import the
data in your views instead you would have a dedicated task running that
updates your database every now an again. In the simplest case this might be a
cronjob that just runs every minute. You might also use a more advanced
solution like celery, in either case you might end up with slightly outdated
data in your database and therefore your website, there isn't really a way
around this.

For information on how to integrate a database into your application see the
[official flask documentation](http://flask.pocoo.org/docs/0.10/tutorial/).
For more information on celery (for regular database updates) take a look at
the [flask documentation on the
topic](http://flask.pocoo.org/docs/0.10/patterns/celery/). The slightly more
hacky way to do this would be cron jobs, as explained
[here](https://help.ubuntu.com/community/CronHowto).

All that being said: if this is just a simple home project and you are not in
for the learning experience your current solution will 100% suffice. If you
are interested in learing more or want lots of people to use

