<p>Are there any good programs for dealing with reading large CSV files?  Some of the datafiles I deal with are in the 1 GB range. They have too many lines for Excel to even deal with. Using Access can be a little slow, as you have to actually import them into a database to work with them directly.  Is there a program that can open large CSV files and give you a simple spreadsheet layout to help you easily and quickly scan through the data?</p>

[{'answer_id': 44290, 'author': 'kemiller2002', 'author_id': 1942, 'author_profile': 'https://Stackoverflow.com/users/1942', 'pm_score': 2, 'selected': False, 'text': '<p><a href="http://www.vedit.com/" rel="nofollow noreferrer">vEdit</a> is great for this.  I routinely open up 100+ meg (i know you said up to one gig, I think they advertise on their site it can handle twice that) files with it.  It has regex support and loads of other features.  70 dollars is cheap for the amount you can do with it.</p>\n'}, {'answer_id': 44302, 'author': 'jason saldo', 'author_id': 1293, 'author_profile': 'https://Stackoverflow.com/users/1293', 'pm_score': 0, 'selected': False, 'text': '<p>vEdit is great but don\'t forget you can always go back to "basics" check out <a href="http://www.cygwin.com/" rel="nofollow noreferrer">Cygwin</a> and start greping.</p>\n\n<p>Helpfull commands</p>\n\n<ul>\n<li>grep  </li>\n<li>head</li>\n<li>tail</li>\n<li>of course perl!</li>\n</ul>\n'}, {'answer_id': 44306, 'author': 'EBGreen', 'author_id': 1358, 'author_profile': 'https://Stackoverflow.com/users/1358', 'pm_score': 1, 'selected': False, 'text': '<p>GVim can handle files that large for free if you are not attached to a true spreadsheet static field size view.</p>\n'}, {'answer_id': 44318, 'author': 'Andrew', 'author_id': 826, 'author_profile': 'https://Stackoverflow.com/users/826', 'pm_score': 0, 'selected': False, 'text': "<p>It depends on what you actually want to do with the data. Given a large text file like that you typically only want a smaller subset of the data at any one time, so don't overlook tools like 'grep' for pulling out the pieces you want to look for and work with.</p>\n"}, {'answer_id': 44955, 'author': 'David Locke', 'author_id': 1447, 'author_profile': 'https://Stackoverflow.com/users/1447', 'pm_score': 0, 'selected': False, 'text': '<p>If you can fit the data into memory and you like python then I recommend checking out the UniTable portion of <a href="http://sourceforge.net/projects/augustus" rel="nofollow noreferrer" title="Augustus">Augustus</a>.  (Disclaimer: Augustus is open source (GPLv2) but I work for the company that writes it.)</p>\n\n<p>It\'s not very well documented but this should help you get going.</p>\n\n<pre><code>from augustus.kernel.unitable import *\na = UniTable().from_csv_file(\'filename\')\nb = a.subtbl(a[\'key\'] == some_value) #creates a subtable\n</code></pre>\n\n<p>It won\'t directly give you an excel like interface but with a little bit of work you can get many statistics out quickly.</p>\n'}, {'answer_id': 45018, 'author': 'Jordi Bunster', 'author_id': 4272, 'author_profile': 'https://Stackoverflow.com/users/4272', 'pm_score': 4, 'selected': True, 'text': '<p>MySQL can import CSV files very quickly onto tables using the <a href="http://dev.mysql.com/doc/refman/5.0/en/load-data.html" rel="noreferrer"><code>LOAD DATA INFILE</code></a> command. It can also read from CSV files directly, bypassing any import procedures, by using the <a href="http://dev.mysql.com/doc/refman/5.0/en/csv-storage-engine.html" rel="noreferrer">CSV storage engine</a>.</p>\n\n<p>Importing it onto native tables with <code>LOAD DATA INFILE</code> has a start up cost, but after that you can <code>INSERT/UPDATE</code> much faster, as well as index fields. Using the CSV storage engine is almost instantaneous at first, but only sequential scan will be fast.</p>\n\n<p><strong>Update</strong>: <a href="http://dev.mysql.com/tech-resources/articles/csv-storage-engine.html" rel="noreferrer">This article</a> (scroll down to the section titled <em>Instant Data Loads</em>) talks about using both approaches to loading CSV data onto MySQL, and gives examples.</p>\n'}, {'answer_id': 12475241, 'author': 'David Sealey', 'author_id': 1679984, 'author_profile': 'https://Stackoverflow.com/users/1679984', 'pm_score': 3, 'selected': False, 'text': '<p>I\'ve found <a href="http://recsveditor.sourceforge.net/" rel="noreferrer">reCSVeditor</a> is a great program for editing large CSV files. It\'s ideal for stripping out unnecessary columns. I\'ve used it for files 1,000,000 record files quite easily.</p>\n'}]
['https://Stackoverflow.com/questions/44286', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/1862/']