folder_paths = %w(abortion  creation  gayRights  god  guns  healthcare).map {|f|"debates/#{f}/*"}

folder_paths.each do |path|
  puts path
  Dir.glob(path) do |file|
    file = File.open(file)
    contents = file.readlines.reject { |x| x[0] == '#' }
    contents.map! { |l| l.chars.select(&:valid_encoding?).join }
    contents.map! { |l| l.gsub("/>", "") }
    contents.map! { |l| l.gsub(" / ", "") }
    File.open(file, "w").write(contents.join("\n"))
  end
end

Add corpora processing script to cut invalid chars
Add corpora processing script to cut invalid chars

Motivation:
-the encoding / chars in the corpus are a little all over the place, now
that I'm doing all the topics it made sense to automate this.
-had only done the abortion folder before

Change Explanation:
-iterate all the files for each topic
-remove the tags / meta data comments
-remove invalid chars
