问:
Improve speed of complex postgres query in rails app
I have a view in my app that visualizes a lot of data, and in the backend the data is produced using this query:

DataPoint Load (20394.8ms)
SELECT communities.id as com,
       consumers.name as con,
       array_agg(timestamp ORDER BY data_points.timestamp asc) as tims,
       array_agg(consumption ORDER BY data_points.timestamp ASC) as cons
FROM "data_points"
     INNER JOIN "consumers" ON "consumers"."id" = "data_points"."consumer_id"
     INNER JOIN "communities_consumers" ON "communities_consumers"."consumer_id" = "consumers"."id"
     INNER JOIN "communities" ON "communities"."id" = "communities_consumers"."community_id"
     INNER JOIN "clusterings" ON "clusterings"."id" = "communities"."clustering_id"
WHERE ("data_points"."timestamp" BETWEEN $1 AND $2)
   AND "data_points"."interval_id" = $3
   AND "clusterings"."id" = 1
GROUP BY communities.id, consumers.id
[["timestamp", "2015-11-20 09:23:00"], ["timestamp", "2015-11-27 09:23:00"], ["interval_id", 2]]
```

<p>The query takes about 20 seconds to execute, which seems a bit excessive. </p>

<p>The code for generating the query is this:</p>

<pre class="lang-rb prettyprint-override"><code>res = {}
DataPoint.joins(consumer: {communities: :clustering} )
         .where('clusterings.id': self,
               timestamp: chart_cookies[:start_date] .. chart_cookies[:end_date],
               interval_id: chart_cookies[:interval_id])
         .group('communities.id')
         .group('consumers.id')
         .select('communities.id as com, consumers.name as con',
                'array_agg(timestamp ORDER BY data_points.timestamp asc) as tims',
                'array_agg(consumption ORDER BY data_points.timestamp ASC) as cons')
         .each do |d|
      res[d.com] ||= {}
      res[d.com][d.con] = d.tims.zip(d.cons)
      res[d.com]["aggregate"] ||= d.tims.map{|t| [t,0]}
      res[d.com]["aggregate"]  = res[d.com]["aggregate"].zip(d.cons).map{|(a,b),d| [a,(b+d)]}
end
res
</code>```

And the relevant database models are the following:

  create_table "data_points", force: :cascade do |t|
    t.bigint "consumer_id"
    t.bigint "interval_id"
    t.datetime "timestamp"
    t.float "consumption"
    t.float "flexibility"
    t.datetime "created_at", null: false
    t.datetime "updated_at", null: false
    t.index ["consumer_id"], name: "index_data_points_on_consumer_id"
    t.index ["interval_id"], name: "index_data_points_on_interval_id"
    t.index ["timestamp", "consumer_id", "interval_id"], name: "index_data_points_on_timestamp_and_consumer_id_and_interval_id", unique: true
    t.index ["timestamp"], name: "index_data_points_on_timestamp"
  end

  create_table "consumers", force: :cascade do |t|
    t.string "name"
    t.string "location"
    t.string "edms_id"
    t.bigint "building_type_id"
    t.bigint "connection_type_id"
    t.float "location_x"
    t.float "location_y"
    t.string "feeder_id"
    t.bigint "consumer_category_id"
    t.datetime "created_at", null: false
    t.datetime "updated_at", null: false
    t.index ["building_type_id"], name: "index_consumers_on_building_type_id"
    t.index ["connection_type_id"], name: "index_consumers_on_connection_type_id"
    t.index ["consumer_category_id"], name: "index_consumers_on_consumer_category_id"
  end

  create_table "communities_consumers", id: false, force: :cascade do |t|
    t.bigint "consumer_id", null: false
    t.bigint "community_id", null: false
    t.index ["community_id", "consumer_id"], name: "index_communities_consumers_on_community_id_and_consumer_id"
    t.index ["consumer_id", "community_id"], name: "index_communities_consumers_on_consumer_id_and_community_id"
  end

  create_table "communities", force: :cascade do |t|
    t.string "name"
    t.text "description"
    t.bigint "clustering_id"
    t.datetime "created_at", null: false
    t.datetime "updated_at", null: false
    t.index ["clustering_id"], name: "index_communities_on_clustering_id"
  end

  create_table "clusterings", force: :cascade do |t|
    t.string "name"
    t.text "description"
    t.datetime "created_at", null: false
    t.datetime "updated_at", null: false
  end
```

<p>How can I make the query execute faster? Is it possible to refactor the query to simplify it, or to add some extra index to the database schema so that it takes a shorter time?</p>

<p>Interestingly, a slightly simplified version of the query, which I use in another view, runs much faster, in only 1161.4ms for the first request and 41.6ms for the following requests:</p>

<pre class="lang-sql prettyprint-override"><code>DataPoint Load (1161.4ms)
SELECT consumers.name as con,
       array_agg(timestamp ORDER BY data_points.timestamp asc) as tims,
       array_agg(consumption ORDER BY data_points.timestamp ASC) as cons
FROM "data_points"
    INNER JOIN "consumers" ON "consumers"."id" = "data_points"."consumer_id"
    INNER JOIN "communities_consumers" ON "communities_consumers"."consumer_id" = "consumers"."id"
    INNER JOIN "communities" ON "communities"."id" = "communities_consumers"."community_id"
WHERE ("data_points"."timestamp" BETWEEN $1 AND $2)
   AND "data_points"."interval_id" = $3
   AND "communities"."id" = 100 GROUP BY communities.id, consumers.name
[["timestamp", "2015-11-20 09:23:00"], ["timestamp", "2015-11-27 09:23:00"], ["interval_id", 2]]
</code>```

Using command EXPLAIN (ANALYZE, BUFFERS) with query in dbconsole, I get the following output:

```<code>----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 GroupAggregate  (cost=12.31..7440.69 rows=246 width=57) (actual time=44.139..20474.015 rows=296 loops=1)
   Group Key: communities.id, consumers.id
   Buffers: shared hit=159692 read=6148105 written=209
   ->  Nested Loop  (cost=12.31..7434.54 rows=246 width=57) (actual time=20.944..20436.806 rows=49728 loops=1)
         Buffers: shared hit=159685 read=6148105 written=209
         ->  Nested Loop  (cost=11.88..49.30 rows=1 width=49) (actual time=0.102..6.374 rows=296 loops=1)
               Buffers: shared hit=988 read=208
               ->  Nested Loop  (cost=11.73..41.12 rows=1 width=57) (actual time=0.084..4.443 rows=296 loops=1)
                     Buffers: shared hit=396 read=208
                     ->  Merge Join  (cost=11.58..40.78 rows=1 width=24) (actual time=0.075..1.365 rows=296 loops=1)
                           Merge Cond: (communities_consumers.community_id = communities.id)
                           Buffers: shared hit=5 read=7
                           ->  Index Only Scan using index_communities_consumers_on_community_id_and_consumer_id on communities_consumers  (cost=0.27..28.71 rows=296 width=16) (actual time=0.039..0.446 rows=296 loops=1)
                                 Heap Fetches: 4
                                 Buffers: shared hit=1 read=6
                           ->  Sort  (cost=11.31..11.31 rows=3 width=16) (actual time=0.034..0.213 rows=247 loops=1)
                                 Sort Key: communities.id
                                 Sort Method: quicksort  Memory: 25kB
                                 Buffers: shared hit=4 read=1
                                 ->  Bitmap Heap Scan on communities  (cost=4.17..11.28 rows=3 width=16) (actual time=0.026..0.027 rows=6 loops=1)
                                       Recheck Cond: (clustering_id = 1)
                                       Heap Blocks: exact=1
                                       Buffers: shared hit=4 read=1
                                       ->  Bitmap Index Scan on index_communities_on_clustering_id  (cost=0.00..4.17 rows=3 width=0) (actual time=0.020..0.020 rows=8 loops=1)
                                             Index Cond: (clustering_id = 1)
                                             Buffers: shared hit=3 read=1
                     ->  Index Scan using consumers_pkey on consumers  (cost=0.15..0.33 rows=1 width=33) (actual time=0.007..0.008 rows=1 loops=296)
                           Index Cond: (id = communities_consumers.consumer_id)
                           Buffers: shared hit=391 read=201
               ->  Index Only Scan using clusterings_pkey on clusterings  (cost=0.15..8.17 rows=1 width=8) (actual time=0.004..0.005 rows=1 loops=296)
                     Index Cond: (id = 1)
                     Heap Fetches: 296
                     Buffers: shared hit=592
         ->  Index Scan using index_data_points_on_consumer_id on data_points  (cost=0.44..7383.44 rows=180 width=24) (actual time=56.128..68.995 rows=168 loops=296)
               Index Cond: (consumer_id = consumers.id)
               Filter: (("timestamp" >= '2015-11-20 09:23:00'::timestamp without time zone) AND ("timestamp" <= '2015-11-27 09:23:00'::timestamp without time zone) AND (interval_id = 2))
               Rows Removed by Filter: 76610
               Buffers: shared hit=158697 read=6147897 written=209
 Planning time: 1.811 ms
 Execution time: 20474.330 ms
(40 rows)
</code>```

The bullet gem returns the following warnings:

```<code>USE eager loading detected
  Community => [:communities_consumers]
  Add to your finder: :includes => [:communities_consumers]

USE eager loading detected
  Community => [:consumers]
  Add to your finder: :includes => [:consumers]
</code>```

After removing the join with the clusterings table, the new query plan is the following:

```<code>EXPLAIN for: SELECT communities.id as com, consumers.name as con, array_agg(timestamp ORDER BY data_points.timestamp asc) as tims, array_agg(consumption ORDER BY data_points.timestamp ASC) as cons FROM "data_points" INNER JOIN "consumers" ON "consumers"."id" = "data_points"."consumer_id" INNER JOIN "communities_consumers" ON "communities_consumers"."consumer_id" = "consumers"."id" INNER JOIN "communities" ON "communities"."id" = "communities_consumers"."community_id" WHERE ("data_points"."timestamp" BETWEEN $1 AND $2) AND "data_points"."interval_id" = $3 AND "communities"."clustering_id" = 1 GROUP BY communities.id, consumers.id [["timestamp", "2015-11-29 20:52:30.926247"], ["timestamp", "2015-12-06 20:52:30.926468"], ["interval_id", 2]]
                                                                                QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 GroupAggregate  (cost=10839.79..10846.42 rows=241 width=57)
   ->  Sort  (cost=10839.79..10840.39 rows=241 width=57)
         Sort Key: communities.id, consumers.id
         ->  Nested Loop  (cost=7643.11..10830.26 rows=241 width=57)
               ->  Nested Loop  (cost=11.47..22.79 rows=1 width=49)
                     ->  Hash Join  (cost=11.32..17.40 rows=1 width=16)
                           Hash Cond: (communities_consumers.community_id = communities.id)
                           ->  Seq Scan on communities_consumers  (cost=0.00..4.96 rows=296 width=16)
                           ->  Hash  (cost=11.28..11.28 rows=3 width=8)
                                 ->  Bitmap Heap Scan on communities  (cost=4.17..11.28 rows=3 width=8)
                                       Recheck Cond: (clustering_id = 1)
                                       ->  Bitmap Index Scan on index_communities_on_clustering_id  (cost=0.00..4.17 rows=3 width=0)
                                             Index Cond: (clustering_id = 1)
                     ->  Index Scan using consumers_pkey on consumers  (cost=0.15..5.38 rows=1 width=33)
                           Index Cond: (id = communities_consumers.consumer_id)
               ->  Bitmap Heap Scan on data_points  (cost=7631.64..10805.72 rows=174 width=24)
                     Recheck Cond: ((consumer_id = consumers.id) AND ("timestamp" >= '2015-11-29 20:52:30.926247'::timestamp without time zone) AND ("timestamp" <= '2015-12-06 20:52:30.926468'::timestamp without time zone))
                     Filter: (interval_id = 2::bigint)
                     ->  BitmapAnd  (cost=7631.64..7631.64 rows=861 width=0)
                           ->  Bitmap Index Scan on index_data_points_on_consumer_id  (cost=0.00..1589.92 rows=76778 width=0)
                                 Index Cond: (consumer_id = consumers.id)
                           ->  Bitmap Index Scan on index_data_points_on_timestamp  (cost=0.00..6028.58 rows=254814 width=0)
                                 Index Cond: (("timestamp" >= '2015-11-29 20:52:30.926247'::timestamp without time zone) AND ("timestamp" <= '2015-12-06 20:52:30.926468'::timestamp without time zone))
(23 rows)
</code>```

As requested in the comments, these are the query plans for the simplified query, with and without the restriction on communities.id

```<code> DataPoint Load (1563.3ms)  SELECT consumers.name as con, array_agg(timestamp ORDER BY data_points.timestamp asc) as tims, array_agg(consumption ORDER BY data_points.timestamp ASC) as cons FROM "data_points" INNER JOIN "consumers" ON "consumers"."id" = "data_points"."consumer_id" INNER JOIN "communities_consumers" ON "communities_consumers"."consumer_id" = "consumers"."id" INNER JOIN "communities" ON "communities"."id" = "communities_consumers"."community_id" WHERE ("data_points"."timestamp" BETWEEN $1 AND $2) AND "data_points"."interval_id" = $3 GROUP BY communities.id, consumers.name  [["timestamp", "2015-11-29 20:52:30.926000"], ["timestamp", "2015-12-06 20:52:30.926000"], ["interval_id", 2]]
EXPLAIN for: SELECT consumers.name as con, array_agg(timestamp ORDER BY data_points.timestamp asc) as tims, array_agg(consumption ORDER BY data_points.timestamp ASC) as cons FROM "data_points" INNER JOIN "consumers" ON "consumers"."id" = "data_points"."consumer_id" INNER JOIN "communities_consumers" ON "communities_consumers"."consumer_id" = "consumers"."id" INNER JOIN "communities" ON "communities"."id" = "communities_consumers"."community_id" WHERE ("data_points"."timestamp" BETWEEN $1 AND $2) AND "data_points"."interval_id" = $3 GROUP BY communities.id, consumers.name [["timestamp", "2015-11-29 20:52:30.926000"], ["timestamp", "2015-12-06 20:52:30.926000"], ["interval_id", 2]]
                                                                                QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 GroupAggregate  (cost=140992.34..142405.51 rows=51388 width=49)
   ->  Sort  (cost=140992.34..141120.81 rows=51388 width=49)
         Sort Key: communities.id, consumers.name
         ->  Hash Join  (cost=10135.44..135214.45 rows=51388 width=49)
               Hash Cond: (data_points.consumer_id = consumers.id)
               ->  Bitmap Heap Scan on data_points  (cost=10082.58..134455.00 rows=51388 width=24)
                     Recheck Cond: (("timestamp" >= '2015-11-29 20:52:30.926'::timestamp without time zone) AND ("timestamp" <= '2015-12-06 20:52:30.926'::timestamp without time zone) AND (interval_id = 2::bigint))
                     ->  Bitmap Index Scan on index_data_points_on_timestamp_and_consumer_id_and_interval_id  (cost=0.00..10069.74 rows=51388 width=0)
                           Index Cond: (("timestamp" >= '2015-11-29 20:52:30.926'::timestamp without time zone) AND ("timestamp" <= '2015-12-06 20:52:30.926'::timestamp without time zone) AND (interval_id = 2::bigint))
               ->  Hash  (cost=49.16..49.16 rows=296 width=49)
                     ->  Hash Join  (cost=33.06..49.16 rows=296 width=49)
                           Hash Cond: (communities_consumers.community_id = communities.id)
                           ->  Hash Join  (cost=8.66..20.69 rows=296 width=49)
                                 Hash Cond: (consumers.id = communities_consumers.consumer_id)
                                 ->  Seq Scan on consumers  (cost=0.00..7.96 rows=296 width=33)
                                 ->  Hash  (cost=4.96..4.96 rows=296 width=16)
                                       ->  Seq Scan on communities_consumers  (cost=0.00..4.96 rows=296 width=16)
                           ->  Hash  (cost=16.40..16.40 rows=640 width=8)
                                 ->  Seq Scan on communities  (cost=0.00..16.40 rows=640 width=8)
(19 rows)
</code>```

and

```<code>  DataPoint Load (1479.0ms)  SELECT consumers.name as con, array_agg(timestamp ORDER BY data_points.timestamp asc) as tims, array_agg(consumption ORDER BY data_points.timestamp ASC) as cons FROM "data_points" INNER JOIN "consumers" ON "consumers"."id" = "data_points"."consumer_id" INNER JOIN "communities_consumers" ON "communities_consumers"."consumer_id" = "consumers"."id" INNER JOIN "communities" ON "communities"."id" = "communities_consumers"."community_id" WHERE ("data_points"."timestamp" BETWEEN $1 AND $2) AND "data_points"."interval_id" = $3 GROUP BY communities.id, consumers.name  [["timestamp", "2015-11-29 20:52:30.926000"], ["timestamp", "2015-12-06 20:52:30.926000"], ["interval_id", 2]]
EXPLAIN for: SELECT consumers.name as con, array_agg(timestamp ORDER BY data_points.timestamp asc) as tims, array_agg(consumption ORDER BY data_points.timestamp ASC) as cons FROM "data_points" INNER JOIN "consumers" ON "consumers"."id" = "data_points"."consumer_id" INNER JOIN "communities_consumers" ON "communities_consumers"."consumer_id" = "consumers"."id" INNER JOIN "communities" ON "communities"."id" = "communities_consumers"."community_id" WHERE ("data_points"."timestamp" BETWEEN $1 AND $2) AND "data_points"."interval_id" = $3 GROUP BY communities.id, consumers.name [["timestamp", "2015-11-29 20:52:30.926000"], ["timestamp", "2015-12-06 20:52:30.926000"], ["interval_id", 2]]
                                                                                QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 GroupAggregate  (cost=140992.34..142405.51 rows=51388 width=49)
   ->  Sort  (cost=140992.34..141120.81 rows=51388 width=49)
         Sort Key: communities.id, consumers.name
         ->  Hash Join  (cost=10135.44..135214.45 rows=51388 width=49)
               Hash Cond: (data_points.consumer_id = consumers.id)
               ->  Bitmap Heap Scan on data_points  (cost=10082.58..134455.00 rows=51388 width=24)
                     Recheck Cond: (("timestamp" >= '2015-11-29 20:52:30.926'::timestamp without time zone) AND ("timestamp" <= '2015-12-06 20:52:30.926'::timestamp without time zone) AND (interval_id = 2::bigint))
                     ->  Bitmap Index Scan on index_data_points_on_timestamp_and_consumer_id_and_interval_id  (cost=0.00..10069.74 rows=51388 width=0)
                           Index Cond: (("timestamp" >= '2015-11-29 20:52:30.926'::timestamp without time zone) AND ("timestamp" <= '2015-12-06 20:52:30.926'::timestamp without time zone) AND (interval_id = 2::bigint))
               ->  Hash  (cost=49.16..49.16 rows=296 width=49)
                     ->  Hash Join  (cost=33.06..49.16 rows=296 width=49)
                           Hash Cond: (communities_consumers.community_id = communities.id)
                           ->  Hash Join  (cost=8.66..20.69 rows=296 width=49)
                                 Hash Cond: (consumers.id = communities_consumers.consumer_id)
                                 ->  Seq Scan on consumers  (cost=0.00..7.96 rows=296 width=33)
                                 ->  Hash  (cost=4.96..4.96 rows=296 width=16)
                                       ->  Seq Scan on communities_consumers  (cost=0.00..4.96 rows=296 width=16)
                           ->  Hash  (cost=16.40..16.40 rows=640 width=8)
                                 ->  Seq Scan on communities  (cost=0.00..16.40 rows=640 width=8)
(19 rows)
</code>```

答:
The index on data_points.timestamp is not being used, perhaps due to the ::timestamp conversion.

I wonder if altering the column datatype or creating a funtional index would help.

EDIT - the datetime in your CREATE TABLE is how Rails chooses to display the Postgres timestamp data type, I guess, and so there may be no conversion taking place after all.

Nevertheless, the index on timestamp is not being used but depending on your data distribution this could be a very smart choice by the optimizer.

Body:
I have a view in my app that visualizes a lot of data, and in the backend the data is produced using this query:

DataPoint Load (20394.8ms)
SELECT communities.id as com,
       consumers.name as con,
       array_agg(timestamp ORDER BY data_points.timestamp asc) as tims,
       array_agg(consumption ORDER BY data_points.timestamp ASC) as cons
FROM "data_points"
     INNER JOIN "consumers" ON "consumers"."id" = "data_points"."consumer_id"
     INNER JOIN "communities_consumers" ON "communities_consumers"."consumer_id" = "consumers"."id"
     INNER JOIN "communities" ON "communities"."id" = "communities_consumers"."community_id"
     INNER JOIN "clusterings" ON "clusterings"."id" = "communities"."clustering_id"
WHERE ("data_points"."timestamp" BETWEEN $1 AND $2)
   AND "data_points"."interval_id" = $3
   AND "clusterings"."id" = 1
GROUP BY communities.id, consumers.id
[["timestamp", "2015-11-20 09:23:00"], ["timestamp", "2015-11-27 09:23:00"], ["interval_id", 2]]

The query takes about 20 seconds to execute, which seems a bit excessive.

The code for generating the query is this:

res = {}
DataPoint.joins(consumer: {communities: :clustering} )
         .where('clusterings.id': self,
               timestamp: chart_cookies[:start_date] .. chart_cookies[:end_date],
               interval_id: chart_cookies[:interval_id])
         .group('communities.id')
         .group('consumers.id')
         .select('communities.id as com, consumers.name as con',
                'array_agg(timestamp ORDER BY data_points.timestamp asc) as tims',
                'array_agg(consumption ORDER BY data_points.timestamp ASC) as cons')
         .each do |d|
      res[d.com] ||= {}
      res[d.com][d.con] = d.tims.zip(d.cons)
      res[d.com]["aggregate"] ||= d.tims.map{|t| [t,0]}
      res[d.com]["aggregate"]  = res[d.com]["aggregate"].zip(d.cons).map{|(a,b),d| [a,(b+d)]}
end
res

And the relevant database models are the following:

  create_table "data_points", force: :cascade do |t|
    t.bigint "consumer_id"
    t.bigint "interval_id"
    t.datetime "timestamp"
    t.float "consumption"
    t.float "flexibility"
    t.datetime "created_at", null: false
    t.datetime "updated_at", null: false
    t.index ["consumer_id"], name: "index_data_points_on_consumer_id"
    t.index ["interval_id"], name: "index_data_points_on_interval_id"
    t.index ["timestamp", "consumer_id", "interval_id"], name: "index_data_points_on_timestamp_and_consumer_id_and_interval_id", unique: true
    t.index ["timestamp"], name: "index_data_points_on_timestamp"
  end

  create_table "consumers", force: :cascade do |t|
    t.string "name"
    t.string "location"
    t.string "edms_id"
    t.bigint "building_type_id"
    t.bigint "connection_type_id"
    t.float "location_x"
    t.float "location_y"
    t.string "feeder_id"
    t.bigint "consumer_category_id"
    t.datetime "created_at", null: false
    t.datetime "updated_at", null: false
    t.index ["building_type_id"], name: "index_consumers_on_building_type_id"
    t.index ["connection_type_id"], name: "index_consumers_on_connection_type_id"
    t.index ["consumer_category_id"], name: "index_consumers_on_consumer_category_id"
  end

  create_table "communities_consumers", id: false, force: :cascade do |t|
    t.bigint "consumer_id", null: false
    t.bigint "community_id", null: false
    t.index ["community_id", "consumer_id"], name: "index_communities_consumers_on_community_id_and_consumer_id"
    t.index ["consumer_id", "community_id"], name: "index_communities_consumers_on_consumer_id_and_community_id"
  end

  create_table "communities", force: :cascade do |t|
    t.string "name"
    t.text "description"
    t.bigint "clustering_id"
    t.datetime "created_at", null: false
    t.datetime "updated_at", null: false
    t.index ["clustering_id"], name: "index_communities_on_clustering_id"
  end

  create_table "clusterings", force: :cascade do |t|
    t.string "name"
    t.text "description"
    t.datetime "created_at", null: false
    t.datetime "updated_at", null: false
  end

How can I make the query execute faster? Is it possible to refactor the query to simplify it, or to add some extra index to the database schema so that it takes a shorter time?

Interestingly, a slightly simplified version of the query, which I use in another view, runs much faster, in only 1161.4ms for the first request and 41.6ms for the following requests:

DataPoint Load (1161.4ms)
SELECT consumers.name as con,
       array_agg(timestamp ORDER BY data_points.timestamp asc) as tims,
       array_agg(consumption ORDER BY data_points.timestamp ASC) as cons
FROM "data_points"
    INNER JOIN "consumers" ON "consumers"."id" = "data_points"."consumer_id"
    INNER JOIN "communities_consumers" ON "communities_consumers"."consumer_id" = "consumers"."id"
    INNER JOIN "communities" ON "communities"."id" = "communities_consumers"."community_id"
WHERE ("data_points"."timestamp" BETWEEN $1 AND $2)
   AND "data_points"."interval_id" = $3
   AND "communities"."id" = 100 GROUP BY communities.id, consumers.name
[["timestamp", "2015-11-20 09:23:00"], ["timestamp", "2015-11-27 09:23:00"], ["interval_id", 2]]

Using command EXPLAIN (ANALYZE, BUFFERS) with query in dbconsole, I get the following output:

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 GroupAggregate  (cost=12.31..7440.69 rows=246 width=57) (actual time=44.139..20474.015 rows=296 loops=1)
   Group Key: communities.id, consumers.id
   Buffers: shared hit=159692 read=6148105 written=209
   ->  Nested Loop  (cost=12.31..7434.54 rows=246 width=57) (actual time=20.944..20436.806 rows=49728 loops=1)
         Buffers: shared hit=159685 read=6148105 written=209
         ->  Nested Loop  (cost=11.88..49.30 rows=1 width=49) (actual time=0.102..6.374 rows=296 loops=1)
               Buffers: shared hit=988 read=208
               ->  Nested Loop  (cost=11.73..41.12 rows=1 width=57) (actual time=0.084..4.443 rows=296 loops=1)
                     Buffers: shared hit=396 read=208
                     ->  Merge Join  (cost=11.58..40.78 rows=1 width=24) (actual time=0.075..1.365 rows=296 loops=1)
                           Merge Cond: (communities_consumers.community_id = communities.id)
                           Buffers: shared hit=5 read=7
                           ->  Index Only Scan using index_communities_consumers_on_community_id_and_consumer_id on communities_consumers  (cost=0.27..28.71 rows=296 width=16) (actual time=0.039..0.446 rows=296 loops=1)
                                 Heap Fetches: 4
                                 Buffers: shared hit=1 read=6
                           ->  Sort  (cost=11.31..11.31 rows=3 width=16) (actual time=0.034..0.213 rows=247 loops=1)
                                 Sort Key: communities.id
                                 Sort Method: quicksort  Memory: 25kB
                                 Buffers: shared hit=4 read=1
                                 ->  Bitmap Heap Scan on communities  (cost=4.17..11.28 rows=3 width=16) (actual time=0.026..0.027 rows=6 loops=1)
                                       Recheck Cond: (clustering_id = 1)
                                       Heap Blocks: exact=1
                                       Buffers: shared hit=4 read=1
                                       ->  Bitmap Index Scan on index_communities_on_clustering_id  (cost=0.00..4.17 rows=3 width=0) (actual time=0.020..0.020 rows=8 loops=1)
                                             Index Cond: (clustering_id = 1)
                                             Buffers: shared hit=3 read=1
                     ->  Index Scan using consumers_pkey on consumers  (cost=0.15..0.33 rows=1 width=33) (actual time=0.007..0.008 rows=1 loops=296)
                           Index Cond: (id = communities_consumers.consumer_id)
                           Buffers: shared hit=391 read=201
               ->  Index Only Scan using clusterings_pkey on clusterings  (cost=0.15..8.17 rows=1 width=8) (actual time=0.004..0.005 rows=1 loops=296)
                     Index Cond: (id = 1)
                     Heap Fetches: 296
                     Buffers: shared hit=592
         ->  Index Scan using index_data_points_on_consumer_id on data_points  (cost=0.44..7383.44 rows=180 width=24) (actual time=56.128..68.995 rows=168 loops=296)
               Index Cond: (consumer_id = consumers.id)
               Filter: (("timestamp" >= '2015-11-20 09:23:00'::timestamp without time zone) AND ("timestamp" <= '2015-11-27 09:23:00'::timestamp without time zone) AND (interval_id = 2))
               Rows Removed by Filter: 76610
               Buffers: shared hit=158697 read=6147897 written=209
 Planning time: 1.811 ms
 Execution time: 20474.330 ms
(40 rows)

The bullet gem returns the following warnings:

USE eager loading detected
  Community => [:communities_consumers]
  Add to your finder: :includes => [:communities_consumers]

USE eager loading detected
  Community => [:consumers]
  Add to your finder: :includes => [:consumers]

After removing the join with the clusterings table, the new query plan is the following:

EXPLAIN for: SELECT communities.id as com, consumers.name as con, array_agg(timestamp ORDER BY data_points.timestamp asc) as tims, array_agg(consumption ORDER BY data_points.timestamp ASC) as cons FROM "data_points" INNER JOIN "consumers" ON "consumers"."id" = "data_points"."consumer_id" INNER JOIN "communities_consumers" ON "communities_consumers"."consumer_id" = "consumers"."id" INNER JOIN "communities" ON "communities"."id" = "communities_consumers"."community_id" WHERE ("data_points"."timestamp" BETWEEN $1 AND $2) AND "data_points"."interval_id" = $3 AND "communities"."clustering_id" = 1 GROUP BY communities.id, consumers.id [["timestamp", "2015-11-29 20:52:30.926247"], ["timestamp", "2015-12-06 20:52:30.926468"], ["interval_id", 2]]
                                                                                QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 GroupAggregate  (cost=10839.79..10846.42 rows=241 width=57)
   ->  Sort  (cost=10839.79..10840.39 rows=241 width=57)
         Sort Key: communities.id, consumers.id
         ->  Nested Loop  (cost=7643.11..10830.26 rows=241 width=57)
               ->  Nested Loop  (cost=11.47..22.79 rows=1 width=49)
                     ->  Hash Join  (cost=11.32..17.40 rows=1 width=16)
                           Hash Cond: (communities_consumers.community_id = communities.id)
                           ->  Seq Scan on communities_consumers  (cost=0.00..4.96 rows=296 width=16)
                           ->  Hash  (cost=11.28..11.28 rows=3 width=8)
                                 ->  Bitmap Heap Scan on communities  (cost=4.17..11.28 rows=3 width=8)
                                       Recheck Cond: (clustering_id = 1)
                                       ->  Bitmap Index Scan on index_communities_on_clustering_id  (cost=0.00..4.17 rows=3 width=0)
                                             Index Cond: (clustering_id = 1)
                     ->  Index Scan using consumers_pkey on consumers  (cost=0.15..5.38 rows=1 width=33)
                           Index Cond: (id = communities_consumers.consumer_id)
               ->  Bitmap Heap Scan on data_points  (cost=7631.64..10805.72 rows=174 width=24)
                     Recheck Cond: ((consumer_id = consumers.id) AND ("timestamp" >= '2015-11-29 20:52:30.926247'::timestamp without time zone) AND ("timestamp" <= '2015-12-06 20:52:30.926468'::timestamp without time zone))
                     Filter: (interval_id = 2::bigint)
                     ->  BitmapAnd  (cost=7631.64..7631.64 rows=861 width=0)
                           ->  Bitmap Index Scan on index_data_points_on_consumer_id  (cost=0.00..1589.92 rows=76778 width=0)
                                 Index Cond: (consumer_id = consumers.id)
                           ->  Bitmap Index Scan on index_data_points_on_timestamp  (cost=0.00..6028.58 rows=254814 width=0)
                                 Index Cond: (("timestamp" >= '2015-11-29 20:52:30.926247'::timestamp without time zone) AND ("timestamp" <= '2015-12-06 20:52:30.926468'::timestamp without time zone))
(23 rows)

As requested in the comments, these are the query plans for the simplified query, with and without the restriction on communities.id

 DataPoint Load (1563.3ms)  SELECT consumers.name as con, array_agg(timestamp ORDER BY data_points.timestamp asc) as tims, array_agg(consumption ORDER BY data_points.timestamp ASC) as cons FROM "data_points" INNER JOIN "consumers" ON "consumers"."id" = "data_points"."consumer_id" INNER JOIN "communities_consumers" ON "communities_consumers"."consumer_id" = "consumers"."id" INNER JOIN "communities" ON "communities"."id" = "communities_consumers"."community_id" WHERE ("data_points"."timestamp" BETWEEN $1 AND $2) AND "data_points"."interval_id" = $3 GROUP BY communities.id, consumers.name  [["timestamp", "2015-11-29 20:52:30.926000"], ["timestamp", "2015-12-06 20:52:30.926000"], ["interval_id", 2]]
EXPLAIN for: SELECT consumers.name as con, array_agg(timestamp ORDER BY data_points.timestamp asc) as tims, array_agg(consumption ORDER BY data_points.timestamp ASC) as cons FROM "data_points" INNER JOIN "consumers" ON "consumers"."id" = "data_points"."consumer_id" INNER JOIN "communities_consumers" ON "communities_consumers"."consumer_id" = "consumers"."id" INNER JOIN "communities" ON "communities"."id" = "communities_consumers"."community_id" WHERE ("data_points"."timestamp" BETWEEN $1 AND $2) AND "data_points"."interval_id" = $3 GROUP BY communities.id, consumers.name [["timestamp", "2015-11-29 20:52:30.926000"], ["timestamp", "2015-12-06 20:52:30.926000"], ["interval_id", 2]]
                                                                                QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 GroupAggregate  (cost=140992.34..142405.51 rows=51388 width=49)
   ->  Sort  (cost=140992.34..141120.81 rows=51388 width=49)
         Sort Key: communities.id, consumers.name
         ->  Hash Join  (cost=10135.44..135214.45 rows=51388 width=49)
               Hash Cond: (data_points.consumer_id = consumers.id)
               ->  Bitmap Heap Scan on data_points  (cost=10082.58..134455.00 rows=51388 width=24)
                     Recheck Cond: (("timestamp" >= '2015-11-29 20:52:30.926'::timestamp without time zone) AND ("timestamp" <= '2015-12-06 20:52:30.926'::timestamp without time zone) AND (interval_id = 2::bigint))
                     ->  Bitmap Index Scan on index_data_points_on_timestamp_and_consumer_id_and_interval_id  (cost=0.00..10069.74 rows=51388 width=0)
                           Index Cond: (("timestamp" >= '2015-11-29 20:52:30.926'::timestamp without time zone) AND ("timestamp" <= '2015-12-06 20:52:30.926'::timestamp without time zone) AND (interval_id = 2::bigint))
               ->  Hash  (cost=49.16..49.16 rows=296 width=49)
                     ->  Hash Join  (cost=33.06..49.16 rows=296 width=49)
                           Hash Cond: (communities_consumers.community_id = communities.id)
                           ->  Hash Join  (cost=8.66..20.69 rows=296 width=49)
                                 Hash Cond: (consumers.id = communities_consumers.consumer_id)
                                 ->  Seq Scan on consumers  (cost=0.00..7.96 rows=296 width=33)
                                 ->  Hash  (cost=4.96..4.96 rows=296 width=16)
                                       ->  Seq Scan on communities_consumers  (cost=0.00..4.96 rows=296 width=16)
                           ->  Hash  (cost=16.40..16.40 rows=640 width=8)
                                 ->  Seq Scan on communities  (cost=0.00..16.40 rows=640 width=8)
(19 rows)

and

  DataPoint Load (1479.0ms)  SELECT consumers.name as con, array_agg(timestamp ORDER BY data_points.timestamp asc) as tims, array_agg(consumption ORDER BY data_points.timestamp ASC) as cons FROM "data_points" INNER JOIN "consumers" ON "consumers"."id" = "data_points"."consumer_id" INNER JOIN "communities_consumers" ON "communities_consumers"."consumer_id" = "consumers"."id" INNER JOIN "communities" ON "communities"."id" = "communities_consumers"."community_id" WHERE ("data_points"."timestamp" BETWEEN $1 AND $2) AND "data_points"."interval_id" = $3 GROUP BY communities.id, consumers.name  [["timestamp", "2015-11-29 20:52:30.926000"], ["timestamp", "2015-12-06 20:52:30.926000"], ["interval_id", 2]]
EXPLAIN for: SELECT consumers.name as con, array_agg(timestamp ORDER BY data_points.timestamp asc) as tims, array_agg(consumption ORDER BY data_points.timestamp ASC) as cons FROM "data_points" INNER JOIN "consumers" ON "consumers"."id" = "data_points"."consumer_id" INNER JOIN "communities_consumers" ON "communities_consumers"."consumer_id" = "consumers"."id" INNER JOIN "communities" ON "communities"."id" = "communities_consumers"."community_id" WHERE ("data_points"."timestamp" BETWEEN $1 AND $2) AND "data_points"."interval_id" = $3 GROUP BY communities.id, consumers.name [["timestamp", "2015-11-29 20:52:30.926000"], ["timestamp", "2015-12-06 20:52:30.926000"], ["interval_id", 2]]
                                                                                QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 GroupAggregate  (cost=140992.34..142405.51 rows=51388 width=49)
   ->  Sort  (cost=140992.34..141120.81 rows=51388 width=49)
         Sort Key: communities.id, consumers.name
         ->  Hash Join  (cost=10135.44..135214.45 rows=51388 width=49)
               Hash Cond: (data_points.consumer_id = consumers.id)
               ->  Bitmap Heap Scan on data_points  (cost=10082.58..134455.00 rows=51388 width=24)
                     Recheck Cond: (("timestamp" >= '2015-11-29 20:52:30.926'::timestamp without time zone) AND ("timestamp" <= '2015-12-06 20:52:30.926'::timestamp without time zone) AND (interval_id = 2::bigint))
                     ->  Bitmap Index Scan on index_data_points_on_timestamp_and_consumer_id_and_interval_id  (cost=0.00..10069.74 rows=51388 width=0)
                           Index Cond: (("timestamp" >= '2015-11-29 20:52:30.926'::timestamp without time zone) AND ("timestamp" <= '2015-12-06 20:52:30.926'::timestamp without time zone) AND (interval_id = 2::bigint))
               ->  Hash  (cost=49.16..49.16 rows=296 width=49)
                     ->  Hash Join  (cost=33.06..49.16 rows=296 width=49)
                           Hash Cond: (communities_consumers.community_id = communities.id)
                           ->  Hash Join  (cost=8.66..20.69 rows=296 width=49)
                                 Hash Cond: (consumers.id = communities_consumers.consumer_id)
                                 ->  Seq Scan on consumers  (cost=0.00..7.96 rows=296 width=33)
                                 ->  Hash  (cost=4.96..4.96 rows=296 width=16)
                                       ->  Seq Scan on communities_consumers  (cost=0.00..4.96 rows=296 width=16)
                           ->  Hash  (cost=16.40..16.40 rows=640 width=8)
                                 ->  Seq Scan on communities  (cost=0.00..16.40 rows=640 width=8)
(19 rows)


来源: stackexchange

问:
Calculating distance over a list
I have two list of coordinates, mapped_coords, unmapped_coords that are both a list of coordinates.

I want to take the unmapped_coords and for every element return the index of the point with the min distance in mapped_coord.

```<code>> head(mapped_coords)
[[1]]
[1] -79.2939  43.8234

[[2]]
[1] -79.7598  43.4381

[[3]]
[1] -79.4569  43.6693

[[4]]
[1] -81.2472  42.9688

[[5]]
[1] -79.1649  43.8073

[[6]]
[1] -79.7388  43.6753

 str(mapped_coords)
List of 62815
 $ : num [1:2] -79.3 43.8
 $ : num [1:2] -79.8 43.4
 $ : num [1:2] -79.5 43.7
</code>```

Using the geosphere package I can use distHaversine to calculate the distance of a a pair but I'm not sure of how to do it over the entire list.

```<code>> distHaversine(unlist(unmapped_coords[1]), unlist(mapped_coords[1]))
[1] 100594.6
</code>```

答:
You can use geosphere::distm to make a distance matrix, of which you can find the minimum column (besides the diagonal, which is not useful) with which.min:

l <- list(c(-79.2939, 43.8234),
          c(-79.7598, 43.4381),
          c(-79.4569, 43.6693),
          c(-81.2472, 42.9688),
          c(-79.1649, 43.8073),
          c(-79.7388, 43.6753))

m <- geosphere::distm(do.call(rbind, l))
diag(m) <- NA

apply(m, 1, which.min)
#> [1] 5 6 1 2 1 3
```

If you have a second list of distances, pass that as the second parameter to distm, making the diagonal useful. Since there will be no NAs, the minimum column can be calculated with max.col(-m).

ContentLicense: CC BY-SA 3.0

Title: Calculating distance over a list

Body:
I have two list of coordinates, mapped_coords, unmapped_coords that are both a list of coordinates.

I want to take the unmapped_coords and for every element return the index of the point with the min distance in mapped_coord.

> head(mapped_coords)
[[1]]
[1] -79.2939  43.8234

[[2]]
[1] -79.7598  43.4381

[[3]]
[1] -79.4569  43.6693

[[4]]
[1] -81.2472  42.9688

[[5]]
[1] -79.1649  43.8073

[[6]]
[1] -79.7388  43.6753

 str(mapped_coords)
List of 62815
 $ : num [1:2] -79.3 43.8
 $ : num [1:2] -79.8 43.4
 $ : num [1:2] -79.5 43.7

Using the geosphere package I can use distHaversine to calculate the distance of a a pair but I'm not sure of how to do it over the entire list.

> distHaversine(unlist(unmapped_coords[1]), unlist(mapped_coords[1]))
[1] 100594.6

Body:
You can use geosphere::distm to make a distance matrix, of which you can find the minimum column (besides the diagonal, which is not useful) with which.min:

l <- list(c(-79.2939, 43.8234),
          c(-79.7598, 43.4381),
          c(-79.4569, 43.6693),
          c(-81.2472, 42.9688),
          c(-79.1649, 43.8073),
          c(-79.7388, 43.6753))

m <- geosphere::distm(do.call(rbind, l))
diag(m) <- NA

apply(m, 1, which.min)
#> [1] 5 6 1 2 1 3

If you have a second list of distances, pass that as the second parameter to distm, making the diagonal useful. Since there will be no NAs, the minimum column can be calculated with max.col(-m).


问:
Given a table with multiple pages on a webpage how can I access data for all pages from chrome?
I have a table with 100 rows which is displayed 20 rows at a time on a webpage. I can view the next 20 rows by clicking "next page" which updated the table view on my webpage. I can access the data for each batch of 20 rows via the developer/inspect browser tools. Is there anyway to get this for all 100 rows at once? The webpage is only accessible after logging in, i.e. the data is not publicly available.</p>

答:
It depends what the page is doing.

If the 100 rows aren't loaded on the browser at once then no you can't get them. For example the page may be calling a web service to load 20 at a time.

You would have to understand how the page acquires the rows and then replicate that with some javascript in order to pull back more at once.

You may also be able to use the console/debugging tools to edit the code on the page to make it load more at once. Really it's very dependent on the specifics of what the page is doing.


问:
Magento - product edit page is not working after being upgraded to 1.9.3.7
after upgrading my site to 1.9.3.7, I want to edit the product, but it seems that the blocks are not loaded successfully.
Please see the image:
Screenshot
Please help.</p>

答: Try clearing all your caches including JS/CSS after you update versions. Your screenshot appears to be missing CSS files. If the problem persists, try recompiling. </p>


问:
Asp.net MVC Custom Model Binder for decimals - adds model error when decimal action param is null
I've created a custom model binder based on an article from Haacked. Here's the code:

```<code>namespace MyNamespace.Project.ModelBinders
{
    public class DecimalModelBinder : DefaultModelBinder
    {
        public override object BindModel(ControllerContext controllerContext, ModelBindingContext bindingContext)
        {
            string modelName = bindingContext.ModelName;
            ValueProviderResult valueResult = bindingContext.ValueProvider.GetValue(modelName);

            ModelState modelState = new ModelState { Value = valueResult };

            object actualValue = null;

            try
            {
                //replace commas with periods
                actualValue = Convert.ToDecimal(valueResult.AttemptedValue.Replace(",", "."));
            }
            catch (Exception ex)
            {
                modelState.Errors.Add(ex);
            }

            bindingContext.ModelState.Add(modelName, modelState);

            return actualValue;
        }
    }
}
</code>```

When MVC loads a view where the controller action is something like this:

```<code>public ActionResult Index(decimal amount)
</code>```

It seems to fire the model binding and add the error and this is because amount at this point is null because I have a valid use case where index can be loaded with or without parameters (QueryString). As far as I know MVC doesn't support typical OO method overloading such that you have:

```<code>public ActionResult Index(decimal amount) {}
public ActionResult Index() {}
</code>```

So, would it be valid to add a null check to my custom model binder to avoid the error that is raised in the try block, or would this interfere with validation?

答:
I see multiple points here in general. First one is regarding this:

As far as I know MVC doesn't support typical OO method overloading...

That's right, but it has a very flexible routes configuration to help with such problems.

You could configure separate routes for calls having the parameter or not having one. I didn't try that, but this would be a sample using Attribute Routing.

```<code>[Route("index/{amount}"]
public ActionResult IndexWithAmount(decimal amount) {}

[Route("index")]
public ActionResult Index() {}
</code>```

Another thing you can do, as described here is to not use the Model Binder globally but rather enable it only on specific routes:

```<code>public ActionResult Index(
    [ModelBinder(typeof(DecimalModelBinder))]decimal amount) {}
</code>```

Body:
I've created a custom model binder based on an article from Haacked. Here's the code:

namespace MyNamespace.Project.ModelBinders
{
    public class DecimalModelBinder : DefaultModelBinder
    {
        public override object BindModel(ControllerContext controllerContext, ModelBindingContext bindingContext)
        {
            string modelName = bindingContext.ModelName;
            ValueProviderResult valueResult = bindingContext.ValueProvider.GetValue(modelName);

            ModelState modelState = new ModelState { Value = valueResult };

            object actualValue = null;

            try
            {
                //replace commas with periods
                actualValue = Convert.ToDecimal(valueResult.AttemptedValue.Replace(",", "."));
            }
            catch (Exception ex)
            {
                modelState.Errors.Add(ex);
            }

            bindingContext.ModelState.Add(modelName, modelState);

            return actualValue;
        }
    }
}

When MVC loads a view where the controller action is something like this:

public ActionResult Index(decimal amount)

It seems to fire the model binding and add the error and this is because amount at this point is null because I have a valid use case where index can be loaded with or without parameters (QueryString). As far as I know MVC doesn't support typical OO method overloading such that you have:

public ActionResult Index(decimal amount) {}
public ActionResult Index() {}

So, would it be valid to add a null check to my custom model binder to avoid the error that is raised in the try block, or would this interfere with validation?

Body:
I see multiple points here in general. First one is regarding this:

As far as I know MVC doesn't support typical OO method overloading...

That's right, but it has a very flexible routes configuration to help with such problems.

You could configure separate routes for calls having the parameter or not having one. I didn't try that, but this would be a sample using Attribute Routing.

[Route("index/{amount}"]
public ActionResult IndexWithAmount(decimal amount) {}

[Route("index")]
public ActionResult Index() {}

Another thing you can do, as described here is to not use the Model Binder globally but rather enable it only on specific routes:

public ActionResult Index(
    [ModelBinder(typeof(DecimalModelBinder))]decimal amount) {}


问:
gnu cflow - not recognizing "typedef struct"
GNU cflow analyzes a collection of C source files and prints a graph, charting control flow within the program.

My .c or .cpp file

```<code>typedef struct _type_1{
    int a;
} type_1_t;

typedef struct _type_2{
    int a;
} type_2_t;

int main()
{
    type_1_t t1;
    type_2_t t2;

    t1.a = 55;
    t2.a = 99;

    return 0;
}
</code>```

The command is cflow.exe test.c -i s -i x > test.graph 2>&1 and the output is:

```<code>cflow.exe:test.c:7: a redefined
cflow.exe:test.c:2: this is the place of previous definition
main() <int main () at test.c:11>:
    type_1_t <type_1_t at test.c:3>
    t1
    type_2_t <type_2_t at test.c:8>
    t2
</code>```

QUESTION

Why does it say "a redefined"?

It can only be because it doesn't recognize the typedef struct construct so how do I fix it?

UPDATE

I ran cflow again with --debug=1 and it gave me this:

```<code>test.c:3: type _type_1
test.c:3: a/-1 defined to int a
test.c:3: type_1_t/-1 defined to type_1_t
test.c:8: type _type_2
cflow.exe:test.c:7: a redefined
cflow.exe:test.c:2: this is the place of previous definition
main() <int main () at test.c:15>:
    type_1_t <type_1_t at test.c:3>
    t1
    type_2_t <type_2_t at test.c:8>
    t2
    f1() <int f1 () at test.c:10>
test.c:8: a/-1 defined to int a
test.c:8: type_2_t/-1 defined to type_2_t
test.c:11: f1/0 defined to int f1 ()
test.c:16: main/0 defined to int main ()
</code>```

Just as we suspected: it's not treating each struct . . . as a struct i.e. the ability to have the exact same identifier in two different structs.

So how to fix this? I'm emailing cflow mailing list. Hopefully will hear back soon. Until then, I'm going to play around with the syntactic classes and see if I can't trigger the correct behavior.

I'll post my own answer if I get an answer back from mailing list.

答:
This is clearly a bug in cflow.

I just built cflow versions 1.3, 1.4, and 1.5 on my system (Ubuntu 17.04). Versions 1.3 and 1.4 do not exhibit the problem you describe. Version 1.5 does.

Here's a simpler test case that exhibits the problem:

```<code>$ cat c.c
typedef struct type1 { int a; } type1;
typedef struct type2 { int a; } type2;
$ cflow --version | head -n 1
cflow (GNU cflow) 1.5
$ cflow c.c
cflow:c.c:2: a redefined
cflow:c.c:1: this is the place of previous definition
$
</code>```

(typedef struct and separate namespaces for different structure types have been features of C for about three decades. It's not plausible that cflow would fail to support them -- and in fact earlier versions handle them with no problem.)

As a workaround, use cflow 1.4. I also suggest submitting a bug report. It doesn't appear that this one has been reported. (The OP has now reported it to the bug-cflow@gnu.org mailing list and received an acknowledgement.)

Body:
GNU cflow analyzes a collection of C source files and prints a graph, charting control flow within the program.

My .c or .cpp file

typedef struct _type_1{
    int a;
} type_1_t;

typedef struct _type_2{
    int a;
} type_2_t;

int main()
{
    type_1_t t1;
    type_2_t t2;

    t1.a = 55;
    t2.a = 99;

    return 0;
}

The command is cflow.exe test.c -i s -i x > test.graph 2>&1 and the output is:

cflow.exe:test.c:7: a redefined
cflow.exe:test.c:2: this is the place of previous definition
main() <int main () at test.c:11>:
    type_1_t <type_1_t at test.c:3>
    t1
    type_2_t <type_2_t at test.c:8>
    t2

QUESTION

Why does it say "a redefined"?

It can only be because it doesn't recognize the typedef struct construct so how do I fix it?

UPDATE

I ran cflow again with --debug=1 and it gave me this:

test.c:3: type _type_1
test.c:3: a/-1 defined to int a
test.c:3: type_1_t/-1 defined to type_1_t
test.c:8: type _type_2
cflow.exe:test.c:7: a redefined
cflow.exe:test.c:2: this is the place of previous definition
main() <int main () at test.c:15>:
    type_1_t <type_1_t at test.c:3>
    t1
    type_2_t <type_2_t at test.c:8>
    t2
    f1() <int f1 () at test.c:10>
test.c:8: a/-1 defined to int a
test.c:8: type_2_t/-1 defined to type_2_t
test.c:11: f1/0 defined to int f1 ()
test.c:16: main/0 defined to int main ()

Just as we suspected: it's not treating each struct . . . as a struct i.e. the ability to have the exact same identifier in two different structs.

So how to fix this? I'm emailing cflow mailing list. Hopefully will hear back soon. Until then, I'm going to play around with the syntactic classes and see if I can't trigger the correct behavior.

I'll post my own answer if I get an answer back from mailing list.

Body:
This is clearly a bug in cflow.

I just built cflow versions 1.3, 1.4, and 1.5 on my system (Ubuntu 17.04). Versions 1.3 and 1.4 do not exhibit the problem you describe. Version 1.5 does.

Here's a simpler test case that exhibits the problem:

$ cat c.c
typedef struct type1 { int a; } type1;
typedef struct type2 { int a; } type2;
$ cflow --version | head -n 1
cflow (GNU cflow) 1.5
$ cflow c.c
cflow:c.c:2: a redefined
cflow:c.c:1: this is the place of previous definition
$

(typedef struct and separate namespaces for different structure types have been features of C for about three decades. It's not plausible that cflow would fail to support them -- and in fact earlier versions handle them with no problem.)

As a workaround, use cflow 1.4. I also suggest submitting a bug report. It doesn't appear that this one has been reported. (The OP has now reported it to the bug-cflow@gnu.org mailing list and received an acknowledgement.)


问:
How would I model 2 different vendors on an online marketplace selling the same product; i.e. with the same productID?
Take Ebay for example. Ebay has an unlimited about of vendors. However, what if two vendors were to sell the same product. That product would have the same productID right? How would I go about modelling this using RDF(s) or OWL?</p>

答:
In OWL you can have a Product class which is a subclass of a DataProperty hasID some string. Then you define that Product hasKey hasID. If you now have 2 individuals of type Product with the same value for hasID, the reasoner will infer that the 2 individuals are the same individuals. Here is the example ontology:

```<code>Datatype: xsd:string

DataProperty: hasID
    Range: xsd:string

Class: Product
    SubClassOf: hasID some xsd:string
    HasKey: hasID

Individual: productA
    Types: Product
    Facts: hasID  "\"123\""^^xsd:string

Individual: productB
    Types: Product
    Facts:  hasID  "\"123\""^^xsd:string
</code>```

Body:
In OWL you can have a Product class which is a subclass of a DataProperty hasID some string. Then you define that Product hasKey hasID. If you now have 2 individuals of type Product with the same value for hasID, the reasoner will infer that the 2 individuals are the same individuals. Here is the example ontology:

Datatype: xsd:string

DataProperty: hasID
    Range: xsd:string

Class: Product
    SubClassOf: hasID some xsd:string
    HasKey: hasID

Individual: productA
    Types: Product
    Facts: hasID  "\"123\""^^xsd:string

Individual: productB
    Types: Product
    Facts:  hasID  "\"123\""^^xsd:string


问:
How can I extract all values of a particular attribute within elements inside a JSON array in MySQL?
I've the below data stored in a column of MySQL table which is of "json" data type (MySQL v5.7.9)

```<code>[{"id": "26", "title": "similar1", "score": "0.97"}, {"id": "27", "title": "similar2", "score": "0.87"}, {"id": "28", "title": "similar2", "score": "0.87"}]
</code>```

I need a MySQL query which will return me all the "title" values i.e. similar1, similar2 and similar3 and so on

答: This looks like a database design issue. It seems that you're using a json column when you should really be using another table to store these results as individual columns.</p>

Body:
I've the below data stored in a column of MySQL table which is of "json" data type (MySQL v5.7.9)

[{"id": "26", "title": "similar1", "score": "0.97"}, {"id": "27", "title": "similar2", "score": "0.87"}, {"id": "28", "title": "similar2", "score": "0.87"}]

I need a MySQL query which will return me all the "title" values i.e. similar1, similar2 and similar3 and so on


问:
Linker error: How can there be a duplicate symbol in just one file?
From what I understand, a linker error due to a duplicate symbol means that:

a symbol was defined in more than one source file

resulting in the same symbol in two different object files after compilation

so the linker does not know which of the two symbols he should link to.

While trying to compile a demo project from www.ugfx.io I came across this error:

```<code>duplicate symbol _main in:
    .build/obj/GFXLIB/demos/modules/gdisp/basics/main.o
ld: 1 duplicate symbol for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [.build/demotest.elf] Error 1
</code>```

In contrast to every duplicate symbol error I could find on the web there is just one file listed as part of this error. And the source file belonging to this object file has just one main symbol. You can view it here.

How shall I even understand this error? And what could be its cause?

答:
It happened to me when I included the same source file twice in the compiling command, e.g. :

```<code>$gcc -o main main.c becool.c foo.c bar.c becool.c end.c
</code>```

The repeated becool.c caused a duplicate symbol error.

So, if this is the problem, just review your compiling command and remove such duplicates

Body:
From what I understand, a linker error due to a duplicate symbol means that:

a symbol was defined in more than one source file

resulting in the same symbol in two different object files after compilation

so the linker does not know which of the two symbols he should link to.

While trying to compile a demo project from www.ugfx.io I came across this error:

duplicate symbol _main in:
    .build/obj/GFXLIB/demos/modules/gdisp/basics/main.o
ld: 1 duplicate symbol for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [.build/demotest.elf] Error 1

In contrast to every duplicate symbol error I could find on the web there is just one file listed as part of this error. And the source file belonging to this object file has just one main symbol. You can view it here.

How shall I even understand this error? And what could be its cause?

Body:
It happened to me when I included the same source file twice in the compiling command, e.g. :

$gcc -o main main.c becool.c foo.c bar.c becool.c end.c

The repeated becool.c caused a duplicate symbol error.

So, if this is the problem, just review your compiling command and remove such duplicates


问:
Laravel eloquent chunk does not work
I have tried the following in laravel-5.4 application to use chunk:

```<code>$out = [];
\App\User::chunk(5,function($users) use($out){
   foreach($users as $user){
       //$out[] = $user;
       $out[] = $user->id;
   }
});
dd($out);
</code>```

However, the output of dd($out) is still empty array. What's the problem here? $out is in global scope!

答:
See, use in PHP doesn't just make a variable from outer scope available within the function (similar to lexical scope in JS etc.). It copies the value of that variable (and here's more explanation, for some reason still missing in the official doc).

Take use as a mean to pass some value inside the function without actually making it a formal parameter, a part of function's signature.

The bottom line is, you should pass the array as reference:

```<code>\App\User::chunk(5,function($users) use(&$out){
   foreach($users as $user){
       //$out[] = $user;
       $out[] = $user->id;
   }
});
</code>```

Title: Laravel eloquent chunk does not work

Body:
I have tried the following in laravel-5.4 application to use chunk:

$out = [];
\App\User::chunk(5,function($users) use($out){
   foreach($users as $user){
       //$out[] = $user;
       $out[] = $user->id;
   }
});
dd($out);

However, the output of dd($out) is still empty array. What's the problem here? $out is in global scope!

Body:
See, use in PHP doesn't just make a variable from outer scope available within the function (similar to lexical scope in JS etc.). It copies the value of that variable (and here's more explanation, for some reason still missing in the official doc).

Take use as a mean to pass some value inside the function without actually making it a formal parameter, a part of function's signature.

The bottom line is, you should pass the array as reference:

\App\User::chunk(5,function($users) use(&$out){
   foreach($users as $user){
       //$out[] = $user;
       $out[] = $user->id;
   }
});


问:
populate drop down list with month/year
I have a date and I need to populate a drop-down with the months/years between that date and today. For instance, if that date is 10/14/2010 then the drop-down should contain October 2010, November 2010, December 2010, January 2011.

The way I'm thinking of doing this is to pass that date to a function, loop from today backwards step 1 month while adding each month to a collection until we reach that date and finally return a collection of strings. Then, populate the drop-down control on page load. Finally, use some ajax with a page method to parse back the string and trigger a partial page reload.

I'm just wondering if there's an easy way to do it.

Thanks.

答:
```<code>public static List<string> GetMonths(DateTime StartDate)
  {
   List<string> MonthList = new List<string>();
   DateTime ThisMonth = DateTime.Now.Date;

   while (ThisMonth.Date > StartDate.Date)
   {
    MonthList.Add(ThisMonth.ToString("MMMM") + " " + ThisMonth.Year.ToString());
    ThisMonth = ThisMonth.AddMonths(-1);
   }

   return MonthList;
  }
</code>```

Title: populate drop down list with month/year

ContentLicense: CC BY-SA 2.5


答:
Arrays are copy on write. You are just creating a new array in that scope when you try to write to the array you are trying to import from the parent scope.

You need to be using the reference to the array.

```<code>function ($users) use (&$out) { ... }
</code>```

Title: Laravel eloquent chunk does not work

Body:
Arrays are copy on write. You are just creating a new array in that scope when you try to write to the array you are trying to import from the parent scope.

You need to be using the reference to the array.

function ($users) use (&$out) { ... }


问:
Formula to find part of a cell's value as exact match to a value from other cells
On tab 1, I have data down column B that contains string values like this:

\\ABC\VOL1\DATA\Dan\Personal Folders

  \\ABC\VOL1\DATA\Mike\My Stuff\Docs\Support

  \\ABC\VOL1\DATA\Mike\My Stuff\Photos

  \\ABC\VOL1\DATA\Bob\Plans

On tab 2, I have data down column C that I want to use as a lookup table like this:

\\ABC\VOL1\DATA\Adam

  \\ABC\VOL1\DATA\Steve

  \\ABC\VOL1\DATA\Mike

  \\ABC\VOL1\DATA\Ronnie

I need a formula I can put on tab 1 down column C to see if any part of each text string from tab 1, column B matches any complete text string from tab 2, column C.  So for example, I would place this formula to show Yes or No like this:

Column B......................................................................Column C

  \\ABC\VOL1\DATA\Dan\Personal Folders....................No

  \\ABC\VOL1\DATA\Mike\My Stuff\Docs\Support..........Yes

  \\ABC\VOL1\DATA\Mike\My Stuff\Photos.....................Yes

  \\ABC\VOL1\DATA\Bob\Plans.......................................No

I have tried VLOOKUP, INDEX/MATCH, ISNUMBER/SEARCH, and COUNTIF as an array with wildcards, but I just cannot seem to figure it out.  Currently, with COUNTIF as array, I have this:

```<code>=COUNTIF(B2,"*" & 'Tab 2'!$C$2:$C$1000 & "*")
</code>```

This is the start to return the array results, but the results aren't as I would expect as I get "0" when I should get "1" for certain rows.

If anyone can help me out with a formula (not VBA) using any of the functions mentioned above or any others, I would greatly appreciate it.  Thanks.

答:
This will search the cell for any of your criteria and return the number of matches if there is a match (which unless you have duplicates in the criteria it will be a 1).

Enter as array.

```<code>{=COUNT(SEARCH($C$1:$C$4,B1))}
</code>```

Body:
On tab 1, I have data down column B that contains string values like this:

\\ABC\VOL1\DATA\Dan\Personal Folders

  \\ABC\VOL1\DATA\Mike\My Stuff\Docs\Support

  \\ABC\VOL1\DATA\Mike\My Stuff\Photos

  \\ABC\VOL1\DATA\Bob\Plans

On tab 2, I have data down column C that I want to use as a lookup table like this:

\\ABC\VOL1\DATA\Adam

  \\ABC\VOL1\DATA\Steve

  \\ABC\VOL1\DATA\Mike

  \\ABC\VOL1\DATA\Ronnie

I need a formula I can put on tab 1 down column C to see if any part of each text string from tab 1, column B matches any complete text string from tab 2, column C.  So for example, I would place this formula to show Yes or No like this:

Column B......................................................................Column C

  \\ABC\VOL1\DATA\Dan\Personal Folders....................No

  \\ABC\VOL1\DATA\Mike\My Stuff\Docs\Support..........Yes

  \\ABC\VOL1\DATA\Mike\My Stuff\Photos.....................Yes

  \\ABC\VOL1\DATA\Bob\Plans.......................................No

I have tried VLOOKUP, INDEX/MATCH, ISNUMBER/SEARCH, and COUNTIF as an array with wildcards, but I just cannot seem to figure it out.  Currently, with COUNTIF as array, I have this:

=COUNTIF(B2,"*" & 'Tab 2'!$C$2:$C$1000 & "*")

This is the start to return the array results, but the results aren't as I would expect as I get "0" when I should get "1" for certain rows.

If anyone can help me out with a formula (not VBA) using any of the functions mentioned above or any others, I would greatly appreciate it.  Thanks.

Body:
This will search the cell for any of your criteria and return the number of matches if there is a match (which unless you have duplicates in the criteria it will be a 1).

Enter as array.

{=COUNT(SEARCH($C$1:$C$4,B1))}


问:
Django 2.0 path error ?: (2_0.W001) has a route that contains '(?P<', begins with a '^', or ends with a '$'
I'm trying to create the back-end code for a music application on my website.

I have created the correct view in my views.py file (in the correct directory) as shown below:

```<code>def detail(request, album_id):
    return HttpResponse("<h1>Details for Album ID:" + str(album_id) + "</h1>")
</code>```

However, when creating the URL or path for this (shown below)

```<code>#/music/71/ (pk)
path(r'^(?P<album_id>[0-9])/$', views.detail, name='detail'),
</code>```

I am experiencing a warning on my terminal stating:

```<code>?: (2_0.W001) Your URL pattern '^(?P<album_id>[0-9])/$'
[name='detail'] has a route that contains '(?P<', begins with a '^',
or ends with a '$'. This was likely an oversight when migrating to
django.urls.path().
</code>```

And whenever the /music/ (for which the path works) is followed by a number, such as /music/1 (which is what I want to be able to do) the page cannot be found and the terminal gives the above warning.

答:
The new path() syntax in Django 2.0 does not use regular expressions. You want something like:

```<code>path('<int:album_id>/', views.detail, name='detail'),
</code>```

If you want to use a regular expression, you can use re_path().

```<code>re_path(r'^(?P<album_id>[0-9])/$', views.detail, name='detail'),
</code>```

The old url() still works and is now an alias to re_path, but it is likely to be deprecated in future.

```<code>url(r'^(?P<album_id>[0-9])/$', views.detail, name='detail'),
</code>```

ContentLicense: CC BY-SA 4.0

Body:
I'm trying to create the back-end code for a music application on my website.

I have created the correct view in my views.py file (in the correct directory) as shown below:

def detail(request, album_id):
    return HttpResponse("
Details for Album ID:" + str(album_id) + "
")

However, when creating the URL or path for this (shown below)

#/music/71/ (pk)
path(r'^(?P<album_id>[0-9])/$', views.detail, name='detail'),

I am experiencing a warning on my terminal stating:

?: (2_0.W001) Your URL pattern '^(?P<album_id>[0-9])/$'
[name='detail'] has a route that contains '(?P<', begins with a '^',
or ends with a '$'. This was likely an oversight when migrating to
django.urls.path().

And whenever the /music/ (for which the path works) is followed by a number, such as /music/1 (which is what I want to be able to do) the page cannot be found and the terminal gives the above warning.

Body:
The new path() syntax in Django 2.0 does not use regular expressions. You want something like:

path('<int:album_id>/', views.detail, name='detail'),

If you want to use a regular expression, you can use re_path().

re_path(r'^(?P<album_id>[0-9])/$', views.detail, name='detail'),

The old url() still works and is now an alias to re_path, but it is likely to be deprecated in future.

url(r'^(?P<album_id>[0-9])/$', views.detail, name='detail'),


问:
Headers with Dashes in powershell
Is there a way to get rid of the header name and the dashes? For example, instead of having

```<code>Display Name
-------------
Test, Directory
</code>```

It would just display the First and Last Name Test, Directory

this is what i have in my write-output:

```<code>write-output $DataAdapter.Fill($Dataset) | Out-null
</code>```

I thought the out-null would take care of it but it didn't.

Can someone help me with that please .

Thank you.

答:
In this specific example, what you want to do is use Select-Object with the ExpandProperty parameter, specifying the property you want displayed.

An example of this is (without expand):

```<code>[1] PS G:\> Get-ADUser -Filter * | Select Name

Name
----
Administrator
Joe Bloggs
...
</code>```

And with expand:

```<code>[2] PS G:\> Get-ADUser -Filter * | Select-Object -ExpandProperty Name
Administrator
Joe Bloggs
...
</code>```

I guess the real follow up question here, though, is "why?"

By doing this you're stripping away the object based paradigm that makes PowerShell powerful. This is now just giving you an array of strings.

Title: Headers with Dashes in powershell

Body:
Is there a way to get rid of the header name and the dashes? For example, instead of having

Display Name
-------------
Test, Directory

It would just display the First and Last Name Test, Directory

this is what i have in my write-output:

write-output $DataAdapter.Fill($Dataset) | Out-null

I thought the out-null would take care of it but it didn't.

Can someone help me with that please .

Thank you.

Body:
In this specific example, what you want to do is use Select-Object with the ExpandProperty parameter, specifying the property you want displayed.

An example of this is (without expand):

[1] PS G:\> Get-ADUser -Filter * | Select Name

Name
----
Administrator
Joe Bloggs
...

And with expand:

[2] PS G:\> Get-ADUser -Filter * | Select-Object -ExpandProperty Name
Administrator
Joe Bloggs
...

I guess the real follow up question here, though, is "why?"

By doing this you're stripping away the object based paradigm that makes PowerShell powerful. This is now just giving you an array of strings.


问:
Shiny splitLayout not wrapping HTML text
I'm using splitLayout in a Shiny R app to create a two-panel structure that has a plot on the left (a googleVis gVisGeoChart object) and HTML on the right.

This is a mockup of the problematic code:

```<code>splitLayout(

    htmlOutput("mychart"), #This winds up rendering properly to the left side of the window

    div(
        h1("A descriptive heading"),
        p("Here's some text that takes more than one line on the screen. I want it to wrap, but it just keeps going to the right out of view in its container.")
    ) #This text will not wrap to the right side of the splitLayout frame
)
</code>```

The text inside the p() command will not wrap to the visible (1/2) of the page created by splitLayout. Instead, it keeps going to the (invisible) right, requiring the user to scroll to see it all. This is not the desired behavior (quite frankly, I have a hard time imagining a situation where it is).

How to ensure that the text wraps properly to the splitLayout area?

答:
Add style = "word-wrap: break-word;":

```<code>p("Here's some text that takes more than one line on the screen. I want it to wrap, but it just keeps going to the right out of view in its container.",
   style = "word-wrap: break-word;")
</code>```

Body:
I'm using splitLayout in a Shiny R app to create a two-panel structure that has a plot on the left (a googleVis gVisGeoChart object) and HTML on the right.

This is a mockup of the problematic code:

splitLayout(

    htmlOutput("mychart"), #This winds up rendering properly to the left side of the window

    div(
        h1("A descriptive heading"),
        p("Here's some text that takes more than one line on the screen. I want it to wrap, but it just keeps going to the right out of view in its container.")
    ) #This text will not wrap to the right side of the splitLayout frame
)

The text inside the p() command will not wrap to the visible (1/2) of the page created by splitLayout. Instead, it keeps going to the (invisible) right, requiring the user to scroll to see it all. This is not the desired behavior (quite frankly, I have a hard time imagining a situation where it is).

How to ensure that the text wraps properly to the splitLayout area?

Body:
Add style = "word-wrap: break-word;":

p("Here's some text that takes more than one line on the screen. I want it to wrap, but it just keeps going to the right out of view in its container.",
   style = "word-wrap: break-word;")


问:
Getting collection of most general type from a GenericType<T>.class filter
I apologize in advance for the terrible title, suggestions for improvement are eagerly accepted.

Let's say I have a method which filters a List<T> of arbitrary type based on a class, returning a new List whose elements are those of the input list that are instances of the given class. Here's a straightforward implementation (yes, you can also do it in a 1-liner with streams):

```<code>public static <T> List<T> filterByClass(List<?> list, Class<T> clazz) {
    List<T> filtered = new ArrayList<>();
    for (Object o : list) {
        if (clazz.isInstance(o)) {
            filtered.add(clazz.cast(o));
        }
    }
    return filtered;
}
</code>```

This works great if you pass it a list of a non-generic type like String as in (prints foo bar):

```<code>List<Object> l = Arrays.<Object>asList(1, "foo ", 2, "bar ");
filterByClass(l, String.class).stream().forEach(System.out::println);
</code>```

Now I want to pass to filter on a class of a generic type, say Optional<T>:

```<code>List<Object> l = Arrays.<Object>asList(1, Optional.of("foo"), 2, Optional.of("bar"));
filterByClass(l, Optional.class).stream().forEach(System.out::println);
</code>```

That works fine and prints:

```<code>Optional[foo]
Optional[bar]
</code>```

The problem is there is a raw type hidden there. The return type of the filterByClass call above is List<Optional>, not List<Optional<...>>. Many uses will trigger warnings about raw types, etc.

Now I understand type erasure, and I know that a class object will never carry generic type information - there is no such Optional<String>.class or Optional<Integer>.class - there is only Optional.class.

However, there is still a better return value than the raw type: I would like instead the fully-generic wildcard version: List<Optional<?>>. This should be totally type safe, since any Optional is an Optional<?>, right?

Assignment doesn't work, since List<Optional> is not convertible to List<Optional<?>>

```<code>List<Optional<?>> r = filterByClass(l, Optional.class);
</code>```

A cast doesn't work either because the types aren't even castable (in the same way that Foo<T> can never be directly cast to Foo no matter the relationship between different types T and U).

The only solution seems to be to cast all the way down to a raw List and then back up to the list with wildcard-paramaterized type-parameter:

```<code>List<Optional<?>> r = (List)filterByClass(l, Optional.class);
</code>```

Now obviously such casts aren't safe in the general case, and this would be totally unsafe if the type parameter in the assignment Optional<?> didn't match the class object passed to filterByClass - although I think they are safe in the specific case that the class matches the type parameter with unbounded wildcards.

Is there some way to do this without the potentially unsafe casts, either by changing the filterByClass method or some safe casting of the result?

Another possible answer would be that this is not safe (i.e., the result of filterByClass(..., Optional.class) cannot safely be converted to List<Optional<?>>, so the question is ill-formed.

答:
You could change the signature of filterByClass to:

```<code>public static <T> List<T> filterByClass(List<?> list, Class<? extends T> clazz) {...}
</code>```

Then you can have T be inferred from the assignee:

```<code>List<Object> l = Arrays.<Object>asList(1, Optional.of("foo"), 2, Optional.of("bar"));
List<Optional<?>> result = filterByClass(l, Optional.class);
</code>```

This works since a raw Optional is assignable to Optional<?> (which is safe), so Class<Optional> satisfies Class<? extends Optional<?>>, and you can add the Optional returned by cast to the list of Optional<?>.

Body:
I apologize in advance for the terrible title, suggestions for improvement are eagerly accepted.

Let's say I have a method which filters a List<T> of arbitrary type based on a class, returning a new List whose elements are those of the input list that are instances of the given class. Here's a straightforward implementation (yes, you can also do it in a 1-liner with streams):

public static <T> List<T> filterByClass(List<?> list, Class<T> clazz) {
    List<T> filtered = new ArrayList<>();
    for (Object o : list) {
        if (clazz.isInstance(o)) {
            filtered.add(clazz.cast(o));
        }
    }
    return filtered;
}

This works great if you pass it a list of a non-generic type like String as in (prints foo bar):

List<Object> l = Arrays.<Object>asList(1, "foo ", 2, "bar ");
filterByClass(l, String.class).stream().forEach(System.out::println);

Now I want to pass to filter on a class of a generic type, say Optional<T>:

List<Object> l = Arrays.<Object>asList(1, Optional.of("foo"), 2, Optional.of("bar"));
filterByClass(l, Optional.class).stream().forEach(System.out::println);

That works fine and prints:

Optional[foo]
Optional[bar]

The problem is there is a raw type hidden there. The return type of the filterByClass call above is List<Optional>, not List<Optional<...>>. Many uses will trigger warnings about raw types, etc.

Now I understand type erasure, and I know that a class object will never carry generic type information - there is no such Optional<String>.class or Optional<Integer>.class - there is only Optional.class.

However, there is still a better return value than the raw type: I would like instead the fully-generic wildcard version: List<Optional<?>>. This should be totally type safe, since any Optional is an Optional<?>, right?

Assignment doesn't work, since List<Optional> is not convertible to List<Optional<?>>

List<Optional<?>> r = filterByClass(l, Optional.class);

A cast doesn't work either because the types aren't even castable (in the same way that Foo<T> can never be directly cast to Foo no matter the relationship between different types T and U).

The only solution seems to be to cast all the way down to a raw List and then back up to the list with wildcard-paramaterized type-parameter:

List<Optional<?>> r = (List)filterByClass(l, Optional.class);

Now obviously such casts aren't safe in the general case, and this would be totally unsafe if the type parameter in the assignment Optional<?> didn't match the class object passed to filterByClass - although I think they are safe in the specific case that the class matches the type parameter with unbounded wildcards.

Is there some way to do this without the potentially unsafe casts, either by changing the filterByClass method or some safe casting of the result?

Another possible answer would be that this is not safe (i.e., the result of filterByClass(..., Optional.class) cannot safely be converted to List<Optional<?>>, so the question is ill-formed.

Body:
You could change the signature of filterByClass to:

public static <T> List<T> filterByClass(List<?> list, Class<? extends T> clazz) {...}

Then you can have T be inferred from the assignee:

List<Object> l = Arrays.<Object>asList(1, Optional.of("foo"), 2, Optional.of("bar"));
List<Optional<?>> result = filterByClass(l, Optional.class);

This works since a raw Optional is assignable to Optional<?> (which is safe), so Class<Optional> satisfies Class<? extends Optional<?>>, and you can add the Optional returned by cast to the list of Optional<?>.


问:
knitr tables appear as one long column
I pretty much copied word-for-word the example from this site on how to create multiple PDFs in knitr from one for-loop: http://www.reed.edu/data-at-reed/software/R/markdown_multiple_reports.html

When I try calling knitr::kable(cars) or kable(cars) the output of each PDF appears as one long column instead of an actual table, as shown in the photo.

Any idea what causes this? Here's the .R and the .Rmd code called in each iteration of the loop:

.R

```<code>library(knitr)
library(markdown)
library(rmarkdown)
mt <- mtcars[1:5,]
for (car in unique(rownames(mt))){
     rmarkdown::render('test.Rmd',
                output_format = "pdf_document",
                output_file =  paste(car, ' report.pdf', sep=''),
                output_dir = '~/')
</code>```

test.Rmd

```<code>library(knitr)
library(markdown)
library(rmarkdown)
cars <- mtcars[rownames(mtcars)==car,]

# create example data
x <- sample(1:10, 1)
cars <- do.call("rbind", replicate(x, cars, simplify = FALSE))

# create hypothetical lat and lon data
cars$lat <- sapply(rownames(cars), function(x) round(runif(1, 30, 46), 3))
cars$lon <- sapply(rownames(cars), function(x) round(runif(1, -115, -80),3))
knitr::kable(cars)
kable(cars)
</code>```

I was able to get are relatively normal looking table by using kable(cars, format = "latex")

I was hoping to get some quick and decent html tables, and now when I try to format in LaTeX with something like kable(cars, format = "latex", booktabs = T), I get this error:

```<code>pandoc: Error producing PDF
Error: pandoc document conversion failed with error 43
</code>```

答:
You have to modify the .Rmd file to include the package in output
Create a YAML header with

```<code>header-includes:
   - \usepackage{booktabs}
</code>```

modified .Rmd file which now works for me (I was also reproducing your error)

```<code>---
title: "Title"
author: "Me"
header-includes:
   - \usepackage{booktabs}
output:
    pdf_document
---
```{r kable}
library(knitr)
library(markdown)
library(rmarkdown)
cars <-mtcars[rownames(mtcars)==car,]

# create example data
x <- sample(1:10, 1)
cars <- do.call("rbind", replicate(x, cars, simplify =FALSE))

# create hypothetical lat and lon data
cars$lat <- sapply(rownames(cars), function(x) round(runif(1, 30, 46), 3))
cars$lon <-sapply(rownames(cars), function(x) round(runif(1, -115, -80),3))
knitr::kable(cars, format = "latex", booktabs = T)
```
</code>```

Now this produces four files

Title: knitr tables appear as one long column

Body:
I pretty much copied word-for-word the example from this site on how to create multiple PDFs in knitr from one for-loop: http://www.reed.edu/data-at-reed/software/R/markdown_multiple_reports.html

When I try calling knitr::kable(cars) or kable(cars) the output of each PDF appears as one long column instead of an actual table, as shown in the photo.

Any idea what causes this? Here's the .R and the .Rmd code called in each iteration of the loop:

.R

library(knitr)
library(markdown)
library(rmarkdown)
mt <- mtcars[1:5,]
for (car in unique(rownames(mt))){
     rmarkdown::render('test.Rmd',
                output_format = "pdf_document",
                output_file =  paste(car, ' report.pdf', sep=''),
                output_dir = '~/')

test.Rmd

library(knitr)
library(markdown)
library(rmarkdown)
cars <- mtcars[rownames(mtcars)==car,]

# create example data
x <- sample(1:10, 1)
cars <- do.call("rbind", replicate(x, cars, simplify = FALSE))

# create hypothetical lat and lon data
cars$lat <- sapply(rownames(cars), function(x) round(runif(1, 30, 46), 3))
cars$lon <- sapply(rownames(cars), function(x) round(runif(1, -115, -80),3))
knitr::kable(cars)
kable(cars)

I was able to get are relatively normal looking table by using kable(cars, format = "latex")

I was hoping to get some quick and decent html tables, and now when I try to format in LaTeX with something like kable(cars, format = "latex", booktabs = T), I get this error:

pandoc: Error producing PDF
Error: pandoc document conversion failed with error 43

Body:
You have to modify the .Rmd file to include the package in output
Create a YAML header with

header-includes:
   - \usepackage{booktabs}

modified .Rmd file which now works for me (I was also reproducing your error)

---
title: "Title"
author: "Me"
header-includes:
   - \usepackage{booktabs}
output:
    pdf_document
---
```{r kable}
library(knitr)
library(markdown)
library(rmarkdown)
cars <-mtcars[rownames(mtcars)==car,]

# create example data
x <- sample(1:10, 1)
cars <- do.call("rbind", replicate(x, cars, simplify =FALSE))

# create hypothetical lat and lon data
cars$lat <- sapply(rownames(cars), function(x) round(runif(1, 30, 46), 3))
cars$lon <-sapply(rownames(cars), function(x) round(runif(1, -115, -80),3))
knitr::kable(cars, format = "latex", booktabs = T)
```

Now this produces four files


问:
What are Sockets afterall? Why can't we use sockets instead of HTTP?
Are sockets just a connection between 2 machines?

And if sockets can be established why do we even use HTTP connection?

答:
I assume this is a very general question about the relationship between sockets and HTTP connections. I also assume that "HTTPConnection" does not refer to something involving a specific API/runtime/environment even though the way it's a camel cased term with spaces removed could suggest otherwise.

Now that that's out of the way, I present to you, the OSI model:

The OSI Model describes levels of abstraction for network communication. A socket is a concept which would exist somewhere on layer 3, the Network Layer, as part of the Internet Protocol (IP).

HTTP is higher abstraction than IP, usually regarded as being up in the Application Layer, at the "top" of the OSI model.

An Analogy

You could define a city's transportation and traffic at different "layers" the same way we define network stuffs.

At its simplest, a city is a bunch of buildings.

As the city grows people need to travel from building to building, so they develop roads. The roads are a new "layer" to the city.

As more people use the roads, they begin to need a system of rules and laws to help keep everyone safe.

Once people are safe on the roads though, they want the roads to be efficient and quick, so a system of lights and signs help coordinate people on the roads.

Two important things:

First, each layer depends on the one "below" it. Without buildings (destinations) roads become silly. Without roads, traffic laws are silly. Without traffic laws, traffic lights are silly.

Second, the specifics of the higher layers vary depending on the city you're in: sometimes you find yourself in a country where people drive on the left, sometimes they drive on the right. Sometimes you can turn on a red, sometimes not. Sometimes there are roads, but they are without laws.

End of Analogy

So on the Internet, sometimes you communicate with different kinds of servers. Underneath, they may all rely on sockets (the "roads" of he internet) but they all have their own "traffic laws" that you have to respect - protocols like HTTP or FTP or SOAP.