问:
How to find week date ranges for a given month
I need to display a range of dates in corresponding week for selected month.
Suppose the selected values are $month=2 and $year=2017.
The output should show a list of ranged dates for weeks in that particular month selected.

```<code>$month = $_GET['month'];
$year = $_GET['year'];
 ...

Output:
Week 1: 01/02/2017 - 05/02/2017
Week 2: 06/02/2017 - 12/02/2017
Week 3: 13/02/2017 - 19/02/2017
Week 4: 20/02/2017 - 26/02/2017
Week 5: 27/02/2017 - 28/02/2017
</code>```

Ive look through Split the current month in to weeks in php , Get week number in month from date in PHP? and Split date ranges into corresponding weeks , but none of them working for me.

Thanks.

Body:
I need to display a range of dates in corresponding week for selected month.
Suppose the selected values are $month=2 and $year=2017.
The output should show a list of ranged dates for weeks in that particular month selected.

$month = $_GET['month'];
$year = $_GET['year'];
 ...

Output:
Week 1: 01/02/2017 - 05/02/2017
Week 2: 06/02/2017 - 12/02/2017
Week 3: 13/02/2017 - 19/02/2017
Week 4: 20/02/2017 - 26/02/2017
Week 5: 27/02/2017 - 28/02/2017

Ive look through Split the current month in to weeks in php , Get week number in month from date in PHP? and Split date ranges into corresponding weeks , but none of them working for me.

Thanks.

Body:
Here is your solution

$month = $_GET['month'];
if($month<10)$month = '0'.$month;
$year = $_GET['year'];
$days = cal_days_in_month(CAL_GREGORIAN,$month,$year);
$start = 1; $end = 5;
function get_week_array($start,$end){
    global $month, $year,$days;
    for($i=0;$i<$end;$i++){
        if($start<10)$array[] = '0'.$start.'/'.$month.'/'.$year;
        else $array[] = $start.'/'.$month.'/'.$year;
        $start = $start+1;
        if($start==$days+1)break;
    }
    return $array;
}
$total = 0;
while($days>$total){
    $week[] = get_week_array($start,$end);
    $total = $total+$end;
    $start = $total+1;
    $end = 7;
}
echo "<pre>";print_r($week);

Output

Array
(
    [0] => Array
        (
            [0] => 01/02/2017
            [1] => 02/02/2017
            [2] => 03/02/2017
            [3] => 04/02/2017
            [4] => 05/02/2017
        )

    [1] => Array
        (
            [0] => 06/02/2017
            [1] => 07/02/2017
            [2] => 08/02/2017
            [3] => 09/02/2017
            [4] => 10/02/2017
            [5] => 11/02/2017
            [6] => 12/02/2017
        )

    [2] => Array
        (
            [0] => 13/02/2017
            [1] => 14/02/2017
            [2] => 15/02/2017
            [3] => 16/02/2017
            [4] => 17/02/2017
            [5] => 18/02/2017
            [6] => 19/02/2017
        )

    [3] => Array
        (
            [0] => 20/02/2017
            [1] => 21/02/2017
            [2] => 22/02/2017
            [3] => 23/02/2017
            [4] => 24/02/2017
            [5] => 25/02/2017
            [6] => 26/02/2017
        )

    [4] => Array
        (
            [0] => 27/02/2017
            [1] => 28/02/2017
        )

)


来源: stackexchange

问:
ClassCastException when need to parse httpServeletRequest
I run my code from tomcat but getting error

```<code>HashMap<String, Object> params;
params = (HashMap) HttpUtils.getParamsFromRequest(request);
</code>```

HttpUtils code below

```<code>    public static Map<String, ? extends Object> getParamsFromRequest(HttpServletRequest httpServletRequest) {
    return httpServletRequest.getParameterMap();
}
</code>```

but getting exception like this

```<code>java.lang.ClassCastException: org.apache.catalina.util.ParameterMap cannot be cast to java.util.HashMap
at se.mindspot.tender.backend.servlet.service.file.FileServiceServlet.doGet(FileServiceServlet.java:47)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
</code>```

答:
Map is an interface, HashMap is a concrete implementation. ParameterMap implements Map but does not sub-class HashMap so it cannot be cast to a HashMap. The proper thing to do is usually

```<code>Map<String, Object> params;
params = HttpUtils.getParamsFromRequest(request);
</code>```

and program to the Map interface. However, you can do

```<code>params = new HashMap<>(HttpUtils.getParamsFromRequest(request));
</code>```

if you need a HashMap and have some other kind of Map, but that involves a copy.

ContentLicense: CC BY-SA 3.0

Body:
I run my code from tomcat but getting error

HashMap<String, Object> params;
params = (HashMap) HttpUtils.getParamsFromRequest(request);

HttpUtils code below

    public static Map<String, ? extends Object> getParamsFromRequest(HttpServletRequest httpServletRequest) {
    return httpServletRequest.getParameterMap();
}

but getting exception like this

java.lang.ClassCastException: org.apache.catalina.util.ParameterMap cannot be cast to java.util.HashMap
at se.mindspot.tender.backend.servlet.service.file.FileServiceServlet.doGet(FileServiceServlet.java:47)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

Body:
Map is an interface, HashMap is a concrete implementation. ParameterMap implements Map but does not sub-class HashMap so it cannot be cast to a HashMap. The proper thing to do is usually

Map<String, Object> params;
params = HttpUtils.getParamsFromRequest(request);

and program to the Map interface. However, you can do

params = new HashMap<>(HttpUtils.getParamsFromRequest(request));

if you need a HashMap and have some other kind of Map, but that involves a copy.


问:
Create a Map with specific parent node value and all of its child node values of an XML
I have a sample LEDES XML file https://codebeautify.org/xmlviewer/cbdc79e7 and I am trying to create a Map with Invoice node's inv_id value as key and all of its child elements file_item_nbr values as below

```<code>['Invoice 31' : [10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33]
 'Invoice 32' : [50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73]
]
</code>```

can someone please help me with it?

答:
You should be able to get it with sudo code:

Use XmlSlurper to parse xml

Read all invoice elements

Build the required map

Here is the script:

```<code>//Not putting entire xml here, just pass the xml as string to parseText method
def xml = new XmlSlurper().parseText(xmlString)
//Get the invoices
def invoices = xml.'**'.findAll{it.name() == 'invoice'}
//Build the desired result
println invoices.collectEntries {inv -> [(inv.inv_id): inv.'**'.findAll{it.name() == 'file_item_nbr'}*.text()] }
</code>```

You may quickly try it online demo

Body:
I have a sample LEDES XML file https://codebeautify.org/xmlviewer/cbdc79e7 and I am trying to create a Map with Invoice node's inv_id value as key and all of its child elements file_item_nbr values as below

['Invoice 31' : [10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33]
 'Invoice 32' : [50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73]
]

can someone please help me with it?

Body:
You should be able to get it with sudo code:

Use XmlSlurper to parse xml

Read all invoice elements

Build the required map

Here is the script:

//Not putting entire xml here, just pass the xml as string to parseText method
def xml = new XmlSlurper().parseText(xmlString)
//Get the invoices
def invoices = xml.'**'.findAll{it.name() == 'invoice'}
//Build the desired result
println invoices.collectEntries {inv -> [(inv.inv_id): inv.'**'.findAll{it.name() == 'file_item_nbr'}*.text()] }

You may quickly try it online demo


问:
Tooltip not working in angular component test (from ngx-bootstrap)
I was using TooltipModule from ng2-bootstrap and recently switched to ngx-bootstrap. Tooltips work fine in my application - no issues. However, in my Angular component test, when I import the following:

```<code>import { TooltipModule } from 'ngx-bootstrap';
</code>```

and then in the TestBed, Import the following:

```<code>TestBed.configureTestingModule({imports: [TooltipModule.forRoot(),
</code>```

Tooltips are not rendered. I dont see the tooltip in dev console under Elements tab in my browser.
When I use the import from ng2-bootstrap as:

```<code>import { TooltipModule } from 'ng2-bootstrap';
</code>```

everything works fine. I see the tooltip rendered.

What is the issue here? Tooltip works fine in my app. Only in Angular test, I am having issues. Do I need to do anything differently or extra in my tests to make tooltips work from ngx-bootstrap?

答:
you have to import import
{ TooltipModule } from 'ngx-bootstrap';

in your relative module like app.module not in your component and then in your modules import array add TooltipModule.forRoot().

that's it you can use tooltip now in your app.modules components.

If you are using lazy-loading then add { TooltipModule } from 'ngx-bootstrap'; it  in your other module.

ContentLicense: CC BY-SA 4.0

Body:
I was using TooltipModule from ng2-bootstrap and recently switched to ngx-bootstrap. Tooltips work fine in my application - no issues. However, in my Angular component test, when I import the following:

import { TooltipModule } from 'ngx-bootstrap';

and then in the TestBed, Import the following:

TestBed.configureTestingModule({imports: [TooltipModule.forRoot(),

Tooltips are not rendered. I dont see the tooltip in dev console under Elements tab in my browser.
When I use the import from ng2-bootstrap as:

import { TooltipModule } from 'ng2-bootstrap';

everything works fine. I see the tooltip rendered.

What is the issue here? Tooltip works fine in my app. Only in Angular test, I am having issues. Do I need to do anything differently or extra in my tests to make tooltips work from ngx-bootstrap?


问:
For Stream.generate(foo).parallel(), does foo have to be thread safe?
I'm working with Java 8 Streams. I have a custom function, foo(), that yields an object, and I would like to parallel stream the objects it creates. I know foo() is not thread safe.

If I write Stream.generate(foo).parallel(), will foo() be called in an asynchronous manner? I.e. will objects be generated serially, and passed to parallel threads, or will multiple threads each generate objects as needed by calling foo()?

答:
The supplier will be called from multiple threads as you can observe with a quick experiment:</p>

```<code>Stream.generate(() -> Thread.currentThread().getId())
    .parallel()
    .limit(100000)
    .distinct()
    .forEach(System.out::println);
</code>```

Body:
The supplier will be called from multiple threads as you can observe with a quick experiment:

Stream.generate(() -> Thread.currentThread().getId())
    .parallel()
    .limit(100000)
    .distinct()
    .forEach(System.out::println);


问:
Pandas create column conditional on others
I have a dataframe like

```<code>df = pd.DataFrame({'tag':['x','x','z','z','x','z'], 'val':[1,2,3,1,2,3]})
</code>```

Is there a one-liner to add a column 'res' that shows -val if tag=='x' otherwise +val, so it would look like

```<code>  tag  val  res
0   x    1   -1
1   x    2   -2
2   z    3    3
3   z    1    1
4   x    2   -2
5   z    3    3
</code>```

答:
By manipulating the boolean Series df.tag != 'x'

```<code>df['res'] = df.tag.ne('x').mul(2).sub(1).mul(df.val)

df
#  tag  val  res
#0   x    1   -1
#1   x    2   -2
#2   z    3    3
#3   z    1    1
#4   x    2   -2
#5   z    3    3
</code>```

Body:
I have a dataframe like

df = pd.DataFrame({'tag':['x','x','z','z','x','z'], 'val':[1,2,3,1,2,3]})

Is there a one-liner to add a column 'res' that shows -val if tag=='x' otherwise +val, so it would look like

  tag  val  res
0   x    1   -1
1   x    2   -2
2   z    3    3
3   z    1    1
4   x    2   -2
5   z    3    3

Body:
By manipulating the boolean Series df.tag != 'x'

df['res'] = df.tag.ne('x').mul(2).sub(1).mul(df.val)

df
#  tag  val  res
#0   x    1   -1
#1   x    2   -2
#2   z    3    3
#3   z    1    1
#4   x    2   -2
#5   z    3    3


问:
Select the number of days a record is present within a certain year/month
In sql server I have a table with start and end dates for job postings. Given a month and a year by the user I need to find the amount of days between the start and end date that fall in the given year/month for that posting.

So if the posting Start Date is 2010/11/15 and End Date is 2010/12/05
Then output should be:

```<code>November  16 days
December  5 days
Total     21 days
</code>```

I've been beating my head against the wall with this one and am fresh out of ideas.

答:
This is probably the most concise answer.

```<code>declare @start datetime, @end datetime
select @start = '20101115', @end = '20101205'

select datename(month,@start+number), count(*)
from master..spt_values
where type='P'
  and number between 0 and datediff(d,@start,@end)
group by datename(month,@start+number), convert(char(6),@start+number,112)
order by convert(char(6),@start+number,112)
</code>```

It will work for ranges up to 2048 days (7-8 years) but can by extended for longer if you need (on request only - that will looks more complicated).

The only reason for the convert(char portion is to make November come up before December, and also before January of the next year.

ContentLicense: CC BY-SA 2.5

Body:
In sql server I have a table with start and end dates for job postings. Given a month and a year by the user I need to find the amount of days between the start and end date that fall in the given year/month for that posting.

So if the posting Start Date is 2010/11/15 and End Date is 2010/12/05
Then output should be:

November  16 days
December  5 days
Total     21 days

I've been beating my head against the wall with this one and am fresh out of ideas.

Body:
This is probably the most concise answer.

declare @start datetime, @end datetime
select @start = '20101115', @end = '20101205'

select datename(month,@start+number), count(*)
from master..spt_values
where type='P'
  and number between 0 and datediff(d,@start,@end)
group by datename(month,@start+number), convert(char(6),@start+number,112)
order by convert(char(6),@start+number,112)

It will work for ranges up to 2048 days (7-8 years) but can by extended for longer if you need (on request only - that will looks more complicated).

The only reason for the convert(char portion is to make November come up before December, and also before January of the next year.


问:
Trying to join three tables to get a specific result
I am trying to write an SQL query to return the name of the person who adopted the cat 'seashell'

I have a adopters table with people who will or have adopted an animal.

```<code>pets=# SELECT * FROM adopters;
 id | first_name | last_name |       address        | phone_number
----+------------+-----------+----------------------+--------------
  1 | jack       | smith     | 1 ave                | 123-123-1234
  2 | james      | dude      | 2 ave                | 221-234-4444
  3 | Forrest    | Stone     | 3 lane               | 234-667-6543
  4 | craig      | list      | 44 bulvard           | 555-444-3243
  5 | kap        | seven     | no team              | 999-000-0909
  6 | Miles      | Garrett   | First energy stadium | 123-555-2424
(6 rows)
</code>```

A cats table with the cats on file

```<code>pets=# SELECT * FROM cats;
 id |   name   | gender | age |     intake_date     |    adoption_date
----+----------+--------+-----+---------------------+---------------------
  1 | Mushi    | M      |   1 | 2016-01-09 00:00:00 | 2016-03-22 00:00:00
  2 | Seashell | F      |   7 | 2016-01-09 00:00:00 |
  3 | Azul     | M      |   3 | 2016-01-11 00:00:00 | 2016-04-17 00:00:00
  4 | Victoire | M      |   7 | 2016-01-11 00:00:00 | 2016-09-01 00:00:00
  5 | Nala     | F      |   1 | 2016-01-12 00:00:00 |
(5 rows)
</code>```

And an adoptions table with a record of all the adoptions that have taken place.

```<code>pets=# SELECT * FROM adoptions;
 id | adopters | cat |  dog  |  fee  |    date
----+----------+-----+-------+-------+------------
  1 |        1 |     | 10001 | 10.50 | 2017-11-22
  2 |        2 |     | 10007 | 10.50 | 2017-11-22
  3 |        3 |   5 |       | 10.50 | 2017-11-22
  4 |        5 |   2 |       | 10.50 | 2016-11-22
(4 rows)
</code>```

I am having trouble with the concept of joining three tables and even if you don't have a direct answer to the problem I would greatly appreciate being pointed in the direction of a resource that would help.

答:
```<code>SELECT a.first_name, a.last_name
FROM adopters a, cats c, adoptions ad
WHERE a.id = ad.adopters AND ad.cat = c.id
AND c.name = "Seashell";
</code>```

Please Try This

Body:
I am trying to write an SQL query to return the name of the person who adopted the cat 'seashell'

I have a adopters table with people who will or have adopted an animal.

pets=# SELECT * FROM adopters;
 id | first_name | last_name |       address        | phone_number
----+------------+-----------+----------------------+--------------
  1 | jack       | smith     | 1 ave                | 123-123-1234
  2 | james      | dude      | 2 ave                | 221-234-4444
  3 | Forrest    | Stone     | 3 lane               | 234-667-6543
  4 | craig      | list      | 44 bulvard           | 555-444-3243
  5 | kap        | seven     | no team              | 999-000-0909
  6 | Miles      | Garrett   | First energy stadium | 123-555-2424
(6 rows)

A cats table with the cats on file

pets=# SELECT * FROM cats;
 id |   name   | gender | age |     intake_date     |    adoption_date
----+----------+--------+-----+---------------------+---------------------
  1 | Mushi    | M      |   1 | 2016-01-09 00:00:00 | 2016-03-22 00:00:00
  2 | Seashell | F      |   7 | 2016-01-09 00:00:00 |
  3 | Azul     | M      |   3 | 2016-01-11 00:00:00 | 2016-04-17 00:00:00
  4 | Victoire | M      |   7 | 2016-01-11 00:00:00 | 2016-09-01 00:00:00
  5 | Nala     | F      |   1 | 2016-01-12 00:00:00 |
(5 rows)

And an adoptions table with a record of all the adoptions that have taken place.

pets=# SELECT * FROM adoptions;
 id | adopters | cat |  dog  |  fee  |    date
----+----------+-----+-------+-------+------------
  1 |        1 |     | 10001 | 10.50 | 2017-11-22
  2 |        2 |     | 10007 | 10.50 | 2017-11-22
  3 |        3 |   5 |       | 10.50 | 2017-11-22
  4 |        5 |   2 |       | 10.50 | 2016-11-22
(4 rows)

I am having trouble with the concept of joining three tables and even if you don't have a direct answer to the problem I would greatly appreciate being pointed in the direction of a resource that would help.

Body:
SELECT a.first_name, a.last_name
FROM adopters a, cats c, adoptions ad
WHERE a.id = ad.adopters AND ad.cat = c.id
AND c.name = "Seashell";

Please Try This


问:
how to see if value entered is a number and not chars?

I want to know if someone entered a number and if they did the program continues but if they didn't the program terminates.
</p>

```<code>int main(void) {
    int i;

    printf("Enter a number: ");
    scanf("%d",&i);
    /*If input is not a number than terminate*/

    /*Otherwise continue*/
    else {
        if (test_prime(i))
            printf("Prime.\n");
        else
            printf("Not prime.\n");
    }
    return 0;
}
</code>```

<p>
If the user enters "hello" than I want program to terminate.
</p>

答:
We have a function isdigit() to identify the variable and the code goes like this:</p>

```<code>int main(void) {
   int i;

   printf("Enter a number: ");
   scanf("%d",&i);
   if( isdigit(i) ) {
      printf("It is a digit.");
   }
   else {
      if (test_prime(i))
        printf("Prime.\n");
      else
        printf("Not prime.\n");
   }
   return 0;
}
</code>```

Body:
I want to know if someone entered a number and if they did the program continues but if they didn't the program terminates.

int main(void) {
    int i;

    printf("Enter a number: ");
    scanf("%d",&i);
    /*If input is not a number than terminate*/

    /*Otherwise continue*/
    else {
        if (test_prime(i))
            printf("Prime.\n");
        else
            printf("Not prime.\n");
    }
    return 0;
}

If the user enters "hello" than I want program to terminate.

Body:
We have a function isdigit() to identify the variable and the code goes like this:

int main(void) {
   int i;

   printf("Enter a number: ");
   scanf("%d",&i);
   if( isdigit(i) ) {
      printf("It is a digit.");
   }
   else {
      if (test_prime(i))
        printf("Prime.\n");
      else
        printf("Not prime.\n");
   }
   return 0;
}


问:
Tensorflow: function.defun with a a while loop in the body is throwing shape error

I am using a while loop to calculate a cost function for memory reasons. When calculating the gradient, tensorflow will store Nm tensors where Nm is the number of iterations in my while loop (this cuases the same memory issues I had with the original energy functions). I do not want that as I don't have enough memory. So I want to register a new op along with a gradient function that both use a while loop. However I am having issues with using function.defun and a while loop. To simplify things, I have a small test example below:

```<code>import numpy as np
import tensorflow as tf
from tensorflow.python.framework import ops
from tensorflow.python.ops import array_ops
from tensorflow.python.ops import sparse_ops
from tensorflow.python.framework import function

def _run(tensor):
    with tf.Session() as sess:
         sess.run(tf.global_variables_initializer())
         res = sess.run(tensor)
    return res

@function.Defun(tf.float32,tf.float32,func_name ='tf_test_log')#,grad_func=tf_test_logGrad)
def tf_test_log(t_x,t_y):
    #N = t_x.shape[0].value
    condition = lambda i,m1: i<N

    def body(index,x):
        #return[(index+1),tf.concat([x, tf.expand_dims(tf.exp( tf.add( t_x[:,index],t_y[:,index]) ),1)   ],1 ) ]
        return[(index+1),tf.add(x, tf.exp( tf.add( t_x[:,0],t_y[:,0]) )    ) ]

    i0      = tf.constant(0,dtype=tf.int32)
    m0      = tf.zeros([N,1],dType)

    ijk_0   = [i0,m0]

    L,t_log_x = tf.while_loop(condition,body,ijk_0,
                     shape_invariants=[i0.get_shape(),
                                       tf.TensorShape([N,None])]
                                        )
    return t_log_x

dType = tf.float32
N     = np.int32(100)

t_N   = tf.constant(N,dtype = tf.int32)
t_x   = tf.constant(np.random.randn(N,N),dtype = dType)
t_y   = tf.constant(np.random.randn(N,N),dtype = dType)
ys = _run(tf_test_log(t_x,t_y))
</code>```

I then try to test the new op:

I get a Value error: The shape for while/Merge_1:0 is not an invariant for the loop. It enters the loop with shape (100, ?), but has shape  after one iteration. Provide shape invariants using either the shape_invariants argument of tf.while_loop or set_shape() on the loop variables.

Note that calling

If i use a concatenate operation (instead of the add operation that gets returned by my while loop), I do not get any issues.

However, If I do not set N as a global variable (i.e. I do N = t_x.shape[0]) inside the body of the tf_test_log function, I get a Value error.

ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 1)

What is wrong with my code? Any help is greatly appreciated!

I am using python 3.5 on ubuntu 16.04 and tensorflow 1.4

full output:

```<code>ValueError                                Traceback (most recent call last)
~/Documents/TheEffingPhDHatersGonnaHate/PAM/defun_while.py in <module>()
     51 t_x   = tf.constant(np.random.randn(N,N),dtype = dType)
     52 t_y   = tf.constant(np.random.randn(N,N),dtype = dType)
---> 53 ys = _run(tf_test_log(t_x,t_y))
     54
     55

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/function.py in __call__(self, *args, **kwargs)
    503
    504   def __call__(self, *args, **kwargs):
--> 505     self.add_to_graph(ops.get_default_graph())
    506     args = [ops.convert_to_tensor(_) for _ in args] + self._extra_inputs
    507     ret, op = _call(self._signature, *args, **kwargs)

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/function.py in add_to_graph(self, g)
    484   def add_to_graph(self, g):
    485     """Adds this function into the graph g."""
--> 486     self._create_definition_if_needed()
    487
    488     # Adds this function into 'g'.

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/function.py in _create_definition_if_needed(self)
    319     """Creates the function definition if it's not created yet."""
    320     with context.graph_mode():
--> 321       self._create_definition_if_needed_impl()
    322
    323   def _create_definition_if_needed_impl(self):

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/function.py in _create_definition_if_needed_impl(self)
    336       # Call func and gather the output tensors.
    337       with vs.variable_scope("", custom_getter=temp_graph.getvar):
--> 338         outputs = self._func(*inputs)
    339
    340       # There is no way of distinguishing between a function not returning

~/Documents/TheEffingPhDHatersGonnaHate/PAM/defun_while.py in tf_test_log(t_x, t_y)
     39     L,t_log_x = tf.while_loop(condition,body,ijk_0,
     40                          shape_invariants=[i0.get_shape(),
---> 41                                            tf.TensorShape([N,None])]
     42                                             )
     43     return t_log_x

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py in while_loop(cond, body, loop_vars, shape_invariants, parallel_iterations, back_prop, swap_memory, name)
   2814     loop_context = WhileContext(parallel_iterations, back_prop, swap_memory)  # pylint: disable=redefined-outer-name
   2815     ops.add_to_collection(ops.GraphKeys.WHILE_CONTEXT, loop_context)
-> 2816     result = loop_context.BuildLoop(cond, body, loop_vars, shape_invariants)
   2817     return result
   2818

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py in BuildLoop(self, pred, body, loop_vars, shape_invariants)
   2638       self.Enter()
   2639       original_body_result, exit_vars = self._BuildLoop(
-> 2640           pred, body, original_loop_vars, loop_vars, shape_invariants)
   2641     finally:
   2642       self.Exit()

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py in _BuildLoop(self, pred, body, original_loop_vars, loop_vars, shape_invariants)
   2619     for m_var, n_var in zip(merge_vars, next_vars):
   2620       if isinstance(m_var, ops.Tensor):
-> 2621         _EnforceShapeInvariant(m_var, n_var)
   2622
   2623     # Exit the loop.

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py in _EnforceShapeInvariant(merge_var, next_var)
    576           "Provide shape invariants using either the `shape_invariants` "
    577           "argument of tf.while_loop or set_shape() on the loop variables."
--> 578           % (merge_var.name, m_shape, n_shape))
    579   else:
    580     if not isinstance(var, (ops.IndexedSlices, sparse_tensor.SparseTensor)):

ValueError: The shape for while/Merge_1:0 is not an invariant for the loop. It enters the loop with shape (100, ?), but has shape <unknown> after one iteration. Provide shape invariants using either the `shape_invariants` argument of tf.while_loop or set_shape() on the loop variables.
</code>```

答:
Thanks @Alexandre Passos for the suggestion in the comment above!
The following piece of code is a modification of the original with a set_shape function added inside the body.

```<code>import numpy as np
import tensorflow as tf
from tensorflow.python.framework import ops
from tensorflow.python.ops import array_ops
from tensorflow.python.ops import sparse_ops
from tensorflow.python.framework import function

def _run(tensor):
    with tf.Session() as sess:
         sess.run(tf.global_variables_initializer())
         res = sess.run(tensor)
    return res

@function.Defun(tf.float32,tf.float32,tf.float32,func_name ='tf_test_logGrad')
def tf_test_logGrad(t_x,t_y,grad):
     return grad

@function.Defun(tf.float32,tf.float32,func_name     ='tf_test_log')#,grad_func=tf_test_logGrad)
def tf_test_log(t_x,t_y):
    #N = t_x.shape[0].value
    condition = lambda i,m1: i<N

    def body(index,x):
        #return[(index+1),tf.concat([x, tf.expand_dims(tf.exp( tf.add(     t_x[:,index],t_y[:,index]) ),1)   ],1 ) ]
        x = tf.add(x, tf.exp( tf.add( t_x[:,0],t_y[:,0]) )    )
        x.set_shape([N])
        return[(index+1), x]

    i0      = tf.constant(0,dtype=tf.int32)
    m0      = tf.zeros([N],dType)

    ijk_0   = [i0,m0]

    L,t_log_x = tf.while_loop(condition,body,ijk_0,
                     shape_invariants=[i0.get_shape(),
                                       tf.TensorShape([N])]
                                        )
    return t_log_x

dType = tf.float32
N     = np.int32(100)

t_N   = tf.constant(N,dtype = tf.int32)
t_x   = tf.constant(np.random.randn(N,N),dtype = dType)
t_y   = tf.constant(np.random.randn(N,N),dtype = dType)
ys = _run(tf_test_log(t_x,t_y))
</code>```

The Issue of global N still persists.

You still need to set the shape of the loop tensors as a global variable outside of the defun decorator. If you try to get it from the shape of the inputs of the defun decorator, you get:

```<code>TypeError                                 Traceback (most recent call last)
~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py in zeros(shape, dtype, name)
   1438       shape = tensor_shape.as_shape(shape)
-> 1439       output = constant(zero, shape=shape, dtype=dtype, name=name)
   1440     except (TypeError, ValueError):

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/constant_op.py in constant(value, dtype, shape, name, verify_shape)
    207       tensor_util.make_tensor_proto(
--> 208           value, dtype=dtype, shape=shape, verify_shape=verify_shape))
    209   dtype_value = attr_value_pb2.AttrValue(type=tensor_value.tensor.dtype)

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/tensor_util.py in make_tensor_proto(values, dtype, shape, verify_shape)
    379     # exception when dtype is set to np.int64
--> 380     if shape is not None and np.prod(shape, dtype=np.int64) == 0:
    381       nparray = np.empty(shape, dtype=np_dt)

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/numpy/core/fromnumeric.py in prod(a, axis, dtype, out, keepdims)
   2517     return _methods._prod(a, axis=axis, dtype=dtype,
-> 2518                           out=out, **kwargs)
   2519

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/numpy/core/_methods.py in _prod(a, axis, dtype, out, keepdims)
     34 def _prod(a, axis=None, dtype=None, out=None, keepdims=False):
---> 35     return umr_prod(a, axis, dtype, out, keepdims)
     36

TypeError: __int__ returned non-int (type NoneType)

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
~/Documents/TheEffingPhDHatersGonnaHate/PAM/defun_while.py in <module>()
     52 t_x   = tf.constant(np.random.randn(N,N),dtype = dType)
     53 t_y   = tf.constant(np.random.randn(N,N),dtype = dType)
---> 54 ys = _run(tf_test_log(t_x,t_y))
     55
     56

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/function.py in __call__(self, *args, **kwargs)
    503
    504   def __call__(self, *args, **kwargs):
--> 505     self.add_to_graph(ops.get_default_graph())
    506     args = [ops.convert_to_tensor(_) for _ in args] + self._extra_inputs
    507     ret, op = _call(self._signature, *args, **kwargs)

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/function.py in add_to_graph(self, g)
    484   def add_to_graph(self, g):
    485     """Adds this function into the graph g."""
--> 486     self._create_definition_if_needed()
    487
    488     # Adds this function into 'g'.

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/function.py in _create_definition_if_needed(self)
    319     """Creates the function definition if it's not created yet."""
    320     with context.graph_mode():
--> 321       self._create_definition_if_needed_impl()
    322
    323   def _create_definition_if_needed_impl(self):

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/function.py in _create_definition_if_needed_impl(self)
    336       # Call func and gather the output tensors.
    337       with vs.variable_scope("", custom_getter=temp_graph.getvar):
--> 338         outputs = self._func(*inputs)
    339
    340       # There is no way of distinguishing between a function not returning

~/Documents/TheEffingPhDHatersGonnaHate/PAM/defun_while.py in tf_test_log(t_x, t_y)
     33
     34     i0      = tf.constant(0,dtype=tf.int32)
---> 35     m0      = tf.zeros([N],dType)
     36
     37

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py in zeros(shape, dtype, name)
   1439       output = constant(zero, shape=shape, dtype=dtype, name=name)
   1440     except (TypeError, ValueError):
-> 1441       shape = ops.convert_to_tensor(shape, dtype=dtypes.int32, name="shape")
   1442       output = fill(shape, constant(zero, dtype=dtype), name=name)
   1443   assert output.dtype.base_dtype == dtype

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/ops.py in convert_to_tensor(value, dtype, name, preferred_dtype)
    834       name=name,
    835       preferred_dtype=preferred_dtype,
--> 836       as_ref=False)
    837
    838

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/ops.py in internal_convert_to_tensor(value, dtype, name, as_ref, preferred_dtype, ctx)
    924
    925     if ret is None:
--> 926       ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
    927
    928     if ret is NotImplemented:

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/constant_op.py in _tensor_shape_tensor_conversion_function(s, dtype, name, as_ref)
    248   if not s.is_fully_defined():
    249     raise ValueError(
--> 250         "Cannot convert a partially known TensorShape to a Tensor: %s" % s)
    251   s_list = s.as_list()
    252   int64_value = 0

ValueError: Cannot convert a partially known TensorShape to a Tensor: (?,)
</code>```

Body:
I am using a while loop to calculate a cost function for memory reasons. When calculating the gradient, tensorflow will store Nm tensors where Nm is the number of iterations in my while loop (this cuases the same memory issues I had with the original energy functions). I do not want that as I don't have enough memory. So I want to register a new op along with a gradient function that both use a while loop. However I am having issues with using function.defun and a while loop. To simplify things, I have a small test example below:

import numpy as np
import tensorflow as tf
from tensorflow.python.framework import ops
from tensorflow.python.ops import array_ops
from tensorflow.python.ops import sparse_ops
from tensorflow.python.framework import function

def _run(tensor):
    with tf.Session() as sess:
         sess.run(tf.global_variables_initializer())
         res = sess.run(tensor)
    return res

@function.Defun(tf.float32,tf.float32,func_name ='tf_test_log')#,grad_func=tf_test_logGrad)
def tf_test_log(t_x,t_y):
    #N = t_x.shape[0].value
    condition = lambda i,m1: i<N

    def body(index,x):
        #return[(index+1),tf.concat([x, tf.expand_dims(tf.exp( tf.add( t_x[:,index],t_y[:,index]) ),1)   ],1 ) ]
        return[(index+1),tf.add(x, tf.exp( tf.add( t_x[:,0],t_y[:,0]) )    ) ]

    i0      = tf.constant(0,dtype=tf.int32)
    m0      = tf.zeros([N,1],dType)

    ijk_0   = [i0,m0]

    L,t_log_x = tf.while_loop(condition,body,ijk_0,
                     shape_invariants=[i0.get_shape(),
                                       tf.TensorShape([N,None])]
                                        )
    return t_log_x

dType = tf.float32
N     = np.int32(100)

t_N   = tf.constant(N,dtype = tf.int32)
t_x   = tf.constant(np.random.randn(N,N),dtype = dType)
t_y   = tf.constant(np.random.randn(N,N),dtype = dType)
ys = _run(tf_test_log(t_x,t_y))

I then try to test the new op:

I get a Value error: The shape for while/Merge_1:0 is not an invariant for the loop. It enters the loop with shape (100, ?), but has shape  after one iteration. Provide shape invariants using either the shape_invariants argument of tf.while_loop or set_shape() on the loop variables.

Note that calling

If i use a concatenate operation (instead of the add operation that gets returned by my while loop), I do not get any issues.

However, If I do not set N as a global variable (i.e. I do N = t_x.shape[0]) inside the body of the tf_test_log function, I get a Value error.

ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 1)

What is wrong with my code? Any help is greatly appreciated!

I am using python 3.5 on ubuntu 16.04 and tensorflow 1.4

full output:

ValueError                                Traceback (most recent call last)
~/Documents/TheEffingPhDHatersGonnaHate/PAM/defun_while.py in <module>()
     51 t_x   = tf.constant(np.random.randn(N,N),dtype = dType)
     52 t_y   = tf.constant(np.random.randn(N,N),dtype = dType)
---> 53 ys = _run(tf_test_log(t_x,t_y))
     54
     55

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/function.py in __call__(self, *args, **kwargs)
    503
    504   def __call__(self, *args, **kwargs):
--> 505     self.add_to_graph(ops.get_default_graph())
    506     args = [ops.convert_to_tensor(_) for _ in args] + self._extra_inputs
    507     ret, op = _call(self._signature, *args, **kwargs)

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/function.py in add_to_graph(self, g)
    484   def add_to_graph(self, g):
    485     """Adds this function into the graph g."""
--> 486     self._create_definition_if_needed()
    487
    488     # Adds this function into 'g'.

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/function.py in _create_definition_if_needed(self)
    319     """Creates the function definition if it's not created yet."""
    320     with context.graph_mode():
--> 321       self._create_definition_if_needed_impl()
    322
    323   def _create_definition_if_needed_impl(self):

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/function.py in _create_definition_if_needed_impl(self)
    336       # Call func and gather the output tensors.
    337       with vs.variable_scope("", custom_getter=temp_graph.getvar):
--> 338         outputs = self._func(*inputs)
    339
    340       # There is no way of distinguishing between a function not returning

~/Documents/TheEffingPhDHatersGonnaHate/PAM/defun_while.py in tf_test_log(t_x, t_y)
     39     L,t_log_x = tf.while_loop(condition,body,ijk_0,
     40                          shape_invariants=[i0.get_shape(),
---> 41                                            tf.TensorShape([N,None])]
     42                                             )
     43     return t_log_x

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py in while_loop(cond, body, loop_vars, shape_invariants, parallel_iterations, back_prop, swap_memory, name)
   2814     loop_context = WhileContext(parallel_iterations, back_prop, swap_memory)  # pylint: disable=redefined-outer-name
   2815     ops.add_to_collection(ops.GraphKeys.WHILE_CONTEXT, loop_context)
-> 2816     result = loop_context.BuildLoop(cond, body, loop_vars, shape_invariants)
   2817     return result
   2818

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py in BuildLoop(self, pred, body, loop_vars, shape_invariants)
   2638       self.Enter()
   2639       original_body_result, exit_vars = self._BuildLoop(
-> 2640           pred, body, original_loop_vars, loop_vars, shape_invariants)
   2641     finally:
   2642       self.Exit()

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py in _BuildLoop(self, pred, body, original_loop_vars, loop_vars, shape_invariants)
   2619     for m_var, n_var in zip(merge_vars, next_vars):
   2620       if isinstance(m_var, ops.Tensor):
-> 2621         _EnforceShapeInvariant(m_var, n_var)
   2622
   2623     # Exit the loop.

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py in _EnforceShapeInvariant(merge_var, next_var)
    576           "Provide shape invariants using either the `shape_invariants` "
    577           "argument of tf.while_loop or set_shape() on the loop variables."
--> 578           % (merge_var.name, m_shape, n_shape))
    579   else:
    580     if not isinstance(var, (ops.IndexedSlices, sparse_tensor.SparseTensor)):

ValueError: The shape for while/Merge_1:0 is not an invariant for the loop. It enters the loop with shape (100, ?), but has shape <unknown> after one iteration. Provide shape invariants using either the `shape_invariants` argument of tf.while_loop or set_shape() on the loop variables.

Body:
Thanks @Alexandre Passos for the suggestion in the comment above!
The following piece of code is a modification of the original with a set_shape function added inside the body.

import numpy as np
import tensorflow as tf
from tensorflow.python.framework import ops
from tensorflow.python.ops import array_ops
from tensorflow.python.ops import sparse_ops
from tensorflow.python.framework import function

def _run(tensor):
    with tf.Session() as sess:
         sess.run(tf.global_variables_initializer())
         res = sess.run(tensor)
    return res

@function.Defun(tf.float32,tf.float32,tf.float32,func_name ='tf_test_logGrad')
def tf_test_logGrad(t_x,t_y,grad):
     return grad

@function.Defun(tf.float32,tf.float32,func_name     ='tf_test_log')#,grad_func=tf_test_logGrad)
def tf_test_log(t_x,t_y):
    #N = t_x.shape[0].value
    condition = lambda i,m1: i<N

    def body(index,x):
        #return[(index+1),tf.concat([x, tf.expand_dims(tf.exp( tf.add(     t_x[:,index],t_y[:,index]) ),1)   ],1 ) ]
        x = tf.add(x, tf.exp( tf.add( t_x[:,0],t_y[:,0]) )    )
        x.set_shape([N])
        return[(index+1), x]

    i0      = tf.constant(0,dtype=tf.int32)
    m0      = tf.zeros([N],dType)

    ijk_0   = [i0,m0]

    L,t_log_x = tf.while_loop(condition,body,ijk_0,
                     shape_invariants=[i0.get_shape(),
                                       tf.TensorShape([N])]
                                        )
    return t_log_x

dType = tf.float32
N     = np.int32(100)

t_N   = tf.constant(N,dtype = tf.int32)
t_x   = tf.constant(np.random.randn(N,N),dtype = dType)
t_y   = tf.constant(np.random.randn(N,N),dtype = dType)
ys = _run(tf_test_log(t_x,t_y))

The Issue of global N still persists.

You still need to set the shape of the loop tensors as a global variable outside of the defun decorator. If you try to get it from the shape of the inputs of the defun decorator, you get:

TypeError                                 Traceback (most recent call last)
~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py in zeros(shape, dtype, name)
   1438       shape = tensor_shape.as_shape(shape)
-> 1439       output = constant(zero, shape=shape, dtype=dtype, name=name)
   1440     except (TypeError, ValueError):

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/constant_op.py in constant(value, dtype, shape, name, verify_shape)
    207       tensor_util.make_tensor_proto(
--> 208           value, dtype=dtype, shape=shape, verify_shape=verify_shape))
    209   dtype_value = attr_value_pb2.AttrValue(type=tensor_value.tensor.dtype)

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/tensor_util.py in make_tensor_proto(values, dtype, shape, verify_shape)
    379     # exception when dtype is set to np.int64
--> 380     if shape is not None and np.prod(shape, dtype=np.int64) == 0:
    381       nparray = np.empty(shape, dtype=np_dt)

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/numpy/core/fromnumeric.py in prod(a, axis, dtype, out, keepdims)
   2517     return _methods._prod(a, axis=axis, dtype=dtype,
-> 2518                           out=out, **kwargs)
   2519

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/numpy/core/_methods.py in _prod(a, axis, dtype, out, keepdims)
     34 def _prod(a, axis=None, dtype=None, out=None, keepdims=False):
---> 35     return umr_prod(a, axis, dtype, out, keepdims)
     36

TypeError: __int__ returned non-int (type NoneType)

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
~/Documents/TheEffingPhDHatersGonnaHate/PAM/defun_while.py in <module>()
     52 t_x   = tf.constant(np.random.randn(N,N),dtype = dType)
     53 t_y   = tf.constant(np.random.randn(N,N),dtype = dType)
---> 54 ys = _run(tf_test_log(t_x,t_y))
     55
     56

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/function.py in __call__(self, *args, **kwargs)
    503
    504   def __call__(self, *args, **kwargs):
--> 505     self.add_to_graph(ops.get_default_graph())
    506     args = [ops.convert_to_tensor(_) for _ in args] + self._extra_inputs
    507     ret, op = _call(self._signature, *args, **kwargs)

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/function.py in add_to_graph(self, g)
    484   def add_to_graph(self, g):
    485     """Adds this function into the graph g."""
--> 486     self._create_definition_if_needed()
    487
    488     # Adds this function into 'g'.

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/function.py in _create_definition_if_needed(self)
    319     """Creates the function definition if it's not created yet."""
    320     with context.graph_mode():
--> 321       self._create_definition_if_needed_impl()
    322
    323   def _create_definition_if_needed_impl(self):

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/function.py in _create_definition_if_needed_impl(self)
    336       # Call func and gather the output tensors.
    337       with vs.variable_scope("", custom_getter=temp_graph.getvar):
--> 338         outputs = self._func(*inputs)
    339
    340       # There is no way of distinguishing between a function not returning

~/Documents/TheEffingPhDHatersGonnaHate/PAM/defun_while.py in tf_test_log(t_x, t_y)
     33
     34     i0      = tf.constant(0,dtype=tf.int32)
---> 35     m0      = tf.zeros([N],dType)
     36
     37

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py in zeros(shape, dtype, name)
   1439       output = constant(zero, shape=shape, dtype=dtype, name=name)
   1440     except (TypeError, ValueError):
-> 1441       shape = ops.convert_to_tensor(shape, dtype=dtypes.int32, name="shape")
   1442       output = fill(shape, constant(zero, dtype=dtype), name=name)
   1443   assert output.dtype.base_dtype == dtype

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/ops.py in convert_to_tensor(value, dtype, name, preferred_dtype)
    834       name=name,
    835       preferred_dtype=preferred_dtype,
--> 836       as_ref=False)
    837
    838

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/ops.py in internal_convert_to_tensor(value, dtype, name, as_ref, preferred_dtype, ctx)
    924
    925     if ret is None:
--> 926       ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
    927
    928     if ret is NotImplemented:

~/environments/tf_1_4_gpu/lib/python3.5/site-packages/tensorflow/python/framework/constant_op.py in _tensor_shape_tensor_conversion_function(s, dtype, name, as_ref)
    248   if not s.is_fully_defined():
    249     raise ValueError(
--> 250         "Cannot convert a partially known TensorShape to a Tensor: %s" % s)
    251   s_list = s.as_list()
    252   int64_value = 0

ValueError: Cannot convert a partially known TensorShape to a Tensor: (?,)


答:
By using np.where

```<code>df['res']=np.where(df.tag=='x',-df.val,df.val)
df
Out[271]:
  tag  val  res
0   x    1   -1
1   x    2   -2
2   z    3    3
3   z    1    1
4   x    2   -2
5   z    3    3
</code>```

Title: Pandas create column conditional on others

Body:
By using np.where

df['res']=np.where(df.tag=='x',-df.val,df.val)
df
Out[271]:
  tag  val  res
0   x    1   -1
1   x    2   -2
2   z    3    3
3   z    1    1
4   x    2   -2
5   z    3    3


问:
React deleteing from state array
I'm working on my first to do app with React. Trying to create delete function that allows a user to click the task and have it removed from the array and UI.

I've been trying a couple of different things following other's tutorials, but no luck. (Check commented out sections)
My console.log within the deleteItem function is showing undefined, so I know that it's not receiving the right data. I'm lost as to why.

App.js:

```<code>import React, { Component } from 'react';
import { List } from './list';

export class Todo extends Component {
    constructor(props) {
        super(props);
        this.state = {
            term: '',
            items: []
        };
        this.deleteItem = this.deleteItem.bind(this);
    }
    onChange = (e) => {
        this.setState({term: e.target.value});
    }
    onSubmit = (e) => {
        e.preventDefault()
        if (this.state.term === "") {
            alert("You must enter something!");

        } else {
            this.setState({
                term: '',
                items: [...this.state.items, this.state.term]
            })
        }
    }

    deleteItem(index){
        // var filteredItems = this.state.items.filter(function (item) {
        // return(item.index !== index);
        // });
        // console.log(filteredItems);
        // this.setState({
        // itmes: filteredItems
        // })

        this.setState({items: this.state.items.filter(function(item){
            console.log(item.index)
            return item !== item.index
        })})
        console.log(this.state.items)
        // if (item.index === index) {
        //     var indArr = this.state.items.indexOf(item.index);
        //     this.state.items.splice(indArr, 1);
        // }

    }

    render() {
        return(
            <div>
                <h3>You currently have {this.state.items.length} things to do.</h3>
                <form className="Todo" onSubmit={this.onSubmit}>
                    <input placeholder="Enter task" value={this.state.term} onChange={this.onChange} />
                    <button>add</button>
                </form>
                <List items={this.state.items} delete={this.deleteItem}/>
            </div>
        )
    }
}
</code>```

List.js

```<code>import React, { Component } from 'react';

export class List extends Component {
    constructor(props) {
        super(props);
        this.delete = this.delete.bind(this);
    }
    delete(key) {
        this.props.delete(key);
    }
    render() {
        return(
            <ul>
                {
                    this.props.items.map((item, key) => <div class="listItems"
                        onClick={(e) => this.delete(item.key)}
                        key={key}>{item}</div>)
                }
            </ul>
        )
    }
}
</code>```

Any help would be deeply appreciated.

答: I believe you're binding the delete function to the wrong component. Try removing this.delete = this.delete.bind(this); in the List.js class and just calling this.props.delete(key) directly from the onClick function

Title: React deleteing from state array

Body:
I'm working on my first to do app with React. Trying to create delete function that allows a user to click the task and have it removed from the array and UI.

I've been trying a couple of different things following other's tutorials, but no luck. (Check commented out sections)
My console.log within the deleteItem function is showing undefined, so I know that it's not receiving the right data. I'm lost as to why.

App.js:

import React, { Component } from 'react';
import { List } from './list';

export class Todo extends Component {
    constructor(props) {
        super(props);
        this.state = {
            term: '',
            items: []
        };
        this.deleteItem = this.deleteItem.bind(this);
    }
    onChange = (e) => {
        this.setState({term: e.target.value});
    }
    onSubmit = (e) => {
        e.preventDefault()
        if (this.state.term === "") {
            alert("You must enter something!");

        } else {
            this.setState({
                term: '',
                items: [...this.state.items, this.state.term]
            })
        }
    }

    deleteItem(index){
        // var filteredItems = this.state.items.filter(function (item) {
        // return(item.index !== index);
        // });
        // console.log(filteredItems);
        // this.setState({
        // itmes: filteredItems
        // })

        this.setState({items: this.state.items.filter(function(item){
            console.log(item.index)
            return item !== item.index
        })})
        console.log(this.state.items)
        // if (item.index === index) {
        //     var indArr = this.state.items.indexOf(item.index);
        //     this.state.items.splice(indArr, 1);
        // }

    }

    render() {
        return(

You currently have {this.state.items.length} things to do.

                <form className="Todo" onSubmit={this.onSubmit}>
                    <input placeholder="Enter task" value={this.state.term} onChange={this.onChange} />
                    <button>add</button>
                </form>
                <List items={this.state.items} delete={this.deleteItem}/>

        )
    }
}

List.js

import React, { Component } from 'react';

export class List extends Component {
    constructor(props) {
        super(props);
        this.delete = this.delete.bind(this);
    }
    delete(key) {
        this.props.delete(key);
    }
    render() {
        return(

                {
                    this.props.items.map((item, key) =>
 this.delete(item.key)}
                        key={key}>{item}
)
                }

        )
    }
}

Any help would be deeply appreciated.


问:
Unable to run webapi solution in vs2017(.net core)
Hi I started working on visual studio 2017 and trying to create small webapi application using .net core. I followed https://learn.microsoft.com/en-us/aspnet/core/tutorials/first-web-api. My application name is RestPrototype. When i click on the arrow of run i can see below options. 1. RestPrototype 2. IISExpress. In visual studio  browser options were coming. When i click on the name of the application(RestProtocol) program begins to run. First one black screen appears next IE will open and close. I have attached below screen shot.

Can someone help me to run the above application? Any help would be greatly appreciated. Thank you

答:
After clicking on your project name RestProtocol under Run option your project gets started and it gets self hosted.

So you able to see whether the port number assigned to your project is active or not using Resource Monitor in your system.

If it is active use Postman or other rest client to give Web API request that you had created in your project. You can see your rest API will works.

I tried this it works for me.


问:
When I create a new website in visual studio 2012, I got this error
When I create a new website in visual studio 2012, I got this error...

No exports were found that match the constraint:

ContractName
Microsoft.VisualStudio.Text.ITextDocumentFactoryService
RequiredTypeIdentity
Microsoft.VisualStudio.Text.ITextDocumentFactoryService

Please tell me how to fix this error???

答: The fix is to rename or delete the ComponentModelCache in your local folder.


问:
select an element based off a select value
I am trying to take the value of a select element and match it to a dataset of another element to affect a transition on that element.

So I have two tags with data in them:

```<code><select data-name="name goes here" updateElement()></select>

<select value="what ever option is picked" applyTransition()></select>
</code>```

When ever you change the option of data-name it will update a variable that holds the element I am trying to apply the transition affect on.

The goal is when you change the select with the data-name it updates and element the other select applies a transition too.

I haven't shared any of the actual code because it is a lot and would like to just share what is needed.

答:
Okay so I figured this out after banging my head against the console.

```<code>const yourElements = document.querySelectorAll(".your-class-name");
let selectedElement;
</code>```

So if your elements have a data-property set you can "pluck" that one out.

```<code>for(let i = 0; i < yourElements.length; i++){
if(select.value === your-class-name[i].dataset.property){
selectedElement = document.querySelector(`[data-property=
${locationHands[i].dataset.property}]`);}
</code>```

Then you should be able to see selectedElement as the element you want.

You can then use that variable where ever.

If you know of a better or more efficient way to achieve this, please let me know.

Body:
I am trying to take the value of a select element and match it to a dataset of another element to affect a transition on that element.

So I have two tags with data in them:

<select data-name="name goes here" updateElement()></select>

<select value="what ever option is picked" applyTransition()></select>

When ever you change the option of data-name it will update a variable that holds the element I am trying to apply the transition affect on.

The goal is when you change the select with the data-name it updates and element the other select applies a transition too.

I haven't shared any of the actual code because it is a lot and would like to just share what is needed.

Body:
Okay so I figured this out after banging my head against the console.

const yourElements = document.querySelectorAll(".your-class-name");
let selectedElement;

So if your elements have a data-property set you can "pluck" that one out.

for(let i = 0; i < yourElements.length; i++){
if(select.value === your-class-name[i].dataset.property){
selectedElement = document.querySelector(`[data-property=
${locationHands[i].dataset.property}]`);}

Then you should be able to see selectedElement as the element you want.

You can then use that variable where ever.

If you know of a better or more efficient way to achieve this, please let me know.


问:
How to create an empty folder on Google Storage with Google API?
How to create an empty folder on Google Storage with Google API? (Assume that / is path separator.)

答:
@SheRey - looking at folders created via the GCS web interface, the Content-Type is set to application/x-www-form-urlencoded;charset=UTF-8 but it doesn't really matter. Here's what worked for me in python:

```<code># pip install google-cloud-storage

from google.cloud import storage

gcs_client = storage.Client(project='some_project')
bucket = gcs_client.get_bucket('some_bucket')
blob = bucket.blob('some/folder/name/')

blob.upload_from_string('', content_type='application/x-www-form-urlencoded;charset=UTF-8')
</code>```

Body:
@SheRey - looking at folders created via the GCS web interface, the Content-Type is set to application/x-www-form-urlencoded;charset=UTF-8 but it doesn't really matter. Here's what worked for me in python:

# pip install google-cloud-storage

from google.cloud import storage

gcs_client = storage.Client(project='some_project')
bucket = gcs_client.get_bucket('some_bucket')
blob = bucket.blob('some/folder/name/')

blob.upload_from_string('', content_type='application/x-www-form-urlencoded;charset=UTF-8')


问:
How to fix this query? Using GROUP_CONCAT
I have a query that would get all product names and concatenate the names to 1 column. But I can't figure out how to make it work. Here is the query.

```<code>SELECT package_id,
GROUP_CONCAT
(
    SELECT c.product_name FROM tbl_product c WHERE c.product_id IN
    (
        SELECT a.product_id FROM tbl_package_detail a WHERE a.package_id = b.package_id
    )
    ORDER BY c.product_id
)
FROM tbl_package b
GROUP BY package_id
</code>```

Note that:

```<code>SELECT c.product_name FROM tbl_product c WHERE c.product_id IN
(
    SELECT a.product_id FROM tbl_package_detail a WHERE a.package_id = b.package_id
)
ORDER BY c.product_id
</code>```

Returns all the product names I need. I am not sure if I am using GROUP_CONCAT correctly.

ERROR MESSAGE:

You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'SELECT c.product_name FROM tbl_product c WHERE c.product_id IN '

答:
Try the following

```<code>SELECT
  b.package_id,
  GROUP_CONCAT(c.product_name ORDER BY c.product_id) ProductList
FROM tbl_package b
LEFT JOIN tbl_package_detail a ON a.package_id = b.package_id
LEFT JOIN tbl_product c ON c.product_id=a.product_id
GROUP BY b.package_id
</code>```

GROUP_CONCAT is an aggregate function like SUM/AVG/MIN/MAX/COUNT.

Body:
I have a query that would get all product names and concatenate the names to 1 column. But I can't figure out how to make it work. Here is the query.

SELECT package_id,
GROUP_CONCAT
(
    SELECT c.product_name FROM tbl_product c WHERE c.product_id IN
    (
        SELECT a.product_id FROM tbl_package_detail a WHERE a.package_id = b.package_id
    )
    ORDER BY c.product_id
)
FROM tbl_package b
GROUP BY package_id

Note that:

SELECT c.product_name FROM tbl_product c WHERE c.product_id IN
(
    SELECT a.product_id FROM tbl_package_detail a WHERE a.package_id = b.package_id
)
ORDER BY c.product_id

Returns all the product names I need. I am not sure if I am using GROUP_CONCAT correctly.

ERROR MESSAGE:

You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'SELECT c.product_name FROM tbl_product c WHERE c.product_id IN '

Body:
Try the following

SELECT
  b.package_id,
  GROUP_CONCAT(c.product_name ORDER BY c.product_id) ProductList
FROM tbl_package b
LEFT JOIN tbl_package_detail a ON a.package_id = b.package_id
LEFT JOIN tbl_product c ON c.product_id=a.product_id
GROUP BY b.package_id

GROUP_CONCAT is an aggregate function like SUM/AVG/MIN/MAX/COUNT.


答:
This'll do: </p>

```<code># ...
import numpy as np
df['res'] = np.where(df['tag'] == 'x', -df['val'], df['val'])
</code>```

Body:
I have a dataframe like

df = pd.DataFrame({'tag':['x','x','z','z','x','z'], 'val':[1,2,3,1,2,3]})

Is there a one-liner to add a column 'res' that shows -val if tag=='x' otherwise +val, so it would look like

  tag  val  res
0   x    1   -1
1   x    2   -2
2   z    3    3
3   z    1    1
4   x    2   -2
5   z    3    3

Body:
This'll do:

# ...
import numpy as np
df['res'] = np.where(df['tag'] == 'x', -df['val'], df['val'])


问:
Couldn't retrieve data from my database
I have a problem in my system. Last time i checked, my system was working fine but now i get this error and it seems like my system cannot retrieve data from my database. I am getting a Token mismatch exception even though i have csrf token in all of my forms. Somebody Help.</p>

答:
You have two ways to resolve this issue that are following:-

```<code>1) Add CSRF token in form i.e. like 2 other answer suggest you
   <input type="hidden" name="_token" value="{{ csrf_token() }}">
 if you still get the error of Csrf token then you have one more solution

2) Open app/http/Middleware/VerifyCsrfToken.php
   protected $except = [
    //
    '/login',
   ];
 Add your route here suppose you  have one login route you can define here and remove csrf_field() or
<input type="hidden" name="_token" value="{{ csrf_token() }}">
 from your login form
</code>```

Try to resolve with first method if not works go for 2nd one. Hope it helps!

Title: Couldn't retrieve data from my database

Body:
You have two ways to resolve this issue that are following:-

1) Add CSRF token in form i.e. like 2 other answer suggest you
   <input type="hidden" name="_token" value="{{ csrf_token() }}">
 if you still get the error of Csrf token then you have one more solution

2) Open app/http/Middleware/VerifyCsrfToken.php
   protected $except = [
    //
    '/login',
   ];
 Add your route here suppose you  have one login route you can define here and remove csrf_field() or
<input type="hidden" name="_token" value="{{ csrf_token() }}">
 from your login form

Try to resolve with first method if not works go for 2nd one. Hope it helps!


问:
Efficiently selecting rows where a row in a related table exists
I've got a recurring pattern in a system that I'm currently working on, where, for example, I need to select all the users that have orders under a list of possible companies. Or needing to select users if a record of this user being flagged exists.

My users table contains 430,825 records, so this shouldn't really be that difficult to deal with. Right now I'm close, I have a query that get's that .047s execution time that I look for, but if I add one more piece to this, it gets real slow.

Here's my current query, the fast one:

```<code>select`UserID`
from`users`
where(`CompanyID`in('3e55d1bb-d8b6-11e4-b38f-b8ca3a83b4c8')
    or`UserID`in(select*
        from(select`UserID`
            from`invoices`
            where`CompanyID`in('3e55d1bb-d8b6-11e4-b38f-b8ca3a83b4c8')
            and`__Active`=1)`a`)
    or`UserID`in(select*
        from(select`UserID`
            from`quoterequests`
            where`CompanyID`in('3e55d1bb-d8b6-11e4-b38f-b8ca3a83b4c8')
            and`__Active`=1)`a`))
and(`UserID`in(select*
        from(select`UserID`
            from`userassociations`
            where`_Email`='brian@yeet.com'
            and`__Active`=1)`a`))
and(`UserID`in(select*
        from(select`UserID`
            from`usercustomerflags`
            where`CustomerFlagID`in(10,27,17,1,2,3,4,5,6)
            and`__Active`=1)`a`)
    or not exists(select 1
        from`usercustomerflags`
        where`__Active`=1
        and`users`.`UserID`=`UserID`))
and`Deleted`=0
order by`DateTimeAdded`desc
limit 50;
</code>```

(The extra select*from(...) is because of this https://stackoverflow.com/a/1434712/728236)

In the middle there, I'm pulling users by further by email address, while checking other related tables for emails that might be related to this user. Like, the next piece searches users including their CC addresses when quotes are sent out to customers.

```<code>select`UserID`
from`users`
where(`CompanyID`in('3e55d1bb-d8b6-11e4-b38f-b8ca3a83b4c8')
    or`UserID`in(select*
        from(select`UserID`
            from`invoices`
            where`CompanyID`in('3e55d1bb-d8b6-11e4-b38f-b8ca3a83b4c8')
            and`__Active`=1)`a`)
    or`UserID`in(select*
        from(select`UserID`
            from`quoterequests`
            where`CompanyID`in('3e55d1bb-d8b6-11e4-b38f-b8ca3a83b4c8')
            and`__Active`=1)`a`))
and(`UserID`in(select*
        from(select`UserID`
            from`userassociations`
            where`_Email`='brian@yeet.com'
            and`__Active`=1)`a`)
    or`UserID`in(select*
        from(select`UserID`
            from`userquotesemails`
            where`Email`='brian@yeet.com'
            and`__Active`=1)`a`))
and(`UserID`in(select*
        from(select`UserID`
            from`usercustomerflags`
            where`CustomerFlagID`in(10,27,17,1,2,3,4,5,6)
            and`__Active`=1)`a`)
    or not exists(select 1
        from`usercustomerflags`
        where`__Active`=1
        and`users`.`UserID`=`UserID`))
and`Deleted`=0
order by`DateTimeAdded`desc
limit 50;
</code>```

I've added the alternate table to search for emails, but now the query takes 3.016s, which is way, way slower. It seems odd that, as I was building this query, that last part seemed to be the tipping point in performance here, what would be the cause of this?

The first and second explains, respectively

```<code>+----+--------------------+-------------------+--+----------------+---------------------------------------------------------------------------------------------+------------------------------+------+-----------------------+---+-------+---------------------------------+
|  1 | PRIMARY            | <subquery6>       |  | ALL            |                                                                                |                              |      |                       |   | 0.00  | Using temporary; Using filesort |
|  1 | PRIMARY            | users             |  | eq_ref         | PRIMARY,UserID_UNIQUE,fk_users_1_idx,users_Customers                                        | PRIMARY                      |  144 | <subquery6>.UserID    | 1 | 50.00 | Using where                     |
|  6 | MATERIALIZED       | userassociations  |  | ref            | userassociations_UserID,userassociations__Email                                             | userassociations__Email      | 1026 | const                 | 3 | 10.00 | Using where                     |
| 10 | DEPENDENT SUBQUERY | usercustomerflags |  | ref            | usercustomerflags_UserID_idx                                                                | usercustomerflags_UserID_idx |  144 | sterling.users.UserID | 1 | 10.00 | Using where                     |
|  8 | DEPENDENT SUBQUERY | usercustomerflags |  | index_subquery | usercustomerflags_CustomerFlagID_idx,usercustomerflags_UserID_idx                           | usercustomerflags_UserID_idx |  144 | func                  | 1 | 4.95  | Using where                     |
|  4 | DEPENDENT SUBQUERY | quoterequests     |  | index_subquery | quoterequests_CompanyID,quoterequests_UserID,quoterequests__Latest,quoterequests_UserQuotes | quoterequests__Latest        |  145 | func                  | 2 | 5.00  | Using where                     |
|  2 | DEPENDENT SUBQUERY | invoices          |  | index_subquery | Invoice_UserID_idx,Invoice_CompanyID_idx,invoices_SampleRequests,invoices_LateOrdersBubble  | Invoice_UserID_idx           |  145 | func                  | 1 | 3.33  | Using where                     |
+----+--------------------+-------------------+--+----------------+---------------------------------------------------------------------------------------------+------------------------------+------+-----------------------+---+-------+---------------------------------+

+----+--------------------+-------------------+--+-----+---------------------------------------------------------------------------------------------+--------------------------------+------+-----------------------+--------+--------+-------------+
|  1 | PRIMARY            | users             |  | ref | fk_users_1_idx,users_Customers                                                              | users_Customers                |    4 | const                 | 227515 | 100.00 | Using where |
| 12 | DEPENDENT SUBQUERY | usercustomerflags |  | ref | usercustomerflags_UserID_idx                                                                | usercustomerflags_UserID_idx   |  144 | sterling.users.UserID |      1 | 10.00  | Using where |
| 10 | SUBQUERY           | usercustomerflags |  | ALL | usercustomerflags_CustomerFlagID_idx,usercustomerflags_UserID_idx                           |                                |      |                       |   3509 | 4.94   | Using where |
|  8 | SUBQUERY           | userquotesemails  |  | ref | userquotesemails_Email__Active,userquotesemails_UserID                                      | userquotesemails_Email__Active | 1027 | const,const           |      1 | 100.00 |             |
|  6 | SUBQUERY           | userassociations  |  | ref | userassociations_UserID,userassociations__Email                                             | userassociations__Email        | 1026 | const                 |      3 | 10.00  | Using where |
|  4 | SUBQUERY           | quoterequests     |  | ref | quoterequests_CompanyID,quoterequests_UserID,quoterequests__Latest,quoterequests_UserQuotes | quoterequests_CompanyID        |  144 | const                 |  16702 | 10.00  | Using where |
|  2 | SUBQUERY           | invoices          |  | ref | Invoice_UserID_idx,Invoice_CompanyID_idx,invoices_SampleRequests,invoices_LateOrdersBubble  | Invoice_CompanyID_idx          |  144 | const                 |  17678 | 10.00  | Using where |
+----+--------------------+-------------------+--+-----+---------------------------------------------------------------------------------------------+--------------------------------+------+-----------------------+--------+--------+-------------+
</code>```

Also, I've tried using joins, e.g. joining the invoices table, etc. but then I get the issue of having duplicate user rows per each invoice or quoterequest the joins receive, and grouping/distinct & ordering the resulting data became immensely slow, in the minutes.

I've also tried just the "exists" version of the first query, as suggesting by the doc https://dev.mysql.com/doc/refman/5.7/en/subquery-optimization-with-exists.html like so

```<code>select`UserID`
from`users`
where(`CompanyID`in('3e55d1bb-d8b6-11e4-b38f-b8ca3a83b4c8')
    or exists(select 1
        from`invoices`
        where`CompanyID`in('3e55d1bb-d8b6-11e4-b38f-b8ca3a83b4c8')
        and`__Active`=1
        and`users`.`UserID`=`UserID`)
    or exists(select 1
        from`quoterequests`
        where`CompanyID`in('3e55d1bb-d8b6-11e4-b38f-b8ca3a83b4c8')
        and`__Active`=1
        and`users`.`UserID`=`UserID`))
and(exists(select 1
        from`userassociations`
        where`_Email`='brian@yeet.com'
        and`__Active`=1
        and`users`.`UserID`=`UserID`))
and(exists(select 1
        from`usercustomerflags`
        where`CustomerFlagID`in(10,27,17,1,2,3,4,5,6)
        and`__Active`=1
        and`users`.`UserID`=`UserID`)
    or not exists(select 1
        from`usercustomerflags`
        where`__Active`=1
        and`users`.`UserID`=`UserID`))
and`Deleted`=0
order by`DateTimeAdded`desc
limit 50;
</code>```

But that gets me up to 5.516s, so that definitely doesn't seem like the right direction to make.

What is the most efficient way to select data the way that I'm trying? Or do I need to restructure some of my tables to get the performance I'm looking for?

I've isolated the smallest sub-problem and bottle neck I think I have. Here's my lighter query

```<code>select`users`.`UserID`,`users`.`_Customer`
from`users`
left join`userassociations`on`userassociations`.`UserID`=`users`.`UserID`
and`userassociations`.`__Active`=1
where(`users`.`Email`='brian@stumpyinc.com'
    or`userassociations`.`_Email`='brian@stumpyinc.com')
and`users`.`Deleted`=0
order by`users`.`DateTimeAdded`desc
limit 50;
</code>```

And the explain

```<code>+---+--------+------------------+--+-----+--------------------------------------------------------+-------------------------+-----+-----------------------+--------+--------+-------------+
| 1 | SIMPLE | users            |  | ref | users_getemail_INDEX,unify_email_INDEX,users_Customers | users_Customers         |   4 | const                 | 221463 | 100.00 | Using where |
| 1 | SIMPLE | userassociations |  | ref | userassociations_UserID                                | userassociations_UserID | 144 | sterling.users.UserID |      1 | 100.00 | Using where |
+---+--------+------------------+--+-----+--------------------------------------------------------+-------------------------+-----+-----------------------+--------+--------+-------------+
</code>```

This query takes about 1.5 seconds to execute

```<code>CREATE TABLE `users` (
  `UserID` char(36) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '',
    ...
  `Email` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
    ...
  `DateTimeAdded` datetime DEFAULT NULL,
    ...
  `Deleted` int(1) NOT NULL DEFAULT '0',
    ...
  `_LatestInvoiceDateTimeAdded` datetime DEFAULT NULL,
  `_InvoiceCount` int(11) NOT NULL DEFAULT '0',
  `_Customer` varchar(512) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
    ...
  PRIMARY KEY (`UserID`),
  UNIQUE KEY `UserID_UNIQUE` (`UserID`),
    ...
  KEY `users_getemail_INDEX` (`Email`(191),`_InvoiceCount`,`_LatestInvoiceDateTimeAdded`,`DateTimeAdded`),
  KEY `unify_email_INDEX` (`Email`(191),`UserID`),
    ...
  KEY `users_Customers` (`Deleted`,`DateTimeAdded`),
    ...
  KEY `users_DateTimeAdded` (`DateTimeAdded`,`UserID`),
  FULLTEXT KEY `users_FULLTEXT__Customer` (`_Customer`),
    ...
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

CREATE TABLE `userassociations` (
   `UserAssociationID` binary(16) NOT NULL,
   `UserID` char(36) COLLATE utf8mb4_unicode_ci NOT NULL,
   `AssociatedUserID` char(36) COLLATE utf8mb4_unicode_ci NOT NULL,
   `_Email` varchar(256) COLLATE utf8mb4_unicode_ci NOT NULL,
   `__UserID` char(36) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
   `__Active` tinyint(1) NOT NULL DEFAULT '1',
   `__Added` timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
   `__Updated` timestamp(6) NULL DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP(6),
   PRIMARY KEY (`UserAssociationID`),
   KEY `userassociations_UserID` (`UserID`),
   KEY `userassociations_AssociatedUserID` (`AssociatedUserID`),
   KEY `userassociations___UserID` (`__UserID`),
   KEY `userassociations__Email` (`_Email`),
   CONSTRAINT `userassociations_AssociatedUserID` FOREIGN KEY (`AssociatedUserID`) REFERENCES `users` (`UserID`) ON DELETE NO ACTION ON UPDATE NO ACTION,
   CONSTRAINT `userassociations_UserID` FOREIGN KEY (`UserID`) REFERENCES `users` (`UserID`) ON DELETE NO ACTION ON UPDATE NO ACTION,
   CONSTRAINT `userassociations___UserID` FOREIGN KEY (`__UserID`) REFERENCES `users` (`UserID`) ON DELETE NO ACTION ON UPDATE NO ACTION
 ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
</code>```

Hmm... So it seemed like it was working, but I've found a pair of tables that it doesn't seem to be as efficient with, which is my users and invoices tables.

I've have these indexes:

```<code>users:    INDEX(`CompanyID`, `Deleted`, `DateTimeAdded`)
invoices: INDEX(`UserID`, `__Active`)
invoices: INDEX(`CompanyID`)
users:    INDEX(`UserID`, `Deleted`)
</code>```

and the query

```<code>select`users`.`UserID`,`users`.`DateTimeAdded`
from`users`
join`invoices`on`invoices`.`UserID`=`users`.`UserID`
and`invoices`.`__Active`=1
where`invoices`.`CompanyID`='3e55c8b4-d8b6-11e4-b38f-b8ca3a83b4c8'
and`users`.`Deleted`=0
order by`DateTimeAdded`desc
limit 200;
</code>```

This query alone takes .3 seconds, which feels slow to me, like it isn't making the best use of the indexes, especially because users only has 430,997 rows and invoices only has 194,180, and this looks like it should be a pretty simple query.

EDIT: Actually it's much worse than that, if the CompanyID given only includes ~4 rows, then this query takes 3.5 seconds

```<code>+---+--------+----------+--+-----+------------------------------------------------+-----------------------+-----+--------------------------------+------+--------+----------------------------------------------+
| 1 | SIMPLE | invoices |  | ref | Invoice_CompanyID_idx,invoices_UserID___Active | Invoice_CompanyID_idx | 144 | const                          | 7750 | 10.00  | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | users    |  | ref | users_UserID_Deleted                           | users_UserID_Deleted  | 148 | sterling.invoices.UserID,const |    1 | 100.00 |                                              |
+---+--------+----------+--+-----+------------------------------------------------+-----------------------+-----+--------------------------------+------+--------+----------------------------------------------+
</code>```

答:
For that smaller problem:

```<code>( select u.`UserID`, u.`_Customer`, u.DateTimeAdded
    from  `users` AS u
    where  u.`Email` = 'brian@stumpyinc.com'
      and  u.`Deleted` = 0
      AND EXISTS ( SELECT * FROM `userassociations`
                       WHERE UserId = u.UserID
                         AND __Active = 1 )
    order by  u.`DateTimeAdded` desc
    limit  50
)
UNION DISTINCT
( select u.`UserID`, u.`_Customer`, u.DateTimeAdded
    from  `users` AS u
    JOIN  `userassociations` AS ua
         ON  ua.`UserID` = u.`UserID`
        and  ua.`__Active` = 1
    where  ua.`_Email` = 'brian@stumpyinc.com'
      and  u.`Deleted`=0
    order by  u.`DateTimeAdded` desc
    limit  50
)
order by `DateTimeAdded` desc
limit  50
</code>```

These will be needed:

```<code>u:  INDEX(Email, Deleted, DateTimeAdded)  -- date last
ua: INDEX(UserId, __Active)   -- either order
ua: INDEX(_Email)
u:  INDEX(UserID, Deleted)
</code>```

(Let me know if you get syntax errors.  If it is too slow, please provide EXPLAIN.)

Index prefixing (Email(191)) is usually useless.  Get rid if it.  Here are 5 ways to avoid it:  http://mysql.rjweb.org/doc.php/limits#767_limit_in_innodb_indexes

A PK is a UNIQUE key, so get rid of the second of:

```<code>PRIMARY KEY (`UserID`),
UNIQUE KEY `UserID_UNIQUE` (`UserID`),
</code>```

Smells like a UUID; use ascii (ascii_general_ci), not utf8mb4:

```<code>... char(36) COLLATE utf8mb4_unicode_ci
</code>```

INT(1) takes 4 bytes; use TINYINT for flags.

Body:
I've got a recurring pattern in a system that I'm currently working on, where, for example, I need to select all the users that have orders under a list of possible companies. Or needing to select users if a record of this user being flagged exists.

My users table contains 430,825 records, so this shouldn't really be that difficult to deal with. Right now I'm close, I have a query that get's that .047s execution time that I look for, but if I add one more piece to this, it gets real slow.

Here's my current query, the fast one:

select`UserID`
from`users`
where(`CompanyID`in('3e55d1bb-d8b6-11e4-b38f-b8ca3a83b4c8')
    or`UserID`in(select*
        from(select`UserID`
            from`invoices`
            where`CompanyID`in('3e55d1bb-d8b6-11e4-b38f-b8ca3a83b4c8')
            and`__Active`=1)`a`)
    or`UserID`in(select*
        from(select`UserID`
            from`quoterequests`
            where`CompanyID`in('3e55d1bb-d8b6-11e4-b38f-b8ca3a83b4c8')
            and`__Active`=1)`a`))
and(`UserID`in(select*
        from(select`UserID`
            from`userassociations`
            where`_Email`='brian@yeet.com'
            and`__Active`=1)`a`))
and(`UserID`in(select*
        from(select`UserID`
            from`usercustomerflags`
            where`CustomerFlagID`in(10,27,17,1,2,3,4,5,6)
            and`__Active`=1)`a`)
    or not exists(select 1
        from`usercustomerflags`
        where`__Active`=1
        and`users`.`UserID`=`UserID`))
and`Deleted`=0
order by`DateTimeAdded`desc
limit 50;

(The extra select*from(...) is because of this https://stackoverflow.com/a/1434712/728236)

In the middle there, I'm pulling users by further by email address, while checking other related tables for emails that might be related to this user. Like, the next piece searches users including their CC addresses when quotes are sent out to customers.

select`UserID`
from`users`
where(`CompanyID`in('3e55d1bb-d8b6-11e4-b38f-b8ca3a83b4c8')
    or`UserID`in(select*
        from(select`UserID`
            from`invoices`
            where`CompanyID`in('3e55d1bb-d8b6-11e4-b38f-b8ca3a83b4c8')
            and`__Active`=1)`a`)
    or`UserID`in(select*
        from(select`UserID`
            from`quoterequests`
            where`CompanyID`in('3e55d1bb-d8b6-11e4-b38f-b8ca3a83b4c8')
            and`__Active`=1)`a`))
and(`UserID`in(select*
        from(select`UserID`
            from`userassociations`
            where`_Email`='brian@yeet.com'
            and`__Active`=1)`a`)
    or`UserID`in(select*
        from(select`UserID`
            from`userquotesemails`
            where`Email`='brian@yeet.com'
            and`__Active`=1)`a`))
and(`UserID`in(select*
        from(select`UserID`
            from`usercustomerflags`
            where`CustomerFlagID`in(10,27,17,1,2,3,4,5,6)
            and`__Active`=1)`a`)
    or not exists(select 1
        from`usercustomerflags`
        where`__Active`=1
        and`users`.`UserID`=`UserID`))
and`Deleted`=0
order by`DateTimeAdded`desc
limit 50;

I've added the alternate table to search for emails, but now the query takes 3.016s, which is way, way slower. It seems odd that, as I was building this query, that last part seemed to be the tipping point in performance here, what would be the cause of this?

The first and second explains, respectively

+----+--------------------+-------------------+--+----------------+---------------------------------------------------------------------------------------------+------------------------------+------+-----------------------+---+-------+---------------------------------+
|  1 | PRIMARY            | <subquery6>       |  | ALL            |                                                                                |                              |      |                       |   | 0.00  | Using temporary; Using filesort |
|  1 | PRIMARY            | users             |  | eq_ref         | PRIMARY,UserID_UNIQUE,fk_users_1_idx,users_Customers                                        | PRIMARY                      |  144 | <subquery6>.UserID    | 1 | 50.00 | Using where                     |
|  6 | MATERIALIZED       | userassociations  |  | ref            | userassociations_UserID,userassociations__Email                                             | userassociations__Email      | 1026 | const                 | 3 | 10.00 | Using where                     |
| 10 | DEPENDENT SUBQUERY | usercustomerflags |  | ref            | usercustomerflags_UserID_idx                                                                | usercustomerflags_UserID_idx |  144 | sterling.users.UserID | 1 | 10.00 | Using where                     |
|  8 | DEPENDENT SUBQUERY | usercustomerflags |  | index_subquery | usercustomerflags_CustomerFlagID_idx,usercustomerflags_UserID_idx                           | usercustomerflags_UserID_idx |  144 | func                  | 1 | 4.95  | Using where                     |
|  4 | DEPENDENT SUBQUERY | quoterequests     |  | index_subquery | quoterequests_CompanyID,quoterequests_UserID,quoterequests__Latest,quoterequests_UserQuotes | quoterequests__Latest        |  145 | func                  | 2 | 5.00  | Using where                     |
|  2 | DEPENDENT SUBQUERY | invoices          |  | index_subquery | Invoice_UserID_idx,Invoice_CompanyID_idx,invoices_SampleRequests,invoices_LateOrdersBubble  | Invoice_UserID_idx           |  145 | func                  | 1 | 3.33  | Using where                     |
+----+--------------------+-------------------+--+----------------+---------------------------------------------------------------------------------------------+------------------------------+------+-----------------------+---+-------+---------------------------------+

+----+--------------------+-------------------+--+-----+---------------------------------------------------------------------------------------------+--------------------------------+------+-----------------------+--------+--------+-------------+
|  1 | PRIMARY            | users             |  | ref | fk_users_1_idx,users_Customers                                                              | users_Customers                |    4 | const                 | 227515 | 100.00 | Using where |
| 12 | DEPENDENT SUBQUERY | usercustomerflags |  | ref | usercustomerflags_UserID_idx                                                                | usercustomerflags_UserID_idx   |  144 | sterling.users.UserID |      1 | 10.00  | Using where |
| 10 | SUBQUERY           | usercustomerflags |  | ALL | usercustomerflags_CustomerFlagID_idx,usercustomerflags_UserID_idx                           |                                |      |                       |   3509 | 4.94   | Using where |
|  8 | SUBQUERY           | userquotesemails  |  | ref | userquotesemails_Email__Active,userquotesemails_UserID                                      | userquotesemails_Email__Active | 1027 | const,const           |      1 | 100.00 |             |
|  6 | SUBQUERY           | userassociations  |  | ref | userassociations_UserID,userassociations__Email                                             | userassociations__Email        | 1026 | const                 |      3 | 10.00  | Using where |
|  4 | SUBQUERY           | quoterequests     |  | ref | quoterequests_CompanyID,quoterequests_UserID,quoterequests__Latest,quoterequests_UserQuotes | quoterequests_CompanyID        |  144 | const                 |  16702 | 10.00  | Using where |
|  2 | SUBQUERY           | invoices          |  | ref | Invoice_UserID_idx,Invoice_CompanyID_idx,invoices_SampleRequests,invoices_LateOrdersBubble  | Invoice_CompanyID_idx          |  144 | const                 |  17678 | 10.00  | Using where |
+----+--------------------+-------------------+--+-----+---------------------------------------------------------------------------------------------+--------------------------------+------+-----------------------+--------+--------+-------------+

Also, I've tried using joins, e.g. joining the invoices table, etc. but then I get the issue of having duplicate user rows per each invoice or quoterequest the joins receive, and grouping/distinct & ordering the resulting data became immensely slow, in the minutes.

I've also tried just the "exists" version of the first query, as suggesting by the doc https://dev.mysql.com/doc/refman/5.7/en/subquery-optimization-with-exists.html like so

select`UserID`
from`users`
where(`CompanyID`in('3e55d1bb-d8b6-11e4-b38f-b8ca3a83b4c8')
    or exists(select 1
        from`invoices`
        where`CompanyID`in('3e55d1bb-d8b6-11e4-b38f-b8ca3a83b4c8')
        and`__Active`=1
        and`users`.`UserID`=`UserID`)
    or exists(select 1
        from`quoterequests`
        where`CompanyID`in('3e55d1bb-d8b6-11e4-b38f-b8ca3a83b4c8')
        and`__Active`=1
        and`users`.`UserID`=`UserID`))
and(exists(select 1
        from`userassociations`
        where`_Email`='brian@yeet.com'
        and`__Active`=1
        and`users`.`UserID`=`UserID`))
and(exists(select 1
        from`usercustomerflags`
        where`CustomerFlagID`in(10,27,17,1,2,3,4,5,6)
        and`__Active`=1
        and`users`.`UserID`=`UserID`)
    or not exists(select 1
        from`usercustomerflags`
        where`__Active`=1
        and`users`.`UserID`=`UserID`))
and`Deleted`=0
order by`DateTimeAdded`desc
limit 50;

But that gets me up to 5.516s, so that definitely doesn't seem like the right direction to make.

What is the most efficient way to select data the way that I'm trying? Or do I need to restructure some of my tables to get the performance I'm looking for?

I've isolated the smallest sub-problem and bottle neck I think I have. Here's my lighter query

select`users`.`UserID`,`users`.`_Customer`
from`users`
left join`userassociations`on`userassociations`.`UserID`=`users`.`UserID`
and`userassociations`.`__Active`=1
where(`users`.`Email`='brian@stumpyinc.com'
    or`userassociations`.`_Email`='brian@stumpyinc.com')
and`users`.`Deleted`=0
order by`users`.`DateTimeAdded`desc
limit 50;

And the explain

+---+--------+------------------+--+-----+--------------------------------------------------------+-------------------------+-----+-----------------------+--------+--------+-------------+
| 1 | SIMPLE | users            |  | ref | users_getemail_INDEX,unify_email_INDEX,users_Customers | users_Customers         |   4 | const                 | 221463 | 100.00 | Using where |
| 1 | SIMPLE | userassociations |  | ref | userassociations_UserID                                | userassociations_UserID | 144 | sterling.users.UserID |      1 | 100.00 | Using where |
+---+--------+------------------+--+-----+--------------------------------------------------------+-------------------------+-----+-----------------------+--------+--------+-------------+

This query takes about 1.5 seconds to execute

CREATE TABLE `users` (
  `UserID` char(36) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '',
    ...
  `Email` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
    ...
  `DateTimeAdded` datetime DEFAULT NULL,
    ...
  `Deleted` int(1) NOT NULL DEFAULT '0',
    ...
  `_LatestInvoiceDateTimeAdded` datetime DEFAULT NULL,
  `_InvoiceCount` int(11) NOT NULL DEFAULT '0',
  `_Customer` varchar(512) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
    ...
  PRIMARY KEY (`UserID`),
  UNIQUE KEY `UserID_UNIQUE` (`UserID`),
    ...
  KEY `users_getemail_INDEX` (`Email`(191),`_InvoiceCount`,`_LatestInvoiceDateTimeAdded`,`DateTimeAdded`),
  KEY `unify_email_INDEX` (`Email`(191),`UserID`),
    ...
  KEY `users_Customers` (`Deleted`,`DateTimeAdded`),
    ...
  KEY `users_DateTimeAdded` (`DateTimeAdded`,`UserID`),
  FULLTEXT KEY `users_FULLTEXT__Customer` (`_Customer`),
    ...
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

CREATE TABLE `userassociations` (
   `UserAssociationID` binary(16) NOT NULL,
   `UserID` char(36) COLLATE utf8mb4_unicode_ci NOT NULL,
   `AssociatedUserID` char(36) COLLATE utf8mb4_unicode_ci NOT NULL,
   `_Email` varchar(256) COLLATE utf8mb4_unicode_ci NOT NULL,
   `__UserID` char(36) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
   `__Active` tinyint(1) NOT NULL DEFAULT '1',
   `__Added` timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
   `__Updated` timestamp(6) NULL DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP(6),
   PRIMARY KEY (`UserAssociationID`),
   KEY `userassociations_UserID` (`UserID`),
   KEY `userassociations_AssociatedUserID` (`AssociatedUserID`),
   KEY `userassociations___UserID` (`__UserID`),
   KEY `userassociations__Email` (`_Email`),
   CONSTRAINT `userassociations_AssociatedUserID` FOREIGN KEY (`AssociatedUserID`) REFERENCES `users` (`UserID`) ON DELETE NO ACTION ON UPDATE NO ACTION,
   CONSTRAINT `userassociations_UserID` FOREIGN KEY (`UserID`) REFERENCES `users` (`UserID`) ON DELETE NO ACTION ON UPDATE NO ACTION,
   CONSTRAINT `userassociations___UserID` FOREIGN KEY (`__UserID`) REFERENCES `users` (`UserID`) ON DELETE NO ACTION ON UPDATE NO ACTION
 ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci

Hmm... So it seemed like it was working, but I've found a pair of tables that it doesn't seem to be as efficient with, which is my users and invoices tables.

I've have these indexes:

users:    INDEX(`CompanyID`, `Deleted`, `DateTimeAdded`)
invoices: INDEX(`UserID`, `__Active`)
invoices: INDEX(`CompanyID`)
users:    INDEX(`UserID`, `Deleted`)

and the query

select`users`.`UserID`,`users`.`DateTimeAdded`
from`users`
join`invoices`on`invoices`.`UserID`=`users`.`UserID`
and`invoices`.`__Active`=1
where`invoices`.`CompanyID`='3e55c8b4-d8b6-11e4-b38f-b8ca3a83b4c8'
and`users`.`Deleted`=0
order by`DateTimeAdded`desc
limit 200;

This query alone takes .3 seconds, which feels slow to me, like it isn't making the best use of the indexes, especially because users only has 430,997 rows and invoices only has 194,180, and this looks like it should be a pretty simple query.

EDIT: Actually it's much worse than that, if the CompanyID given only includes ~4 rows, then this query takes 3.5 seconds

+---+--------+----------+--+-----+------------------------------------------------+-----------------------+-----+--------------------------------+------+--------+----------------------------------------------+
| 1 | SIMPLE | invoices |  | ref | Invoice_CompanyID_idx,invoices_UserID___Active | Invoice_CompanyID_idx | 144 | const                          | 7750 | 10.00  | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | users    |  | ref | users_UserID_Deleted                           | users_UserID_Deleted  | 148 | sterling.invoices.UserID,const |    1 | 100.00 |                                              |
+---+--------+----------+--+-----+------------------------------------------------+-----------------------+-----+--------------------------------+------+--------+----------------------------------------------+

Body:
For that smaller problem:

( select u.`UserID`, u.`_Customer`, u.DateTimeAdded
    from  `users` AS u
    where  u.`Email` = 'brian@stumpyinc.com'
      and  u.`Deleted` = 0
      AND EXISTS ( SELECT * FROM `userassociations`
                       WHERE UserId = u.UserID
                         AND __Active = 1 )
    order by  u.`DateTimeAdded` desc
    limit  50
)
UNION DISTINCT
( select u.`UserID`, u.`_Customer`, u.DateTimeAdded
    from  `users` AS u
    JOIN  `userassociations` AS ua
         ON  ua.`UserID` = u.`UserID`
        and  ua.`__Active` = 1
    where  ua.`_Email` = 'brian@stumpyinc.com'
      and  u.`Deleted`=0
    order by  u.`DateTimeAdded` desc
    limit  50
)
order by `DateTimeAdded` desc
limit  50

These will be needed:

u:  INDEX(Email, Deleted, DateTimeAdded)  -- date last
ua: INDEX(UserId, __Active)   -- either order
ua: INDEX(_Email)
u:  INDEX(UserID, Deleted)

(Let me know if you get syntax errors.  If it is too slow, please provide EXPLAIN.)

Index prefixing (Email(191)) is usually useless.  Get rid if it.  Here are 5 ways to avoid it:  http://mysql.rjweb.org/doc.php/limits#767_limit_in_innodb_indexes

A PK is a UNIQUE key, so get rid of the second of:

PRIMARY KEY (`UserID`),
UNIQUE KEY `UserID_UNIQUE` (`UserID`),

Smells like a UUID; use ascii (ascii_general_ci), not utf8mb4:

... char(36) COLLATE utf8mb4_unicode_ci

INT(1) takes 4 bytes; use TINYINT for flags.