问:
could not broadcast input array from shape (20,310,310) into shape (20)
I'm trying to detect lung cancer nodules using DICOM files. The main steps in cancer detection included following steps.

```<code>1) Preprocessing
  * Converting the pixel values to Hounsfield Units (HU)
  * Resampling to an isomorphic resolution to remove variance in scanner resolution
  *Lung segmentation
2) Training the data set using preprocessed images in Tensorflow CNN
3) Testing and validation
</code>```

I followed few online tutorials to do this.

I need to combine the given solutions in

1) https://www.kaggle.com/gzuidhof/full-preprocessing-tutorial

2) https://www.kaggle.com/sentdex/first-pass-through-data-w-3d-convnet.

I could implement the example in link two. But since it is lack ok lung segmentation and few other preprocessing steps I need to combine the steps in link one with link two. But I'm getting number of errors while doing it. Since I'm new to python can someone please help me in solving it.

There are 20 patient folders and each patient folder has number of slices, which are dicom files.

For the process_data method , slices_path of each patient and patient number was sent.

```<code>def process_data(slices,patient,labels_df,img_px_size,hm_slices):

try:
    label=labels_df.get_value(patient,'cancer')

    patient_pixels = get_pixels_hu(slices)

    segmented_lungs2, spacing = resample(patient_pixels, slices, [1,1,1])

    new_slices=[]

    segmented_lung = segment_lung_mask(segmented_lungs2, False)
    segmented_lungs_fill = segment_lung_mask(segmented_lungs2, True)
    segmented_lungs=segmented_lungs_fill-segmented_lung

    #This method returns smallest integer not less than x.
    chunk_sizes =math.ceil(len(segmented_lungs)/HM_SLICES)

    for slice_chunk in chunks(segmented_lungs,chunk_sizes):

            slice_chunk=list(map(mean,zip(*slice_chunk))) #list - []

            #print (slice_chunk)
            new_slices.append(slice_chunk)

    print(len(segmented_lungs), len(new_slices))

    if len(new_slices)==HM_SLICES-1:
            new_slices.append(new_slices[-1])

    if len(new_slices)==HM_SLICES-2:
            new_slices.append(new_slices[-1])
            new_slices.append(new_slices[-1])

    if len(new_slices)==HM_SLICES+2:
            new_val =list(map(mean, zip(*[new_slices[HM_SLICES-1],new_slices[HM_SLICES],])))
            del new_slices[HM_SLICES]
            new_slices[HM_SLICES-1]=new_val

    if len(new_slices)==HM_SLICES+1:
            new_val =list(map(mean, zip(*[new_slices[HM_SLICES-1],new_slices[HM_SLICES],])))
            del new_slices[HM_SLICES]
            new_slices[HM_SLICES-1]=new_val

    print('LENGTH ',len(segmented_lungs), len(new_slices))
except Exception as e:
    # again, some patients are not labeled, but JIC we still want the error if something
    # else is wrong with our code
    print(str(e))

#print(len(new_slices))

if label==1: label=np.array([0,1])
elif label==0: label=np.array([1,0])
return np.array(new_slices),label
</code>```

Main method

```<code>    # Some constants
    #data_dir = '../../CT_SCAN_IMAGE_SET/IMAGES/'
    #patients = os.listdir(data_dir)
    #labels_df=pd.read_csv('../../CT_SCAN_IMAGE_SET/stage1_labels.csv',index_col=0)
    #patients.sort()
    #print (labels_df.head())

    much_data=[]
    much_data2=[]
    for num,patient in enumerate(patients):
        if num%100==0:
            print (num)

        try:

            slices = load_scan(data_dir + patients[num])
            img_data,label=process_data(slices,patients[num],labels_df,IMG_PX_SIZE,HM_SLICES)
            much_data.append([img_data,label])
            #much_data2.append([processed,label])
        except:
            print ('This is unlabeled data')

    np.save('muchdata-{}-{}-{}.npy'.format(IMG_PX_SIZE,IMG_PX_SIZE,HM_SLICES),much_data)
    #np.save('muchdata-{}-{}-{}.npy'.format(IMG_PX_SIZE,IMG_PX_SIZE,HM_SLICES),much_data2)
</code>```

The preprocessing part works fine but when I'm trying to enter the final out put to a Convolutional NN and train the data set , Following is the error I'm receiving including some of the comments that I had put

```<code>    0
    shape hu
    (113, 512, 512)
    Resize factor
    [ 2.49557522  0.6015625   0.6015625 ]
    shape
    (282, 308, 308)
    chunk size
    15
    282 19
    LENGTH  282 20
    Tensor("Placeholder:0", dtype=float32)
    ..........1.........
    ..........2.........
    ..........3.........
    ..........4.........
    WARNING:tensorflow:From C:\Research\Python_installation\lib\site-packages\tensorflow\python\util\tf_should_use.py:170: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
    Instructions for updating:
    Use `tf.global_variables_initializer` instead.
    ..........5.........
    ..........6.........
    Epoch 1 completed out of 20 loss: 0
    ..........7.........
    Traceback (most recent call last):
    File "C:\Research\LungCancerDetaction\sendbox2.py", line 436, in <module>
    train_neural_network(x)
    File "C:\Research\LungCancerDetaction\sendbox2.py", line 424, in train_neural_network
    print('Accuracy:',accuracy.eval({x:[i[0] for i in validation_data], y:[i[1] for i in validation_data]}))
    File "C:\Research\Python_installation\lib\site-packages\tensorflow\python\framework\ops.py", line 606, in eval
    return _eval_using_default_session(self, feed_dict, self.graph, session)
    File "C:\Research\Python_installation\lib\site-packages\tensorflow\python\framework\ops.py", line 3928, in _eval_using_default_session
    return session.run(tensors, feed_dict)
    File "C:\Research\Python_installation\lib\site-packages\tensorflow\python\client\session.py", line 789, in run
    run_metadata_ptr)
    File "C:\Research\Python_installation\lib\site-packages\tensorflow\python\client\session.py", line 968, in _run
    np_val = np.asarray(subfeed_val, dtype=subfeed_dtype)
    File "C:\Research\Python_installation\lib\site-packages\numpy\core\numeric.py", line 531, in asarray
    return array(a, dtype, copy=False, order=order)
    ValueError: could not broadcast input array from shape (20,310,310) into shape (20)
</code>```

I think it is the issue with the 'segmented_lungs=segmented_lungs_fill-segmented_lung'

In the working example,

```<code>segmented_lungs=[cv2.resize(each_slice,(IMG_PX_SIZE,IMG_PX_SIZE)) for each_slice in patient_pixels]
</code>```

Please help me in solving this. I'm unable to proceed since some time. If anything is not clear please let me know.

Following is the whole code that had tried.

```<code>    import numpy as np # linear algebra
    import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
    import dicom
    import os
    import scipy.ndimage
    import matplotlib.pyplot as plt
    import cv2
    import math
    import tensorflow as tf

    from skimage import measure, morphology
    from mpl_toolkits.mplot3d.art3d import Poly3DCollection

     # Some constants
    data_dir = '../../CT_SCAN_IMAGE_SET/IMAGES/'
    patients = os.listdir(data_dir)
    labels_df=pd.read_csv('../../CT_SCAN_IMAGE_SET/stage1_labels.csv',index_col=0)
    patients.sort()
    print (labels_df.head())

    #Image pixel array watching

    for patient in patients[:10]:
        #label is to get the label  of the patient. This is what done in the .get_value method.
        label=labels_df.get_value(patient,'cancer')
        path=data_dir+patient
        slices = [dicom.read_file(path + '/' + s) for s in os.listdir(path)]
        #You have dicom files and they have attributes.
        slices.sort(key = lambda x: float(x.ImagePositionPatient[2]))
        print (len(slices),slices[0].pixel_array.shape)

    #If u need to see many slices and resize the large pixelated 2D images into 150*150 pixelated images

    IMG_PX_SIZE=50
    HM_SLICES=20

    for patient in patients[:1]:
        #label is to get the label of the patient. This is what done in the .get_value method.
        label=labels_df.get_value(patient,'cancer')
        path=data_dir+patient
        slices = [dicom.read_file(path + '/' + s) for s in os.listdir(path)]
        #You have dicom files and they have attributes.
        slices.sort(key = lambda x: float(x.ImagePositionPatient[2]))
        #This shows the pixel arrayed image related to the second slice of each patient

        #subplot
        fig=plt.figure()
        for num,each_slice in enumerate(slices[:16]):
            print (num)
            y=fig.add_subplot(4,4,num+1)
            #down sizing everything. Resize the imag size as their pixel values are 512*512
            new_image=cv2.resize(np.array(each_slice.pixel_array),(IMG_PX_SIZE,IMG_PX_SIZE))
            y.imshow(new_image)
        plt.show()

    print (len(patients))

    ###################################################################################

    def get_pixels_hu(slices):
        image = np.array([s.pixel_array for s in slices])
        # Convert to int16 (from sometimes int16),
        # should be possible as values should always be low enough (<32k)
        image = image.astype(np.int16)

        # Set outside-of-scan pixels to 0
        # The intercept is usually -1024, so air is approximately 0
        image[image == -2000] = 0

        # Convert to Hounsfield units (HU)
        for slice_number in range(len(slices)):

            intercept = slices[slice_number].RescaleIntercept
            slope = slices[slice_number].RescaleSlope

            if slope != 1:
                image[slice_number] = slope * image[slice_number].astype(np.float64)
                image[slice_number] = image[slice_number].astype(np.int16)

            image[slice_number] += np.int16(intercept)

        return np.array(image, dtype=np.int16)

    #The next problem is each patient is got different number of slices . This is a performance issue.
    # Take the slices and put that into a list of slices and chunk that list of slices into fixed numer of
    #chunk of slices and averaging those chunks.

    #yield is like 'return'. It returns a generator

    def chunks(l,n):
        for i in range(0,len(l),n):
            #print ('Inside yield')
            #print (i)
            yield l[i:i+n]

    def mean(l):
        return sum(l)/len(l)

    def largest_label_volume(im, bg=-1):
        vals, counts = np.unique(im, return_counts=True)

        counts = counts[vals != bg]
        vals = vals[vals != bg]

        if len(counts) > 0:
            return vals[np.argmax(counts)]
        else:
            return None

    def segment_lung_mask(image, fill_lung_structures=True):

        # not actually binary, but 1 and 2.
        # 0 is treated as background, which we do not want
        binary_image = np.array(image > -320, dtype=np.int8)+1
        labels = measure.label(binary_image)

        # Pick the pixel in the very corner to determine which label is air.
        #   Improvement: Pick multiple background labels from around the patient
        #   More resistant to "trays" on which the patient lays cutting the air
        #   around the person in half
        background_label = labels[0,0,0]
        #Fill the air around the person
        binary_image[background_label == labels] = 2

        # Method of filling the lung structures (that is superior to something like
        # morphological closing)

        if fill_lung_structures:
            # For every slice we determine the largest solid structure

            for i, axial_slice in enumerate(binary_image):
                axial_slice = axial_slice - 1
                labeling = measure.label(axial_slice)
                l_max = largest_label_volume(labeling, bg=0)

                if l_max is not None: #This slice contains some lung
                    binary_image[i][labeling != l_max] = 1

        binary_image -= 1 #Make the image actual binary
        binary_image = 1-binary_image # Invert it, lungs are now 1
        # Remove other air pockets insided body
        labels = measure.label(binary_image, background=0)
        l_max = largest_label_volume(labels, bg=0)

        if l_max is not None: # There are air pockets
            binary_image[labels != l_max] = 0

        return binary_image

    #Loading the files
    #Load the scans in given folder path
    def load_scan(path):
        slices = [dicom.read_file(path + '/' + s) for s in os.listdir(path)]
        slices.sort(key = lambda x: float(x.ImagePositionPatient[2]))
        try:
            slice_thickness = np.abs(slices[0].ImagePositionPatient[2] - slices[1].ImagePositionPatient[2])
        except:
            slice_thickness = np.abs(slices[0].SliceLocation - slices[1].SliceLocation)

        for s in slices:
            s.SliceThickness = slice_thickness

        return slices

    def resample(image, scan, new_spacing=[1,1,1]):
        # Determine current pixel spacing

        spacing = np.array([scan[0].SliceThickness] + scan[0].PixelSpacing, dtype=np.float32)

        resize_factor = spacing / new_spacing
        new_real_shape = image.shape * resize_factor
        new_shape = np.round(new_real_shape)
        real_resize_factor = new_shape / image.shape
        new_spacing = spacing / real_resize_factor

        print ('Resize factor')
        print (real_resize_factor)

        image = scipy.ndimage.interpolation.zoom(image, real_resize_factor, mode='nearest')
        print ('shape')
        print (image.shape)

        return image, new_spacing

    '''def chunks(l,n):
        for i in range(0,len(l),n):
            #print ('Inside yield')
            #print (i)
            yield l[i:i+n]

    def mean(l):
        return sum(l)/len(l)'''

    #processing data
    def process_data(slices,patient,labels_df,img_px_size,hm_slices):
         #for patient in patients[:10]:
                #label is to get the label of the patient. This is what done in the .get_value method.
        try:
            label=labels_df.get_value(patient,'cancer')
            print ('label process data')
            print (label)
            #path=data_dir+patient
            #slices = [dicom.read_file(path + '/' + s) for s in os.listdir(path)]
            #You have dicom files and they have attributes.
            slices.sort(key = lambda x: float(x.ImagePositionPatient[2]))
            #This shows the pixel arrayed image related to the second slice of each patient
            patient_pixels = get_pixels_hu(slices)
            print ('shape hu')
            print (patient_pixels.shape)
            segmented_lungs2, spacing = resample(patient_pixels, slices, [1,1,1])
            #print ('Pix shape')
            #print (segmented_lungs2.shape)

            #segmented_lungs=np.array(segmented_lungs2).tolist()

            new_slices=[]

            segmented_lung = segment_lung_mask(segmented_lungs2, False)
            segmented_lungs_fill = segment_lung_mask(segmented_lungs2, True)
            segmented_lungs=segmented_lungs_fill-segmented_lung

            #print ('length of segmented lungs')
            #print (len(segmented_lungs))
            #print ('Shape of segmented lungs......................................')
            #print (segmented_lungs.shape)
            #print ('hiiii')
            #segmented_lungs=[cv2.resize(each_slice,(IMG_PX_SIZE,IMG_PX_SIZE)) for each_slice in segmented_lungs3]
            #print ('bye')
            #print ('length of slices')
            #print (len(slices))
            #print ('shape of slices')
            #print (slices.shape)

            #print (each_slice.pixel_array)

            #This method returns smallest integer not less than x.
            chunk_sizes =math.ceil(len(segmented_lungs)/HM_SLICES)

            print ('chunk size ')
            print (chunk_sizes)

            for slice_chunk in chunks(segmented_lungs,chunk_sizes):

                    slice_chunk=list(map(mean,zip(*slice_chunk))) #list - []

                    #print (slice_chunk)
                    new_slices.append(slice_chunk)

            print(len(segmented_lungs), len(new_slices))

            if len(new_slices)==HM_SLICES-1:
                    new_slices.append(new_slices[-1])

            if len(new_slices)==HM_SLICES-2:
                    new_slices.append(new_slices[-1])
                    new_slices.append(new_slices[-1])

            if len(new_slices)==HM_SLICES-3:
                    new_slices.append(new_slices[-1])
                    new_slices.append(new_slices[-1])
                    new_slices.append(new_slices[-1])

            if len(new_slices)==HM_SLICES+2:
                    new_val =list(map(mean, zip(*[new_slices[HM_SLICES-1],new_slices[HM_SLICES],])))
                    del new_slices[HM_SLICES]
                    new_slices[HM_SLICES-1]=new_val

            if len(new_slices)==HM_SLICES+1:
                    new_val =list(map(mean, zip(*[new_slices[HM_SLICES-1],new_slices[HM_SLICES],])))
                    del new_slices[HM_SLICES]
                    new_slices[HM_SLICES-1]=new_val

            if len(new_slices)==HM_SLICES+3:
                    new_val =list(map(mean, zip(*[new_slices[HM_SLICES-1],new_slices[HM_SLICES],])))
                    del new_slices[HM_SLICES]
                    new_slices[HM_SLICES-1]=new_val

            print('LENGTH ',len(segmented_lungs), len(new_slices))
        except Exception as e:
            # again, some patients are not labeled, but JIC we still want the error if something
            # else is wrong with our code
            print(str(e))

        #print(len(new_slices))

        if label==1: label=np.array([0,1])
        elif label==0: label=np.array([1,0])
        return np.array(new_slices),label

    # Some constants
    #data_dir = '../../CT_SCAN_IMAGE_SET/IMAGES/'
    #patients = os.listdir(data_dir)
    #labels_df=pd.read_csv('../../CT_SCAN_IMAGE_SET/stage1_labels.csv',index_col=0)
    #patients.sort()
    #print (labels_df.head())

    much_data=[]
    much_data2=[]
    for num,patient in enumerate(patients):
        if num%100==0:
            print (num)

        try:

            slices = load_scan(data_dir + patients[num])
            img_data,label=process_data(slices,patients[num],labels_df,IMG_PX_SIZE,HM_SLICES)
            much_data.append([img_data,label])
            #much_data2.append([processed,label])
        except:
            print ('This is unlabeled data')

    np.save('muchdata-{}-{}-{}.npy'.format(IMG_PX_SIZE,IMG_PX_SIZE,HM_SLICES),much_data)
    #np.save('muchdata-{}-{}-{}.npy'.format(IMG_PX_SIZE,IMG_PX_SIZE,HM_SLICES),much_data2)

    IMG_SIZE_PX = 50
    SLICE_COUNT = 20

    n_classes=2
    batch_size=10

    x = tf.placeholder('float')
    y = tf.placeholder('float')

    keep_rate = 0.8

    def conv3d(x, W):
        return tf.nn.conv3d(x, W, strides=[1,1,1,1,1], padding='SAME')

    def maxpool3d(x):
        #                        size of window         movement of window as you slide about
        return tf.nn.max_pool3d(x, ksize=[1,2,2,2,1], strides=[1,2,2,2,1], padding='SAME')

    def convolutional_neural_network(x):
        #                # 5 x 5 x 5 patches, 1 channel, 32 features to compute.
        weights = {'W_conv1':tf.Variable(tf.random_normal([3,3,3,1,32])),
                   #       5 x 5 x 5 patches, 32 channels, 64 features to compute.
                   'W_conv2':tf.Variable(tf.random_normal([3,3,3,32,64])),
                   #                                  64 features
                   'W_fc':tf.Variable(tf.random_normal([54080,1024])),
                   'out':tf.Variable(tf.random_normal([1024, n_classes]))}

        biases = {'b_conv1':tf.Variable(tf.random_normal([32])),
                   'b_conv2':tf.Variable(tf.random_normal([64])),
                   'b_fc':tf.Variable(tf.random_normal([1024])),
                   'out':tf.Variable(tf.random_normal([n_classes]))}

        #                            image X      image Y        image Z
        x = tf.reshape(x, shape=[-1, IMG_SIZE_PX, IMG_SIZE_PX, SLICE_COUNT, 1])

        conv1 = tf.nn.relu(conv3d(x, weights['W_conv1']) + biases['b_conv1'])
        conv1 = maxpool3d(conv1)

        conv2 = tf.nn.relu(conv3d(conv1, weights['W_conv2']) + biases['b_conv2'])
        conv2 = maxpool3d(conv2)

        fc = tf.reshape(conv2,[-1, 54080])
        fc = tf.nn.relu(tf.matmul(fc, weights['W_fc'])+biases['b_fc'])
        fc = tf.nn.dropout(fc, keep_rate)

        output = tf.matmul(fc, weights['out'])+biases['out']

        return output

    much_data = np.load('muchdata-50-50-20.npy')
    # If you are working with the basic sample data, use maybe 2 instead of 100 here... you don't have enough data to really do this
    train_data = much_data[:-4]
    validation_data = much_data[-4:]

    def train_neural_network(x):
        print ('..........1.........')
        prediction = convolutional_neural_network(x)
        print ('..........2.........')
        #cost = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits(prediction,y) )
        cost = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits(logits=prediction,labels=y))
        print ('..........3.........')
        optimizer = tf.train.AdamOptimizer(learning_rate=1e-3).minimize(cost)
        print ('..........4.........')
        hm_epochs = 20
        with tf.Session() as sess:
            sess.run(tf.initialize_all_variables())

            successful_runs = 0
            total_runs = 0
            print ('..........5.........')
            for epoch in range(hm_epochs):
                epoch_loss = 0
                for data in train_data:
                    total_runs += 1
                    try:
                        X = data[0]
                        Y = data[1]
                        _, c = sess.run([optimizer, cost], feed_dict={x: X, y: Y})
                        epoch_loss += c
                        successful_runs += 1
                    except Exception as e:
                        # I am passing for the sake of notebook space, but we are getting 1 shaping issue from one
                        # input tensor. Not sure why, will have to look into it. Guessing it's
                        # one of the depths that doesn't come to 20.
                        pass
                        #print(str(e))
                print ('..........6.........')
                print('Epoch', epoch+1, 'completed out of',hm_epochs,'loss:',epoch_loss)
                print ('..........7.........')
                correct = tf.equal(tf.argmax(prediction, 1), tf.argmax(y, 1))
                accuracy = tf.reduce_mean(tf.cast(correct, 'float'))

                print('Accuracy:',accuracy.eval({x:[i[0] for i in validation_data], y:[i[1] for i in validation_data]}))

            print('Done. Finishing accuracy:')
            print('Accuracy:',accuracy.eval({x:[i[0] for i in validation_data], y:[i[1] for i in validation_data]}))

            print('fitment percent:',successful_runs/total_runs)

    print (x)

    # Run this locally:
    train_neural_network(x)
</code>```

P.S : resample() , segment_lung_mask() methods can be found from link 1.

答:
For training you have

```<code>for data in train_data:
    total_runs += 1
    try:
        X = data[0]
        Y = data[1]
        _, c = sess.run([optimizer, cost], feed_dict={x: X, y: Y})
</code>```

So x and y are, respectively, the first two elements of a single row of train_data.

However, when calculating the accuracy you have

```<code>print('Accuracy:',accuracy.eval({x:[i[0] for i in validation_data], y:[i[1] for i in validation_data]}))
</code>```

So x is the first element of all rows of validation_data, which gives it dimensions of (20,310,310), which can't be broadcast to a placeholder of dimension (20).  Ditto for y.  (Broadcasting means that if you gave it a tensor of dimensions (20, 310) it would know to take each of the 310 columns and feed it to the placeholder separately.  It can't figure out what to do with a tensor of (20, 310, 310).)

Incidentally, when you declare your placeholders it's a good idea to specify their dimensions, using None for the dimension depending on the number of separate examples.  This way the program can warn you when dimensions don't match up.

Body:
I'm trying to detect lung cancer nodules using DICOM files. The main steps in cancer detection included following steps.

1) Preprocessing
  * Converting the pixel values to Hounsfield Units (HU)
  * Resampling to an isomorphic resolution to remove variance in scanner resolution
  *Lung segmentation
2) Training the data set using preprocessed images in Tensorflow CNN
3) Testing and validation

I followed few online tutorials to do this.

I need to combine the given solutions in

1) https://www.kaggle.com/gzuidhof/full-preprocessing-tutorial

2) https://www.kaggle.com/sentdex/first-pass-through-data-w-3d-convnet.

I could implement the example in link two. But since it is lack ok lung segmentation and few other preprocessing steps I need to combine the steps in link one with link two. But I'm getting number of errors while doing it. Since I'm new to python can someone please help me in solving it.

There are 20 patient folders and each patient folder has number of slices, which are dicom files.

For the process_data method , slices_path of each patient and patient number was sent.

def process_data(slices,patient,labels_df,img_px_size,hm_slices):

try:
    label=labels_df.get_value(patient,'cancer')

    patient_pixels = get_pixels_hu(slices)

    segmented_lungs2, spacing = resample(patient_pixels, slices, [1,1,1])

    new_slices=[]

    segmented_lung = segment_lung_mask(segmented_lungs2, False)
    segmented_lungs_fill = segment_lung_mask(segmented_lungs2, True)
    segmented_lungs=segmented_lungs_fill-segmented_lung

    #This method returns smallest integer not less than x.
    chunk_sizes =math.ceil(len(segmented_lungs)/HM_SLICES)

    for slice_chunk in chunks(segmented_lungs,chunk_sizes):

            slice_chunk=list(map(mean,zip(*slice_chunk))) #list - []

            #print (slice_chunk)
            new_slices.append(slice_chunk)

    print(len(segmented_lungs), len(new_slices))

    if len(new_slices)==HM_SLICES-1:
            new_slices.append(new_slices[-1])

    if len(new_slices)==HM_SLICES-2:
            new_slices.append(new_slices[-1])
            new_slices.append(new_slices[-1])

    if len(new_slices)==HM_SLICES+2:
            new_val =list(map(mean, zip(*[new_slices[HM_SLICES-1],new_slices[HM_SLICES],])))
            del new_slices[HM_SLICES]
            new_slices[HM_SLICES-1]=new_val

    if len(new_slices)==HM_SLICES+1:
            new_val =list(map(mean, zip(*[new_slices[HM_SLICES-1],new_slices[HM_SLICES],])))
            del new_slices[HM_SLICES]
            new_slices[HM_SLICES-1]=new_val

    print('LENGTH ',len(segmented_lungs), len(new_slices))
except Exception as e:
    # again, some patients are not labeled, but JIC we still want the error if something
    # else is wrong with our code
    print(str(e))

#print(len(new_slices))

if label==1: label=np.array([0,1])
elif label==0: label=np.array([1,0])
return np.array(new_slices),label

Main method

    # Some constants
    #data_dir = '../../CT_SCAN_IMAGE_SET/IMAGES/'
    #patients = os.listdir(data_dir)
    #labels_df=pd.read_csv('../../CT_SCAN_IMAGE_SET/stage1_labels.csv',index_col=0)
    #patients.sort()
    #print (labels_df.head())

    much_data=[]
    much_data2=[]
    for num,patient in enumerate(patients):
        if num%100==0:
            print (num)

        try:

            slices = load_scan(data_dir + patients[num])
            img_data,label=process_data(slices,patients[num],labels_df,IMG_PX_SIZE,HM_SLICES)
            much_data.append([img_data,label])
            #much_data2.append([processed,label])
        except:
            print ('This is unlabeled data')

    np.save('muchdata-{}-{}-{}.npy'.format(IMG_PX_SIZE,IMG_PX_SIZE,HM_SLICES),much_data)
    #np.save('muchdata-{}-{}-{}.npy'.format(IMG_PX_SIZE,IMG_PX_SIZE,HM_SLICES),much_data2)

The preprocessing part works fine but when I'm trying to enter the final out put to a Convolutional NN and train the data set , Following is the error I'm receiving including some of the comments that I had put

    0
    shape hu
    (113, 512, 512)
    Resize factor
    [ 2.49557522  0.6015625   0.6015625 ]
    shape
    (282, 308, 308)
    chunk size
    15
    282 19
    LENGTH  282 20
    Tensor("Placeholder:0", dtype=float32)
    ..........1.........
    ..........2.........
    ..........3.........
    ..........4.........
    WARNING:tensorflow:From C:\Research\Python_installation\lib\site-packages\tensorflow\python\util\tf_should_use.py:170: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
    Instructions for updating:
    Use `tf.global_variables_initializer` instead.
    ..........5.........
    ..........6.........
    Epoch 1 completed out of 20 loss: 0
    ..........7.........
    Traceback (most recent call last):
    File "C:\Research\LungCancerDetaction\sendbox2.py", line 436, in <module>
    train_neural_network(x)
    File "C:\Research\LungCancerDetaction\sendbox2.py", line 424, in train_neural_network
    print('Accuracy:',accuracy.eval({x:[i[0] for i in validation_data], y:[i[1] for i in validation_data]}))
    File "C:\Research\Python_installation\lib\site-packages\tensorflow\python\framework\ops.py", line 606, in eval
    return _eval_using_default_session(self, feed_dict, self.graph, session)
    File "C:\Research\Python_installation\lib\site-packages\tensorflow\python\framework\ops.py", line 3928, in _eval_using_default_session
    return session.run(tensors, feed_dict)
    File "C:\Research\Python_installation\lib\site-packages\tensorflow\python\client\session.py", line 789, in run
    run_metadata_ptr)
    File "C:\Research\Python_installation\lib\site-packages\tensorflow\python\client\session.py", line 968, in _run
    np_val = np.asarray(subfeed_val, dtype=subfeed_dtype)
    File "C:\Research\Python_installation\lib\site-packages\numpy\core\numeric.py", line 531, in asarray
    return array(a, dtype, copy=False, order=order)
    ValueError: could not broadcast input array from shape (20,310,310) into shape (20)

I think it is the issue with the 'segmented_lungs=segmented_lungs_fill-segmented_lung'

In the working example,

segmented_lungs=[cv2.resize(each_slice,(IMG_PX_SIZE,IMG_PX_SIZE)) for each_slice in patient_pixels]

Please help me in solving this. I'm unable to proceed since some time. If anything is not clear please let me know.

Following is the whole code that had tried.

    import numpy as np # linear algebra
    import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
    import dicom
    import os
    import scipy.ndimage
    import matplotlib.pyplot as plt
    import cv2
    import math
    import tensorflow as tf

    from skimage import measure, morphology
    from mpl_toolkits.mplot3d.art3d import Poly3DCollection

     # Some constants
    data_dir = '../../CT_SCAN_IMAGE_SET/IMAGES/'
    patients = os.listdir(data_dir)
    labels_df=pd.read_csv('../../CT_SCAN_IMAGE_SET/stage1_labels.csv',index_col=0)
    patients.sort()
    print (labels_df.head())

    #Image pixel array watching

    for patient in patients[:10]:
        #label is to get the label  of the patient. This is what done in the .get_value method.
        label=labels_df.get_value(patient,'cancer')
        path=data_dir+patient
        slices = [dicom.read_file(path + '/' + s) for s in os.listdir(path)]
        #You have dicom files and they have attributes.
        slices.sort(key = lambda x: float(x.ImagePositionPatient[2]))
        print (len(slices),slices[0].pixel_array.shape)

    #If u need to see many slices and resize the large pixelated 2D images into 150*150 pixelated images

    IMG_PX_SIZE=50
    HM_SLICES=20

    for patient in patients[:1]:
        #label is to get the label of the patient. This is what done in the .get_value method.
        label=labels_df.get_value(patient,'cancer')
        path=data_dir+patient
        slices = [dicom.read_file(path + '/' + s) for s in os.listdir(path)]
        #You have dicom files and they have attributes.
        slices.sort(key = lambda x: float(x.ImagePositionPatient[2]))
        #This shows the pixel arrayed image related to the second slice of each patient

        #subplot
        fig=plt.figure()
        for num,each_slice in enumerate(slices[:16]):
            print (num)
            y=fig.add_subplot(4,4,num+1)
            #down sizing everything. Resize the imag size as their pixel values are 512*512
            new_image=cv2.resize(np.array(each_slice.pixel_array),(IMG_PX_SIZE,IMG_PX_SIZE))
            y.imshow(new_image)
        plt.show()

    print (len(patients))

    ###################################################################################

    def get_pixels_hu(slices):
        image = np.array([s.pixel_array for s in slices])
        # Convert to int16 (from sometimes int16),
        # should be possible as values should always be low enough (<32k)
        image = image.astype(np.int16)

        # Set outside-of-scan pixels to 0
        # The intercept is usually -1024, so air is approximately 0
        image[image == -2000] = 0

        # Convert to Hounsfield units (HU)
        for slice_number in range(len(slices)):

            intercept = slices[slice_number].RescaleIntercept
            slope = slices[slice_number].RescaleSlope

            if slope != 1:
                image[slice_number] = slope * image[slice_number].astype(np.float64)
                image[slice_number] = image[slice_number].astype(np.int16)

            image[slice_number] += np.int16(intercept)

        return np.array(image, dtype=np.int16)

    #The next problem is each patient is got different number of slices . This is a performance issue.
    # Take the slices and put that into a list of slices and chunk that list of slices into fixed numer of
    #chunk of slices and averaging those chunks.

    #yield is like 'return'. It returns a generator

    def chunks(l,n):
        for i in range(0,len(l),n):
            #print ('Inside yield')
            #print (i)
            yield l[i:i+n]

    def mean(l):
        return sum(l)/len(l)

    def largest_label_volume(im, bg=-1):
        vals, counts = np.unique(im, return_counts=True)

        counts = counts[vals != bg]
        vals = vals[vals != bg]

        if len(counts) > 0:
            return vals[np.argmax(counts)]
        else:
            return None

    def segment_lung_mask(image, fill_lung_structures=True):

        # not actually binary, but 1 and 2.
        # 0 is treated as background, which we do not want
        binary_image = np.array(image > -320, dtype=np.int8)+1
        labels = measure.label(binary_image)

        # Pick the pixel in the very corner to determine which label is air.
        #   Improvement: Pick multiple background labels from around the patient
        #   More resistant to "trays" on which the patient lays cutting the air
        #   around the person in half
        background_label = labels[0,0,0]
        #Fill the air around the person
        binary_image[background_label == labels] = 2

        # Method of filling the lung structures (that is superior to something like
        # morphological closing)

        if fill_lung_structures:
            # For every slice we determine the largest solid structure

            for i, axial_slice in enumerate(binary_image):
                axial_slice = axial_slice - 1
                labeling = measure.label(axial_slice)
                l_max = largest_label_volume(labeling, bg=0)

                if l_max is not None: #This slice contains some lung
                    binary_image[i][labeling != l_max] = 1

        binary_image -= 1 #Make the image actual binary
        binary_image = 1-binary_image # Invert it, lungs are now 1
        # Remove other air pockets insided body
        labels = measure.label(binary_image, background=0)
        l_max = largest_label_volume(labels, bg=0)

        if l_max is not None: # There are air pockets
            binary_image[labels != l_max] = 0

        return binary_image

    #Loading the files
    #Load the scans in given folder path
    def load_scan(path):
        slices = [dicom.read_file(path + '/' + s) for s in os.listdir(path)]
        slices.sort(key = lambda x: float(x.ImagePositionPatient[2]))
        try:
            slice_thickness = np.abs(slices[0].ImagePositionPatient[2] - slices[1].ImagePositionPatient[2])
        except:
            slice_thickness = np.abs(slices[0].SliceLocation - slices[1].SliceLocation)

        for s in slices:
            s.SliceThickness = slice_thickness

        return slices

    def resample(image, scan, new_spacing=[1,1,1]):
        # Determine current pixel spacing

        spacing = np.array([scan[0].SliceThickness] + scan[0].PixelSpacing, dtype=np.float32)

        resize_factor = spacing / new_spacing
        new_real_shape = image.shape * resize_factor
        new_shape = np.round(new_real_shape)
        real_resize_factor = new_shape / image.shape
        new_spacing = spacing / real_resize_factor

        print ('Resize factor')
        print (real_resize_factor)

        image = scipy.ndimage.interpolation.zoom(image, real_resize_factor, mode='nearest')
        print ('shape')
        print (image.shape)

        return image, new_spacing

    '''def chunks(l,n):
        for i in range(0,len(l),n):
            #print ('Inside yield')
            #print (i)
            yield l[i:i+n]

    def mean(l):
        return sum(l)/len(l)'''

    #processing data
    def process_data(slices,patient,labels_df,img_px_size,hm_slices):
         #for patient in patients[:10]:
                #label is to get the label of the patient. This is what done in the .get_value method.
        try:
            label=labels_df.get_value(patient,'cancer')
            print ('label process data')
            print (label)
            #path=data_dir+patient
            #slices = [dicom.read_file(path + '/' + s) for s in os.listdir(path)]
            #You have dicom files and they have attributes.
            slices.sort(key = lambda x: float(x.ImagePositionPatient[2]))
            #This shows the pixel arrayed image related to the second slice of each patient
            patient_pixels = get_pixels_hu(slices)
            print ('shape hu')
            print (patient_pixels.shape)
            segmented_lungs2, spacing = resample(patient_pixels, slices, [1,1,1])
            #print ('Pix shape')
            #print (segmented_lungs2.shape)

            #segmented_lungs=np.array(segmented_lungs2).tolist()

            new_slices=[]

            segmented_lung = segment_lung_mask(segmented_lungs2, False)
            segmented_lungs_fill = segment_lung_mask(segmented_lungs2, True)
            segmented_lungs=segmented_lungs_fill-segmented_lung

            #print ('length of segmented lungs')
            #print (len(segmented_lungs))
            #print ('Shape of segmented lungs......................................')
            #print (segmented_lungs.shape)
            #print ('hiiii')
            #segmented_lungs=[cv2.resize(each_slice,(IMG_PX_SIZE,IMG_PX_SIZE)) for each_slice in segmented_lungs3]
            #print ('bye')
            #print ('length of slices')
            #print (len(slices))
            #print ('shape of slices')
            #print (slices.shape)

            #print (each_slice.pixel_array)

            #This method returns smallest integer not less than x.
            chunk_sizes =math.ceil(len(segmented_lungs)/HM_SLICES)

            print ('chunk size ')
            print (chunk_sizes)

            for slice_chunk in chunks(segmented_lungs,chunk_sizes):

                    slice_chunk=list(map(mean,zip(*slice_chunk))) #list - []

                    #print (slice_chunk)
                    new_slices.append(slice_chunk)

            print(len(segmented_lungs), len(new_slices))

            if len(new_slices)==HM_SLICES-1:
                    new_slices.append(new_slices[-1])

            if len(new_slices)==HM_SLICES-2:
                    new_slices.append(new_slices[-1])
                    new_slices.append(new_slices[-1])

            if len(new_slices)==HM_SLICES-3:
                    new_slices.append(new_slices[-1])
                    new_slices.append(new_slices[-1])
                    new_slices.append(new_slices[-1])

            if len(new_slices)==HM_SLICES+2:
                    new_val =list(map(mean, zip(*[new_slices[HM_SLICES-1],new_slices[HM_SLICES],])))
                    del new_slices[HM_SLICES]
                    new_slices[HM_SLICES-1]=new_val

            if len(new_slices)==HM_SLICES+1:
                    new_val =list(map(mean, zip(*[new_slices[HM_SLICES-1],new_slices[HM_SLICES],])))
                    del new_slices[HM_SLICES]
                    new_slices[HM_SLICES-1]=new_val

            if len(new_slices)==HM_SLICES+3:
                    new_val =list(map(mean, zip(*[new_slices[HM_SLICES-1],new_slices[HM_SLICES],])))
                    del new_slices[HM_SLICES]
                    new_slices[HM_SLICES-1]=new_val

            print('LENGTH ',len(segmented_lungs), len(new_slices))
        except Exception as e:
            # again, some patients are not labeled, but JIC we still want the error if something
            # else is wrong with our code
            print(str(e))

        #print(len(new_slices))

        if label==1: label=np.array([0,1])
        elif label==0: label=np.array([1,0])
        return np.array(new_slices),label

    # Some constants
    #data_dir = '../../CT_SCAN_IMAGE_SET/IMAGES/'
    #patients = os.listdir(data_dir)
    #labels_df=pd.read_csv('../../CT_SCAN_IMAGE_SET/stage1_labels.csv',index_col=0)
    #patients.sort()
    #print (labels_df.head())

    much_data=[]
    much_data2=[]
    for num,patient in enumerate(patients):
        if num%100==0:
            print (num)

        try:

            slices = load_scan(data_dir + patients[num])
            img_data,label=process_data(slices,patients[num],labels_df,IMG_PX_SIZE,HM_SLICES)
            much_data.append([img_data,label])
            #much_data2.append([processed,label])
        except:
            print ('This is unlabeled data')

    np.save('muchdata-{}-{}-{}.npy'.format(IMG_PX_SIZE,IMG_PX_SIZE,HM_SLICES),much_data)
    #np.save('muchdata-{}-{}-{}.npy'.format(IMG_PX_SIZE,IMG_PX_SIZE,HM_SLICES),much_data2)

    IMG_SIZE_PX = 50
    SLICE_COUNT = 20

    n_classes=2
    batch_size=10

    x = tf.placeholder('float')
    y = tf.placeholder('float')

    keep_rate = 0.8

    def conv3d(x, W):
        return tf.nn.conv3d(x, W, strides=[1,1,1,1,1], padding='SAME')

    def maxpool3d(x):
        #                        size of window         movement of window as you slide about
        return tf.nn.max_pool3d(x, ksize=[1,2,2,2,1], strides=[1,2,2,2,1], padding='SAME')

    def convolutional_neural_network(x):
        #                # 5 x 5 x 5 patches, 1 channel, 32 features to compute.
        weights = {'W_conv1':tf.Variable(tf.random_normal([3,3,3,1,32])),
                   #       5 x 5 x 5 patches, 32 channels, 64 features to compute.
                   'W_conv2':tf.Variable(tf.random_normal([3,3,3,32,64])),
                   #                                  64 features
                   'W_fc':tf.Variable(tf.random_normal([54080,1024])),
                   'out':tf.Variable(tf.random_normal([1024, n_classes]))}

        biases = {'b_conv1':tf.Variable(tf.random_normal([32])),
                   'b_conv2':tf.Variable(tf.random_normal([64])),
                   'b_fc':tf.Variable(tf.random_normal([1024])),
                   'out':tf.Variable(tf.random_normal([n_classes]))}

        #                            image X      image Y        image Z
        x = tf.reshape(x, shape=[-1, IMG_SIZE_PX, IMG_SIZE_PX, SLICE_COUNT, 1])

        conv1 = tf.nn.relu(conv3d(x, weights['W_conv1']) + biases['b_conv1'])
        conv1 = maxpool3d(conv1)

        conv2 = tf.nn.relu(conv3d(conv1, weights['W_conv2']) + biases['b_conv2'])
        conv2 = maxpool3d(conv2)

        fc = tf.reshape(conv2,[-1, 54080])
        fc = tf.nn.relu(tf.matmul(fc, weights['W_fc'])+biases['b_fc'])
        fc = tf.nn.dropout(fc, keep_rate)

        output = tf.matmul(fc, weights['out'])+biases['out']

        return output

    much_data = np.load('muchdata-50-50-20.npy')
    # If you are working with the basic sample data, use maybe 2 instead of 100 here... you don't have enough data to really do this
    train_data = much_data[:-4]
    validation_data = much_data[-4:]

    def train_neural_network(x):
        print ('..........1.........')
        prediction = convolutional_neural_network(x)
        print ('..........2.........')
        #cost = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits(prediction,y) )
        cost = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits(logits=prediction,labels=y))
        print ('..........3.........')
        optimizer = tf.train.AdamOptimizer(learning_rate=1e-3).minimize(cost)
        print ('..........4.........')
        hm_epochs = 20
        with tf.Session() as sess:
            sess.run(tf.initialize_all_variables())

            successful_runs = 0
            total_runs = 0
            print ('..........5.........')
            for epoch in range(hm_epochs):
                epoch_loss = 0
                for data in train_data:
                    total_runs += 1
                    try:
                        X = data[0]
                        Y = data[1]
                        _, c = sess.run([optimizer, cost], feed_dict={x: X, y: Y})
                        epoch_loss += c
                        successful_runs += 1
                    except Exception as e:
                        # I am passing for the sake of notebook space, but we are getting 1 shaping issue from one
                        # input tensor. Not sure why, will have to look into it. Guessing it's
                        # one of the depths that doesn't come to 20.
                        pass
                        #print(str(e))
                print ('..........6.........')
                print('Epoch', epoch+1, 'completed out of',hm_epochs,'loss:',epoch_loss)
                print ('..........7.........')
                correct = tf.equal(tf.argmax(prediction, 1), tf.argmax(y, 1))
                accuracy = tf.reduce_mean(tf.cast(correct, 'float'))

                print('Accuracy:',accuracy.eval({x:[i[0] for i in validation_data], y:[i[1] for i in validation_data]}))

            print('Done. Finishing accuracy:')
            print('Accuracy:',accuracy.eval({x:[i[0] for i in validation_data], y:[i[1] for i in validation_data]}))

            print('fitment percent:',successful_runs/total_runs)

    print (x)

    # Run this locally:
    train_neural_network(x)

P.S : resample() , segment_lung_mask() methods can be found from link 1.

Body:
For training you have

for data in train_data:
    total_runs += 1
    try:
        X = data[0]
        Y = data[1]
        _, c = sess.run([optimizer, cost], feed_dict={x: X, y: Y})

So x and y are, respectively, the first two elements of a single row of train_data.

However, when calculating the accuracy you have

print('Accuracy:',accuracy.eval({x:[i[0] for i in validation_data], y:[i[1] for i in validation_data]}))

So x is the first element of all rows of validation_data, which gives it dimensions of (20,310,310), which can't be broadcast to a placeholder of dimension (20).  Ditto for y.  (Broadcasting means that if you gave it a tensor of dimensions (20, 310) it would know to take each of the 310 columns and feed it to the placeholder separately.  It can't figure out what to do with a tensor of (20, 310, 310).)

Incidentally, when you declare your placeholders it's a good idea to specify their dimensions, using None for the dimension depending on the number of separate examples.  This way the program can warn you when dimensions don't match up.


来源: stackexchange

问:
numpy array float number division leads to strange behavior
Please follow the code lines:

```<code>A = np.array([0 for i in range(10)])

A[:] = 0.1/2

A

Out[36]: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0])

A[:] = float(0.2/2)

A[:]

Out[38]: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
</code>```

I can't understand, why numpy is giving such a strange behavior?

I am using numpy 1.13.3

答:
Because when you define your array it is of dtype=int. Define it as float and it will work as you expect.

example:

```<code>a = np.arange(10) # dtype = int
a[:] = 0.1 # converts to int so to 0
print a

a = np.arange(10.) # dtype = float
# or equivalently a = np.arange(10, dtype=float)
a[:] = 0.1 # works as you expect
print a
</code>```

ContentLicense: CC BY-SA 3.0

Body:
Please follow the code lines:

A = np.array([0 for i in range(10)])

A[:] = 0.1/2

A

Out[36]: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0])

A[:] = float(0.2/2)

A[:]

Out[38]: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0])

I can't understand, why numpy is giving such a strange behavior?

I am using numpy 1.13.3

Body:
Because when you define your array it is of dtype=int. Define it as float and it will work as you expect.

example:

a = np.arange(10) # dtype = int
a[:] = 0.1 # converts to int so to 0
print a

a = np.arange(10.) # dtype = float
# or equivalently a = np.arange(10, dtype=float)
a[:] = 0.1 # works as you expect
print a


问:
DataInputStream for input text files?
I am learning to read and write in Java and am stuck with a simple exercise. The program reads from 2 txt files that each contain numbers in rows. It writes to an output file the result of the multiplication of each row of numbers. eg. file 1 row 1 : 10, file 2 row 1: 2 , the program should write 20 to the output file. My code seems to have something missing somewhere. The output file is created but nothing is written to it. Any ideas?</p>

```<code>import java.io.*;
import java.util.*;

class ReadWriteData
{
    public static void main(String[] args) throws Exception
    {

        //create ouput file
        PrintWriter output = new PrintWriter("output2.txt");

        DataInputStream file1 = new DataInputStream(new FileInputStream(args[0]));
        DataInputStream file2 = new DataInputStream(new FileInputStream(args[1]));

        try
        {

        // read data from file
        while (true)
        {
            double number1 = file1.readDouble();
            double number2 = file2.readDouble();
            double result = number1 * number2 ;
            output.println(result);

        }

        }

        catch (IOException e)
        {
            System.err.println("Error");
            System.exit(1);
        }

        output.close() ;

    }
}
</code>```

答: You need to call output.flush just before closing the stream. Also, you should close the streams to the files in a finally block, this will make sure that the close command wil always be executed.</p>

Title: DataInputStream for input text files?

Body:
I am learning to read and write in Java and am stuck with a simple exercise. The program reads from 2 txt files that each contain numbers in rows. It writes to an output file the result of the multiplication of each row of numbers. eg. file 1 row 1 : 10, file 2 row 1: 2 , the program should write 20 to the output file. My code seems to have something missing somewhere. The output file is created but nothing is written to it. Any ideas?

import java.io.*;
import java.util.*;

class ReadWriteData
{
    public static void main(String[] args) throws Exception
    {

        //create ouput file
        PrintWriter output = new PrintWriter("output2.txt");

        DataInputStream file1 = new DataInputStream(new FileInputStream(args[0]));
        DataInputStream file2 = new DataInputStream(new FileInputStream(args[1]));

        try
        {

        // read data from file
        while (true)
        {
            double number1 = file1.readDouble();
            double number2 = file2.readDouble();
            double result = number1 * number2 ;
            output.println(result);

        }

        }

        catch (IOException e)
        {
            System.err.println("Error");
            System.exit(1);
        }

        output.close() ;

    }
}

ContentLicense: CC BY-SA 2.5


问:
How can I properly use a Pandas Dataframe with a multiindex that includes Intervals?
I'm trying to slice into a DataFrame that has a MultiIndex composed of an IntervalIndex and a regular Index. Example code:

```<code>from pandas import Interval as ntv

df = pd.DataFrame.from_records([
   {'id': 1, 'var1': 0.1, 'ntv': ntv(0,10), 'E': 1},
   {'id':2, 'var1': 0.5, 'ntv': ntv(0,12), 'E': 0}
], index=('ntv', 'id'))
</code>```

Looks like this:

```<code>            E  var1
ntv     id
(0, 10] 1   1   0.1
(0, 12] 2   0   0.5
</code>```

What I would like to do is to slice into the DataFrame at a specific value and return all rows that has an interval that contains the value. Ex:

```<code>df.loc[4]
</code>```

should return (trivially)

```<code>    E  var1
id
1   1   0.1
2   0   0.5
</code>```

The problem is I keep getting a TypeError about the index, and the docs show a similar operation (but on a single-level index) that does produce what I'm looking for.

```<code>TypeError: only integer scalar arrays can be converted to a scalar index
</code>```

I've tried many things, nothing seems to work normally. I could include the id column inside the dataframe, but I'd rather keep my index unique, and I would constantly be calling set_index('id').

I feel like either a) I'm missing something about MultiIndexes or b) there is a bug / ambiguity with using an IntervalIndex in a MultiIndex.

答:
So I did a bit of digging to try and understand the problem. If I try to run your code the following happens.
You try to index into the index label with
"slice(array([0, 1], dtype=int64), array([1, 2], dtype=int64), None)"

(when I say index_type I mean the Pandas datatype)

An index_type's label is a list of indices that map to the index_type's levels array. Here is an example from the documentation.

```<code>   >>> arrays = [[1, 1, 2, 2], ['red', 'blue', 'red', 'blue']]
    >>> pd.MultiIndex.from_arrays(arrays, names=('number', 'color'))
    MultiIndex(levels=[[1, 2], ['blue', 'red']],
           labels=[[0, 0, 1, 1], [1, 0, 1, 0]],
           names=['number', 'color'])
</code>```

Notice how the second list in labels connects to the order of levels. level[1][1] is equal to red, and level[1][0] is equal to blue.

Anyhow, this is all to say that I don't believe intervalindex is meant to be used in an overlapping fashion. If you look at the orginal proposal for it
https://github.com/pandas-dev/pandas/issues/7640

"A IntervalIndex would be a monotonic and non-overlapping one-dimensional array of intervals."

My suggestion is to move the interval into a column. You could probably write up a simple function with numba to test if a number is in each interval. Do you mind explaining the way you're benefiting from the interval?

Body:
I'm trying to slice into a DataFrame that has a MultiIndex composed of an IntervalIndex and a regular Index. Example code:

from pandas import Interval as ntv

df = pd.DataFrame.from_records([
   {'id': 1, 'var1': 0.1, 'ntv': ntv(0,10), 'E': 1},
   {'id':2, 'var1': 0.5, 'ntv': ntv(0,12), 'E': 0}
], index=('ntv', 'id'))

Looks like this:

            E  var1
ntv     id
(0, 10] 1   1   0.1
(0, 12] 2   0   0.5

What I would like to do is to slice into the DataFrame at a specific value and return all rows that has an interval that contains the value. Ex:

df.loc[4]

should return (trivially)

    E  var1
id
1   1   0.1
2   0   0.5

The problem is I keep getting a TypeError about the index, and the docs show a similar operation (but on a single-level index) that does produce what I'm looking for.

TypeError: only integer scalar arrays can be converted to a scalar index

I've tried many things, nothing seems to work normally. I could include the id column inside the dataframe, but I'd rather keep my index unique, and I would constantly be calling set_index('id').

I feel like either a) I'm missing something about MultiIndexes or b) there is a bug / ambiguity with using an IntervalIndex in a MultiIndex.

Body:
So I did a bit of digging to try and understand the problem. If I try to run your code the following happens.
You try to index into the index label with
"slice(array([0, 1], dtype=int64), array([1, 2], dtype=int64), None)"

(when I say index_type I mean the Pandas datatype)

An index_type's label is a list of indices that map to the index_type's levels array. Here is an example from the documentation.

   >>> arrays = [[1, 1, 2, 2], ['red', 'blue', 'red', 'blue']]
    >>> pd.MultiIndex.from_arrays(arrays, names=('number', 'color'))
    MultiIndex(levels=[[1, 2], ['blue', 'red']],
           labels=[[0, 0, 1, 1], [1, 0, 1, 0]],
           names=['number', 'color'])

Notice how the second list in labels connects to the order of levels. level[1][1] is equal to red, and level[1][0] is equal to blue.

Anyhow, this is all to say that I don't believe intervalindex is meant to be used in an overlapping fashion. If you look at the orginal proposal for it
https://github.com/pandas-dev/pandas/issues/7640

"A IntervalIndex would be a monotonic and non-overlapping one-dimensional array of intervals."

My suggestion is to move the interval into a column. You could probably write up a simple function with numba to test if a number is in each interval. Do you mind explaining the way you're benefiting from the interval?


问:
Input Validation for GUI
My program works fine as is, I just need to add an input validation "invalid input, number has to be positive" dialog, after an input of 0 or a negative number for length: tLength and width: tWidth.

Here is my code which runs fine, just needs the validation added:

```<code>import java.awt.event.*;
import javax.swing.*;
import java.awt.*;

public class FloorsRUs extends JFrame implements ActionListener, ItemListener
{

    public JTabbedPane jTP;
    public JRadioButton flrWood = null;
    public JRadioButton flrCarpet = null;
    public JTextField tCName;
    public JTextField tAddress;
    public JTextField tLength;
    public JTextField tWidth;
    public JButton calcArea;
    public JButton calcCost;
    public JButton subOrder;
    public JButton orderSum;
    public JButton ordList;
    public JPanel flrPan;
    public JPanel lwPan;
    public JPanel cPan;
    public JLabel lab1;
    public JLabel lab2;
    public JLabel lab3;
    public JLabel lab4;
    public String flrType;
    public double area = 0;
    public double cost = 0;

    FloorsRUs(String title)
    {
        super(title);
        jTP=new JTabbedPane();
        flrPan=new JPanel();
        lwPan=new JPanel();
        cPan=new JPanel();

        flrPan.add(flrWood=new JRadioButton("Choose Wood"));
        flrPan.add(flrCarpet=new JRadioButton("Choose Carpet"));

        ButtonGroup grp = new ButtonGroup();
        grp.add(flrWood);
        grp.add(flrCarpet);

        lwPan.add(lab1=new JLabel("Enter the Length : "));
        lwPan.add(tLength=new JTextField(10));

        lwPan.add(lab2=new JLabel("Enter the Width"));
        lwPan.add(tWidth=new JTextField(10));

        lwPan.add(calcArea=new JButton("Calculate the Area"));
        lwPan.add(calcCost=new JButton("Calculate the Cost"));
        lwPan.add(subOrder=new JButton("Submit the Order"));

        cPan.add(lab3=new JLabel("Enter the Customers Name : "));
        cPan.add(tCName=new JTextField(10));

        cPan.add(lab4=new JLabel("Enter the Customers Address : "));
        cPan.add(tAddress=new JTextField(10));

        cPan.add(orderSum=new JButton("The Order Summary"));
        cPan.add(ordList=new JButton("The Order List"));

        jTP.addTab("Select Floor Type ",flrPan);
        jTP.addTab("Enter Length and Width ",lwPan);
        jTP.addTab("Customer Info",cPan);
        add(jTP);

        calcArea.addActionListener(this);
        calcCost.addActionListener(this);
        subOrder.addActionListener(this);
        orderSum.addActionListener(this);
        ordList.addActionListener(this);
        flrWood.addItemListener(this);
        flrCarpet.addItemListener(this);

    }
    @Override
    public void itemStateChanged(ItemEvent ie)

    {
        if(flrWood.isSelected()){
            flrType="Wood";

        }

        else if(flrCarpet.isSelected()){
            flrType="Carpet";
        }

    }

    @Override
    public void actionPerformed(ActionEvent ae)
    {

        String source=ae.getActionCommand();

        switch (source) {

        case "Calculate the Area":
            System.out.println("Calc 1 Complete");
            area=Double.parseDouble(tLength.getText())
            *Double.parseDouble(tWidth.getText());
            break;

        case "Calculate the Cost":
            System.out.println("Calc 2 Complete");
            if(flrType.equals("Wood"))
            cost = area *20;
            else if(flrType.equals("Carpet"))
            cost = area *10;
            break;

        case "The Order Summary":
            JOptionPane.showMessageDialog(null,"The Order Summary - Area is :" +
            ""+area+""+ " The Cost is :  "+cost," Information",
            JOptionPane.INFORMATION_MESSAGE);
            break;

        case "The Order List":
            JOptionPane.showMessageDialog(null,"The Order List : \n Customer's name :
"+tCName.getText()
            +"\nAddress : "+tAddress.getText()
            +"\n The Area is :"+area
            +"\n The Cost is : "+cost,"Information",
            JOptionPane.INFORMATION_MESSAGE);
            break;

        default:

            break;
        }

    }

    public static void main(String args[])

    {
        FloorsRUs f1=new FloorsRUs("FloorsRUs Easy App");
        f1.setSize(new Dimension(650,450));
        f1.setVisible(true);
    }

}
</code>```

I've tried:

```<code>if(tLength.getText()<=0 && tWidth.getText()<=0)
{
    System.out.println("invalid input, number has to be positive");
}
</code>```

And it received errors for  "Bad operand for binary operator '<='". I believe because tLength and tWidth are both String.

So then I tried:

```<code>case "Calculate the Area":
System.out.println("Calc 1 Complete");
area=Double.parseDouble(tLength.getText())
*Double.parseDouble(tWidth.getText());
if((Double.parseDouble(tLength.getText())<=0) &&
((Double.parseDouble(tWidth.getText())<=0)))
{
   System.out.println("invalid input, number has to be positive");
}
break;
</code>```

It accepted it but didn't change the output when I put a 0 or negative number into the tLength or tWidth field.

So finally I tried:

```<code>case "Calculate the Area":
        System.out.println("Calc 1 Complete");
        try {
            if ((Double.parseDouble(tLength.getText()) < 0)
                    && ((Double.parseDouble(tWidth.getText()) < 0))) {
                System.out.println("invalid input, number has to be
positive");
            } else {
                area = Double.parseDouble(tLength.getText()) *
Double.parseDouble(tWidth.getText());
            }

        } catch (NumberFormatException e) {
            System.out.println("invalid input, number has to be positive");
        }
        break;
</code>```

Also, no errors but it didn't work when negative numbers or 0 were entered.

I have tried switching '&&' to '||' as well and no luck.

答:
I ended up using a JOptionPane and used this code which worked. Thank you all for the help though, I appreciate it.</p>

```<code> String source = ae.getActionCommand();
    switch (source) {
        case "Calculate the Area":
            System.out.println("Calc 1 Complete");
            try {
                if ((Double.parseDouble(tLength.getText()) <= 0)
                        || ((Double.parseDouble(tWidth.getText()) <= 0))) {
                    JOptionPane.showMessageDialog(null, "invalid input,
             number has to be positive", "Error", JOptionPane.ERROR_MESSAGE);
                } else {
                    area = Double.parseDouble(tLength.getText()) *
                           Double.parseDouble(tWidth.getText());
                }
            } catch (NumberFormatException e) {
                JOptionPane.showMessageDialog(null, "invalid input, number
                 has to be positive", "Error", JOptionPane.ERROR_MESSAGE);
            }
            break;

        case "Calculate the Cost":
            System.out.println("Calc 2 Complete");
            if (flrType.equals("Wood")) {
                cost = area * 20;
            } else if (flrType.equals("Carpet")) {
                cost = area * 10;
            }
            break;
</code>```

Title: Input Validation for GUI

Body:
My program works fine as is, I just need to add an input validation "invalid input, number has to be positive" dialog, after an input of 0 or a negative number for length: tLength and width: tWidth.

Here is my code which runs fine, just needs the validation added:

import java.awt.event.*;
import javax.swing.*;
import java.awt.*;

public class FloorsRUs extends JFrame implements ActionListener, ItemListener
{

    public JTabbedPane jTP;
    public JRadioButton flrWood = null;
    public JRadioButton flrCarpet = null;
    public JTextField tCName;
    public JTextField tAddress;
    public JTextField tLength;
    public JTextField tWidth;
    public JButton calcArea;
    public JButton calcCost;
    public JButton subOrder;
    public JButton orderSum;
    public JButton ordList;
    public JPanel flrPan;
    public JPanel lwPan;
    public JPanel cPan;
    public JLabel lab1;
    public JLabel lab2;
    public JLabel lab3;
    public JLabel lab4;
    public String flrType;
    public double area = 0;
    public double cost = 0;

    FloorsRUs(String title)
    {
        super(title);
        jTP=new JTabbedPane();
        flrPan=new JPanel();
        lwPan=new JPanel();
        cPan=new JPanel();

        flrPan.add(flrWood=new JRadioButton("Choose Wood"));
        flrPan.add(flrCarpet=new JRadioButton("Choose Carpet"));

        ButtonGroup grp = new ButtonGroup();
        grp.add(flrWood);
        grp.add(flrCarpet);

        lwPan.add(lab1=new JLabel("Enter the Length : "));
        lwPan.add(tLength=new JTextField(10));

        lwPan.add(lab2=new JLabel("Enter the Width"));
        lwPan.add(tWidth=new JTextField(10));

        lwPan.add(calcArea=new JButton("Calculate the Area"));
        lwPan.add(calcCost=new JButton("Calculate the Cost"));
        lwPan.add(subOrder=new JButton("Submit the Order"));

        cPan.add(lab3=new JLabel("Enter the Customers Name : "));
        cPan.add(tCName=new JTextField(10));

        cPan.add(lab4=new JLabel("Enter the Customers Address : "));
        cPan.add(tAddress=new JTextField(10));

        cPan.add(orderSum=new JButton("The Order Summary"));
        cPan.add(ordList=new JButton("The Order List"));

        jTP.addTab("Select Floor Type ",flrPan);
        jTP.addTab("Enter Length and Width ",lwPan);
        jTP.addTab("Customer Info",cPan);
        add(jTP);

        calcArea.addActionListener(this);
        calcCost.addActionListener(this);
        subOrder.addActionListener(this);
        orderSum.addActionListener(this);
        ordList.addActionListener(this);
        flrWood.addItemListener(this);
        flrCarpet.addItemListener(this);

    }
    @Override
    public void itemStateChanged(ItemEvent ie)

    {
        if(flrWood.isSelected()){
            flrType="Wood";

        }

        else if(flrCarpet.isSelected()){
            flrType="Carpet";
        }

    }

    @Override
    public void actionPerformed(ActionEvent ae)
    {

        String source=ae.getActionCommand();

        switch (source) {

        case "Calculate the Area":
            System.out.println("Calc 1 Complete");
            area=Double.parseDouble(tLength.getText())
            *Double.parseDouble(tWidth.getText());
            break;

        case "Calculate the Cost":
            System.out.println("Calc 2 Complete");
            if(flrType.equals("Wood"))
            cost = area *20;
            else if(flrType.equals("Carpet"))
            cost = area *10;
            break;

        case "The Order Summary":
            JOptionPane.showMessageDialog(null,"The Order Summary - Area is :" +
            ""+area+""+ " The Cost is :  "+cost," Information",
            JOptionPane.INFORMATION_MESSAGE);
            break;

        case "The Order List":
            JOptionPane.showMessageDialog(null,"The Order List : \n Customer's name :
"+tCName.getText()
            +"\nAddress : "+tAddress.getText()
            +"\n The Area is :"+area
            +"\n The Cost is : "+cost,"Information",
            JOptionPane.INFORMATION_MESSAGE);
            break;

        default:

            break;
        }

    }

    public static void main(String args[])

    {
        FloorsRUs f1=new FloorsRUs("FloorsRUs Easy App");
        f1.setSize(new Dimension(650,450));
        f1.setVisible(true);
    }

}

I've tried:

if(tLength.getText()<=0 && tWidth.getText()<=0)
{
    System.out.println("invalid input, number has to be positive");
}

And it received errors for  "Bad operand for binary operator '<='". I believe because tLength and tWidth are both String.

So then I tried:

case "Calculate the Area":
System.out.println("Calc 1 Complete");
area=Double.parseDouble(tLength.getText())
*Double.parseDouble(tWidth.getText());
if((Double.parseDouble(tLength.getText())<=0) &&
((Double.parseDouble(tWidth.getText())<=0)))
{
   System.out.println("invalid input, number has to be positive");
}
break;

It accepted it but didn't change the output when I put a 0 or negative number into the tLength or tWidth field.

So finally I tried:

case "Calculate the Area":
        System.out.println("Calc 1 Complete");
        try {
            if ((Double.parseDouble(tLength.getText()) < 0)
                    && ((Double.parseDouble(tWidth.getText()) < 0))) {
                System.out.println("invalid input, number has to be
positive");
            } else {
                area = Double.parseDouble(tLength.getText()) *
Double.parseDouble(tWidth.getText());
            }

        } catch (NumberFormatException e) {
            System.out.println("invalid input, number has to be positive");
        }
        break;

Also, no errors but it didn't work when negative numbers or 0 were entered.

I have tried switching '&&' to '||' as well and no luck.

Body:
I ended up using a JOptionPane and used this code which worked. Thank you all for the help though, I appreciate it.

 String source = ae.getActionCommand();
    switch (source) {
        case "Calculate the Area":
            System.out.println("Calc 1 Complete");
            try {
                if ((Double.parseDouble(tLength.getText()) <= 0)
                        || ((Double.parseDouble(tWidth.getText()) <= 0))) {
                    JOptionPane.showMessageDialog(null, "invalid input,
             number has to be positive", "Error", JOptionPane.ERROR_MESSAGE);
                } else {
                    area = Double.parseDouble(tLength.getText()) *
                           Double.parseDouble(tWidth.getText());
                }
            } catch (NumberFormatException e) {
                JOptionPane.showMessageDialog(null, "invalid input, number
                 has to be positive", "Error", JOptionPane.ERROR_MESSAGE);
            }
            break;

        case "Calculate the Cost":
            System.out.println("Calc 2 Complete");
            if (flrType.equals("Wood")) {
                cost = area * 20;
            } else if (flrType.equals("Carpet")) {
                cost = area * 10;
            }
            break;


问:
How to replace several words in javascript
I wanna replace several words in a text using replace() in javascript, how can I do that?

For example, if I wanna replace, 'Dave Chambers, David Chambers, Will Smith' with 'Jackie Chan', no matter if they're in upper-case or lower-case, do I have to keep repeating the replace() method on the same string variable everytime, i.e.

```<code>var str = sometext.innerHTML;
str.replace('Dave Chambers', 'Jackie Chan');
str.replace('David Chambers', 'Jackie Chan');
str.replace('Will Smith', 'Jackie Chan');
</code>```

答:
If you want to make some array to array replace, respecting the separators like "." or ",", I have created something like that that may help you.

function arrayReplace( text, arrFrom, arrTo, caseSensitive ) {
  return text.
    replace(/</g," <<").
    replace(/>/g,">> ").
    replace(/([\.\;\,])/g," <$1> ").
    split(" ").
    map(
      function(value) {
        var pos = arrFrom.indexOf( caseSensitive ? value : value.toLowerCase() );
        if( pos == -1 ) {
          return value;
        } else {
          return arrTo[pos % arrTo.length];
        }
      }
    ).
    join(" ").
    replace(/( \<|\> )/g,"");
};

console.log(
  arrayReplace(
    "First example. Trivial case",
    [ "example", "case"],
    [ "demo", "test" ]
  )
); // First demo. Trivial test

console.log(
  arrayReplace(
    "Leaving earth, passing close to the sun, going to the moon.",
    [ "earth", "sun", "moon"],
    [ "EARTH", "SUN", "MOON"]
  )
); // Leaving EARTH, passing close to the SUN, going to the MOON.

console.log(
  arrayReplace(
    "Leaving earth, passing close to the sun, going to the moon.",
    [ "earth", "sun", "moon"],
    [ "PLANET"]
  )
); // Leaving PLANET, passing close to the PLANET, going to the PLANET.

console.log(
  arrayReplace(
    "Leaving earth, passing close to the sun, going to the moon.",
    [ "earth", "sun", "moon"],
    [ "PLANET", "STAR" ]
  )
); // Leaving PLANET, passing close to the STAR, going to the PLANET.

console.log(
  arrayReplace(
    "Rain rain, goes away, no one wants you any way.",
    [ "rain", "a"],
    [ "pig", "x"]
  )
);
// pig pig, goes away, no one wants you any way.

console.log(
  arrayReplace(
    "Testing the <<function>>. Replacing, in <any> case. Even <the ;funny; ones>.",
    [ "funny", "even", "function" ],
    [ "complex", "including", "code" ]
  )
); // Testing the <>. Replacing, in <any> case. including <the ;complex; ones>.```

Body:
I wanna replace several words in a text using replace() in javascript, how can I do that?

For example, if I wanna replace, 'Dave Chambers, David Chambers, Will Smith' with 'Jackie Chan', no matter if they're in upper-case or lower-case, do I have to keep repeating the replace() method on the same string variable everytime, i.e.

var str = sometext.innerHTML;
str.replace('Dave Chambers', 'Jackie Chan');
str.replace('David Chambers', 'Jackie Chan');
str.replace('Will Smith', 'Jackie Chan');

Body:
If you want to make some array to array replace, respecting the separators like "." or ",", I have created something like that that may help you.

function arrayReplace( text, arrFrom, arrTo, caseSensitive ) {
  return text.
    replace(/</g," <<").
    replace(/>/g,">> ").
    replace(/([\.\;\,])/g," <$1> ").
    split(" ").
    map(
      function(value) {
        var pos = arrFrom.indexOf( caseSensitive ? value : value.toLowerCase() );
        if( pos == -1 ) {
          return value;
        } else {
          return arrTo[pos % arrTo.length];
        }
      }
    ).
    join(" ").
    replace(/( \<|\> )/g,"");
};

console.log(
  arrayReplace(
    "First example. Trivial case",
    [ "example", "case"],
    [ "demo", "test" ]
  )
); // First demo. Trivial test

console.log(
  arrayReplace(
    "Leaving earth, passing close to the sun, going to the moon.",
    [ "earth", "sun", "moon"],
    [ "EARTH", "SUN", "MOON"]
  )
); // Leaving EARTH, passing close to the SUN, going to the MOON.

console.log(
  arrayReplace(
    "Leaving earth, passing close to the sun, going to the moon.",
    [ "earth", "sun", "moon"],
    [ "PLANET"]
  )
); // Leaving PLANET, passing close to the PLANET, going to the PLANET.

console.log(
  arrayReplace(
    "Leaving earth, passing close to the sun, going to the moon.",
    [ "earth", "sun", "moon"],
    [ "PLANET", "STAR" ]
  )
); // Leaving PLANET, passing close to the STAR, going to the PLANET.

console.log(
  arrayReplace(
    "Rain rain, goes away, no one wants you any way.",
    [ "rain", "a"],
    [ "pig", "x"]
  )
);
// pig pig, goes away, no one wants you any way.

console.log(
  arrayReplace(
    "Testing the <<function>>. Replacing, in <any> case. Even <the ;funny; ones>.",
    [ "funny", "even", "function" ],
    [ "complex", "including", "code" ]
  )
); // Testing the <<code>>. Replacing, in <any> case. including <the ;complex; ones>.


问:
Docker: disable caching for a specific stage
I have a multi-stage Dockerfile. In stage one, I git clone from a github repo. In later stage, I do other stuff like pip etc and use a file from stage 1. I'd like to only disable caching for the first stage.

It looks like docker build --target stage1 --no-cache doesn't do what I want.

Is there a way to disable only a certain stage?

My Dockerfile looks like this:

```<code>FROM yijian/git-alpine
WORKDIR /tmp
RUN git clone https://github.com/abc/abc.git

FROM python:3.5.3-slim

RUN mkdir /app
ADD requirements.txt /app
ADD pip/pip.conf /root/.pip/pip.conf
WORKDIR /app
RUN pip3 install --upgrade pip && \
    pip3 install pbr && \
    pip3 install -r requirements.txt
ADD server.py /app
ADD docker/start.sh /app
RUN chmod a+x /app/start.sh
COPY --from=0 /tmp/abc/directory /usr/local/lib/python3.5/site-packages/abc/directory

EXPOSE 9092
ENTRYPOINT ["./start.sh"]
</code>```

答:
I don't believe that a single Dockerfile can have caching disabled for a specific state.  That might make a nice feature request but I would rather see that as a declarative statement in the file rather than on the command line.

According to Docker's reference site:
https://docs.docker.com/engine/reference/commandline/build/#usage

The "--target" flag allows you to select a target stage from a Dockerfile, meaning that it would only run that part of the Dockerfile.  I would expect that the --no-cache flag would work in conjunction with this flag, however I wouldn't expect the other sections of the Dockerfile to run.

I believe that what you want to occur would take multiple commands which may defeat the purpose of having a multistage Dockerfile.

It would take more work, but depending on what you want to cache, you could possible include a script, such as bash or powershell, which can accomplish this goal.

Another option (depending on your needs) may be to use a separate Docker container which caches just what you need.  For instance, I created a CI build which uses a Dockerfile that only imports dependencies and then my main build happens in a container that references that first container.  I have done this with "dotnet restore" commands, so that the dependencies are preloaded and also have done this using "npm install".  This method would work with any package management tool which allows you to specify a source.  so where you have a project.json, you can extract the common dependencies and call it cache.package.json, then build a base image that has already done the heavy downloading for you, then ideally when you run this again during your more frequent builds it needs to pull less.  Take advantage of the layered approach Docker offers!

Body:
I have a multi-stage Dockerfile. In stage one, I git clone from a github repo. In later stage, I do other stuff like pip etc and use a file from stage 1. I'd like to only disable caching for the first stage.

It looks like docker build --target stage1 --no-cache doesn't do what I want.

Is there a way to disable only a certain stage?

My Dockerfile looks like this:

FROM yijian/git-alpine
WORKDIR /tmp
RUN git clone https://github.com/abc/abc.git

FROM python:3.5.3-slim

RUN mkdir /app
ADD requirements.txt /app
ADD pip/pip.conf /root/.pip/pip.conf
WORKDIR /app
RUN pip3 install --upgrade pip && \
    pip3 install pbr && \
    pip3 install -r requirements.txt
ADD server.py /app
ADD docker/start.sh /app
RUN chmod a+x /app/start.sh
COPY --from=0 /tmp/abc/directory /usr/local/lib/python3.5/site-packages/abc/directory

EXPOSE 9092
ENTRYPOINT ["./start.sh"]


问:
angular component replacement feature
I am trying to use the angular component replacement feature.

```<code><ul>
  <li></li>
  <li class="com"></li>
  <li class="com"></li>
</ul>
</code>```

For complicated interface, if I want to make the middle .com "li" into a whole new component, and write this code like:

```<code><ul>
  <li></li>
  <com-component></com-component>
</ul>
</code>```

How can I make sure that the finally compiled html is exactly what the original html template looks like?

It is not possible as far as I have searched from angular docs.

In angularjs, there is replacement feature. For a "compiled" angular framework, this is bit of off-putting not to have this.

答:
View Encapsulation is what you need as you want the styles to bleed in from the parent component to the child component.

ViewEncapsulation.None - No Shadow DOM at all. Therefore, also no
  style encapsulation. ViewEncapsulation.Emulated - No Shadow DOM but
  style encapsulation emulation. ViewEncapsulation.Native - Native
  Shadow DOM with all it’s goodness.

More on View Encapsulation

Title: angular component replacement feature

Body:
I am trying to use the angular component replacement feature.

For complicated interface, if I want to make the middle .com "li" into a whole new component, and write this code like:

  <com-component></com-component>

How can I make sure that the finally compiled html is exactly what the original html template looks like?

It is not possible as far as I have searched from angular docs.

In angularjs, there is replacement feature. For a "compiled" angular framework, this is bit of off-putting not to have this.


问:
I have a mysterious PHP SOAP error on my host, but can't duplicate locally
I've run across an interesting PHP/SOAP error that has me stymied. After searching I have not found a plausible explanation, and I'd appreciate your help. Here's the background:

I have a site built in PHP/CodeIgniter which uses SOAP to communicate over SSL with a back-end system provided by a third party (let's call them "Company X" to protect the innocent) not in my control. In the spirit of good MVC, I put the code specific to interacting with that data source in a separate model (system/application/models/company_x.php). I have been developing locally using MAMP on my Mac, and just about everything was relatively smooth through testing and development; including calling Company X's web service over SSL. I should probably mention that their web service had strange WSDL that PHP 5's SOAP didn't like. Things like required parameters that were not there. It's been a little odd to call SOAP methods very explicitly, but I got it going and it worked through testing. I even deployed it to a test site at Mosso, and I could have sworn that it worked for a time up there too.

Imagine my surprise when every call to the SOAP web service starting producing errors like the following:

```
    A PHP Error was encountered
    Severity: Warning
    Message: SoapClient::__doRequest() [soapclient.--dorequest]:
    WARNING: URL fopen access
    Filename: models/company_x.php
    Line Number: 86
```

The error logs didn't give any more information other than the full path to the model file on the server. It works locally, and I thought it worked on Mosso before. Maybe Mosso changed their settings, and disabled SOAP or something. A little phpinfo() later, and they have more than enough. I thought that maybe my Mac is more tolerant of the SSL certificate. After all, it's a GoDaddy *.domain.com cert, maybe fopen is having trouble getting through. I whipped up a test file to connect over SSL, and put it on Mosso and it worked.

I'm wondering why is fopen access suddenly a problem for SOAP? What is it about Mosso that is making this difficult suddenly? Do I need to override some php.ini setting? Could it be, like is so often with unhelpful errors like this, something completely different?

Update: here is the configure command from phpinfo():

'./configure' '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/usr/com' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--cache-file=../config.cache' '--with-libdir=lib64' '--with-config-file-path=/etc' '--with-config-file-scan-dir=/etc/php.d' '--disable-debug' '--with-pic' '--disable-rpath' '--without-pear' '--with-bz2' '--with-curl' '--with-exec-dir=/usr/bin' '--with-freetype-dir=/usr' '--with-png-dir=/usr' '--enable-gd-native-ttf' '--without-gdbm' '--with-gettext' '--with-gmp' '--with-iconv' '--with-jpeg-dir=/usr' '--with-openssl' '--with-png' '--with-pspell' '--with-expat-dir=/usr' '--with-pcre-regex=/usr' '--with-zlib' '--with-layout=GNU' '--enable-exif' '--enable-ftp' '--enable-magic-quotes' '--enable-sockets' '--enable-sysvsem' '--enable-sysvshm' '--enable-sysvmsg' '--enable-track-vars' '--enable-trans-sid' '--enable-yp' '--enable-wddx' '--with-kerberos' '--enable-ucd-snmp-hack' '--with-unixODBC=shared,/usr' '--enable-memory-limit' '--enable-shmop' '--enable-calendar' '--enable-dbx' '--enable-dio' '--with-ming' '--with-mime-magic' '--with-sqlite=shared' '--with-libxml-dir=/usr' '--with-xml' '--with-system-tzdata' '--with-apxs2=/usr/sbin/apxs' '--without-mysql' '--without-gd' '--without-odbc' '--disable-dom' '--disable-dba' '--without-unixODBC' '--disable-pdo' '--disable-xmlreader' '--disable-xmlwriter' '--disable-json'

Note: My comment below that phpinfo() reports that furl is allowed; weird!

答: OK, a little update on this problem: the error mysteriously disappeared without any change from me. I will have to assume that there was some odd configuration error on Mosso that was silently fixed. It is also remotely possible that there was a problem with Company X's setup. That is the worst kind of fix IMO! Thanks anyway to everyone who looked in on this.

OwnerDisplayName: Barnabas Kendall

Body:
I've run across an interesting PHP/SOAP error that has me stymied. After searching I have not found a plausible explanation, and I'd appreciate your help. Here's the background:

I have a site built in PHP/CodeIgniter which uses SOAP to communicate over SSL with a back-end system provided by a third party (let's call them "Company X" to protect the innocent) not in my control. In the spirit of good MVC, I put the code specific to interacting with that data source in a separate model (system/application/models/company_x.php). I have been developing locally using MAMP on my Mac, and just about everything was relatively smooth through testing and development; including calling Company X's web service over SSL. I should probably mention that their web service had strange WSDL that PHP 5's SOAP didn't like. Things like required parameters that were not there. It's been a little odd to call SOAP methods very explicitly, but I got it going and it worked through testing. I even deployed it to a test site at Mosso, and I could have sworn that it worked for a time up there too.

Imagine my surprise when every call to the SOAP web service starting producing errors like the following:

    A PHP Error was encountered
    Severity: Warning
    Message: SoapClient::__doRequest() [soapclient.--dorequest]:
    WARNING: URL fopen access
    Filename: models/company_x.php
    Line Number: 86

The error logs didn't give any more information other than the full path to the model file on the server. It works locally, and I thought it worked on Mosso before. Maybe Mosso changed their settings, and disabled SOAP or something. A little phpinfo() later, and they have more than enough. I thought that maybe my Mac is more tolerant of the SSL certificate. After all, it's a GoDaddy *.domain.com cert, maybe fopen is having trouble getting through. I whipped up a test file to connect over SSL, and put it on Mosso and it worked.

I'm wondering why is fopen access suddenly a problem for SOAP? What is it about Mosso that is making this difficult suddenly? Do I need to override some php.ini setting? Could it be, like is so often with unhelpful errors like this, something completely different?

Update: here is the configure command from phpinfo():

'./configure' '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/usr/com' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--cache-file=../config.cache' '--with-libdir=lib64' '--with-config-file-path=/etc' '--with-config-file-scan-dir=/etc/php.d' '--disable-debug' '--with-pic' '--disable-rpath' '--without-pear' '--with-bz2' '--with-curl' '--with-exec-dir=/usr/bin' '--with-freetype-dir=/usr' '--with-png-dir=/usr' '--enable-gd-native-ttf' '--without-gdbm' '--with-gettext' '--with-gmp' '--with-iconv' '--with-jpeg-dir=/usr' '--with-openssl' '--with-png' '--with-pspell' '--with-expat-dir=/usr' '--with-pcre-regex=/usr' '--with-zlib' '--with-layout=GNU' '--enable-exif' '--enable-ftp' '--enable-magic-quotes' '--enable-sockets' '--enable-sysvsem' '--enable-sysvshm' '--enable-sysvmsg' '--enable-track-vars' '--enable-trans-sid' '--enable-yp' '--enable-wddx' '--with-kerberos' '--enable-ucd-snmp-hack' '--with-unixODBC=shared,/usr' '--enable-memory-limit' '--enable-shmop' '--enable-calendar' '--enable-dbx' '--enable-dio' '--with-ming' '--with-mime-magic' '--with-sqlite=shared' '--with-libxml-dir=/usr' '--with-xml' '--with-system-tzdata' '--with-apxs2=/usr/sbin/apxs' '--without-mysql' '--without-gd' '--without-odbc' '--disable-dom' '--disable-dba' '--without-unixODBC' '--disable-pdo' '--disable-xmlreader' '--disable-xmlwriter' '--disable-json'

Note: My comment below that phpinfo() reports that furl is allowed; weird!

LastEditorDisplayName: Barnabas Kendall


问:
multiple Session requests getting overwritten in nodejs application
I am trying to implement session using nodejs. However i am unable to handle multiple requests in sessions.

Example:

If user1 login, data loads.

If user2 login, user 1 data is getting overwritten with user2.

So on refreshing user1 page, again user2 data getting loaded.

```<code>var session = require('express-session');

app.use(session({
secret: "secterd13",
resave: false,
saveUninitialized: true,
cookie: { secure: true }
}));

//Set the session data in login page.
sessionObj = req.session;
sessionObj.Location = "Location1";

//Get in dashboard.

console.log(sessionObj.Location);
</code>```

答: Where you do sessionObj = req.session you are creating a global variable and reassigning it on every user. The last user to connect to your server will always overwrite this. If you plan to pass the session data to a template engine, you can use 'res.render('view', { session: req.session })simply and the session variable will be available in your template. Or you can set the session as global usingres.locals.myVariable = req.sessionandmyVariable` will be available in your template.

Body:
I am trying to implement session using nodejs. However i am unable to handle multiple requests in sessions.

Example:

If user1 login, data loads.

If user2 login, user 1 data is getting overwritten with user2.

So on refreshing user1 page, again user2 data getting loaded.

var session = require('express-session');

app.use(session({
secret: "secterd13",
resave: false,
saveUninitialized: true,
cookie: { secure: true }
}));

//Set the session data in login page.
sessionObj = req.session;
sessionObj.Location = "Location1";

//Get in dashboard.

console.log(sessionObj.Location);


问:
Azure Logic app and custom DNS
Is is possible, and if so how, to have a custom dns point to my Azure logic app url?  For instance, I have 2 logic apps, one sends an email and one is an unsubscribe logic app.  There is a link in the email to point to my unsubscribe logic app. The URL is not user friendly.  Is there a way to create a DNS alias to point with a more fitting url like https://something.mysiteaddress.com insead of https://azurewebsiteurl.com?</p>

答: You can initiate your logic app from your website. When someone hit your URL call logic app url from that URL.</p>

Title: Azure Logic app and custom DNS


问:
Table of means and sds using dply
Is there a way to print out grouped means, and sds using dplyr in R(i.e. what one would observe in many randomised controlled trial reports where baseline demographics are reported for the placebo and drug arms). At the moment I have:

```<code>group_summary<- df %>%
  group_by(group) %>%
  summarise_at( vars(age, iq), c(mean, sd), na.rm=TRUE)

t(group_summary)
</code>```

which places the sds under the means:

```<code>                                [,1]       [,2]       [,3]
group                           "drug1"     "drug2"       "placebo"
age (x, ...) ...             "22.72436" "25.06164" "23.54394"
weight (x, ...) ...           "57.97143" NA         "64.66667"
age (x, na.rm = FALSE) ...   "3.793276" "3.930634" "4.623405"
weight (x, na.rm = FALSE) ... "16.03617" NA         "15.44099"
</code>```

when ideally what I would like is something more along the lines of:

```<code>                            [,1]       [,2]       [,3]
group                           "drug1"     "drug2"       "placebo"
age (x, ...) ...             "22.7(3.8)" "25.1(3.9)" "23.5(4.6)"
weight (x, ...) ...           "56.0(16.0)" NA         "64.7(15.4)"
</code>```

Grateful for any advice, although a tidyverse method would be ideal if possible

答:
You just have to build on @Pete's answer. If you want to do this for several variables say a, b, c and d, do this..

```<code>library(tidyverse)

df%>%
   group_by(grouping.variable)%>%
     summarise_at(vars(a,b,c,d), funs(paste0(format(round(mean(., na.rm = T), y),
                                                nsmall = y),"(",format(round(sd(., na.rm = T), y), nsmall = y) ,")")))%>%
gather(Variable, Value, -grouping.variable)%>%
spread(grouping.variable, Value)
</code>```

I don't quite like how t() works. I like to use gather and spread from the tidyr package.

I hope this gives you what you want.

Title: Table of means and sds using dply

Body:
Is there a way to print out grouped means, and sds using dplyr in R(i.e. what one would observe in many randomised controlled trial reports where baseline demographics are reported for the placebo and drug arms). At the moment I have:

group_summary<- df %>%
  group_by(group) %>%
  summarise_at( vars(age, iq), c(mean, sd), na.rm=TRUE)

t(group_summary)

which places the sds under the means:

                                [,1]       [,2]       [,3]
group                           "drug1"     "drug2"       "placebo"
age (x, ...) ...             "22.72436" "25.06164" "23.54394"
weight (x, ...) ...           "57.97143" NA         "64.66667"
age (x, na.rm = FALSE) ...   "3.793276" "3.930634" "4.623405"
weight (x, na.rm = FALSE) ... "16.03617" NA         "15.44099"

when ideally what I would like is something more along the lines of:

                            [,1]       [,2]       [,3]
group                           "drug1"     "drug2"       "placebo"
age (x, ...) ...             "22.7(3.8)" "25.1(3.9)" "23.5(4.6)"
weight (x, ...) ...           "56.0(16.0)" NA         "64.7(15.4)"

Grateful for any advice, although a tidyverse method would be ideal if possible

Body:
You just have to build on @Pete's answer. If you want to do this for several variables say a, b, c and d, do this..

library(tidyverse)

df%>%
   group_by(grouping.variable)%>%
     summarise_at(vars(a,b,c,d), funs(paste0(format(round(mean(., na.rm = T), y),
                                                nsmall = y),"(",format(round(sd(., na.rm = T), y), nsmall = y) ,")")))%>%
gather(Variable, Value, -grouping.variable)%>%
spread(grouping.variable, Value)

I don't quite like how t() works. I like to use gather and spread from the tidyr package.

I hope this gives you what you want.