问:
Error: Message: Too many sources provided: 15285. Limit is 10000
I'm currently trying to run a Dataflow (Apache Beam, Python SDK) task to import a >100GB Tweet file into BigQuery, but running into Error: Message: Too many sources provided: 15285. Limit is 10000.

The task takes the tweets (JSON), extracts 5 relevant fields, transforms/sanitizes them a bit with some transforms and then write those values into BigQuery, which will be used for further processing.

There's Cloud Dataflow to BigQuery - too many sources but it seems to be caused by having a lot of different input files, whereas I have a single input file, so it doesn't seem relevant. Also the solutions mentioned there are rather cryptic and I'm not sure if/how I could apply them to my problem.

My guess is that BigQuery writes temporary files for each row or something before persisting them, and that's what's meant by "too many sources" ?

How can I fix this?

[Edit]

Code:

```<code>import argparse
import json
import logging

import apache_beam as beam

class JsonCoder(object):
    """A JSON coder interpreting each line as a JSON string."""

    def encode(self, x):
        return json.dumps(x)

    def decode(self, x):
        return json.loads(x)

def filter_by_nonempty_county(record):
    if 'county_fips' in record and record['county_fips'] is not None:
        yield record

def run(argv=None):

    parser = argparse.ArgumentParser()
    parser.add_argument('--input',
                        default='...',
                        help=('Input twitter json file specified as: '
                            'gs://path/to/tweets.json'))
    parser.add_argument(
        '--output',
        required=True,
        help=
        ('Output BigQuery table for results specified as: PROJECT:DATASET.TABLE '
        'or DATASET.TABLE.'))

    known_args, pipeline_args = parser.parse_known_args(argv)

    p = beam.Pipeline(argv=pipeline_args)

    # read text file

    #Read all tweets from given source file
    read_tweets = "Read Tweet File" >> beam.io.ReadFromText(known_args.input, coder=JsonCoder())

    #Extract the relevant fields of the source file
    extract_fields =  "Project relevant fields" >> beam.Map(lambda row: {'text': row['text'],
                                                                  'user_id': row['user']['id'],
                                                                  'location': row['user']['location'] if 'location' in row['user'] else None,
                                                                  'geo':row['geo'] if 'geo' in row else None,
                                                                  'tweet_id': row['id'],
                                                                  'time': row['created_at']})

    #check what type of geo-location the user has
    has_geo_location_or_not = "partition by has geo or not" >> beam.Partition(lambda element, partitions: 0 if element['geo'] is None else 1, 2)

    check_county_not_empty = lambda element, partitions: 1 if 'county_fips' in element and element['county_fips'] is not None else 0

    #tweet has coordinates partition or not
    coordinate_partition = (p
             | read_tweets
             | extract_fields
             | beam.ParDo(TimeConversion())
             | has_geo_location_or_not)

    #lookup by coordinates
    geo_lookup = (coordinate_partition[1] | "geo coordinates mapping" >> beam.ParDo(BeamGeoLocator())
                           | "filter successful geo coords" >> beam.Partition(check_county_not_empty, 2))

    #lookup by profile
    profile_lookup = ((coordinate_partition[0], geo_lookup[0])
                      | "join streams" >> beam.Flatten()
                      | "Lookup from profile location" >> beam.ParDo(ComputeLocationFromProfile())
                      )

    bigquery_output = "write output to BigQuery" >>  beam.io.Write(
       beam.io.BigQuerySink(known_args.output,
                   schema='text:STRING, user_id:INTEGER, county_fips:STRING, tweet_id:INTEGER, time:TIMESTAMP, county_source:STRING',
                  create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
                 write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE))

    #file_output = "write output" >> beam.io.WriteToText(known_args.output, coder=JsonCoder())

    output = ((profile_lookup, geo_lookup[1]) | "merge streams" >> beam.Flatten()
              | "Filter entries without location" >> beam.FlatMap(filter_by_nonempty_county)
              | "project relevant fields" >> beam.Map(lambda row: {'text': row['text'],
                                                                   'user_id': row['user_id'],
                                                                   'county_fips': row['county_fips'],
                                                                   'tweet_id': row['tweet_id'],
                                                                   'time': row['time'],
                                                                   'county_source': row['county_source']})
              | bigquery_output)

    result = p.run()
    result.wait_until_finish()

if __name__ == '__main__':
    logging.getLogger().setLevel(logging.DEBUG)
    run()
</code>```

It's a little bit complicated, so it would probably take too much time to do it in bigquery directly. The code reads the tweets json, splits the PCollection by whether it's geotagged or not, if not it tries to look it up via profile location, maps to location to what's relevant for our GIS analysis and then writes it to BigQuery.

答:
The number of files correspond to the number of shards the elements were processed in.

One trick to reducing this is to generate some random keys, and group the elements based on that before writing them out.

For example, you could use the following DoFn and PTransform in your pipeline:

```<code>class _RoundRobinKeyFn(beam.DoFn):
  def __init__(self, count):
    self.count = count

  def start_bundle(self):
    self.counter = random.randint(0, self.count - 1)

  def process(self, element):
    self.counter += 1
    if self.counter >= self.count:
      self.counter -= self.count
    yield self.counter, element

class LimitBundles(beam.PTransform):
  def __init__(self, count):
    self.count = count

  def expand(self, input):
    return input
        | beam.ParDo(_RoundRobinKeyFn(self.count))
        | beam.GroupByKey()
        | beam.FlatMap(lambda kv: kv[1])
</code>```

You would just use this before the bigquery_output:

```<code>output = (# ...
         | LimitBundles(10000)
         | bigquery_output)
</code>```

(Note that I just typed this in without testing it, so there are likely some Python typos.)

Body:
I'm currently trying to run a Dataflow (Apache Beam, Python SDK) task to import a >100GB Tweet file into BigQuery, but running into Error: Message: Too many sources provided: 15285. Limit is 10000.

The task takes the tweets (JSON), extracts 5 relevant fields, transforms/sanitizes them a bit with some transforms and then write those values into BigQuery, which will be used for further processing.

There's Cloud Dataflow to BigQuery - too many sources but it seems to be caused by having a lot of different input files, whereas I have a single input file, so it doesn't seem relevant. Also the solutions mentioned there are rather cryptic and I'm not sure if/how I could apply them to my problem.

My guess is that BigQuery writes temporary files for each row or something before persisting them, and that's what's meant by "too many sources" ?

How can I fix this?

[Edit]

Code:

import argparse
import json
import logging

import apache_beam as beam

class JsonCoder(object):
    """A JSON coder interpreting each line as a JSON string."""

    def encode(self, x):
        return json.dumps(x)

    def decode(self, x):
        return json.loads(x)

def filter_by_nonempty_county(record):
    if 'county_fips' in record and record['county_fips'] is not None:
        yield record

def run(argv=None):

    parser = argparse.ArgumentParser()
    parser.add_argument('--input',
                        default='...',
                        help=('Input twitter json file specified as: '
                            'gs://path/to/tweets.json'))
    parser.add_argument(
        '--output',
        required=True,
        help=
        ('Output BigQuery table for results specified as: PROJECT:DATASET.TABLE '
        'or DATASET.TABLE.'))

    known_args, pipeline_args = parser.parse_known_args(argv)

    p = beam.Pipeline(argv=pipeline_args)

    # read text file

    #Read all tweets from given source file
    read_tweets = "Read Tweet File" >> beam.io.ReadFromText(known_args.input, coder=JsonCoder())

    #Extract the relevant fields of the source file
    extract_fields =  "Project relevant fields" >> beam.Map(lambda row: {'text': row['text'],
                                                                  'user_id': row['user']['id'],
                                                                  'location': row['user']['location'] if 'location' in row['user'] else None,
                                                                  'geo':row['geo'] if 'geo' in row else None,
                                                                  'tweet_id': row['id'],
                                                                  'time': row['created_at']})

    #check what type of geo-location the user has
    has_geo_location_or_not = "partition by has geo or not" >> beam.Partition(lambda element, partitions: 0 if element['geo'] is None else 1, 2)

    check_county_not_empty = lambda element, partitions: 1 if 'county_fips' in element and element['county_fips'] is not None else 0

    #tweet has coordinates partition or not
    coordinate_partition = (p
             | read_tweets
             | extract_fields
             | beam.ParDo(TimeConversion())
             | has_geo_location_or_not)

    #lookup by coordinates
    geo_lookup = (coordinate_partition[1] | "geo coordinates mapping" >> beam.ParDo(BeamGeoLocator())
                           | "filter successful geo coords" >> beam.Partition(check_county_not_empty, 2))

    #lookup by profile
    profile_lookup = ((coordinate_partition[0], geo_lookup[0])
                      | "join streams" >> beam.Flatten()
                      | "Lookup from profile location" >> beam.ParDo(ComputeLocationFromProfile())
                      )

    bigquery_output = "write output to BigQuery" >>  beam.io.Write(
       beam.io.BigQuerySink(known_args.output,
                   schema='text:STRING, user_id:INTEGER, county_fips:STRING, tweet_id:INTEGER, time:TIMESTAMP, county_source:STRING',
                  create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
                 write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE))

    #file_output = "write output" >> beam.io.WriteToText(known_args.output, coder=JsonCoder())

    output = ((profile_lookup, geo_lookup[1]) | "merge streams" >> beam.Flatten()
              | "Filter entries without location" >> beam.FlatMap(filter_by_nonempty_county)
              | "project relevant fields" >> beam.Map(lambda row: {'text': row['text'],
                                                                   'user_id': row['user_id'],
                                                                   'county_fips': row['county_fips'],
                                                                   'tweet_id': row['tweet_id'],
                                                                   'time': row['time'],
                                                                   'county_source': row['county_source']})
              | bigquery_output)

    result = p.run()
    result.wait_until_finish()

if __name__ == '__main__':
    logging.getLogger().setLevel(logging.DEBUG)
    run()

It's a little bit complicated, so it would probably take too much time to do it in bigquery directly. The code reads the tweets json, splits the PCollection by whether it's geotagged or not, if not it tries to look it up via profile location, maps to location to what's relevant for our GIS analysis and then writes it to BigQuery.

Body:
The number of files correspond to the number of shards the elements were processed in.

One trick to reducing this is to generate some random keys, and group the elements based on that before writing them out.

For example, you could use the following DoFn and PTransform in your pipeline:

class _RoundRobinKeyFn(beam.DoFn):
  def __init__(self, count):
    self.count = count

  def start_bundle(self):
    self.counter = random.randint(0, self.count - 1)

  def process(self, element):
    self.counter += 1
    if self.counter >= self.count:
      self.counter -= self.count
    yield self.counter, element

class LimitBundles(beam.PTransform):
  def __init__(self, count):
    self.count = count

  def expand(self, input):
    return input
        | beam.ParDo(_RoundRobinKeyFn(self.count))
        | beam.GroupByKey()
        | beam.FlatMap(lambda kv: kv[1])

You would just use this before the bigquery_output:

output = (# ...
         | LimitBundles(10000)
         | bigquery_output)

(Note that I just typed this in without testing it, so there are likely some Python typos.)

ContentLicense: CC BY-SA 4.0


来源: stackexchange

问:
How to get the ip address of a particular adapter in windows xp through command
I have more than one adapter in my system and I want to know the IP of particular adapter. Is it possible through a command in Windows, in Linux it's easy? Or any other way?</p>

答:
The following will let you specify an adapter (something ipconfig doesn't appear to do):

```<code>netsh interface ip show config name="Local Area Connection"
</code>```

You can add | findstr "IP Address" to the end of that command to only show the IP address line.

ContentLicense: CC BY-SA 3.0

Body:
The following will let you specify an adapter (something ipconfig doesn't appear to do):

netsh interface ip show config name="Local Area Connection"

You can add | findstr "IP Address" to the end of that command to only show the IP address line.

ContentLicense: CC BY-SA 2.5


问:
Don't understand, IndexError: too many indices for array
My task is to remove longitude and latitude coordinates if the distance between points are within a specific distance (5km or 10km or 30km). This is for modeling purposes and avoid clustering of points. I am using haversine equation to measure distance.

Below is my initial code:

```<code>load the geometry record from points,
then convert it to an array,
compare each coordinate pairs and measure distance.
After that, remove the longitude and latitude pairs that are
close to each other,
</code>```

but got stuck on this step.

What I am planning is to update the item list of coordinate pairs and iterate again using new set of coordinate pairs.

Running the below script gives me this error:

IndexError: too many indices for array

It seems that the index in the iteration is not updating. It still getting the index during the first pass.

```<code>import math, easygui, shapefile, itertools, os
import pandas as pd
import numpy as np

filepath = easygui.fileopenbox()

input_dist = int(raw_input("Distance Filter Value?: "))
input_crop = raw_input("what crop?: ")

directory = os.path.split(filepath)[0]

def dist_haversine(shp,input_dist,input_crop):
    """
    Calculate the great circle distance between two points
    on the earth (specified in decimal degrees)
    """

    r = shapefile.Reader(shp)
    idx = np.arange(len(r.records()))
    coordinates = []
    for i in idx:
        geom = r.shape(i)
        coordinates.append(geom.points[0])

    acoords = np.array(coordinates)

    for r,n in itertools.izip(acoords[:,0],acoords[:,1]):

        coordinates_ = []

        for i,j in itertools.izip(acoords[:,0],acoords[:,1]):

            lon1=r
            lat1=n
            lon2=i
            lat2=j

            lon1, lat1, lon2, lat2 = map(math.radians, [lon1, lat1, lon2, lat2])

            # haversine formula
            dlon = lon2 - lon1
            dlat = lat2 - lat1
            a = math.sin(dlat/2)**2 + math.cos(lat1) * math.cos(lat2) * math.sin(dlon/2)**2
            c = 2 * math.asin(math.sqrt(a))
            km = c*6371 #/1000.0

            if km > input_dist:
                coordinates_.append([i,j])

        coordinates[:] = coordinates_
        acoords = np.array(coordinates)

    df_coords_ = pd.DataFrame(coordinates).drop_duplicates().values
    df_coords = pd.DataFrame(df_coords_, columns=['Lon','Lat'])

    df_coords.insert(0, 'Crop', input_crop)

    return df_coords.to_csv(os.path.split(directory)[0] + "\\" + "%s_distFilter_%skm.csv" % (input_crop, input_dist), sep=",", index=None)
</code>```

Traceback

```<code>File "<ipython-input-3-8e88eba2ab54>", line 1,  in <module>
  dist_haversine(filepath,input_dist,input_crop)
File "<ipython-input-2-d43a1f1da26a>", line 20, in
  dist_haversine
    for i,j in itertools.izip(acoords[:,0],acoords[:,1]):
IndexError: too many indices for array
</code>```

答:
This is the initial solution that I used to filter points. Its working, and a bit fast for a 1,000 - 3,000 points dataset. However, tried filtering 50,000 points, it took 2.5 - 3 hours to finish.</p>

```<code>def dist_haversine(filepath,input_dist,input_crop):
    """
    Calculate the great circle distance between two points
    on the earth (specified in decimal degrees)
    """

    r = shapefile.Reader(filepath)
    idx = np.arange(len(r.records()))
    coordinates = []
    for i in idx:
        geom = r.shape(i)
        coordinates.append(geom.points[0])

    acoords = np.array(coordinates)

    index = []
    for r,n,l in itertools.izip(acoords[:,0],acoords[:,1],idx):
        if l in index:
            continue
        else:
            for i,j,k in itertools.izip(acoords[:,0],acoords[:,1], idx):
                if k in index:
                    continue

                else:

                    lon1=r
                    lat1=n
                    lon2=i
                    lat2=j

                    coord_check = ((lon1 == lon2) & (lat1 == lat2))*1

                    if coord_check == 1:
                        continue

                    else:
                        lon1, lat1, lon2, lat2 = map(math.radians, [lon1, lat1, lon2, lat2])

                        # haversine formula
                        dlon = lon2 - lon1
                        dlat = lat2 - lat1
                        a = math.sin(dlat/2)**2 + math.cos(lat1) * math.cos(lat2) * math.sin(dlon/2)**2
                        c = 2 * math.asin(math.sqrt(a))
                        km = c*6371 #/1000.0

                    if km < input_dist:
                        if k in index:
                            continue
                        else:
                            index.append(k)

    filterList = [i for j, i in enumerate(coordinates) if j not in index]

    df_coords = pd.DataFrame(filterList, columns=['Lon','Lat'])

    df_coords.insert(0, 'Crop', input_crop)

    return df_coords.to_csv(directory + "\\" + "%s_distFilter_%skm.csv" % (input_crop, input_dist), sep=",", index=None)
</code>```

Body:
My task is to remove longitude and latitude coordinates if the distance between points are within a specific distance (5km or 10km or 30km). This is for modeling purposes and avoid clustering of points. I am using haversine equation to measure distance.

Below is my initial code:

load the geometry record from points,
then convert it to an array,
compare each coordinate pairs and measure distance.
After that, remove the longitude and latitude pairs that are
close to each other,

but got stuck on this step.

What I am planning is to update the item list of coordinate pairs and iterate again using new set of coordinate pairs.

Running the below script gives me this error:

IndexError: too many indices for array

It seems that the index in the iteration is not updating. It still getting the index during the first pass.

import math, easygui, shapefile, itertools, os
import pandas as pd
import numpy as np

filepath = easygui.fileopenbox()

input_dist = int(raw_input("Distance Filter Value?: "))
input_crop = raw_input("what crop?: ")

directory = os.path.split(filepath)[0]

def dist_haversine(shp,input_dist,input_crop):
    """
    Calculate the great circle distance between two points
    on the earth (specified in decimal degrees)
    """

    r = shapefile.Reader(shp)
    idx = np.arange(len(r.records()))
    coordinates = []
    for i in idx:
        geom = r.shape(i)
        coordinates.append(geom.points[0])

    acoords = np.array(coordinates)

    for r,n in itertools.izip(acoords[:,0],acoords[:,1]):

        coordinates_ = []

        for i,j in itertools.izip(acoords[:,0],acoords[:,1]):

            lon1=r
            lat1=n
            lon2=i
            lat2=j

            lon1, lat1, lon2, lat2 = map(math.radians, [lon1, lat1, lon2, lat2])

            # haversine formula
            dlon = lon2 - lon1
            dlat = lat2 - lat1
            a = math.sin(dlat/2)**2 + math.cos(lat1) * math.cos(lat2) * math.sin(dlon/2)**2
            c = 2 * math.asin(math.sqrt(a))
            km = c*6371 #/1000.0

            if km > input_dist:
                coordinates_.append([i,j])

        coordinates[:] = coordinates_
        acoords = np.array(coordinates)

    df_coords_ = pd.DataFrame(coordinates).drop_duplicates().values
    df_coords = pd.DataFrame(df_coords_, columns=['Lon','Lat'])

    df_coords.insert(0, 'Crop', input_crop)

    return df_coords.to_csv(os.path.split(directory)[0] + "\\" + "%s_distFilter_%skm.csv" % (input_crop, input_dist), sep=",", index=None)

Traceback

File "<ipython-input-3-8e88eba2ab54>", line 1,  in <module>
  dist_haversine(filepath,input_dist,input_crop)
File "<ipython-input-2-d43a1f1da26a>", line 20, in
  dist_haversine
    for i,j in itertools.izip(acoords[:,0],acoords[:,1]):
IndexError: too many indices for array

Body:
This is the initial solution that I used to filter points. Its working, and a bit fast for a 1,000 - 3,000 points dataset. However, tried filtering 50,000 points, it took 2.5 - 3 hours to finish.

def dist_haversine(filepath,input_dist,input_crop):
    """
    Calculate the great circle distance between two points
    on the earth (specified in decimal degrees)
    """

    r = shapefile.Reader(filepath)
    idx = np.arange(len(r.records()))
    coordinates = []
    for i in idx:
        geom = r.shape(i)
        coordinates.append(geom.points[0])

    acoords = np.array(coordinates)

    index = []
    for r,n,l in itertools.izip(acoords[:,0],acoords[:,1],idx):
        if l in index:
            continue
        else:
            for i,j,k in itertools.izip(acoords[:,0],acoords[:,1], idx):
                if k in index:
                    continue

                else:

                    lon1=r
                    lat1=n
                    lon2=i
                    lat2=j

                    coord_check = ((lon1 == lon2) & (lat1 == lat2))*1

                    if coord_check == 1:
                        continue

                    else:
                        lon1, lat1, lon2, lat2 = map(math.radians, [lon1, lat1, lon2, lat2])

                        # haversine formula
                        dlon = lon2 - lon1
                        dlat = lat2 - lat1
                        a = math.sin(dlat/2)**2 + math.cos(lat1) * math.cos(lat2) * math.sin(dlon/2)**2
                        c = 2 * math.asin(math.sqrt(a))
                        km = c*6371 #/1000.0

                    if km < input_dist:
                        if k in index:
                            continue
                        else:
                            index.append(k)

    filterList = [i for j, i in enumerate(coordinates) if j not in index]

    df_coords = pd.DataFrame(filterList, columns=['Lon','Lat'])

    df_coords.insert(0, 'Crop', input_crop)

    return df_coords.to_csv(directory + "\\" + "%s_distFilter_%skm.csv" % (input_crop, input_dist), sep=",", index=None)


问:
Input and Text-area won't lineup - CSS
I am practicing CSS and I can't lineup an input and textarea tag for a reply form.

<form action="" method="">
  <input style="width: 50%;" type="text" class="" name="message_name" placeholder=" Your Name">

  <textarea rows="3" cols="5" style="width: 50%;" name="message_comment" class="" placeholder=" Message"></textarea>
</form>```

答:
Im not sure if Im understanding your question at all.

<form action="" method="">
  <input style="width: 50%; margin-bottom:10px;" type="text" class="" name="message_name" placeholder=" Your Name">

  <textarea rows="3" cols="5" style="width: 50%;" name="message_comment" class="" placeholder=" Message"></textarea>
</form>```

Title: Input and Text-area won't lineup - CSS

Body:
Im not sure if Im understanding your question at all.

<form action="" method="">
  <input style="width: 50%; margin-bottom:10px;" type="text" class="" name="message_name" placeholder=" Your Name">
  <br>
  <textarea rows="3" cols="5" style="width: 50%;" name="message_comment" class="" placeholder=" Message"></textarea>
</form>


问:
Gunicorn can't find staticfiles after dockerization
I am using docker 17.05.0-cs edge on Ubuntu 17.04 Zesty. I have a gunicorn/Django app that I built here. It runs fine before being dockerized, but gunicorn can't see the staticfiles after docker build. Here is the Dockerfile:

```<code># Dockerfile

# FROM base image to build upon
FROM phusion/baseimage

# RUN install python and pip
RUN apt-get update
RUN apt-get install -y python python-pip python-dev

# ENV set environment directory tree
ENV PROJECT=mysite
ENV CONTAINER_HOME=/opt
ENV CONTAINER_PROJECT=$CONTAINER_HOME/$PROJECT

# move to project WORKDIR
WORKDIR $CONTAINER_PROJECT

# COPY project to container directory
COPY . $CONTAINER_PROJECT

# RUN pip and install requirements
RUN pip install -r requirements.txt

# COPY start.sh into known container location
COPY start.sh /start.sh

# EXPOSE port 8000 to allow communication to/from server
EXPOSE 8000

# CMD execute start.sh to start the server running.
CMD ["/start.sh"]
# done!
</code>```

The app runs and displays the page, funny, it even applies the css templates while saying it can't find bootstrap.css or bootstrap.js. It does not execute the script to toggle the menu though. The current templates are just an artifact of my initial build, the new ones won't even use bootstrap.js I'm pretty sure. It runs but the toggle doesn't work which I know is a feature of bootstrap.js. This is the out from sudo docker run -it -p 8000:8000 dockerized_site:

```<code>Starting Gunicorn.
[2017-06-07 00:38:56 +0000] [1] [INFO] Starting gunicorn 19.6.0
[2017-06-07 00:38:56 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
[2017-06-07 00:38:56 +0000] [1] [INFO] Using worker: sync
[2017-06-07 00:38:56 +0000] [9] [INFO] Booting worker with pid: 9
[2017-06-07 00:38:56 +0000] [10] [INFO] Booting worker with pid: 10
[2017-06-07 00:38:56 +0000] [11] [INFO] Booting worker with pid: 11
Not Found: /js/bootstrap.min.js
Not Found: /js/jquery.js
Not Found: /js/bootstrap.min.js
Not Found: /favicon.ico
</code>```

I appended the static file url pattern to urls.py, I am pretty sure this has to do with the container build, or perhaps settings.py. Perhaps I need to add an environment directory for the static files? Any help is, of course, appreciated.

答:
This is your problem:

Your css's are working ok:

```<code><link href="{% static 'css/bootstrap.min.css' %}" rel="stylesheet">
</code>```

Your js's are not working (404):

```<code><script src="{% static '/js/jquery.js' %}"></script>
</code>```

See the difference?

The leading / in the path of the /js

This fixes your issue:

```<code><script src="{% static 'js/jquery.js' %}"></script>
</code>```

I cloned your repo and fixed the templates and I've gotten to fix your problem. Please, fix the other

Body:
I am using docker 17.05.0-cs edge on Ubuntu 17.04 Zesty. I have a gunicorn/Django app that I built here. It runs fine before being dockerized, but gunicorn can't see the staticfiles after docker build. Here is the Dockerfile:

# Dockerfile

# FROM base image to build upon
FROM phusion/baseimage

# RUN install python and pip
RUN apt-get update
RUN apt-get install -y python python-pip python-dev

# ENV set environment directory tree
ENV PROJECT=mysite
ENV CONTAINER_HOME=/opt
ENV CONTAINER_PROJECT=$CONTAINER_HOME/$PROJECT

# move to project WORKDIR
WORKDIR $CONTAINER_PROJECT

# COPY project to container directory
COPY . $CONTAINER_PROJECT

# RUN pip and install requirements
RUN pip install -r requirements.txt

# COPY start.sh into known container location
COPY start.sh /start.sh

# EXPOSE port 8000 to allow communication to/from server
EXPOSE 8000

# CMD execute start.sh to start the server running.
CMD ["/start.sh"]
# done!

The app runs and displays the page, funny, it even applies the css templates while saying it can't find bootstrap.css or bootstrap.js. It does not execute the script to toggle the menu though. The current templates are just an artifact of my initial build, the new ones won't even use bootstrap.js I'm pretty sure. It runs but the toggle doesn't work which I know is a feature of bootstrap.js. This is the out from sudo docker run -it -p 8000:8000 dockerized_site:

Starting Gunicorn.
[2017-06-07 00:38:56 +0000] [1] [INFO] Starting gunicorn 19.6.0
[2017-06-07 00:38:56 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
[2017-06-07 00:38:56 +0000] [1] [INFO] Using worker: sync
[2017-06-07 00:38:56 +0000] [9] [INFO] Booting worker with pid: 9
[2017-06-07 00:38:56 +0000] [10] [INFO] Booting worker with pid: 10
[2017-06-07 00:38:56 +0000] [11] [INFO] Booting worker with pid: 11
Not Found: /js/bootstrap.min.js
Not Found: /js/jquery.js
Not Found: /js/bootstrap.min.js
Not Found: /favicon.ico

I appended the static file url pattern to urls.py, I am pretty sure this has to do with the container build, or perhaps settings.py. Perhaps I need to add an environment directory for the static files? Any help is, of course, appreciated.

Body:
This is your problem:

Your css's are working ok:

<link href="{% static 'css/bootstrap.min.css' %}" rel="stylesheet">

Your js's are not working (404):

<script src="{% static '/js/jquery.js' %}"></script>

See the difference?

The leading / in the path of the /js

This fixes your issue:

<script src="{% static 'js/jquery.js' %}"></script>

I cloned your repo and fixed the templates and I've gotten to fix your problem. Please, fix the other <script>s


问:
How to add a column with values corresponding to another column?
I will summarize how I got the dataframes I work with:

```<code>     name abundance
1    joe  1
2    tim  1
3    bob  1
4    joe  1
5    bob  1
</code>```

First I created a new dataframe by aggregating the columns and calculated what the relative freqeuency would be:

```<code>     name  abundance  relative_ab
1    joe   2          0.4
2    tim   1          0.2
3    bob   2          0.4
</code>```

But I want to add a column to the first dataframe so that there are redundant entries with the relative_ab like so (the actual data set has other information in it and I would lose the information by aggregating them).

```<code>     name abundance relative_ab
1    joe  1         0.4
2    tim  1         0.2
3    bob  1         0.4
4    joe  1         0.4
5    bob  1         0.4
</code>```

I think I could brute force this but I am relatively new to R and wondering what slick ways you guys might come up with.

Thanks!

答:
You can do this with match.   Assuming your first data.frame is df1 and the second one is df2, you can use:

```<code>df1$relative_ab = df2$relative_ab[match(df1$name, df2$name)]
df1
  name abundance relative_ab
1  joe         1         0.4
2  tim         1         0.2
3  bob         1         0.4
4  joe         1         0.4
5  bob         1         0.4
</code>```

match uses the name to select which row to use.

```<code>match(df1$name, df2$name)
[1] 1 2 3 1 3
</code>```

Body:
I will summarize how I got the dataframes I work with:

     name abundance
1    joe  1
2    tim  1
3    bob  1
4    joe  1
5    bob  1

First I created a new dataframe by aggregating the columns and calculated what the relative freqeuency would be:

     name  abundance  relative_ab
1    joe   2          0.4
2    tim   1          0.2
3    bob   2          0.4

But I want to add a column to the first dataframe so that there are redundant entries with the relative_ab like so (the actual data set has other information in it and I would lose the information by aggregating them).

     name abundance relative_ab
1    joe  1         0.4
2    tim  1         0.2
3    bob  1         0.4
4    joe  1         0.4
5    bob  1         0.4

I think I could brute force this but I am relatively new to R and wondering what slick ways you guys might come up with.

Thanks!

Body:
You can do this with match.   Assuming your first data.frame is df1 and the second one is df2, you can use:

df1$relative_ab = df2$relative_ab[match(df1$name, df2$name)]
df1
  name abundance relative_ab
1  joe         1         0.4
2  tim         1         0.2
3  bob         1         0.4
4  joe         1         0.4
5  bob         1         0.4

match uses the name to select which row to use.

match(df1$name, df2$name)
[1] 1 2 3 1 3


问:
Destructuring assignment and null coalescing
For a singular assignment in CoffeeScript, you can use the existential operator:

```<code>name = obj?.props?.name
</code>```

This results in a rather lengthy block of code, that checks that obj and props are defined.

```<code>name = typeof obj !== "undefined" && obj !== null ?
    (_ref2 = obj.props) != null ?
    _ref2.name : void 0 : void 0;
</code>```

Consider a more complex, destructuring assignment:

```<code>{name: name, emails: [primary], age: age} = Person.get(id)
</code>```

If the object contains no emails property, that code would throw a TypeError. Is there any way to use the existential operator with these kinds of destructuring assignments?

This is the best alternative I have so far:

```<code>{name: name, emails: emails, age: age} = Person.get(id)
primary = emails?[0]
</code>```

答:
In ES6, you can do this:

```<code>const {name: name, emails: [primary] = [], age: age} = Person.get(id)
</code>```

If Person.get(id) returns an empty object, primary will be undefined (no TypeError is thrown).

The forthcoming CoffeeScript 2 supports this as well, which you can try at http://coffeescript.org/v2/#try

Body:
For a singular assignment in CoffeeScript, you can use the existential operator:

name = obj?.props?.name

This results in a rather lengthy block of code, that checks that obj and props are defined.

name = typeof obj !== "undefined" && obj !== null ?
    (_ref2 = obj.props) != null ?
    _ref2.name : void 0 : void 0;

Consider a more complex, destructuring assignment:

{name: name, emails: [primary], age: age} = Person.get(id)

If the object contains no emails property, that code would throw a TypeError. Is there any way to use the existential operator with these kinds of destructuring assignments?

This is the best alternative I have so far:

{name: name, emails: emails, age: age} = Person.get(id)
primary = emails?[0]

Body:
In ES6, you can do this:

const {name: name, emails: [primary] = [], age: age} = Person.get(id)

If Person.get(id) returns an empty object, primary will be undefined (no TypeError is thrown).

The forthcoming CoffeeScript 2 supports this as well, which you can try at http://coffeescript.org/v2/#try


问:
How to connect to server, listen for arbitrary time and send back data
I'm trying to:

Connect to a server/port

Listen for x seconds

Receive user input

Send user input to server

Go back to step 2

So far, I've written the following code, but it's not working properly receiving input after the first send. Any help would be greatly appreciated.

```<code>import socket
import select

client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.connect(('domain.com', 1234))
client_socket.setblocking(0)

timeout = 5
while True:
    while True:
        ready = select.select([client_socket], [], [], timeout)

        if ready[0]:
            data = client_socket.recv(4096)
            print data
        else:
            break

    data = raw_input("Enter input:")
    client_socket.send(data)
</code>```

答:
You need to have separate server side code and client side code. This article has been referred.

Server binds to a port and listens for clients

server.py

```<code>import select
import socket

# Create a TCP/IP socket
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.setblocking(0)

# Bind the socket to the port
server_address = ('localhost', 1234)
server.bind(server_address)

# Listen for incoming connections
server.listen(5)

# Sockets from which we expect to read
inputs = [ server ]

# Sockets to which we expect to write
outputs = [ ]

while inputs:

    readable, writable, exceptional = select.select(inputs, outputs, inputs)
    # Handle inputs
    for s in readable:

        if s is server:

            # A "readable" server socket is ready to accept a connection
            connection, client_address = s.accept()
            connection.setblocking(0)
            inputs.append(connection)

        else:
            data = s.recv(1024)
            if data:
                print "Receiving data from client"
                print data
            else:
                inputs.remove(s)
                s.close()
</code>```

Client first establishes a connection with the server and then keeps on sending user input to the server.

client.py

```<code>import socket

server_address = ('domain.com', 1234)

# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

sock.connect(server_address)

while True:

    data = raw_input("Enter input:")
    sock.send(data)
</code>```

Open terminal.

Run server in background:

```<code>python server.py &
</code>```

Run client after that:

```<code>python client.py
</code>```

Body:
I'm trying to:

Connect to a server/port

Listen for x seconds

Receive user input

Send user input to server

Go back to step 2

So far, I've written the following code, but it's not working properly receiving input after the first send. Any help would be greatly appreciated.

import socket
import select

client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.connect(('domain.com', 1234))
client_socket.setblocking(0)

timeout = 5
while True:
    while True:
        ready = select.select([client_socket], [], [], timeout)

        if ready[0]:
            data = client_socket.recv(4096)
            print data
        else:
            break

    data = raw_input("Enter input:")
    client_socket.send(data)

Body:
You need to have separate server side code and client side code. This article has been referred.

Server binds to a port and listens for clients

server.py

import select
import socket

# Create a TCP/IP socket
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.setblocking(0)

# Bind the socket to the port
server_address = ('localhost', 1234)
server.bind(server_address)

# Listen for incoming connections
server.listen(5)

# Sockets from which we expect to read
inputs = [ server ]

# Sockets to which we expect to write
outputs = [ ]

while inputs:

    readable, writable, exceptional = select.select(inputs, outputs, inputs)
    # Handle inputs
    for s in readable:

        if s is server:

            # A "readable" server socket is ready to accept a connection
            connection, client_address = s.accept()
            connection.setblocking(0)
            inputs.append(connection)

        else:
            data = s.recv(1024)
            if data:
                print "Receiving data from client"
                print data
            else:
                inputs.remove(s)
                s.close()

Client first establishes a connection with the server and then keeps on sending user input to the server.

client.py

import socket

server_address = ('domain.com', 1234)

# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

sock.connect(server_address)

while True:

    data = raw_input("Enter input:")
    sock.send(data)

Open terminal.

Run server in background:

python server.py &

Run client after that:

python client.py


问:
Can I download Trained Model from Google Cloud ML Engine?
I have a requirement where I need to Deploy a Convolutional Neural Network Model in an Offline device. I know we can use Google Cloud ML to train a model, tune the hyper-hyperparameters and deploy it for prediction.

But my question is if we can download the trained TensorFlow model and deploy it on a custom device for prediction?

Note - The Custom Device will have a lot of processing power but no internet connectivity.

答:
Yes. The training service and the prediction services are completely separate. To train a model for a custom device, you create a TensorFlow script to train the model. The script will typically describe one TensorFlow graph for training and a second for prediction (the prediction graph will be constructed in such a way that it can load the parameters learned during training). The prediction graph will be tailored for your custom hardware.

Be sure to include commands to export the prediction graph to GCS. For an example of how to export a model, see this post.


问:
The statement has been terminated SQL management studio
I have made a query in SQL management studio which is as follows :-

```<code>ALTER TABLE [TransactionNote]
ALTER COLUMN NoteLine [varchar]
COLLATE SQL_Latin1_General_CP1_CI_AS.
</code>```

Why do I see

```<code>Msg 8152, Level 16, State 14, Line 4
String or binary data would be truncated.
The statement has been terminated.
</code>```

in the output window.?

答:
You need to specify the new length for your NoteLine field. If no length is specified for a varchar, it will default to a length of 1. Any current values in that field that are longer than 1 character would be truncated, hence the error message.

Determine how long the longest value is in that field, and then ALTER the column accordingly.

```<code>select max(len(NoteLine) from TransactionNote;
</code>```

Make note of the result.

```<code>ALTER TABLE [TransactionNote]
ALTER COLUMN NoteLine [varchar(VALUE_FROM_BEFORE)]
COLLATE SQL_Latin1_General_CP1_CI_AS
</code>```

Body:
I have made a query in SQL management studio which is as follows :-

ALTER TABLE [TransactionNote]
ALTER COLUMN NoteLine [varchar]
COLLATE SQL_Latin1_General_CP1_CI_AS.

Why do I see

Msg 8152, Level 16, State 14, Line 4
String or binary data would be truncated.
The statement has been terminated.

in the output window.?

Body:
You need to specify the new length for your NoteLine field. If no length is specified for a varchar, it will default to a length of 1. Any current values in that field that are longer than 1 character would be truncated, hence the error message.

Determine how long the longest value is in that field, and then ALTER the column accordingly.

select max(len(NoteLine) from TransactionNote;

Make note of the result.

ALTER TABLE [TransactionNote]
ALTER COLUMN NoteLine [varchar(VALUE_FROM_BEFORE)]
COLLATE SQL_Latin1_General_CP1_CI_AS


问:
user defined commands in verbatim environment
is it possible to create a new command with a parameter in a verbatim environment and some indentation?

```<code>\newcommand{codeblock}[1]{\begin{quote}\begin{verbatim}#1\end{verbatim}\end{quote}}
</code>```

This does not work. Is there an other way?

答: It looks like you want to have code in your document, in which case you're probably better served by the package listings than by verbatim. Listings also gives you nice features like line numbering and syntax highlighting for many common languages. See http://www.ctan.org/tex-archive/macros/latex/contrib/listings/ if it's not already installed with your LaTeX distribution.</p>

Body:
is it possible to create a new command with a parameter in a verbatim environment and some indentation?

\newcommand{codeblock}[1]{\begin{quote}\begin{verbatim}#1\end{verbatim}\end{quote}}

This does not work. Is there an other way?