问:
Kafka consumer showing numbers in unreadable format
I am trying out the kafka streaming. I am reading messages from one topic and doing groupByKey and then doing the count of groups. But the problem is that the messages count is coming as unreadable "boxes".

If I run the console consumer these are coming as empty strings

This is the WordCount code I wrote

```<code>package streams;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.kstream.KStream;

import java.util.Arrays;
import java.util.Properties;

public class WordCount {
    public static void main(String[] args) {

        Properties properties = new Properties();
        properties.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        properties.setProperty(StreamsConfig.APPLICATION_ID_CONFIG, "streams-demo-2");
        properties.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        properties.setProperty(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.StringSerde.class.getName());
        properties.setProperty(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.StringSerde.class.getName());

        // topology
        StreamsBuilder builder = new StreamsBuilder();
        KStream<String, String> input = builder.stream("temp-in");
        KStream<String, Long> fil = input.flatMapValues(val -> Arrays.asList(val.split(" "))) // making stream of text line to stream of words
                .selectKey((k, v) -> v) // changing the key
                .groupByKey().count().toStream(); // getting count after groupBy

        fil.to("temp-out");

        KafkaStreams streams = new KafkaStreams(builder.build(), properties);
        streams.start();

        System.out.println(streams.toString());

        Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
    }
}
</code>```

This is the output I am getting in the consumer. It is there on the right side in image

I had tried casting the long to long again to see if it works. But it's not working

I am attaching the consumer code too if it helps.

```<code>package tutorial;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class Consumer {
    public static void main(String[] args) {
        Properties properties = new Properties();
        properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        properties.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());

        // Once the consumer starts running it keeps running even after we stop in console
        // We should create new consumer to read from earliest because the previous one had already consumed until certain offset
        // when we run the same consumer in two consoles kafka detects it and re balances
        // In this case the consoles split the partitions they consume forming a consumer group
        properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG, "consumer-application-1"); // -> consumer id
        properties.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); // -> From when consumer gets data

        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(properties);
        consumer.subscribe(Collections.singleton("temp-out"));

        while (true) {
            ConsumerRecords<String, String> consumerRecords = consumer.poll(Duration.ofMillis(1000));
            for (ConsumerRecord<String, String> record: consumerRecords) {
                System.out.println(record.key() + " " + record.value());
                System.out.println(record.partition() + " " + record.offset());
            }
        }
    }
}
</code>```

Any help is appreciated. Thanks in advance.

答:
The message value you're writing with Kafka Streams is a Long, and you're consuming it as a String.

If you make the following changes to your Consumer class, you'll be able to see the count printed correctly to stdout:

```<code>        // Change this from StringDeserializer to LongDeserializer.
        properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, LongDeserializer.class.getName());

        ...

        // The value you're consuming here is a Long, not a String.
        KafkaConsumer<String, Long> consumer = new KafkaConsumer<>(properties);
        consumer.subscribe(Collections.singleton("temp-out"));

        while (true) {
            ConsumerRecords<String, Long> consumerRecords = consumer.poll(Duration.ofMillis(1000));
            for (ConsumerRecord<String, Long> record : consumerRecords) {
                System.out.println(record.key() + " " + record.value());
                System.out.println(record.partition() + " " + record.offset());
            }
        }
</code>```

Body:
I am trying out the kafka streaming. I am reading messages from one topic and doing groupByKey and then doing the count of groups. But the problem is that the messages count is coming as unreadable "boxes".

If I run the console consumer these are coming as empty strings

This is the WordCount code I wrote

package streams;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.kstream.KStream;

import java.util.Arrays;
import java.util.Properties;

public class WordCount {
    public static void main(String[] args) {

        Properties properties = new Properties();
        properties.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        properties.setProperty(StreamsConfig.APPLICATION_ID_CONFIG, "streams-demo-2");
        properties.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        properties.setProperty(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.StringSerde.class.getName());
        properties.setProperty(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.StringSerde.class.getName());

        // topology
        StreamsBuilder builder = new StreamsBuilder();
        KStream<String, String> input = builder.stream("temp-in");
        KStream<String, Long> fil = input.flatMapValues(val -> Arrays.asList(val.split(" "))) // making stream of text line to stream of words
                .selectKey((k, v) -> v) // changing the key
                .groupByKey().count().toStream(); // getting count after groupBy

        fil.to("temp-out");

        KafkaStreams streams = new KafkaStreams(builder.build(), properties);
        streams.start();

        System.out.println(streams.toString());

        Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
    }
}

This is the output I am getting in the consumer. It is there on the right side in image

I had tried casting the long to long again to see if it works. But it's not working

I am attaching the consumer code too if it helps.

package tutorial;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class Consumer {
    public static void main(String[] args) {
        Properties properties = new Properties();
        properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        properties.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());

        // Once the consumer starts running it keeps running even after we stop in console
        // We should create new consumer to read from earliest because the previous one had already consumed until certain offset
        // when we run the same consumer in two consoles kafka detects it and re balances
        // In this case the consoles split the partitions they consume forming a consumer group
        properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG, "consumer-application-1"); // -> consumer id
        properties.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); // -> From when consumer gets data

        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(properties);
        consumer.subscribe(Collections.singleton("temp-out"));

        while (true) {
            ConsumerRecords<String, String> consumerRecords = consumer.poll(Duration.ofMillis(1000));
            for (ConsumerRecord<String, String> record: consumerRecords) {
                System.out.println(record.key() + " " + record.value());
                System.out.println(record.partition() + " " + record.offset());
            }
        }
    }
}

Any help is appreciated. Thanks in advance.

Body:
The message value you're writing with Kafka Streams is a Long, and you're consuming it as a String.

If you make the following changes to your Consumer class, you'll be able to see the count printed correctly to stdout:

        // Change this from StringDeserializer to LongDeserializer.
        properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, LongDeserializer.class.getName());

        ...

        // The value you're consuming here is a Long, not a String.
        KafkaConsumer<String, Long> consumer = new KafkaConsumer<>(properties);
        consumer.subscribe(Collections.singleton("temp-out"));

        while (true) {
            ConsumerRecords<String, Long> consumerRecords = consumer.poll(Duration.ofMillis(1000));
            for (ConsumerRecord<String, Long> record : consumerRecords) {
                System.out.println(record.key() + " " + record.value());
                System.out.println(record.partition() + " " + record.offset());
            }
        }


来源: stackexchange

问:
Meteor Svelte can't find files I create on Ubuntu
I've tried following along multiple Meteor tutorials, and I keep getting the same issue, although I haven't been able to find a solution when searching for answers.

When I try to import html files in my client/main.js, it keeps telling me it can't find those modules. For example, I have the following file:

imports/app.svelte

```<code><header>
    <h1>Yo!</h1>
</header>
</code>```

Then in my main.js:

client/main.js

```<code>import { Meteor } from 'meteor/meteor';
import App from '../imports/App.svelte';

Meteor.startup(() => {
  new App({
    target: document.getElementById("app"),
  });
});
</code>```

And now I keep getting this error:

```<code>Uncaught Error: Cannot find module '../imports/App.svelte'
</code>```

I'm following along a LevelUpTuts tutorial, so I'm certain I've installed the right packages etc. And as mentioned, the same happened when I tried to do the Meteor official tutorial as well as the Vue tutorial. I'm suspecting this might have something to do with Ubuntu?

答:
You have a typo there in the filename: Your file is called app.svelte, but you are importing App.svelte.

You need to rename the file to be called App.svelte with capital A.

Your feeling that it has something to do with Ubuntu is sort-of-right because the file system there is by default case-sensitive, while in Windows it is by default case-insensitive. So if you previously did the same mistake in Windows, you may not have noticed it because when the application asked Windows for App.svelte, Windows happily said "yes here is your file" and sent app.svelte instead, while Ubuntu would say "no sorry I don't know about any App.svelte (only an app.svelte but you asked for App and not app, soo...)".

So, Ubuntu made your mistake visible while Windows hid it away. It's good that you ran into this issue now and not only once you were trying to deploy your code to a Linux server.

ContentLicense: CC BY-SA 4.0

Body:
I've tried following along multiple Meteor tutorials, and I keep getting the same issue, although I haven't been able to find a solution when searching for answers.

When I try to import html files in my client/main.js, it keeps telling me it can't find those modules. For example, I have the following file:

imports/app.svelte

Yo!

Then in my main.js:

client/main.js

import { Meteor } from 'meteor/meteor';
import App from '../imports/App.svelte';

Meteor.startup(() => {
  new App({
    target: document.getElementById("app"),
  });
});

And now I keep getting this error:

Uncaught Error: Cannot find module '../imports/App.svelte'

I'm following along a LevelUpTuts tutorial, so I'm certain I've installed the right packages etc. And as mentioned, the same happened when I tried to do the Meteor official tutorial as well as the Vue tutorial. I'm suspecting this might have something to do with Ubuntu?


问:
Receiving CORS policy error when submitting form from Angular to flask application
I have tried several things but I am not able to figure this one out. I have a back-end flask app and a front-end client written in Angular. When I submit my register user form I get a cors error. I have read the documentation for flask_cors and have tried to get it to work but I still get the same error below:

Access to XMLHttpRequest at 'http://localhost:5000/v1/auth/register' from origin 'http://localhost:4200' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Here is my app.py from the flask app.

Any help is greatly appreciated!

```<code>import os, sys
from flask import Flask
import pathlib
from flask_cors import CORS
from flask_restplus import Api, Resource, fields

from werkzeug.middleware.proxy_fix import ProxyFix

import coloredlogs, logging as log
coloredlogs.install()

from main.apis.user import api as User
from main.apis.auth import api as Auth

from main import create_app
from flask_pymongo import PyMongo

# Init app
app = Flask(__name__)

#cors = CORS(app, resources={r"*": {"origins": "*"}})

CORS(app, origins="http://localhost:4200", allow_headers=[
    "Content-Type", "Authorization", "Access-Control-Allow-Credentials","Access-Control-Allow-Origin"],
    supports_credentials=True, intercept_exceptions=False)

authorizations = {
    'token': {
        'type': 'apiKey',
        'in': 'header',
        'name': 'Authorization'
    }
}

config_name = os.getenv('FLASK_CONFIG')
app = create_app(config_name)

api = Api(app, authorizations=authorizations, version='1.0', title='API docs',
    description='A simple REST API with JWT authentication.',
    doc='/docs'
)

app.config['jwt']._set_error_handler_callbacks(api)
app.config['ROOT_DIR'] = pathlib.Path(__file__).parent.absolute()

# @app.before_first_request
# this function is to init the db and realted models
# def create_tables():
    # print("Before first statement")
#     db.create_all()

# Endpoints
api.add_namespace(Auth, path='/v1')
api.add_namespace(User, path='/v1')

# Run Server
if __name__ == '__main__':
    app.run()
</code>```

答:
adding this seemed to fix my issue.</p>

```<code>@app.after_request
def after_request(response):
    response.headers.add('Access-Control-Allow-Origin', '*')
    response.headers.add('Access-Control-Allow-Headers', 'Content-Type')
    return response
</code>```

Body:
I have tried several things but I am not able to figure this one out. I have a back-end flask app and a front-end client written in Angular. When I submit my register user form I get a cors error. I have read the documentation for flask_cors and have tried to get it to work but I still get the same error below:

Access to XMLHttpRequest at 'http://localhost:5000/v1/auth/register' from origin 'http://localhost:4200' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Here is my app.py from the flask app.

Any help is greatly appreciated!

import os, sys
from flask import Flask
import pathlib
from flask_cors import CORS
from flask_restplus import Api, Resource, fields

from werkzeug.middleware.proxy_fix import ProxyFix

import coloredlogs, logging as log
coloredlogs.install()

from main.apis.user import api as User
from main.apis.auth import api as Auth

from main import create_app
from flask_pymongo import PyMongo

# Init app
app = Flask(__name__)

#cors = CORS(app, resources={r"*": {"origins": "*"}})

CORS(app, origins="http://localhost:4200", allow_headers=[
    "Content-Type", "Authorization", "Access-Control-Allow-Credentials","Access-Control-Allow-Origin"],
    supports_credentials=True, intercept_exceptions=False)

authorizations = {
    'token': {
        'type': 'apiKey',
        'in': 'header',
        'name': 'Authorization'
    }
}

config_name = os.getenv('FLASK_CONFIG')
app = create_app(config_name)

api = Api(app, authorizations=authorizations, version='1.0', title='API docs',
    description='A simple REST API with JWT authentication.',
    doc='/docs'
)

app.config['jwt']._set_error_handler_callbacks(api)
app.config['ROOT_DIR'] = pathlib.Path(__file__).parent.absolute()

# @app.before_first_request
# this function is to init the db and realted models
# def create_tables():
    # print("Before first statement")
#     db.create_all()

# Endpoints
api.add_namespace(Auth, path='/v1')
api.add_namespace(User, path='/v1')

# Run Server
if __name__ == '__main__':
    app.run()

Body:
adding this seemed to fix my issue.

@app.after_request
def after_request(response):
    response.headers.add('Access-Control-Allow-Origin', '*')
    response.headers.add('Access-Control-Allow-Headers', 'Content-Type')
    return response


问:
close the application when session is idle for more than 10mins
I want to close the application when my application is idle for more than 10 min.Any one have idea.Give me some solutions.Thanks in adavance</p>

答:
Well, that's difficult: how do you define idle? No user interaction? No button presses? No scrolling by the user?

If you go with button presses as the metric for activity, I would suggest creating a Handler object, scheduling a message to be delivered in 600000 milliseconds and then any time the user presses a button cancel the message and reschedule it. If the message ever goes off you can just call finish().

ContentLicense: CC BY-SA 3.0


问:
Persist data between two pages with Next.js
I would like to refactor my Next.js webapp to have different pages handle different screens. Currently, I have this component holding several states to know in which screen I'm in. In the jsx section, I'm using {value && ... } to render the right component.

But I feel this is not good design, and won't be maintainable when adding more and more screens.

I would also like to avoid Redux as it is overkill for my project.

I was thinking about persisting data in cookies so I can retrieve them with getInitialProps in every component when rendering a new page, but is there a more elegant way?

I've read about tweaking the _app.js but I'm not sure to understand the consequences of doing so, and how it could help me..

Any suggestion?

答:
When multiple of your pages need to make use of same data, you can make use of Context to store the result. It a good way to make a centralized storage without using complex and more self sufficient libraries like redux

You can implement context inside of _app.js file which must reside inside your root folder. This way next.js treats it as a root wrapper and you would just need to use 1 instance of Context

contexts/appContext

```<code>import React from 'react';
const AppContext = React.createContext();
export const AppProvider = AppContext.Provider;
export const AppConsumer = AppContext.Consumer;
export default AppContext;
</code>```

_app.js

```<code>import React from 'react'
import App from 'next/app'
import AppProvider from '../contexts/appContext';
class MyApp extends App {
  state={
    data:[]
  }
  render() {
    const { Component, pageProps } = this.props;
    // You can implement logic in this component to fetch data and update state
    return (
      <div>
        <AppProvider value={this.state.data}> // pass on value to context
          <Component {...pageProps} />
        </AppProvider>
      </div>
    )
  }
}
export default MyApp
</code>```

Now further each component can make use of context value by using AppConsumer or using useContext if you use hooks

Please read more about how to use Context here

Body:
When multiple of your pages need to make use of same data, you can make use of Context to store the result. It a good way to make a centralized storage without using complex and more self sufficient libraries like redux

You can implement context inside of _app.js file which must reside inside your root folder. This way next.js treats it as a root wrapper and you would just need to use 1 instance of Context

contexts/appContext

import React from 'react';
const AppContext = React.createContext();
export const AppProvider = AppContext.Provider;
export const AppConsumer = AppContext.Consumer;
export default AppContext;

_app.js

import React from 'react'
import App from 'next/app'
import AppProvider from '../contexts/appContext';
class MyApp extends App {
  state={
    data:[]
  }
  render() {
    const { Component, pageProps } = this.props;
    // You can implement logic in this component to fetch data and update state
    return (
      <div>
        <AppProvider value={this.state.data}> // pass on value to context
          <Component {...pageProps} />
        </AppProvider>
      </div>
    )
  }
}
export default MyApp

Now further each component can make use of context value by using AppConsumer or using useContext if you use hooks

Please read more about how to use Context here


问:
how to display the dynamic data in html page
how to send data to club.html dynamically changing with user.

how to achieve this. I stuck here.

I don't know where is the mistake please help me.

club.html

```<code><div class='container'>
            <p class="newlycreated">Prisimax Welcomes you {{user.username}}
            <img src={{user.userprofile.userphoto.url}} id="ownimg"></p>
            <br>
            <p style="margin-left: 100px;">Your LEVEL: {{Clubmember.level}} </p>
            <p style="margin-left: 100px;">Your cash <i class="fas fa-wallet"></i> : {{Clubmember.usermoney}}</p>
        </div>
</code>```

views.py

```<code>class club(View):
    template_name = "club.html"
    def get(self,*args,**kwargs):
        if self.request.user.userprofile.Isclubmem:
            club_member = Clubmember.objects.filter(user=self.request.user.userprofile)
            context = {
                'club_member': club_member
            }
            return render(self.request,'club.html',context=context)
        return render(self.request,'club.html')
    def post(self, *args ,**kwargs):
        if self.request.user.userprofile.Isclubmem:
            if self.request.method == 'POST':
                self.request.user.userprofile.userphoto = self.request.FILES.get('image')
                self.request.user.userprofile.phone_number = self.request.POST.get('userphonenumber')
                self.request.user.userprofile.Isclubmem = True
                club_member = Clubmember.objects.create(
                    user = self.request.user.userprofile,
                    refer = refgen(),
                )
                club_member.save()
                self.request.user.userprofile.save()
                return redirect('core:home')
        else:
            return redirect('/paytmpay')
</code>```

答:
In you view there are no Clubmember named object passed to your render. Instead you have club_member.

club.html

Prisimax Welcomes you {{user.username}}

Your LEVEL: {{club_member.level}}

Your cash  : {{club_member.usermoney}}

```

<p>Besides, you are using <code>template_name = "club.html"</code> variable but not using in your view. You should either delete this variable or use it in your code:</p>

<pre class="lang-py prettyprint-override"><code>            ...
            return render(self.request, self.template_name, context=context)
        return render(self.request, self.template_name)
</code>```

Furthermore, the naming self.request.user.userprofile.Isclubmem is not good. You need to checkout python naming conventions aka PEP8.

Body:
how to send data to club.html dynamically changing with user.

how to achieve this. I stuck here.

I don't know where is the mistake please help me.

club.html

Prisimax Welcomes you {{user.username}}

Your LEVEL: {{Clubmember.level}}

Your cash  : {{Clubmember.usermoney}}

views.py

class club(View):
    template_name = "club.html"
    def get(self,*args,**kwargs):
        if self.request.user.userprofile.Isclubmem:
            club_member = Clubmember.objects.filter(user=self.request.user.userprofile)
            context = {
                'club_member': club_member
            }
            return render(self.request,'club.html',context=context)
        return render(self.request,'club.html')
    def post(self, *args ,**kwargs):
        if self.request.user.userprofile.Isclubmem:
            if self.request.method == 'POST':
                self.request.user.userprofile.userphoto = self.request.FILES.get('image')
                self.request.user.userprofile.phone_number = self.request.POST.get('userphonenumber')
                self.request.user.userprofile.Isclubmem = True
                club_member = Clubmember.objects.create(
                    user = self.request.user.userprofile,
                    refer = refgen(),
                )
                club_member.save()
                self.request.user.userprofile.save()
                return redirect('core:home')
        else:
            return redirect('/paytmpay')

Body:
In you view there are no Clubmember named object passed to your render. Instead you have club_member.

club.html

        <div class='container'>
            <p class="newlycreated">Prisimax Welcomes you {{user.username}}
            <img src={{user.userprofile.userphoto.url}} id="ownimg"></p>
            <br>
            <p style="margin-left: 100px;">Your LEVEL: {{club_member.level}} </p>
            <p style="margin-left: 100px;">Your cash <i class="fas fa-wallet"></i> : {{club_member.usermoney}}</p>
        </div>

Besides, you are using template_name = "club.html" variable but not using in your view. You should either delete this variable or use it in your code:

            ...
            return render(self.request, self.template_name, context=context)
        return render(self.request, self.template_name)

Furthermore, the naming self.request.user.userprofile.Isclubmem is not good. You need to checkout python naming conventions aka PEP8.


问:
Loss function and data format for training a ''categorical input' to 'categorical output' model?
I am trying to train a model for autonomous driving that converts input from the front camera, to a bird's eye view image.

The input and output, both are segmentation masks with shape (96, 144) where each pixel has a range from  0 to 12 (each number represents a different class).

Now my question is how should i preprocess my data and which loss function should i use for the model (I am trying to use a Fully convolutional Network).

I tried to convert input and outputs to shape (96, 144, 13) using keras' to_categorical utility so each channel has 0s and 1s of representing a specific mask of a category. I used binary_crossentropy ad sigmoid activation for last layer with this and the model seemed to learn and loss started reducing.

But i am still unsure if this is the correct way or if there are any better ways.

what should be the:

input and ouptput data format

activation of last layer

loss function

答: I found the solution, use categorical crossentropy with softmax activation at last layer. Use the same data format as specified in the question.</p>


问:
Certain text not showing on my webpage, jQuery script to blame?
I'm trying to create a nice simple checkout form with a jquery script I got on the web and I seem to be facing a weird problem that could be due to a couple of things and I'm clueless as to how to fix this.

Maybe you can solve this... Below is my website in development:

[link removed]

If you look at the form (the light gray part that scrolls up and down), I have a list of cupcakes (around 30) that stretch all the way down the frame. My problem is that for no apparent reason the last 5 items don't show up, but if you look at the HTML everything is there.

The script I'm using is featureList : http://jqueryglobe.com/article/feature-list

Is this :

a jQuery limitation?

a
 limitation?

a featureList bug/limitation?

multiple ids tags on html limitation?

is it a stupid error and I've been looking at this code for too long?

How can I solve this?

Thanks

答:
You are setting the z-index of the hidden divs to negative values, most likely due to this brilliant piece of code:

```<code><script type="text/javascript">
    $(function() {
        var zIndexNumber = 1000;
        $('div').each(function() {
        $(this).css('zIndex', zIndexNumber);
        zIndexNumber -= 10;
        });
    });
</script>
</code>```

But hey I guess it's easier to blame a framework as widely used as jQuery instead of your poking around things you don't understand.

Edit: a
 limitation? Seriously?

Body:
You are setting the z-index of the hidden divs to negative values, most likely due to this brilliant piece of code:

<script type="text/javascript">
    $(function() {
        var zIndexNumber = 1000;
        $('div').each(function() {
        $(this).css('zIndex', zIndexNumber);
        zIndexNumber -= 10;
        });
    });
</script>

But hey I guess it's easier to blame a framework as widely used as jQuery instead of your poking around things you don't understand.

Edit: a <li> limitation? Seriously?


问:
Why does useEffect run multiple times?
I'm using useEffect to fetch data from an API. No matter what I do, the useEffect runs multiple times.

```<code>  const [productDetails, setProductDetails] = useState([]);

  useEffect(() => {
    fetch(
      `url`
    )
      .then(results => results.json())
      .then(results => {
        setProductDetails(results);
      });
  }, []);
</code>```

I read that you don't set state in useEffect, so I tried another approach:

```<code>const [productDetails, setProductDetails] = useState([]);

  useEffect(() => {
    fetchAsync();
  }, []);

  async function fetchAsync() {
    // await response of fetch call
    let response = await fetch(
      `url`
    );
    // only proceed once promise is resolved
    let data = await response.json();
    // only proceed once second promise is resolved
    setProductDetails(data); // setting state after fetching data
  }
</code>```

In both cases, when I console.log(productDetails); I get the results back multiple times, so I guess useEffect runs multiple times. I thought about a bug in useEffect and tried with the old componentDidMount() but it's always the same. Results are coming back multiple times.

Is there a 100% correct way to set up an API call and getting the results back only ONCE?

答:
I'm quite sure the problem is outside the code you have posted.

I guess that the component that contains this useEffect is re-created for some reason. Maybe even because the parent component (or the parent-parent ...) is re-created.

Title: Why does useEffect run multiple times?

Body:
I'm using useEffect to fetch data from an API. No matter what I do, the useEffect runs multiple times.

  const [productDetails, setProductDetails] = useState([]);

  useEffect(() => {
    fetch(
      `url`
    )
      .then(results => results.json())
      .then(results => {
        setProductDetails(results);
      });
  }, []);

I read that you don't set state in useEffect, so I tried another approach:

const [productDetails, setProductDetails] = useState([]);

  useEffect(() => {
    fetchAsync();
  }, []);

  async function fetchAsync() {
    // await response of fetch call
    let response = await fetch(
      `url`
    );
    // only proceed once promise is resolved
    let data = await response.json();
    // only proceed once second promise is resolved
    setProductDetails(data); // setting state after fetching data
  }

In both cases, when I console.log(productDetails); I get the results back multiple times, so I guess useEffect runs multiple times. I thought about a bug in useEffect and tried with the old componentDidMount() but it's always the same. Results are coming back multiple times.

Is there a 100% correct way to set up an API call and getting the results back only ONCE?


问:
Trying to mock IntConsumer with Mockito fails
I'm trying to mock IntConsumer:

```<code>class TickerServiceImplTest {
@Test
void testRunIterations() {
    TickerServiceImpl tickerService = new TickerServiceImpl();
    int ticksToRun = 100;
    tickerService.setTicksToRun(ticksToRun);
    IntConsumer intConsumerMock = mock(IntConsumer.class);
    tickerService.run(intConsumerMock);
    verify(intConsumerMock, times(ticksToRun));
}
</code>```

and it fails on the 'verify' with below error code:

Method threw 'org.mockito.exceptions.base.MockitoException' exception.
Cannot evaluate $java.util.function.IntConsumer$$EnhancerByMockitoWithCGLIB$$3ee084c4.toString()

答:
You need to tell Mockito what method it is supposed to verify on the IntConsumer mock. Your verification code should look something like:

verify(intConsumerMock, times(ticksToRun)).accept(anyInt());
```

See for example the tutorial at Baeldung.

Body:
I'm trying to mock IntConsumer:

class TickerServiceImplTest {
@Test
void testRunIterations() {
    TickerServiceImpl tickerService = new TickerServiceImpl();
    int ticksToRun = 100;
    tickerService.setTicksToRun(ticksToRun);
    IntConsumer intConsumerMock = mock(IntConsumer.class);
    tickerService.run(intConsumerMock);
    verify(intConsumerMock, times(ticksToRun));
}

and it fails on the 'verify' with below error code:

Method threw 'org.mockito.exceptions.base.MockitoException' exception.
Cannot evaluate $java.util.function.IntConsumer$$EnhancerByMockitoWithCGLIB$$3ee084c4.toString()

Body:
You need to tell Mockito what method it is supposed to verify on the IntConsumer mock. Your verification code should look something like:

verify(intConsumerMock, times(ticksToRun)).accept(anyInt());

See for example the tutorial at Baeldung.


问:
FLTK 1.4 widget position w.r.t. X11 root window?

context

I am coding with others RefPerSys, a GPLv3+ project in C++17 on gitlab for GNU/Linux/x86-64/Debian/Sid. Its fltk-branch git branch is using FLTK 1.4, compiled from source code, with an Xorg display server.

I have C++ classes like (in file headfltk_rps.hh):

```<code>class RpsGui_Window: public Fl_Double_Window
{
  static std::set<RpsGui_Window*> _set_of_gui_windows_;
public:
  virtual int handle(int);
protected:
  Fl_Menu_Bar *guiwin_menubar;
  std::string guiwin_label;
  virtual void initialize_menubar(void) =0;
  RpsGui_Window(int w, int h, const std::string& lab);
  RpsGui_Window(int x, int y, int w, int h, const std::string& lab);
public:
  virtual ~RpsGui_Window();
  /// .... skipping irrelevant code
  const std::string label_str(void) const {
    return guiwin_label;
  };
}; /// end class RpsGui_Window

class RpsGui_CommandWindow : public RpsGui_Window
{
  static constexpr int right_menu_gap = 16;
  static constexpr int menu_height = 20;
  Fl_Pack* cmdwin_pack;
  friend  void rps_fltk_initialize(int &,char**);
  virtual void initialize_menubar(void);
  virtual void initialize_pack(void);
  static void menu_dump_cb(Fl_Widget*, void*);
  static void menu_exit_cb(Fl_Widget*, void*);
public:
  RpsGui_CommandWindow(int w, int h, const std::string& lab);
  RpsGui_CommandWindow(int x, int y, int w, int h, const std::string& lab);
  virtual ~RpsGui_CommandWindow();
};              // end class RpsGui_CommandWindow
</code>```

and I am debugging with old C++ macros outputting to std::cerr  (defined in refpersys.hh lines 315 and following) such as below:

```<code> RPS_DEBUG_LOG(GUI, "RpsGui_CommandWindow::initialize_pack this:"
               <<  RpsGui_ShowWidget(this)
               << std::endl << "... cmdwin_pack:"
               << RpsGui_ShowWidget(cmdwin_pack));
</code>```

Something is still wrong on the screen.
See RefPerSys issue#33 for even more details (with a screenshot).

I would like to output the position of a given FLTK widget w.r.t. my X11 root window. FWIW xdpyinfo is giving (with a lot of output skipped)

```<code>name of display:    :0
version number:    11.0
vendor string:    The X.Org Foundation
vendor release number:    12008000
X.Org version: 1.20.8

screen #0:
  dimensions:    5360x1440 pixels (1418x381 millimeters)
  resolution:    96x96 dots per inch
</code>```

question

In other words, I want to code (for debugging purposes)

```<code>int RpsGui_Window::x_wrt_root() const;
</code>```

as a member function returning the top left corner horizontal coordinate of this w.r.t. X11 root window but I am not sure to know how to code that.

The call to XGetWindowAttributes  in function fl_handle of FLTK (file src/Fl_x.cxx, near line 2159)  is probably related to my question, and so is the top_window_offset member function of Fl_Widget

答:
There is a function inherited from Fl_Widget: x() and x(); you may call those to know the parent window position:

```<code>class RpsGui_CommandWindow {
    void your_func () {
        int parent_x = RpsGui_Window::x();
    }
};
</code>```

With X11, you may call XQueryTree to obtain the root ID of the window and then call XGetWindowAttributes to know the value you want. You need the X11 window ID of your command window though. For that In FLTK docs there is some documented global variables to access that data. It must be done after calling Fl_Window::make_current()

I understand that this "command" window is the menu, which in the image it seems positioned in the right position but with wrong width or may be the window manager has changed the size. In that case you should have an event handler to resize your widgets.

Body:
context

I am coding with others RefPerSys, a GPLv3+ project in C++17 on gitlab for GNU/Linux/x86-64/Debian/Sid. Its fltk-branch git branch is using FLTK 1.4, compiled from source code, with an Xorg display server.

I have C++ classes like (in file headfltk_rps.hh):

class RpsGui_Window: public Fl_Double_Window
{
  static std::set<RpsGui_Window*> _set_of_gui_windows_;
public:
  virtual int handle(int);
protected:
  Fl_Menu_Bar *guiwin_menubar;
  std::string guiwin_label;
  virtual void initialize_menubar(void) =0;
  RpsGui_Window(int w, int h, const std::string& lab);
  RpsGui_Window(int x, int y, int w, int h, const std::string& lab);
public:
  virtual ~RpsGui_Window();
  /// .... skipping irrelevant code
  const std::string label_str(void) const {
    return guiwin_label;
  };
}; /// end class RpsGui_Window

class RpsGui_CommandWindow : public RpsGui_Window
{
  static constexpr int right_menu_gap = 16;
  static constexpr int menu_height = 20;
  Fl_Pack* cmdwin_pack;
  friend  void rps_fltk_initialize(int &,char**);
  virtual void initialize_menubar(void);
  virtual void initialize_pack(void);
  static void menu_dump_cb(Fl_Widget*, void*);
  static void menu_exit_cb(Fl_Widget*, void*);
public:
  RpsGui_CommandWindow(int w, int h, const std::string& lab);
  RpsGui_CommandWindow(int x, int y, int w, int h, const std::string& lab);
  virtual ~RpsGui_CommandWindow();
};              // end class RpsGui_CommandWindow

and I am debugging with old C++ macros outputting to std::cerr  (defined in refpersys.hh lines 315 and following) such as below:

 RPS_DEBUG_LOG(GUI, "RpsGui_CommandWindow::initialize_pack this:"
               <<  RpsGui_ShowWidget(this)
               << std::endl << "... cmdwin_pack:"
               << RpsGui_ShowWidget(cmdwin_pack));

Something is still wrong on the screen.
See RefPerSys issue#33 for even more details (with a screenshot).

I would like to output the position of a given FLTK widget w.r.t. my X11 root window. FWIW xdpyinfo is giving (with a lot of output skipped)

name of display:    :0
version number:    11.0
vendor string:    The X.Org Foundation
vendor release number:    12008000
X.Org version: 1.20.8

screen #0:
  dimensions:    5360x1440 pixels (1418x381 millimeters)
  resolution:    96x96 dots per inch

question

In other words, I want to code (for debugging purposes)

int RpsGui_Window::x_wrt_root() const;

as a member function returning the top left corner horizontal coordinate of this w.r.t. X11 root window but I am not sure to know how to code that.

The call to XGetWindowAttributes  in function fl_handle of FLTK (file src/Fl_x.cxx, near line 2159)  is probably related to my question, and so is the top_window_offset member function of Fl_Widget

Body:
There is a function inherited from Fl_Widget: x() and x(); you may call those to know the parent window position:

class RpsGui_CommandWindow {
    void your_func () {
        int parent_x = RpsGui_Window::x();
    }
};

With X11, you may call XQueryTree to obtain the root ID of the window and then call XGetWindowAttributes to know the value you want. You need the X11 window ID of your command window though. For that In FLTK docs there is some documented global variables to access that data. It must be done after calling Fl_Window::make_current()

I understand that this "command" window is the menu, which in the image it seems positioned in the right position but with wrong width or may be the window manager has changed the size. In that case you should have an event handler to resize your widgets.


问:
Iterate over class inheritances in C++
Assume I have a some classes architecture (the number of the classes is growing up during the development time), that each class inherit from N classes with the same basic interface. What is the best way (if possible) to create a base function (in the base class OR in the derived class) that will iterate over the inheritances?

Target: Avoid developers mistakes and make sure we won't forget to call all the base functions from all of the inheritances & make the code more clear to read and understandable.

Please see edit notes for updated state

Short Example:

```<code>class shared_base {
public:
    virtual void func() = 0;
}

class base_1 : virtual public shared_base {
public:
    void func() override {}
}

class base_2 : virtual public shared_base {
public:
    void func() override {}
}

class target : virtual public base_1, virtual public base_2 {
public:
    void func() override {
        // Instead of:
        base_1::func();
        base_2::func();
        // ... My func() implementation
        /*
        ~~TODO~~
        for_each(std::begin(inheritances), std::end(inheritances), [](auto& inheritance) -> void { inheritance::func(); })
        ~~TODO~~
        */
    }
}
</code>```

More descriptive & practical example:

```<code>class base {
public:
    virtual void func() = 0;
    /*...Some interface (pure virtual) functions...*/
}

class base_core : virtual public base {
public:
    void func() override {}
    /*...Some base implementations for the rest...*/

protected:
    template <typename FuncT>
    virtual void iterate_over_base_core_inheritances(FuncT function_to_apply) {
        /*~~TODO~~*/
    }
}

template <class Decorator = base_core, typename = typename std::enable_if<std::is_base_of<base_core, Decorator>::value>::type>
class core_1 : virtual public Decorator {
public:
    void func() override {
        // Will iterate (once) over Decorator
        /*iterate_over_base_core_inheritances([](core_base*) -> void {
            // Implementation
        });*/
        // Instead of:
        Decorator::func();
    }
    /*More functions implementations*/
}

template <class Decorator = base_core, typename = typename std::enable_if<std::is_base_of<base_core, Decorator>::value>::type>
class core_2 : virtual public core_1<>, virtual public Decorator {
public:
    void func() override {
        // Will iterate (twice) over core_1 and Decorator
        /*iterate_over_base_core_inheritances([](core_base*) -> void {
            // Implementation
        });*/
        // Instead of:
        Decorator::func();
        core_1::func();
        //... Self func() implementation
    }
    /*More functions implementations*/

protected:
    // If it's not possible doing it in the upper hierarchy level is it possible do it here?
    template <typename FuncT>
    void iterate_over_base_core_inheritances(FuncT function_to_apply) override {
        /*~~TODO~~*/
    }
}
</code>```

Some things to know:

I am working on Linux 64x platform (Ubuntu 16.04)- if it's matter for the answers.

The idea behind this code is to create kind of Decorator DP, which will be easy to extend and to understand, and also will enable the developers to use the protected functions/attributes of the base class.

A practical example (for my actual use) can be found in this commit.

Edit:

Thanks to @RaymondChen I got a working solution, with (so far) only one minor issue: Every time I want to use a class that implemented this way, I need to specify the core_base class in it's template arguments list (before- I was using the default type parameter). I am looking for a way to solve this issue.

The current solution:

```<code>template <class ...Decorators>
class core_2 : virtual public Decorators... {
public:
    static_assert((std::is_base_of<base_core, Decorators>::value && ...), "All decorators must inherit from base_core class.");

    void func() override {
        (Decorators::func(), ...);
        //... Self func() implementation
    }
    /*More functions implementations*/
}
</code>```

Creating an instance example:

Current:

std::shared_ptr<base> base = std::make_shared<core_2<core_1<base_core>, core_3<base_core>>>();

Desired:

std::shared_ptr<base> base = std::make_shared<core_2<core_1<>, core_3<>>>();

A practical example (for my actual use) can be found in this commit.

答:
Thanks to @RaymondChen I got really close to my original target with the following solution [See update section at the bottom]:

```<code>template <class ...Decorators>
class core_2 : virtual public Decorators... {
public:
    static_assert((std::is_base_of<base_core, Decorators>::value && ...), "All decorators must inherit from base_core class.");

    void func() override {
        (Decorators::func(), ...);
        //... Self func() implementation
    }
    /*More functions implementations*/
}
</code>```

Explanation:

Using parameters pack we can create a "list" of classes we inherit from, and using folding expression [c++17] we can implement it in just few lines of code.

Pros compare to my original idea:

The object creation line is more clear and logically now:

Before:
std::shared_ptr<base> base = std::make_shared<core_2<core_1<core_3<>>>>();

After:
std::shared_ptr<base> base = std::make_shared<core_2<core_1<base_core>, core_3<base_core>>>();

Because core_1 & core_3 are independent, but core_2 is using both of them.

No need of new function in the base/derived class, it's just fit within the target line (for example in is_equal function that didn't mention within this post).

Lost functionality:

Template validation of is_base_of (Solved with static_assert & fold expressions).

Default inheritance in case that no inheritance specified is not possible yet (Still trying to solve).

Current:

std::shared_ptr<base> base = std::make_shared<core_2<core_1<base_core>, core_3<base_core>>>();

Desired:

std::shared_ptr<base> base = std::make_shared<core_2<core_1<>, core_3<>>>();

Update

After a lot of research and tries, I came up with the following solution (improved also with C++20 concepts feature):

```<code>template <class T>
        concept Decorator = std::is_base_of_v<base_core, T>;

class empty_inheritance {};

template<typename Base = base_core, typename ...Decorators>
struct base_if_not_exists {
    static constexpr bool value = sizeof...(Decorators);
    using type = typename std::conditional<value, empty_inheritance, Base>::type;
};

template <Decorator ...Decorators>
class core_2 : virtual public base_if_not_exists<base_core, Decorators...>::type, virtual public Decorators... {
public:
    void func() override {
        if constexpr (!base_if_not_exists<base_core, Decorators...>::value) {
            base_core::func();
        }
        (Decorators::func(), ...);
        //... Self func() implementation
    }
    /*More functions implementations*/
}
</code>```

No functionality lost :)

Title: Iterate over class inheritances in C++

Body:
Assume I have a some classes architecture (the number of the classes is growing up during the development time), that each class inherit from N classes with the same basic interface. What is the best way (if possible) to create a base function (in the base class OR in the derived class) that will iterate over the inheritances?

Target: Avoid developers mistakes and make sure we won't forget to call all the base functions from all of the inheritances & make the code more clear to read and understandable.

Please see edit notes for updated state

Short Example:

class shared_base {
public:
    virtual void func() = 0;
}

class base_1 : virtual public shared_base {
public:
    void func() override {}
}

class base_2 : virtual public shared_base {
public:
    void func() override {}
}

class target : virtual public base_1, virtual public base_2 {
public:
    void func() override {
        // Instead of:
        base_1::func();
        base_2::func();
        // ... My func() implementation
        /*
        ~~TODO~~
        for_each(std::begin(inheritances), std::end(inheritances), [](auto& inheritance) -> void { inheritance::func(); })
        ~~TODO~~
        */
    }
}

More descriptive & practical example:

class base {
public:
    virtual void func() = 0;
    /*...Some interface (pure virtual) functions...*/
}

class base_core : virtual public base {
public:
    void func() override {}
    /*...Some base implementations for the rest...*/

protected:
    template <typename FuncT>
    virtual void iterate_over_base_core_inheritances(FuncT function_to_apply) {
        /*~~TODO~~*/
    }
}

template <class Decorator = base_core, typename = typename std::enable_if<std::is_base_of<base_core, Decorator>::value>::type>
class core_1 : virtual public Decorator {
public:
    void func() override {
        // Will iterate (once) over Decorator
        /*iterate_over_base_core_inheritances([](core_base*) -> void {
            // Implementation
        });*/
        // Instead of:
        Decorator::func();
    }
    /*More functions implementations*/
}

template <class Decorator = base_core, typename = typename std::enable_if<std::is_base_of<base_core, Decorator>::value>::type>
class core_2 : virtual public core_1<>, virtual public Decorator {
public:
    void func() override {
        // Will iterate (twice) over core_1 and Decorator
        /*iterate_over_base_core_inheritances([](core_base*) -> void {
            // Implementation
        });*/
        // Instead of:
        Decorator::func();
        core_1::func();
        //... Self func() implementation
    }
    /*More functions implementations*/

protected:
    // If it's not possible doing it in the upper hierarchy level is it possible do it here?
    template <typename FuncT>
    void iterate_over_base_core_inheritances(FuncT function_to_apply) override {
        /*~~TODO~~*/
    }
}

Some things to know:

I am working on Linux 64x platform (Ubuntu 16.04)- if it's matter for the answers.

The idea behind this code is to create kind of Decorator DP, which will be easy to extend and to understand, and also will enable the developers to use the protected functions/attributes of the base class.

A practical example (for my actual use) can be found in this commit.

Edit:

Thanks to @RaymondChen I got a working solution, with (so far) only one minor issue: Every time I want to use a class that implemented this way, I need to specify the core_base class in it's template arguments list (before- I was using the default type parameter). I am looking for a way to solve this issue.

The current solution:

template <class ...Decorators>
class core_2 : virtual public Decorators... {
public:
    static_assert((std::is_base_of<base_core, Decorators>::value && ...), "All decorators must inherit from base_core class.");

    void func() override {
        (Decorators::func(), ...);
        //... Self func() implementation
    }
    /*More functions implementations*/
}

Creating an instance example:

Current:

std::shared_ptr<base> base = std::make_shared<core_2<core_1<base_core>, core_3<base_core>>>();

Desired:

std::shared_ptr<base> base = std::make_shared<core_2<core_1<>, core_3<>>>();

A practical example (for my actual use) can be found in this commit.

Body:
Thanks to @RaymondChen I got really close to my original target with the following solution [See update section at the bottom]:

template <class ...Decorators>
class core_2 : virtual public Decorators... {
public:
    static_assert((std::is_base_of<base_core, Decorators>::value && ...), "All decorators must inherit from base_core class.");

    void func() override {
        (Decorators::func(), ...);
        //... Self func() implementation
    }
    /*More functions implementations*/
}

Explanation:

Using parameters pack we can create a "list" of classes we inherit from, and using folding expression [c++17] we can implement it in just few lines of code.

Pros compare to my original idea:

The object creation line is more clear and logically now:

Before:
std::shared_ptr<base> base = std::make_shared<core_2<core_1<core_3<>>>>();

After:
std::shared_ptr<base> base = std::make_shared<core_2<core_1<base_core>, core_3<base_core>>>();

Because core_1 & core_3 are independent, but core_2 is using both of them.

No need of new function in the base/derived class, it's just fit within the target line (for example in is_equal function that didn't mention within this post).

Lost functionality:

Template validation of is_base_of (Solved with static_assert & fold expressions).

Default inheritance in case that no inheritance specified is not possible yet (Still trying to solve).

Current:

std::shared_ptr<base> base = std::make_shared<core_2<core_1<base_core>, core_3<base_core>>>();

Desired:

std::shared_ptr<base> base = std::make_shared<core_2<core_1<>, core_3<>>>();

Update

After a lot of research and tries, I came up with the following solution (improved also with C++20 concepts feature):

template <class T>
        concept Decorator = std::is_base_of_v<base_core, T>;

class empty_inheritance {};

template<typename Base = base_core, typename ...Decorators>
struct base_if_not_exists {
    static constexpr bool value = sizeof...(Decorators);
    using type = typename std::conditional<value, empty_inheritance, Base>::type;
};

template <Decorator ...Decorators>
class core_2 : virtual public base_if_not_exists<base_core, Decorators...>::type, virtual public Decorators... {
public:
    void func() override {
        if constexpr (!base_if_not_exists<base_core, Decorators...>::value) {
            base_core::func();
        }
        (Decorators::func(), ...);
        //... Self func() implementation
    }
    /*More functions implementations*/
}

No functionality lost :)


问:
Convert CSV files from multiple directory into parquet in PySpark
I have CSV files from multiple paths that are not parent directories in s3 bucket. All the tables have the same partition keys.

the directory of the s3:

```<code>table_name_1/partition_key_1 = <pk_1>/partition_key_2 = <pk_2>/file.csv
table_name_2/partition_key_1 = <pk_1>/partition_key_2 = <pk_2>/file.csv
...
</code>```

I need to convert these csv files into parquet files and store them in another s3 bucket that has the same directory structure.

the directory of another s3:

```<code>table_name_1/partition_key_1 = <pk_1>/partition_key_2 = <pk_2>/file.parquet
table_name_2/partition_key_1 = <pk_1>/partition_key_2 = <pk_2>/file.parquet
...
</code>```

I have a solution is iterating through the s3 bucket and find the CSV file and convert it to parquet and save to the another S3 path. I find this way is not efficient, because i have a loop and did the conversion one file by one file.

I want to utilize the spark library to improve the efficiency.
Then, I tried:

```<code>spark.read.csv('s3n://bucket_name/table_name_1/').write.partitionBy('partition_key_1', 'partition_key_2').parquet('s3n://another_bucket/table_name_1')
</code>```

This way works good for each table, but to optimize it more, I want to take the table_name as a parameter, something like:

```<code>TABLE_NAMES = [table_name_1, table_name_2, ...]
spark.read.csv('s3n://bucket_name/{*TABLE_NAMES}/').write.partitionBy('partition_key_1', 'partition_key_2').parquet('s3n://another_bucket/{*TABLE_NAMES}')
</code>```

Thanks

答:
The mentioned question provides solutions for reading multiple files at once. The method spark.read.csv(...) accepts one or multiple paths as shown here. For reading the files you can apply the same logic. Although, when it comes to writing, Spark will merge all the given dataset/paths into one Dataframe. Therefore it is not possible to generate from one single dataframe multiple dataframes without applying a custom logic first. So to conclude, there is not such a method for extracting the initial dataframe directly into multiple directories i.e df.write.csv(*TABLE_NAMES).

The good news is that Spark provides a dedicated function namely input_file_name() which returns the file path of the current record. You can use it in combination with TABLE_NAMES to filter on the table name.

Here it is one possible untested PySpark solution:

from pyspark.sql.functions import input_file_name

TABLE_NAMES = [table_name_1, table_name_2, ...]

source_path = "s3n://bucket_name/"
input_paths = [f"{source_path}/{t}" for t in TABLE_NAMES]

all_df = spark.read.csv(*input_paths) \
              .withColumn("file_name", input_file_name()) \
              .cache()

dest_path = "s3n://another_bucket/"

def write_table(table_name: string) -> None:
   all_df.where(all_df["file_name"].contains(table_name))
     .write
     .partitionBy('partition_key_1','partition_key_2')
     .parquet(f"{dest_path}/{table_name}")

for t in TABLE_NAMES:
   write_table(t)
```

Explanation:

We generate and store the input paths into input_paths. This will create paths such as: s3n://bucket_name/table1, s3n://bucket_name/table2 ... s3n://bucket_name/tableN.

Then we load all the paths into one dataframe in which we add a new column called file_name, this will hold the path of each row. Notice that we also use cache here, this is important since we have multiple len(TABLE_NAMES) actions in the following code. Using cache will prevent us from loading the datasource again and again.

Next we create the write_table which is responsible for saving the data for the given table. The next step is to filter based on the table name using all_df["file_name"].contains(table_name), this will return only the records that contain the value of the table_name in the file_name column. Finally we save the filtered data as you already did.

In the last step we call write_table for every item of TABLE_NAMES.

Related links

How to import multiple csv files in a single load?

Get HDFS file path in PySpark for files in sequence file format

Body:
I have CSV files from multiple paths that are not parent directories in s3 bucket. All the tables have the same partition keys.

the directory of the s3:

table_name_1/partition_key_1 = <pk_1>/partition_key_2 = <pk_2>/file.csv
table_name_2/partition_key_1 = <pk_1>/partition_key_2 = <pk_2>/file.csv
...

I need to convert these csv files into parquet files and store them in another s3 bucket that has the same directory structure.

the directory of another s3:

table_name_1/partition_key_1 = <pk_1>/partition_key_2 = <pk_2>/file.parquet
table_name_2/partition_key_1 = <pk_1>/partition_key_2 = <pk_2>/file.parquet
...

I have a solution is iterating through the s3 bucket and find the CSV file and convert it to parquet and save to the another S3 path. I find this way is not efficient, because i have a loop and did the conversion one file by one file.

I want to utilize the spark library to improve the efficiency.
Then, I tried:

spark.read.csv('s3n://bucket_name/table_name_1/').write.partitionBy('partition_key_1', 'partition_key_2').parquet('s3n://another_bucket/table_name_1')

This way works good for each table, but to optimize it more, I want to take the table_name as a parameter, something like:

TABLE_NAMES = [table_name_1, table_name_2, ...]
spark.read.csv('s3n://bucket_name/{*TABLE_NAMES}/').write.partitionBy('partition_key_1', 'partition_key_2').parquet('s3n://another_bucket/{*TABLE_NAMES}')

Thanks

Body:
The mentioned question provides solutions for reading multiple files at once. The method spark.read.csv(...) accepts one or multiple paths as shown here. For reading the files you can apply the same logic. Although, when it comes to writing, Spark will merge all the given dataset/paths into one Dataframe. Therefore it is not possible to generate from one single dataframe multiple dataframes without applying a custom logic first. So to conclude, there is not such a method for extracting the initial dataframe directly into multiple directories i.e df.write.csv(*TABLE_NAMES).

The good news is that Spark provides a dedicated function namely input_file_name() which returns the file path of the current record. You can use it in combination with TABLE_NAMES to filter on the table name.

Here it is one possible untested PySpark solution:

from pyspark.sql.functions import input_file_name

TABLE_NAMES = [table_name_1, table_name_2, ...]

source_path = "s3n://bucket_name/"
input_paths = [f"{source_path}/{t}" for t in TABLE_NAMES]

all_df = spark.read.csv(*input_paths) \
              .withColumn("file_name", input_file_name()) \
              .cache()

dest_path = "s3n://another_bucket/"

def write_table(table_name: string) -> None:
   all_df.where(all_df["file_name"].contains(table_name))
     .write
     .partitionBy('partition_key_1','partition_key_2')
     .parquet(f"{dest_path}/{table_name}")

for t in TABLE_NAMES:
   write_table(t)

Explanation:

We generate and store the input paths into input_paths. This will create paths such as: s3n://bucket_name/table1, s3n://bucket_name/table2 ... s3n://bucket_name/tableN.

Then we load all the paths into one dataframe in which we add a new column called file_name, this will hold the path of each row. Notice that we also use cache here, this is important since we have multiple len(TABLE_NAMES) actions in the following code. Using cache will prevent us from loading the datasource again and again.

Next we create the write_table which is responsible for saving the data for the given table. The next step is to filter based on the table name using all_df["file_name"].contains(table_name), this will return only the records that contain the value of the table_name in the file_name column. Finally we save the filtered data as you already did.

In the last step we call write_table for every item of TABLE_NAMES.

Related links

How to import multiple csv files in a single load?

Get HDFS file path in PySpark for files in sequence file format


问:
Image serial is not maintaining in OpenCV
Using OpenCV I have read all the images of a folder then converted it into grayscale and then I have saved all the converted images to a new folder which name is "binary image". But when I again read all the images from the "binary image" folder to show histogram then the serial of images in "binary image" folder is not maintaining. Instead of image number 12th, the histogram is showing is 19th number image in the folder. But the total number of histograms are good as it is supposed to be. How can I maintain serial of images?

How i read images:

```<code>images1=[cv2.imread(file) for file in glob.glob(r"C:\Users\USER\Handcrafted dataset\binary_image/*.jpg")]
</code>```

thats how i save histogram image:

```<code> dir1=r"C:\Users\USER\Handcrafted dataset\histogram"
for i,img in enumerate(images1):
    plt.figure(figsize=(5,5))
    plt.hist(img.ravel(),256,[0,256],label=str(np.mean(img)))
    plt.legend(loc="upper right")
    plt.savefig(dir1+"\\"+str(i)+".png")
    print(i)

    plt.show()
</code>```

to save gray scale image:

```<code>di=r"C:\Users\USER\Handcrafted dataset\binary_image"
for i,img in enumerate(images):
    img = rgb2gray(img)
    plt.figure(figsize=(5,5))
    print(i)
    plt.imshow(img ,cmap='gray')
    io.imsave(di+"\\"+str(i)+".jpg",img)
</code>```

at first when I read images then the image name was random but after converting into grayscale I have saved the images in "binary image" folder as renaming from 0 to 51. That means 0.jpg, 1.jpg serially. After that to show histogram I have used the binay_image folder where images were sorted as I mentioned before. but histogram is showing randomly.

答:
ok, here's a problem. When you use glob.glob() file names come in no specific order. That is, unless you sort them yourself or do something about the order, you cannot assume the file 001.jpg coming before 002.jpg.

So, when you enumerate(images1) -- the results will be like you described. The numbers and file names won't match.

What you can do, the simplest is to add sorted() to the file list:

```<code>images1=[cv2.imread(file) for file in sorted(glob.glob(r"C:\Users\..."))]
</code>```

This however assumes your files are properly numbered without files missing in between, like 006 and then 010 with 007, 008 and 009 missing.

UPDATE: once I saw the file names: 0.jpg, 1.jpg I immediately know what the problem is. File names are sorted lexicographically, that is, 1, 10, 11, 12, ... 2, 20, 21, 22 -- I hope you catch my drift...

When you name your files, please, add leading zeros, they are there for a reason, to keep your files in the order, instead of:

```<code>plt.savefig(dir1+"\\"+str(i)+".png")
</code>```

use:

```<code>plt.savefig(os.path.join( dir1, "%06d.png" % i))
</code>```

Body:
Using OpenCV I have read all the images of a folder then converted it into grayscale and then I have saved all the converted images to a new folder which name is "binary image". But when I again read all the images from the "binary image" folder to show histogram then the serial of images in "binary image" folder is not maintaining. Instead of image number 12th, the histogram is showing is 19th number image in the folder. But the total number of histograms are good as it is supposed to be. How can I maintain serial of images?

How i read images:

images1=[cv2.imread(file) for file in glob.glob(r"C:\Users\USER\Handcrafted dataset\binary_image/*.jpg")]

thats how i save histogram image:

 dir1=r"C:\Users\USER\Handcrafted dataset\histogram"
for i,img in enumerate(images1):
    plt.figure(figsize=(5,5))
    plt.hist(img.ravel(),256,[0,256],label=str(np.mean(img)))
    plt.legend(loc="upper right")
    plt.savefig(dir1+"\\"+str(i)+".png")
    print(i)

    plt.show()

to save gray scale image:

di=r"C:\Users\USER\Handcrafted dataset\binary_image"
for i,img in enumerate(images):
    img = rgb2gray(img)
    plt.figure(figsize=(5,5))
    print(i)
    plt.imshow(img ,cmap='gray')
    io.imsave(di+"\\"+str(i)+".jpg",img)

at first when I read images then the image name was random but after converting into grayscale I have saved the images in "binary image" folder as renaming from 0 to 51. That means 0.jpg, 1.jpg serially. After that to show histogram I have used the binay_image folder where images were sorted as I mentioned before. but histogram is showing randomly.

Body:
ok, here's a problem. When you use glob.glob() file names come in no specific order. That is, unless you sort them yourself or do something about the order, you cannot assume the file 001.jpg coming before 002.jpg.

So, when you enumerate(images1) -- the results will be like you described. The numbers and file names won't match.

What you can do, the simplest is to add sorted() to the file list:

images1=[cv2.imread(file) for file in sorted(glob.glob(r"C:\Users\..."))]

This however assumes your files are properly numbered without files missing in between, like 006 and then 010 with 007, 008 and 009 missing.

UPDATE: once I saw the file names: 0.jpg, 1.jpg I immediately know what the problem is. File names are sorted lexicographically, that is, 1, 10, 11, 12, ... 2, 20, 21, 22 -- I hope you catch my drift...

When you name your files, please, add leading zeros, they are there for a reason, to keep your files in the order, instead of:

plt.savefig(dir1+"\\"+str(i)+".png")

use:

plt.savefig(os.path.join( dir1, "%06d.png" % i))


问:
insertRowBefore() not displaying values
Can someone advise what I am doing wrong with the following code please?
I know it's simple but I think I've been staring at the screen for too long!

```<code>function addDelivery() {
  var ss = SpreadsheetApp.getActiveSpreadsheet();
  var srcSheet = ss.getSheetByName("Sheet7");

  var range = srcSheet.getRange("B3:J3").getValues();
  srcSheet.insertRowBefore(6);
  srcSheet.getRange(6,2,1,4).setValues([['B3', 'C3', 'D3', 'E3' , 'F3' , 'G3' , 'H3' , 'I3' , 'J3']]);
</code>```

I'm obviously trying to append the values in cells B3:J3, not have it show B3, C3, D3 etc in the appended row!

答:
Using getRange() and setValues() together

```<code>function addDelivery() {
  var ss = SpreadsheetApp.getActiveSpreadsheet();
  var srcSheet = ss.getSheetByName("Sheet7");
  srcSheet.insertRowBefore(6);
</code>```

First Parameter is row 3

Second Parameter is column B or 2

Third Parameter is number of rows which is 1

Fourth Parameter is number of columns which 9

```<code>  srcSheet.getRange(3,2,1,9).setValues([['B3', 'C3', 'D3', 'E3' , 'F3' , 'G3' , 'H3' , 'I3' , 'J3']]);
</code>```

You could have also done it this way to keep from having to count

```<code>var data=[['B3', 'C3', 'D3', 'E3' , 'F3' , 'G3' , 'H3' , 'I3' , 'J3']];
srcSheet.getRange(3,2,data.length,data[0].length).setValues(data);
</code>```

Sheet.getRange()

Range.setValues()

Title: insertRowBefore() not displaying values

Body:
Can someone advise what I am doing wrong with the following code please?
I know it's simple but I think I've been staring at the screen for too long!

function addDelivery() {
  var ss = SpreadsheetApp.getActiveSpreadsheet();
  var srcSheet = ss.getSheetByName("Sheet7");

  var range = srcSheet.getRange("B3:J3").getValues();
  srcSheet.insertRowBefore(6);
  srcSheet.getRange(6,2,1,4).setValues([['B3', 'C3', 'D3', 'E3' , 'F3' , 'G3' , 'H3' , 'I3' , 'J3']]);

I'm obviously trying to append the values in cells B3:J3, not have it show B3, C3, D3 etc in the appended row!

Body:
Using getRange() and setValues() together

function addDelivery() {
  var ss = SpreadsheetApp.getActiveSpreadsheet();
  var srcSheet = ss.getSheetByName("Sheet7");
  srcSheet.insertRowBefore(6);

First Parameter is row 3

Second Parameter is column B or 2

Third Parameter is number of rows which is 1

Fourth Parameter is number of columns which 9

  srcSheet.getRange(3,2,1,9).setValues([['B3', 'C3', 'D3', 'E3' , 'F3' , 'G3' , 'H3' , 'I3' , 'J3']]);

You could have also done it this way to keep from having to count

var data=[['B3', 'C3', 'D3', 'E3' , 'F3' , 'G3' , 'H3' , 'I3' , 'J3']];
srcSheet.getRange(3,2,data.length,data[0].length).setValues(data);

Sheet.getRange()

Range.setValues()


问:
How to modify XSD element from string to xml?
I have the following part in my XSD:

```<code> <xs:element name="Parameter">
   <xs:complexType mixed="true">
     <xs:attribute name="Name" use="required" type="xs:string"/>
   </xs:complexType>
 </xs:element>
</code>```

which means that value for a Parameter can be any string. Can I somehow specify that value for Parameter can be  any valid xml like <aaa></aaa> ? Something like type="xs:xml"

答:
You will have to use

```<code><xs:element name="Parameter" type="myType>
</code>```

where

```<code><complexType name="myType>
<choice>
<element "myXml" type="xml">
<element "myStr" type="xs:sting">
</choice>
</complexType>
</code>```

and you will have to define xml like a pattern of xml you want to be your value.

Body:
I have the following part in my XSD:

 <xs:element name="Parameter">
   <xs:complexType mixed="true">
     <xs:attribute name="Name" use="required" type="xs:string"/>
   </xs:complexType>
 </xs:element>

which means that value for a Parameter can be any string. Can I somehow specify that value for Parameter can be  any valid xml like <aaa></aaa> ? Something like type="xs:xml"

Body:
You will have to use

<xs:element name="Parameter" type="myType>

where

<complexType name="myType>
<choice>
<element "myXml" type="xml">
<element "myStr" type="xs:sting">
</choice>
</complexType>

and you will have to define xml like a pattern of xml you want to be your value.


问:
Spring Social Google return after login wrong
I have implemented Spring social for Facebook, Twitter and Google.
For the first 2 it works but Google redirect the user to /signin after accepting on the google page where you allow my application access to your google+ account.
/signin is a 404. If I make a page for /signin the page is shown but the user is not logged in.

I am to familiar with the underlying mechanics of connectController and GoogleConnectionFactory but i believe it should redirect the request back to where it came from and with access tokens. This is /auth/google i believe. I tried a redirect from /signin to /auth/google but that did not work. As mentioned for Twitter and Facebook there was no problem.

The code:
public class SocialConfig implements SocialConfigurer

```<code>@Override
public void addConnectionFactories(ConnectionFactoryConfigurer cfConfig, Environment env) {
    cfConfig.addConnectionFactory(new FacebookConnectionFactory(
            env.getProperty("facebook.app.id"),
            env.getProperty("facebook.app.secret")
    ));
    cfConfig.addConnectionFactory(new TwitterConnectionFactory(
            env.getProperty("twitter.consumer.key"),
            env.getProperty("twitter.consumer.secret")
    ));
    cfConfig.addConnectionFactory(new GoogleConnectionFactory(
            env.getProperty("google.app.id"),
            env.getProperty("google.app.secret")));
}

@Bean
public ConnectController connectController(ConnectionFactoryLocator connectionFactoryLocator, ConnectionRepository connectionRepository) {
    return new ConnectController(connectionFactoryLocator, connectionRepository);
}
</code>```

SecurityConfig.java

```<code>.apply(getSpringSocialConfigurer()); //to http.

private SpringSocialConfigurer getSpringSocialConfigurer() {
    SpringSocialConfigurer config = new SpringSocialConfigurer();
    config.alwaysUsePostLoginUrl(true);
    config.postLoginUrl("/");

    return config;
}
</code>```

the jsp.

```<code>  //facebook
                <a href="${pageContext.request.contextPath}/auth/facebook" class="hidden-xs">
                    <img src="<c:url value="/static/img/fb-login.png"/>" height="32"/>
                </a>

//twitter
                <a href="${pageContext.request.contextPath}/auth/twitter" class="hidden-xs">
                    <img src="<c:url value="/static/img/twitter-login.png"/>" height="20" width="24"/>
                </a>
 //google
            <form name="go_signin" id="go_signin" action="<c:url value="/auth/google"/>" method="POST" class="float-left">
                <div onclick="this.parentNode.submit();">
                    <img src="<c:url value="/static/img/google-login.png"/>" height="32"/>
                </div>
                <input type="hidden" name="${_csrf.parameterName}" value="${_csrf.token}" />
                <input type="hidden" name="scope" value="https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/plus.login" />
            </form>
</code>```

In the google app console i set Redirect URIs to : {my_url}/auth/google
and Javascript Origins just to my url.
I tried almost every combination. anything but /auth/google will result in a redirect uri mismatch error before you get to the google login page

Any idea`s are welcome, thank you in advance.

答: updating version of spring-social-google in pom.xml to 1.0.0.RC1 solved my problem.</p>

Body:
I have implemented Spring social for Facebook, Twitter and Google.
For the first 2 it works but Google redirect the user to /signin after accepting on the google page where you allow my application access to your google+ account.
/signin is a 404. If I make a page for /signin the page is shown but the user is not logged in.

I am to familiar with the underlying mechanics of connectController and GoogleConnectionFactory but i believe it should redirect the request back to where it came from and with access tokens. This is /auth/google i believe. I tried a redirect from /signin to /auth/google but that did not work. As mentioned for Twitter and Facebook there was no problem.

The code:
public class SocialConfig implements SocialConfigurer

@Override
public void addConnectionFactories(ConnectionFactoryConfigurer cfConfig, Environment env) {
    cfConfig.addConnectionFactory(new FacebookConnectionFactory(
            env.getProperty("facebook.app.id"),
            env.getProperty("facebook.app.secret")
    ));
    cfConfig.addConnectionFactory(new TwitterConnectionFactory(
            env.getProperty("twitter.consumer.key"),
            env.getProperty("twitter.consumer.secret")
    ));
    cfConfig.addConnectionFactory(new GoogleConnectionFactory(
            env.getProperty("google.app.id"),
            env.getProperty("google.app.secret")));
}

@Bean
public ConnectController connectController(ConnectionFactoryLocator connectionFactoryLocator, ConnectionRepository connectionRepository) {
    return new ConnectController(connectionFactoryLocator, connectionRepository);
}

SecurityConfig.java

.apply(getSpringSocialConfigurer()); //to http.

private SpringSocialConfigurer getSpringSocialConfigurer() {
    SpringSocialConfigurer config = new SpringSocialConfigurer();
    config.alwaysUsePostLoginUrl(true);
    config.postLoginUrl("/");

    return config;
}

the jsp.

  //facebook

                    <img src="<c:url value="/static/img/fb-login.png"/>" height="32"/>

//twitter

                    <img src="<c:url value="/static/img/twitter-login.png"/>" height="20" width="24"/>

 //google
            <form name="go_signin" id="go_signin" action="<c:url value="/auth/google"/>" method="POST" class="float-left">

                    <img src="<c:url value="/static/img/google-login.png"/>" height="32"/>

                <input type="hidden" name="${_csrf.parameterName}" value="${_csrf.token}" />
                <input type="hidden" name="scope" value="https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/plus.login" />
            </form>

In the google app console i set Redirect URIs to : {my_url}/auth/google
and Javascript Origins just to my url.
I tried almost every combination. anything but /auth/google will result in a redirect uri mismatch error before you get to the google login page

Any idea`s are welcome, thank you in advance.


问:
how do i build gcc on a mac?
I'd like to build the latest version of gcc on a mac. I have the latest xcode but I'm looking for some of the c++0x features that are in more recent versions (the lambda functions, etc).

Are there any good step-by-step tutorials on doing this?

答:
You should look at the Homebrew project.

Homebrew allows you to do things like this:

```<code>brew install gcc
</code>```

Mac homebrew installation instructions are available here.

Title: how do i build gcc on a mac?

Body:
You should look at the Homebrew project.

Homebrew allows you to do things like this:

brew install gcc

Mac homebrew installation instructions are available here.


问:
pyGithub Pull request invalid head (422 API error)
Recently I've been trying to use pyGithub to create some automated PRs after running a make update in a bunch of repos using this snipppet.

```<code>body = 'Test PR 2 please ignore'
pr = repo.create_pull(title="Test PR please ignore",
                          body=body,
                          head="patch/helm-chart-update",
                          base="master")
pr
</code>```

I've consistently gotten this error message back from the API

github.GithubException.GithubException: 422 {"message": "Validation Failed", "errors": [{"resource": "PullRequest", "field": "head", "code": "invalid"}], "documentation_url": "https://developer.github.com/v3/pulls/#create-a-pull-request"}

Naturally, my first guess was that I'd mistyped the repo name, but this wasn't the case. Strangely, I get an identical error when I replace the head name with a nonexistent branch name like 'someGarbage'.

Next, I thought perhaps there were conflicts or other invalidating details about the PR (even though it was extremely unlikely due to the simplicity of the test branch I was working with), but the Github GUI created the PR without any issues.

Ultimately I was able to make a PR using https://pypi.org/project/git-pull-request/ , but unfortunately I can't see a way to avoid it opening vim in the shell (which would create a lot of toil due to the number of repos this will run on). If all else fails, I'll likely just use requests to make a manual call to the Github API, but I'd really like to use pyGithub, both to keep the code simple/maintainable (since I already use the library to search for target repos and clone them into /tmp) and because it generally seems like a nice tool I'd like to learn to use well.

Sorry this isn't the most reproducable due to the nature of the problem, but I'll be happy to elaborate on details. Any thoughts?

Update

After implementing this functionality using a requests call I learned thanks to this post that if you make a call using a 'params' argument to pass a json object to the POST request instead of a 'json' argument, you get the same error code. Beginning to suspect a bug, but I'd love to be proven wrong

答:
[{"resource": "PullRequest", "field": "head", "code": "invalid"}]

The error message is trying to communicate that the field/keyword called "head" contains invalid data. Looking at GitHub's REST endpoint documentation:

head: Required. The name of the branch where your changes are implemented. For cross-repository pull requests in the same network, namespace head with a user like this: username:branch.

The way you currently have it set up (head="patch/helm-chart-update", base="master") it goes to the online GitHub repository, looks for branch patch/helm-chart-update there, and tries to prepare a merge to master. Note this will not work if that branch only exists locally (i.e. you haven't done git push) or if you created a fork on GitHub (in which case you need to prefix it with your username as described)

Body:
Recently I've been trying to use pyGithub to create some automated PRs after running a make update in a bunch of repos using this snipppet.

body = 'Test PR 2 please ignore'
pr = repo.create_pull(title="Test PR please ignore",
                          body=body,
                          head="patch/helm-chart-update",
                          base="master")
pr

I've consistently gotten this error message back from the API

github.GithubException.GithubException: 422 {"message": "Validation Failed", "errors": [{"resource": "PullRequest", "field": "head", "code": "invalid"}], "documentation_url": "https://developer.github.com/v3/pulls/#create-a-pull-request"}

Naturally, my first guess was that I'd mistyped the repo name, but this wasn't the case. Strangely, I get an identical error when I replace the head name with a nonexistent branch name like 'someGarbage'.

Next, I thought perhaps there were conflicts or other invalidating details about the PR (even though it was extremely unlikely due to the simplicity of the test branch I was working with), but the Github GUI created the PR without any issues.

Ultimately I was able to make a PR using https://pypi.org/project/git-pull-request/ , but unfortunately I can't see a way to avoid it opening vim in the shell (which would create a lot of toil due to the number of repos this will run on). If all else fails, I'll likely just use requests to make a manual call to the Github API, but I'd really like to use pyGithub, both to keep the code simple/maintainable (since I already use the library to search for target repos and clone them into /tmp) and because it generally seems like a nice tool I'd like to learn to use well.

Sorry this isn't the most reproducable due to the nature of the problem, but I'll be happy to elaborate on details. Any thoughts?

Update

After implementing this functionality using a requests call I learned thanks to this post that if you make a call using a 'params' argument to pass a json object to the POST request instead of a 'json' argument, you get the same error code. Beginning to suspect a bug, but I'd love to be proven wrong


问:
How do I make impulse noise image efficiently using numpy?
I've been trying to create noisy image using the following piece of code:

```<code>def toImpulse(img, coef=1.2): # img - > 3dim numpy array, returns impulse image
    bmp = np.zeros(shape=(img.shape[0], img.shape[1], 1), dtype=np.float)
    for i in range(img.shape[0]):
        for j in range(img.shape[1]):
            bmp[i][j] = sum(img[i][j]) / 3
</code>```

Here I'm summing over the RGB channel and making the image single-channeled? I guess it can be written as numpy.sum()

```<code>    sum_ = 0
    max_ = np.amax(bmp)
    final = np.zeros(img.shape)
    for i in range(img.shape[0]):
        for j in range(img.shape[1]):
            sum_ += bmp[i][j]
            if sum_ >= coef * max_:
                final[i][j] = 1
                sum_ -= coef * max_
            else:
                final[i][j] = 0

    return final
</code>```

But it occurred to be very slow in real-time video processing. How can I make it more efficient using NumPy? Here is the example of the function's work
The original image
Noisy image

答:
You second for loop can be replaced by np.searchsorted in cumsum of your array.

```<code>def toImpulse2(img, coef=1.2): # img - > 3dim numpy array, returns impulse image
    bmp = img.sum(axis=2)
    max_ = bmp.max() * coef
    cumsum = np.cumsum(bmp)
    repeats = int(cumsum[-1] / max_)
    final = np.zeros((img.shape[0] * img.shape[1], img.shape[2]))
    final[np.searchsorted(cumsum, np.arange(1, repeats + 1) * max_)] = 1
    return final.reshape(img.shape)
</code>```

I use img.sum() instead of img.mean() to keep all calculations in integer type for precision. For big images there is little difference between this code and your code because it operates on float type.

Body:
I've been trying to create noisy image using the following piece of code:

def toImpulse(img, coef=1.2): # img - > 3dim numpy array, returns impulse image
    bmp = np.zeros(shape=(img.shape[0], img.shape[1], 1), dtype=np.float)
    for i in range(img.shape[0]):
        for j in range(img.shape[1]):
            bmp[i][j] = sum(img[i][j]) / 3

Here I'm summing over the RGB channel and making the image single-channeled? I guess it can be written as numpy.sum()

    sum_ = 0
    max_ = np.amax(bmp)
    final = np.zeros(img.shape)
    for i in range(img.shape[0]):
        for j in range(img.shape[1]):
            sum_ += bmp[i][j]
            if sum_ >= coef * max_:
                final[i][j] = 1
                sum_ -= coef * max_
            else:
                final[i][j] = 0

    return final

But it occurred to be very slow in real-time video processing. How can I make it more efficient using NumPy? Here is the example of the function's work
The original image
Noisy image

Body:
You second for loop can be replaced by np.searchsorted in cumsum of your array.

def toImpulse2(img, coef=1.2): # img - > 3dim numpy array, returns impulse image
    bmp = img.sum(axis=2)
    max_ = bmp.max() * coef
    cumsum = np.cumsum(bmp)
    repeats = int(cumsum[-1] / max_)
    final = np.zeros((img.shape[0] * img.shape[1], img.shape[2]))
    final[np.searchsorted(cumsum, np.arange(1, repeats + 1) * max_)] = 1
    return final.reshape(img.shape)

I use img.sum() instead of img.mean() to keep all calculations in integer type for precision. For big images there is little difference between this code and your code because it operates on float type.


问:
MongoDB How to check if all ObjectIds inside the array exist in the database?
I have an array containing multiple mongodb ObjectId, how can I know if all that Ids exist in the specific collection,

Although I can achieve my expected result by for loop like this, it will take too long to load because it will make multiple query for how many index my array has.

const queryData = async objectIds => {
    // objectIds contains an array of ObjectId

    for (let i = 0; i < objectIds.length; i++) {
        const objId = objectIds[i]
        const data = await DataCollection.findOne({ _id: objId })

        if (!data) throw new Error('Data not found!')
    }
}
```

how can I achieve the same result as my sample function without querying multiple times that makes the process so much slower?

答:
You can use .count() and $in for filtering condition:

```<code>let count = await DataCollection.count( { _id: { $in: objectIds } } );
let all = count === objectIds.length;
</code>```

Body:
I have an array containing multiple mongodb ObjectId, how can I know if all that Ids exist in the specific collection,

Although I can achieve my expected result by for loop like this, it will take too long to load because it will make multiple query for how many index my array has.

const queryData = async objectIds => {
    // objectIds contains an array of ObjectId

    for (let i = 0; i < objectIds.length; i++) {
        const objId = objectIds[i]
        const data = await DataCollection.findOne({ _id: objId })

        if (!data) throw new Error('Data not found!')
    }
}

how can I achieve the same result as my sample function without querying multiple times that makes the process so much slower?

Body:
You can use .count() and $in for filtering condition:

let count = await DataCollection.count( { _id: { $in: objectIds } } );
let all = count === objectIds.length;


问:
Scroll text horizontally in a NSTextView inside NSScrollview
I'm using a NSScrollView with a nested NSTextView to let the user insert and edit a large chunk of text in my OSX application.

If i insert some text, the text didn't expand itself into its current line, instead a new line is created and the horizontal scrollbar doesn't expand neither.
How can I resolve this situation?
Thanks.

答:
Here is an extension on NSTextView</p>

```<code>extension NSTextView {
    func wrapText(_ isWrapped: Bool) {
        guard let scrollView = enclosingScrollView else { return }
        scrollView.hasHorizontalScroller = !isWrapped
        isHorizontallyResizable = !isWrapped
        let width = isWrapped ? Int(scrollView.contentSize.width) : Int.max
        maxSize = NSSize(width: width, height: Int.max)
        textContainer?.size = NSSize(width: width, height: Int.max)
        textContainer?.widthTracksTextView = isWrapped
    }
}
</code>```

Body:
Here is an extension on NSTextView

extension NSTextView {
    func wrapText(_ isWrapped: Bool) {
        guard let scrollView = enclosingScrollView else { return }
        scrollView.hasHorizontalScroller = !isWrapped
        isHorizontallyResizable = !isWrapped
        let width = isWrapped ? Int(scrollView.contentSize.width) : Int.max
        maxSize = NSSize(width: width, height: Int.max)
        textContainer?.size = NSSize(width: width, height: Int.max)
        textContainer?.widthTracksTextView = isWrapped
    }
}