问:
Pygame will only play some .wav files
I am currently trying to programme some Christmas lights using the GPIO pins on the raspberry pi 3. and the lights work, so i decided to put some Christmas music to it as well. when using Pygame, Python won't let me use .mp3 files, and it only lets me play certain .wav files, so when i downloaded some christmas songs from http://www.thewavsite.com/christmas.htm, none of them worked. i will attach the code that i used. I hope you can be of help.</p>

```<code>import pygame

pygame.mixer.init()
s = pygame.mixer.Sound("all_i_want_for_christmas.wav")
s.play()
print "Happy christmas!!"
</code>```

答:
I checked one of christmas songs using on Linux commands file and ffprobe and both show that this .wav uses mp3 compression :)

You will have to convert to real wav before you use it with PyGame.

file shows

```<code>A Christmas To Remember - Kenny Rogers & Dolly Parton.wav:
 RIFF (little-endian) data, WAVE audio, MPEG Layer 3, mono 22050 Hz
</code>```

ffprobe shows

```<code>Input #0, wav, from 'A Christmas To Remember - Kenny Rogers & Dolly Parton.wav':
  Duration: 00:03:39.75, bitrate: 32 kb/s
    Stream #0:0: Audio: mp3 (U[0][0][0] / 0x0055), 22050 Hz, mono, s16p, 32 kb/s
</code>```

Title: Pygame will only play some .wav files

Body:
I am currently trying to programme some Christmas lights using the GPIO pins on the raspberry pi 3. and the lights work, so i decided to put some Christmas music to it as well. when using Pygame, Python won't let me use .mp3 files, and it only lets me play certain .wav files, so when i downloaded some christmas songs from http://www.thewavsite.com/christmas.htm, none of them worked. i will attach the code that i used. I hope you can be of help.

import pygame

pygame.mixer.init()
s = pygame.mixer.Sound("all_i_want_for_christmas.wav")
s.play()
print "Happy christmas!!"

Body:
I checked one of christmas songs using on Linux commands file and ffprobe and both show that this .wav uses mp3 compression :)

You will have to convert to real wav before you use it with PyGame.

file shows

A Christmas To Remember - Kenny Rogers & Dolly Parton.wav:
 RIFF (little-endian) data, WAVE audio, MPEG Layer 3, mono 22050 Hz

ffprobe shows

Input #0, wav, from 'A Christmas To Remember - Kenny Rogers & Dolly Parton.wav':
  Duration: 00:03:39.75, bitrate: 32 kb/s
    Stream #0:0: Audio: mp3 (U[0][0][0] / 0x0055), 22050 Hz, mono, s16p, 32 kb/s


来源: stackexchange

问:
Is try-catching actually this clunky in Java or am I just not aware of the elegant approach?
```<code>public void lock() {
    if (this.isLocked()) return;

    try {
        this.dataOut.flush();
        this.dataOut.close();
    } catch (IOException e) {
        throw new RuntimeException(e);
    }

    DataInputStream inputStream =
        new DataInputStream(
        new BufferedInputStream(
        new ByteArrayInputStream(
            this.byteOut.toByteArray())));

    IntStream.Builder intStreamBuilder = IntStream.builder();
    try {
        try {
            while (true) {
                intStreamBuilder.accept(inputStream.readInt());
            }
        } catch (EOFException e) {
            // logic to be executed after stream has been fully read
            int[] pool = intStreamBuilder.build().toArray();
            super.lock(pool);
        } finally {
            inputStream.close();
        }
    } catch (IOException e) {
        throw new RuntimeException(e);
    }
}
</code>```

What I do here is take an DataOutputStream containing Integers, flush its remaining contents into a ByteArrayOutputStream named this.byteOut and then build an IntStream from it.

I'm from the C# domain and in the process of learning Java, so the code here does not have any actual purpose.

Is there any way to do what I do here more elegantly in Java?

My two main concerns are:

The way I determine that the DataInputStream has been fully read is by catching an EOFException and putting the logic to be executed after reading inside a catch block. I don't like that, since I suppose throwing and catching exceptions is somewhat expensive? Is there a better way to determine that the stream doesn't contain any more Integers?

The fact that I have to wrap a try-catch block around a try-catch block just to be able to call inputStream.close() in the inner finally block. Is there a solution that is not so clunky?

答:
If I were you I will change my code to be as the following

```<code> //try-with-resources-close
 try (DataInputStream inputStream =
                 new DataInputStream(
                         new BufferedInputStream(
                                 new ByteArrayInputStream(
                                         this.byteOut.toByteArray())))) {

        IntStream.Builder intStreamBuilder = IntStream.builder();
        byte[] byts = new byte[4];
        while (inputStream.read(byts) > -1) {// read  4 bytes and convert them to int
            int result = ByteBuffer.wrap(byts).getInt();
            intStreamBuilder.accept(result);
        }
        // logic to be executed after stream has been fully read
        int[] pool = intStreamBuilder.build().toArray();
        super.lock(pool);
    } catch (IOException e) {
        throw new RuntimeException(e);
    }
</code>```

try-with-resources this new way added in Java 7 to close any object that implements AutoCloseable automatically when try block is executed

readInt method it works as the following read 4 bteys then convert them to int

```<code>    int ch1 = in.read();
    int ch2 = in.read();
    int ch3 = in.read();
    int ch4 = in.read();
    if ((ch1 | ch2 | ch3 | ch4) < 0)
        throw new EOFException();
    return ((ch1 << 24) + (ch2 << 16) + (ch3 << 8) + (ch4 << 0));
</code>```

//So you also can do the same thing, but instead of throws EOFException
you can add return false or break the loop

for read bytes method if no bytes existed then it reture -1

ContentLicense: CC BY-SA 3.0

Body:
public void lock() {
    if (this.isLocked()) return;

    try {
        this.dataOut.flush();
        this.dataOut.close();
    } catch (IOException e) {
        throw new RuntimeException(e);
    }

    DataInputStream inputStream =
        new DataInputStream(
        new BufferedInputStream(
        new ByteArrayInputStream(
            this.byteOut.toByteArray())));

    IntStream.Builder intStreamBuilder = IntStream.builder();
    try {
        try {
            while (true) {
                intStreamBuilder.accept(inputStream.readInt());
            }
        } catch (EOFException e) {
            // logic to be executed after stream has been fully read
            int[] pool = intStreamBuilder.build().toArray();
            super.lock(pool);
        } finally {
            inputStream.close();
        }
    } catch (IOException e) {
        throw new RuntimeException(e);
    }
}

What I do here is take an DataOutputStream containing Integers, flush its remaining contents into a ByteArrayOutputStream named this.byteOut and then build an IntStream from it.

I'm from the C# domain and in the process of learning Java, so the code here does not have any actual purpose.

Is there any way to do what I do here more elegantly in Java?

My two main concerns are:

The way I determine that the DataInputStream has been fully read is by catching an EOFException and putting the logic to be executed after reading inside a catch block. I don't like that, since I suppose throwing and catching exceptions is somewhat expensive? Is there a better way to determine that the stream doesn't contain any more Integers?

The fact that I have to wrap a try-catch block around a try-catch block just to be able to call inputStream.close() in the inner finally block. Is there a solution that is not so clunky?

Body:
If I were you I will change my code to be as the following

 //try-with-resources-close
 try (DataInputStream inputStream =
                 new DataInputStream(
                         new BufferedInputStream(
                                 new ByteArrayInputStream(
                                         this.byteOut.toByteArray())))) {

        IntStream.Builder intStreamBuilder = IntStream.builder();
        byte[] byts = new byte[4];
        while (inputStream.read(byts) > -1) {// read  4 bytes and convert them to int
            int result = ByteBuffer.wrap(byts).getInt();
            intStreamBuilder.accept(result);
        }
        // logic to be executed after stream has been fully read
        int[] pool = intStreamBuilder.build().toArray();
        super.lock(pool);
    } catch (IOException e) {
        throw new RuntimeException(e);
    }

try-with-resources this new way added in Java 7 to close any object that implements AutoCloseable automatically when try block is executed

readInt method it works as the following read 4 bteys then convert them to int

    int ch1 = in.read();
    int ch2 = in.read();
    int ch3 = in.read();
    int ch4 = in.read();
    if ((ch1 | ch2 | ch3 | ch4) < 0)
        throw new EOFException();
    return ((ch1 << 24) + (ch2 << 16) + (ch3 << 8) + (ch4 << 0));

//So you also can do the same thing, but instead of throws EOFException
you can add return false or break the loop

for read bytes method if no bytes existed then it reture -1


问:
What is the prototype of the String constructor function (String.__proto__)?
```<code>var mString = new String('A');

console.log(typeof mString); // object

console.log(mString instanceof String); // true

console.log(mString instanceof Object); // true

console.log(mString.__proto__ === String.prototype); // true

console.log(mString.__proto__.__proto__ === Object.prototype); // true
</code>```

Now, why

```<code>console.log(String.__proto__.__proto__ === Object.prototype); // true
</code>```

and not

```<code>console.log(String.__proto__ === Object.prototype); // false
</code>```

when walking up the prototype chain?

What is between String and Object prototypes?

答:
When you use instanceof Object it means that the variable is either an Object or "extends" an Object.

As mentioned here "The instanceof operator tests whether the prototype property of a constructor appears anywhere in the prototype chain of an object."

So mString.__proto__ is not equal to Object.prototype, but it Object.prototype is indeed higher on the prototype chain, that's why they are not equal. The String prototype inherits the Object prototype.

Body:
var mString = new String('A');

console.log(typeof mString); // object

console.log(mString instanceof String); // true

console.log(mString instanceof Object); // true

console.log(mString.__proto__ === String.prototype); // true

console.log(mString.__proto__.__proto__ === Object.prototype); // true

Now, why

console.log(String.__proto__.__proto__ === Object.prototype); // true

and not

console.log(String.__proto__ === Object.prototype); // false

when walking up the prototype chain?

What is between String and Object prototypes?


问:
Error of value of polynomial with covariance matrix for coefficents
From polyfit() function I have coefficents and their covariance matrix. Now for some argument I want a value of that polynomial with error. Apparently polyval() function doesn't accept covariance matrix (in matlab it does). From definition I can write my own function:

```<code>def sigma(x,pcov):
    s=0
    for i in range(len(pcov)):
        for j in range(len(pcov[0])):
            s+=x**(len(pcov)-i-1)*pcov[i][j]*x**(len(pcov[0])-j-1)
    return sqrt(s)
</code>```

Where I have to do some funny stuff with indices because of reverse order of coefficents in polynomial. It doesn't seem pythonic though. Am I missing something? Or maybe support for this kind of operation is in some bigger library like SciPy?

答:
Without loops, this is computed by multiplying the covariance matrix C on both sides by the vector x_powers made of the relevant powers of x: for example, [x**5, x**4, ..., x**0]. The setup of polyfit is included for completeness.

```<code>xdata = np.arange(10)
ydata = np.abs(xdata-2)    # some data to fit
degree = 5
p, C = np.polyfit(xdata, ydata, deg=degree, cov=True)
x = 5.4                    # some value of x to estimate error at
x_powers = x**np.arange(degree, -1, -1)
x_error = np.sqrt(x_powers.dot(C).dot(x_powers))
</code>```

Here x_error is the same as what your function returns.

Body:
From polyfit() function I have coefficents and their covariance matrix. Now for some argument I want a value of that polynomial with error. Apparently polyval() function doesn't accept covariance matrix (in matlab it does). From definition I can write my own function:

def sigma(x,pcov):
    s=0
    for i in range(len(pcov)):
        for j in range(len(pcov[0])):
            s+=x**(len(pcov)-i-1)*pcov[i][j]*x**(len(pcov[0])-j-1)
    return sqrt(s)

Where I have to do some funny stuff with indices because of reverse order of coefficents in polynomial. It doesn't seem pythonic though. Am I missing something? Or maybe support for this kind of operation is in some bigger library like SciPy?

Body:
Without loops, this is computed by multiplying the covariance matrix C on both sides by the vector x_powers made of the relevant powers of x: for example, [x**5, x**4, ..., x**0]. The setup of polyfit is included for completeness.

xdata = np.arange(10)
ydata = np.abs(xdata-2)    # some data to fit
degree = 5
p, C = np.polyfit(xdata, ydata, deg=degree, cov=True)
x = 5.4                    # some value of x to estimate error at
x_powers = x**np.arange(degree, -1, -1)
x_error = np.sqrt(x_powers.dot(C).dot(x_powers))

Here x_error is the same as what your function returns.


问:
how to bulk insert in sqlite in android
I am using SQLiteOpenHelper  for data insertion. I need to insert 2500  id and 2500  names, So it takes too much time. Please any one help me how to reduce the insertion time.  can we insert multiple records at a time ? any one help me. thank you in advance.
code:

```<code>public class DatabaseHandler extends SQLiteOpenHelper {
    SQLiteDatabase db;
    private static final int DATABASE_VERSION = 8;
    private static final String TABLE_CITY = "CITYDETAILS";
    public DatabaseHandler(Context context) {
        super(context, DATABASE_NAME, null, DATABASE_VERSION);
        this.mContext = context;
}

public void onCreate(SQLiteDatabase db) {
    db.execSQL("DROP TABLE IF EXISTS " + TABLE_CITY );
    String CREATE_CITY_TABLE = "CREATE TABLE " + TABLE_CITY + "("
                + CityId + " INTEGER," + CityName + " TEXT " + ")";
    db.execSQL(CREATE_CITY_TABLE);
    db.execSQL(CREATE_RechargeTypes_TABLE);
    this.db=db;
 }

 @Override
 public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
     // Drop older table if existed
     // Create tables again
     onCreate(db);
 }

 public  void add_city(String cityid,String cityname){
     SQLiteDatabase db = this.getWritableDatabase();
     db.beginTransaction();
     ContentValues values = new ContentValues();
     values.put(CityId, cityid);
     values.put(CityName, cityname);
     db.insert(TABLE_CITY, null, values);
     db.insertWithOnConflict(TABLE_CITY, null, values, SQLiteDatabase.CONFLICT_IGNORE);
     db.setTransactionSuccessful();
     db.endTransaction();
     }
}
</code>```

activity calss:

```<code>try{
            String Status = result.get("Status").getAsString();
            if (TextUtils.equals(Status, "true")) {
                Gson gson = new Gson();

                JsonArray array = result.get("data")
                        .getAsJsonArray();
                Type type = new TypeToken<ArrayList<Cities>>() {
                }.getType();
                setmCities((ArrayList<Cities>) gson.fromJson(array, type));
                for(int i=0;i< array.size();i++) {

                   db.add_city(mCities.get(i).getCityid(),mCities.get(i).getCityname());

                }
            }
</code>```

答:
try SQLiteStatement

in 2-3 times faster than just beginTransaction()

```<code>public void add_cities(ArrayList<Cities> list) {
    SQLiteDatabase database = this.getWritableDatabase();
    String sql = "INSERT INTO " + TABLE_NAME + " VALUES(?, ?)";
    SQLiteStatement statement = database.compileStatement(sql);
    database.beginTransaction();
    try {
        for (Cities c : list) {
            statement.clearBindings();
            statement.bindLong(1, c.getCityId());
            statement.bindLong(2, c.getCityName());
            statement.execute();
        }
        database.setTransactionSuccessful();
    } finally {
        database.endTransaction();
    }
}
</code>```

Title: how to bulk insert in sqlite in android

Body:
I am using SQLiteOpenHelper  for data insertion. I need to insert 2500  id and 2500  names, So it takes too much time. Please any one help me how to reduce the insertion time.  can we insert multiple records at a time ? any one help me. thank you in advance.
code:

public class DatabaseHandler extends SQLiteOpenHelper {
    SQLiteDatabase db;
    private static final int DATABASE_VERSION = 8;
    private static final String TABLE_CITY = "CITYDETAILS";
    public DatabaseHandler(Context context) {
        super(context, DATABASE_NAME, null, DATABASE_VERSION);
        this.mContext = context;
}

public void onCreate(SQLiteDatabase db) {
    db.execSQL("DROP TABLE IF EXISTS " + TABLE_CITY );
    String CREATE_CITY_TABLE = "CREATE TABLE " + TABLE_CITY + "("
                + CityId + " INTEGER," + CityName + " TEXT " + ")";
    db.execSQL(CREATE_CITY_TABLE);
    db.execSQL(CREATE_RechargeTypes_TABLE);
    this.db=db;
 }

 @Override
 public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
     // Drop older table if existed
     // Create tables again
     onCreate(db);
 }

 public  void add_city(String cityid,String cityname){
     SQLiteDatabase db = this.getWritableDatabase();
     db.beginTransaction();
     ContentValues values = new ContentValues();
     values.put(CityId, cityid);
     values.put(CityName, cityname);
     db.insert(TABLE_CITY, null, values);
     db.insertWithOnConflict(TABLE_CITY, null, values, SQLiteDatabase.CONFLICT_IGNORE);
     db.setTransactionSuccessful();
     db.endTransaction();
     }
}

activity calss:

try{
            String Status = result.get("Status").getAsString();
            if (TextUtils.equals(Status, "true")) {
                Gson gson = new Gson();

                JsonArray array = result.get("data")
                        .getAsJsonArray();
                Type type = new TypeToken<ArrayList<Cities>>() {
                }.getType();
                setmCities((ArrayList<Cities>) gson.fromJson(array, type));
                for(int i=0;i< array.size();i++) {

                   db.add_city(mCities.get(i).getCityid(),mCities.get(i).getCityname());

                }
            }

Body:
try SQLiteStatement

in 2-3 times faster than just beginTransaction()

public void add_cities(ArrayList<Cities> list) {
    SQLiteDatabase database = this.getWritableDatabase();
    String sql = "INSERT INTO " + TABLE_NAME + " VALUES(?, ?)";
    SQLiteStatement statement = database.compileStatement(sql);
    database.beginTransaction();
    try {
        for (Cities c : list) {
            statement.clearBindings();
            statement.bindLong(1, c.getCityId());
            statement.bindLong(2, c.getCityName());
            statement.execute();
        }
        database.setTransactionSuccessful();
    } finally {
        database.endTransaction();
    }
}


问:
Can I do type introspection with trait objects and then downcast it?
I have a collection of Trait, a function that iterates over it and does something, and then I would like to check the implementor type and if it is of type Foo then downcast it and call some Foo method.

Basically, something similar to Go's type-switch and interface conversion.

Searching around I found about the Any trait but it can only be implemented on 'static types.

To help demonstrate what I want:

```<code>let vec: Vec<Box<Trait>> = //

for e in vec.iter() {
    e.trait_method();

    // if typeof e == Foo {
    // let f = e as Foo;
    // f.foo_method();
    //}
}
</code>```

答:
This isn't a Rust-specific problem, although the vocabulary may be a little different. The ideal way to solve a problem like this, not just with traits in Rust but in any language, is to add the desired behavior (foo_method in your example) to the abstract interface (Trait):

trait Trait {
    fn trait_method(&self);
    fn foo_method(&self) {} // does nothing by default
}

struct Foo;

impl Trait for Foo {
    fn trait_method(&self) {
        println!("In trait_method of Foo");
    }

    fn foo_method(&self) {
        // override default behavior
        println!("In foo_method");
    }
}

struct Bar;

impl Trait for Bar {
    fn trait_method(&self) {
        println!("In trait_method of Bar");
    }
}

fn main() {
    let vec: Vec<Box<dyn Trait>> = vec![Box::new(Foo), Box::new(Bar)];

    for e in &vec {
        e.trait_method();
        e.foo_method();
    }
}
```
<p>In this example, I have put a default implementation of <code>foo_method</code> in <code>Trait</code> which does nothing, so that you don't have to define it in every <code>impl</code> but only the one(s) where it applies. You should <em>really</em> attempt to make the above work before you resort to downcasting to a concrete type, which has serious drawbacks that all but erase the advantages of having trait objects in the first place.</p>
<p>That said, there are cases where downcasting may be necessary, and Rust does support it -- although the interface is a little clunky. You can downcast <code>&Trait</code> to <code>&Foo</code> by adding an intermediate <em>up</em>cast to <code>&Any</code>:</p>
<pre class="lang-rust prettyprint-override"><code>use std::any::Any;

trait Trait {
    fn as_any(&self) -> &dyn Any;
}

struct Foo;

impl Trait for Foo {
    fn as_any(&self) -> &dyn Any {
        self
    }
}

fn downcast<T: Trait + 'static>(this: &dyn Trait) -> Option<&T> {
    this.as_any().downcast_ref()
}
</code>```

as_any has to be a method in Trait because it needs access to the concrete type. Now you can attempt to call Foo methods on a Trait trait object like this (complete playground example):

if let Some(r) = downcast::<Foo>(&**e) {
    r.foo_method();
}
```

To make this work, you have to specify what type you expect (::<Foo>) and use if let to handle what happens when the referenced object is not an instance of Foo. You can't downcast a trait object to a concrete type unless you know exactly what concrete type it is.

If you ever need to know the concrete type, trait objects are almost useless anyway! You probably should use an enum instead, so that you will get compile-time errors if you omit to handle a variant somewhere. Furthermore, you can't use Any with non-'static structs, so if any Foo might need to contain a reference, this design is a dead end. The best solution, if you can do it, is to add foo_method to the trait itself.

Body:
I have a collection of Trait, a function that iterates over it and does something, and then I would like to check the implementor type and if it is of type Foo then downcast it and call some Foo method.

Basically, something similar to Go's type-switch and interface conversion.

Searching around I found about the Any trait but it can only be implemented on 'static types.

To help demonstrate what I want:

let vec: Vec<Box<Trait>> = //

for e in vec.iter() {
    e.trait_method();

    // if typeof e == Foo {
    // let f = e as Foo;
    // f.foo_method();
    //}
}

Body:
This isn't a Rust-specific problem, although the vocabulary may be a little different. The ideal way to solve a problem like this, not just with traits in Rust but in any language, is to add the desired behavior (foo_method in your example) to the abstract interface (Trait):

trait Trait {
    fn trait_method(&self);
    fn foo_method(&self) {} // does nothing by default
}

struct Foo;

impl Trait for Foo {
    fn trait_method(&self) {
        println!("In trait_method of Foo");
    }

    fn foo_method(&self) {
        // override default behavior
        println!("In foo_method");
    }
}

struct Bar;

impl Trait for Bar {
    fn trait_method(&self) {
        println!("In trait_method of Bar");
    }
}

fn main() {
    let vec: Vec<Box<dyn Trait>> = vec![Box::new(Foo), Box::new(Bar)];

    for e in &vec {
        e.trait_method();
        e.foo_method();
    }
}

In this example, I have put a default implementation of foo_method in Trait which does nothing, so that you don't have to define it in every impl but only the one(s) where it applies. You should really attempt to make the above work before you resort to downcasting to a concrete type, which has serious drawbacks that all but erase the advantages of having trait objects in the first place.

That said, there are cases where downcasting may be necessary, and Rust does support it -- although the interface is a little clunky. You can downcast &Trait to &Foo by adding an intermediate upcast to &Any:

use std::any::Any;

trait Trait {
    fn as_any(&self) -> &dyn Any;
}

struct Foo;

impl Trait for Foo {
    fn as_any(&self) -> &dyn Any {
        self
    }
}

fn downcast<T: Trait + 'static>(this: &dyn Trait) -> Option<&T> {
    this.as_any().downcast_ref()
}

as_any has to be a method in Trait because it needs access to the concrete type. Now you can attempt to call Foo methods on a Trait trait object like this (complete playground example):

if let Some(r) = downcast::<Foo>(&**e) {
    r.foo_method();
}

To make this work, you have to specify what type you expect (::<Foo>) and use if let to handle what happens when the referenced object is not an instance of Foo. You can't downcast a trait object to a concrete type unless you know exactly what concrete type it is.

If you ever need to know the concrete type, trait objects are almost useless anyway! You probably should use an enum instead, so that you will get compile-time errors if you omit to handle a variant somewhere. Furthermore, you can't use Any with non-'static structs, so if any Foo might need to contain a reference, this design is a dead end. The best solution, if you can do it, is to add foo_method to the trait itself.

ContentLicense: CC BY-SA 4.0


答:
```<code>import re
from machine import UART
uart = UART(2, 9600)

pattern=r"b'[0-9]+"

while True:
  if uart.any():
    data=uart.read()
    all_items=[]
    track=[]
    for i in data.split(','):
        pr=re.search(pattern,i)
        if len(track)==5:
            break
        else:
            if pr!=None:
                track.append(pr.group())
                all_items.append(pr.group())
            else:
                all_items.append(i)

    print(all_items)

this would be the code apllied to my project, but I get this error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 12, in <module>`enter code here`
TypeError: can't convert 'str' object to bytes implicitly
</code>```

Title: Counting a character a number of times

Body:
What I want to do is a break on the program once a character has appeared a number of times in a string in Python.

This is the output I get:

b'30,22,37,293,29,26,55,30D\r\n$GPGSV,3,1,11,01,04,250,,03,21,309,24,08,15,200,,10,20,157,*7B\r\n$GPGSV,3,2,11,14,49,062,37,16'
      b'8,W,001328.00,A,A*70\r\n9,26,55,359,30*75\r\n$GPGSV,3,3,11,27,40,170,38,31,20,022,14,32,42,094,35*4C\r\n$GPGLL,0226.72251,N,0'
      b'A*2E\r\n$GPGGA,001329.00,A,A*70\r\n$GPRMC,001329.00,A,0226.72248,N,07635.92031,W,0.008,,011217,,,A*64\r\n$GPVTG,,T,,M,0.008,N,'
      b'2.25,1.08,1.98*0D\r\n$GPG1329.00,0226.72248,N,07635.92031,W,1,08,1.08,1769.1,M,9.8,M,,*4C\r\n$GPGSA,A,3,16,32,26,31,14,22,27'

So I want to stop once the fifth "b'" shows up.


问:
Having trouble creating an object within a switch statement, "Object cannot be resolved" in later code (java)
I am writing a method that takes in a string variable type that is used to determine which list class to implement. The method uses a switch statement to create the appropriate type of list using the variable name newList.

The switch statement looks like this:

```<code>switch (type) {
    case "unsorted": {
        UnsortedList newList = new UnsortedList();
        break;
    }
    case "sorted": {
        SortedList newList = new SortedList();
        break;
    }
    case "new sorted": {
        NewSortedList newList = new NewSortedList();
        break;
    }
    default: {
        System.out.print("Something went wrong! List type unrecognized. I have initialized newList to be an unsorted list.");
        UnsortedList newList = new UnsortedList();
    }
}
</code>```

According to this, I believe that newList will always be instantiated due to the default case of the switch statement. However I receive the compile error "newList cannot be resolved".

Is there anything I can do to prevent this?

答:
Your newList is being created into too small of a scope. Scopes in Java are denoted by {}. A variable only exists in scopes of the same level or higher of the scope of its creation.

For example:

```<code>public class ScopeClass(){ //this is the start of the class level scope
    private String classString = "this can been seen anywhere in the class";
    public void scopeMethod(){ //this starts the method level scope
        private String methodString = "this is only visible in the method";
        { //start of internal scope block
            private String internalString = "only visible in this scope block";
        }
    }
}
</code>```

To make your newList visible outside of the switch scope, try this:

```<code>List newlist; // make sure all of your various List types are compatible.
switch (type){
    case "unsorted":
    {
        newList = new UnsortedList();
        break;
    }
    case "sorted":
    {
        newList = new SortedList();
        break;
    }
    case "new sorted":
    {
        newList = new NewSortedList();
        break;
    }

    default:
    {
        System.out.print("Something went wrong! List type unrecognized. I have initialized newList to be an unsorted list.");
        newList = new UnsortedList();
    }
}
</code>```

Body:
I am writing a method that takes in a string variable type that is used to determine which list class to implement. The method uses a switch statement to create the appropriate type of list using the variable name newList.

The switch statement looks like this:

switch (type) {
    case "unsorted": {
        UnsortedList newList = new UnsortedList();
        break;
    }
    case "sorted": {
        SortedList newList = new SortedList();
        break;
    }
    case "new sorted": {
        NewSortedList newList = new NewSortedList();
        break;
    }
    default: {
        System.out.print("Something went wrong! List type unrecognized. I have initialized newList to be an unsorted list.");
        UnsortedList newList = new UnsortedList();
    }
}

According to this, I believe that newList will always be instantiated due to the default case of the switch statement. However I receive the compile error "newList cannot be resolved".

Is there anything I can do to prevent this?

Body:
Your newList is being created into too small of a scope. Scopes in Java are denoted by {}. A variable only exists in scopes of the same level or higher of the scope of its creation.

For example:

public class ScopeClass(){ //this is the start of the class level scope
    private String classString = "this can been seen anywhere in the class";
    public void scopeMethod(){ //this starts the method level scope
        private String methodString = "this is only visible in the method";
        { //start of internal scope block
            private String internalString = "only visible in this scope block";
        }
    }
}

To make your newList visible outside of the switch scope, try this:

List newlist; // make sure all of your various List types are compatible.
switch (type){
    case "unsorted":
    {
        newList = new UnsortedList();
        break;
    }
    case "sorted":
    {
        newList = new SortedList();
        break;
    }
    case "new sorted":
    {
        newList = new NewSortedList();
        break;
    }

    default:
    {
        System.out.print("Something went wrong! List type unrecognized. I have initialized newList to be an unsorted list.");
        newList = new UnsortedList();
    }
}


问:
php array. cant get the output i desire. help me
i have this code

```<code>$dir = "img/ori/";
$dir_thumbs = "img/thumbs/";

$images = array();
$d = dir($dir);
while($name = $d->read()){
    $thumb = "thumb_".$name;
    $images[] = array('name' => $name,'thumb_url' => $dir_thumbs.$thumb);
}
$d->close();
$o = array('images'=>$images);
</code>```

i want to output $name and $thumb_url of all the images. how to do it.? help me

答:
Use print_r or var_dump for this:

```<code>print_r($o);
//or
var_dump($o);
</code>```

ContentLicense: CC BY-SA 2.5

Body:
i have this code

$dir = "img/ori/";
$dir_thumbs = "img/thumbs/";

$images = array();
$d = dir($dir);
while($name = $d->read()){
    $thumb = "thumb_".$name;
    $images[] = array('name' => $name,'thumb_url' => $dir_thumbs.$thumb);
}
$d->close();
$o = array('images'=>$images);

i want to output $name and $thumb_url of all the images. how to do it.? help me

Body:
Use print_r or var_dump for this:

print_r($o);
//or
var_dump($o);


问:
unable to understand docker cp command

I am trying to copy file from docker container to host. I have attached the snapshot in which I listed all the containers as well as container file structure. But when I execute docker cp, it says it could not find the file. Can anyone tell me what wrong am I doing or if I am missing something. I am very new to docker so please guide me through.

答:
As documented for the docker cp command:

The docker cp command assumes container paths are relative to the
  container’s / (root) directory

Thus CONTAINER:file.txt is equivalent to CONTAINER:/file.txt as paths are interpreted relative to the root /

Title: unable to understand docker cp command


问:
ctrl+shift+space Not Working in Visual Studio
Starting about a week ago (11/30/2017) I noticed that ctrl+shift+space was no longer working in visual studio (any version) to show the parameter info of a method invocation. I did quite a bit of searching to figure out the problem, especially here on SO, and I couldn't find a solution. I also double-checked my VS settings and even reverted back to an old settings file.</p>

答:
I methodically started closing applications to see if another application was stealing the input. My hunch was right and it turned out to be the Microsoft Teams desktop application. It turns out that Teams already has a similar bug open about shortcut keys related to the Polish language not working correctly when Teams is running. Those details are here: https://microsoftteams.uservoice.com/forums/555103-public/suggestions/32265802-alt-a-and-alt-s-affecting-polish-character-for-%C4%85

Within the above link there is a link to a beta build that fixes the problem for Polish, and it turns out that it fixes the problem for the ctrl+shift+space shortcut too.


问:
How to deduce contiguous memory from iterator
Somehow, the native stl::copy() algorithm on VC++ (Dinkumware) figures out that it can use memcpy() on data that is trivially copy-able. Is it possible for a mere mortal to do that? - assuming each element is_trivially_copyable.

Does random_access_iterator imply contiguous memory? The standard is not clear to me.

So, if all you have in a template is an iterator or two, is it possible to deduce at compile-time that the underlying array can be copied with memcpy(), and if so how?

EDIT - Here's my motivation. I have a routine that moves a block of data rather than copying it. To get speed when the data are memmove-able, I've made it call stl::copy sometimes. I was wondering if that is the only way. (As a kid, I would always try this at home.)

```<code>// Move a range of values
template<class Ptr, class Ptr2>
inline Ptr2 move(Ptr src, Ptr end, Ptr2 dest) {
    using value_type = std::iterator_traits<Ptr>::value_type;
    if constexpr (std::is_trivially_copyable_v<value_type>) {
        return std::copy(src, end, dest);
    } else {
        while (src != end) {
            *dest = std::move(*src);
            ++src; ++dest;
        }
    }
    return dest;
}
</code>```

EDIT: Kudos to zett42 for finding this related question: Contiguous iterator detection I can't explain how I missed it.

EDIT More: After going down many twisty little passages all the same, I discovered that Dinkum uses secret tags for iterators that are in with the in-crowd, e.g. _Really_trivial_ptr_iterator_tag. So prospects look dim.

My $0.02 worth: It was a beginner's mistake to make iterator_category an ad-hoc type rather than busting out the various traits, like "points_into_contiguous_memory," etc... Since random_access_iterator is an ad-hoc type denoted only with a tag, it cannot be subtyped without breaking legacy applications.  So the committee is now kind of stuck. Time to start over, I say.

Oh well.

答:
Does random_access_iterator imply contiguous memory?

Short Answer: No.

Long Answer: depends on the type.

Example: for std::vector (with random access iterator) it's granted that the memory is all values are in a contiguous block of memory that can be accessed from data() method.

For std::deque (also with random access iterator) you know that the memory area is divided in chunks (std::deque is designed do make efficient insert/remove of element in  middle) so it's improbable that the memory is contiguous.

Body:
Somehow, the native stl::copy() algorithm on VC++ (Dinkumware) figures out that it can use memcpy() on data that is trivially copy-able. Is it possible for a mere mortal to do that? - assuming each element is_trivially_copyable.

Does random_access_iterator imply contiguous memory? The standard is not clear to me.

So, if all you have in a template is an iterator or two, is it possible to deduce at compile-time that the underlying array can be copied with memcpy(), and if so how?

EDIT - Here's my motivation. I have a routine that moves a block of data rather than copying it. To get speed when the data are memmove-able, I've made it call stl::copy sometimes. I was wondering if that is the only way. (As a kid, I would always try this at home.)

// Move a range of values
template<class Ptr, class Ptr2>
inline Ptr2 move(Ptr src, Ptr end, Ptr2 dest) {
    using value_type = std::iterator_traits<Ptr>::value_type;
    if constexpr (std::is_trivially_copyable_v<value_type>) {
        return std::copy(src, end, dest);
    } else {
        while (src != end) {
            *dest = std::move(*src);
            ++src; ++dest;
        }
    }
    return dest;
}

EDIT: Kudos to zett42 for finding this related question: Contiguous iterator detection I can't explain how I missed it.

EDIT More: After going down many twisty little passages all the same, I discovered that Dinkum uses secret tags for iterators that are in with the in-crowd, e.g. _Really_trivial_ptr_iterator_tag. So prospects look dim.

My $0.02 worth: It was a beginner's mistake to make iterator_category an ad-hoc type rather than busting out the various traits, like "points_into_contiguous_memory," etc... Since random_access_iterator is an ad-hoc type denoted only with a tag, it cannot be subtyped without breaking legacy applications.  So the committee is now kind of stuck. Time to start over, I say.

Oh well.


问:
Active model Serializers not eager loading relationships
I'm using the Bullet gem to see where there are n+1 queries in my application.  It's telling me to eager load my association taggings when calling my serializer.  My code looks something like this:

```<code>render json: @products, each_serializer: ::V1::ProductSerializer, includes: [:taggings], links: links, status: :ok
</code>```

But after I add that, i'm still getting the same warning from the Bullet gem.  Which looks like this:

```<code>GET /api/v1/product_feed?state=CA&page=1
USE eager loading detected
  Product => [:taggings]
  Add to your finder: :includes => [:taggings]
Call stack
  /home/jay/current_projects/api/app/controllers/api/v1/products_controller.rb:111:in `product_feed'
</code>```

Does anyone have any idea why the taggings table is not being eager loaded.

答:
You need to include taggings in your query first then the serializer will be able to read the loaded records instead of separately asking for the taggings association record by record </p>

```<code>@products = Product.includes(:taggings)
render json: @products, each_serializer: ::V1::ProductSerializer, includes: [:taggings], links: links, status: :ok
</code>```

Body:
I'm using the Bullet gem to see where there are n+1 queries in my application.  It's telling me to eager load my association taggings when calling my serializer.  My code looks something like this:

render json: @products, each_serializer: ::V1::ProductSerializer, includes: [:taggings], links: links, status: :ok

But after I add that, i'm still getting the same warning from the Bullet gem.  Which looks like this:

GET /api/v1/product_feed?state=CA&page=1
USE eager loading detected
  Product => [:taggings]
  Add to your finder: :includes => [:taggings]
Call stack
  /home/jay/current_projects/api/app/controllers/api/v1/products_controller.rb:111:in `product_feed'

Does anyone have any idea why the taggings table is not being eager loaded.

Body:
You need to include taggings in your query first then the serializer will be able to read the loaded records instead of separately asking for the taggings association record by record

@products = Product.includes(:taggings)
render json: @products, each_serializer: ::V1::ProductSerializer, includes: [:taggings], links: links, status: :ok


问:
What's the difference between this and super keywords in java?
Check my coolMethod, babes:

```<code>package zoo;

public class Zoo {

    public String coolMethod() {
        return "Wow baby!";
    }

}
</code>```

My Moo Class, that extends my Zoo class, full of ways to do the same thing, which is calling my coolMethod.

```<code>package zoo;

public class Moo extends Zoo {

    public void useAZoo(){

        Zoo z = new Zoo();

        System.out.println("A Zoo says: "+ z.coolMethod());
    }

    public void useMyCoolMethod(){

        System.out.println("My cool method says: " + super.coolMethod());

    }

     public void useMyCoolMethodAgain(){

        System.out.println("My cool method says: " + this.coolMethod()+ " (again)");

    }

    public void inheritMyMethod(){
        System.out.println("My inhertied method says: " + coolMethod());
    }

}
</code>```

And my main, that calls my Moo class functionalities.

```<code>package javacert1;

import zoo.Moo;

public class Main {

    public static void main(String[] args) {

        Moo moo = new Moo();

        moo.useAZoo();

        moo.useMyCoolMethod();

        moo.useMyCoolMethodAgain();

        moo.inheritMyMethod();
    }

}
</code>```

As you can see, all those four calls get the same results. I'm learning Java and I practiced with an example that used the "this" but I had seen the use of "super" elsewhere so I tested it and the results where the same. Why is this? What's the major difference between a this and super keyword?

答:
In this case, they will do nothing different.

However, if you were to override coolMethod in the Moo class, the call to the super version will call the version in the Zoo class, but all the other versions will call the new overridden one.

Body:
Check my coolMethod, babes:

package zoo;

public class Zoo {

    public String coolMethod() {
        return "Wow baby!";
    }

}

My Moo Class, that extends my Zoo class, full of ways to do the same thing, which is calling my coolMethod.

package zoo;

public class Moo extends Zoo {

    public void useAZoo(){

        Zoo z = new Zoo();

        System.out.println("A Zoo says: "+ z.coolMethod());
    }

    public void useMyCoolMethod(){

        System.out.println("My cool method says: " + super.coolMethod());

    }

     public void useMyCoolMethodAgain(){

        System.out.println("My cool method says: " + this.coolMethod()+ " (again)");

    }

    public void inheritMyMethod(){
        System.out.println("My inhertied method says: " + coolMethod());
    }

}

And my main, that calls my Moo class functionalities.

package javacert1;

import zoo.Moo;

public class Main {

    public static void main(String[] args) {

        Moo moo = new Moo();

        moo.useAZoo();

        moo.useMyCoolMethod();

        moo.useMyCoolMethodAgain();

        moo.inheritMyMethod();
    }

}

As you can see, all those four calls get the same results. I'm learning Java and I practiced with an example that used the "this" but I had seen the use of "super" elsewhere so I tested it and the results where the same. Why is this? What's the major difference between a this and super keyword?


问:
Custom attributes lead to errors when binding Angular templates
We are binding model data in component template markup which works until we attempt to bind data to custom attributes. We see errors such as Can't bind to 'x-version' since it isn't a known property of 'section' and different errors if we apply the bracketed pattern.

Essentially we are placing hidden data in the markup for other scripts to use within their processes. Can someone clarify what is happening and suggest ways to accomplish this?

We have attempted to apply the following two patterns and get errors on each. We are using Angular 2+

```<code><section x-version="{{item?.version}}">
...
</section>

<section [x-version]="item?.version">
...
</section>
</code>```

答:
That error says that x-version should be a component defined @input.
If it is not, you may want to use an attribute handler way like your first line

Body:
We are binding model data in component template markup which works until we attempt to bind data to custom attributes. We see errors such as Can't bind to 'x-version' since it isn't a known property of 'section' and different errors if we apply the bracketed pattern.

Essentially we are placing hidden data in the markup for other scripts to use within their processes. Can someone clarify what is happening and suggest ways to accomplish this?

We have attempted to apply the following two patterns and get errors on each. We are using Angular 2+

...

...


问:
How can I improve the performance of a WordPress WooCommerce website that gets a low speed score from Google PageSpeed Insights?
How can I make my WordPress WooCommerce website performance fast?

My website is http://www.furniturehub.pk/. But it is loading too slow.

I have also checked its performance using Google PageSpeed Insights and it scores 64/100. So how can I make it FASTER?

答:
Your website has images that are 1200x300 when there really only using 300x70 space.
since your website is mainly pictures you should resize all of them to be smaller.</p>


问:
NSNotification NSMetadataQueryDidUpdate getting non-stop Notification messages
I am implementing saving to iCloud Drive in my application. Right now I am testing the app on the simulator. I can save a file on to the iCloud Drive and it appears on the drive as expected. In my code, I have implemented NotificationCenter.default.addObserver. The startQuery() gets called with the application is made active. Here is the code sample:

```<code>@objc func startQuery()
{
    stopQuery()
    query = documentQuery()
    print( "adding notifications" )
    NotificationCenter.default.addObserver(self, selector: #selector(finishGatheringNotification(notifciations:)), name: NSNotification.Name.NSMetadataQueryDidFinishGathering, object:nil )
    NotificationCenter.default.addObserver(self, selector: #selector(didUpdateNotification(notifciations:)), name: NSNotification.Name.NSMetadataQueryDidUpdate, object:nil )
    query?.start()
}

@objc func stopQuery()
{
    guard let query = self.query else{
        return
    }
    NotificationCenter.default.removeObserver(self, name: NSNotification.Name.NSMetadataQueryDidFinishGathering, object:nil )
    NotificationCenter.default.removeObserver(self, name: NSNotification.Name.NSMetadataQueryDidUpdate, object:nil )
    query.stop()
    self.query = nil

}

     @objc func finishGatheringNotification( notifciations:NSNotification ){
        print( "called finished gathering notification" )
        processICloudFiles(notifciations: notifciations )
    }

    @objc func didUpdateNotification( notifciations:NSNotification ){
        // This seems to be getting called non-stop
        print( "called updated notification" )
        processICloudFiles(notifciations: notifciations )
    }

    @objc func processICloudFiles( notifciations:NSNotification )
    {
        guard let query = query else{
            return
        }
        query.disableUpdates()
        var metaDataItems:[NSMetadataItem] = []

        if let queryResults = query.results as? [NSMetadataItem]{
            print( "query results:\(queryResults.count)")
            for result in queryResults{
                if let fileURL = result.value(forAttribute: NSMetadataItemURLKey ) as? NSURL{
                     print( "file found in iCloud:\(String(describing: fileURL.lastPathComponent))")
                    metaDataItems.append( result )
                }
            }
            if metaDataItems.count > 0{
                updateICloudFiles(metaDataItems: metaDataItems)
            }
        }
        query.enableUpdates()
    }
</code>```

However, the selector for NSMetadataQueryDidUpdate is getting called no stop. I thought it was meant to get called only when the content of the iCloud disk changes.

Any suggestions.

Thanks

Reza

答:
Try to deinitialize:</p>

```<code>deinit {
    NotificationCenter.default.removeObserver(self)
}
</code>```

Body:
I am implementing saving to iCloud Drive in my application. Right now I am testing the app on the simulator. I can save a file on to the iCloud Drive and it appears on the drive as expected. In my code, I have implemented NotificationCenter.default.addObserver. The startQuery() gets called with the application is made active. Here is the code sample:

@objc func startQuery()
{
    stopQuery()
    query = documentQuery()
    print( "adding notifications" )
    NotificationCenter.default.addObserver(self, selector: #selector(finishGatheringNotification(notifciations:)), name: NSNotification.Name.NSMetadataQueryDidFinishGathering, object:nil )
    NotificationCenter.default.addObserver(self, selector: #selector(didUpdateNotification(notifciations:)), name: NSNotification.Name.NSMetadataQueryDidUpdate, object:nil )
    query?.start()
}

@objc func stopQuery()
{
    guard let query = self.query else{
        return
    }
    NotificationCenter.default.removeObserver(self, name: NSNotification.Name.NSMetadataQueryDidFinishGathering, object:nil )
    NotificationCenter.default.removeObserver(self, name: NSNotification.Name.NSMetadataQueryDidUpdate, object:nil )
    query.stop()
    self.query = nil

}

     @objc func finishGatheringNotification( notifciations:NSNotification ){
        print( "called finished gathering notification" )
        processICloudFiles(notifciations: notifciations )
    }

    @objc func didUpdateNotification( notifciations:NSNotification ){
        // This seems to be getting called non-stop
        print( "called updated notification" )
        processICloudFiles(notifciations: notifciations )
    }

    @objc func processICloudFiles( notifciations:NSNotification )
    {
        guard let query = query else{
            return
        }
        query.disableUpdates()
        var metaDataItems:[NSMetadataItem] = []

        if let queryResults = query.results as? [NSMetadataItem]{
            print( "query results:\(queryResults.count)")
            for result in queryResults{
                if let fileURL = result.value(forAttribute: NSMetadataItemURLKey ) as? NSURL{
                     print( "file found in iCloud:\(String(describing: fileURL.lastPathComponent))")
                    metaDataItems.append( result )
                }
            }
            if metaDataItems.count > 0{
                updateICloudFiles(metaDataItems: metaDataItems)
            }
        }
        query.enableUpdates()
    }

However, the selector for NSMetadataQueryDidUpdate is getting called no stop. I thought it was meant to get called only when the content of the iCloud disk changes.

Any suggestions.

Thanks

Reza

Body:
Try to deinitialize:

deinit {
    NotificationCenter.default.removeObserver(self)
}


问:
Why does a user filter result in an "Unsupported query"?
I am using the .NET Microsoft Graph API to issue a query against my Azure AD as follows:

```<code>var userRequest = _graphClient.Users.Delta();
var filter = "startswith(displayName,'K')";
var users = await userRequest.Request().Filter(filter).GetAsync();
</code>```

According to Fiddler traffic, this results in a HTTP GET that looks like:

```<code>GET /v1.0/users/microsoft.graph.delta?$filter=startswith(displayName,'K')
</code>```

Response:

```<code>400 Bad Request
Content-Type: application/json
Preference-Applied: odata.track-changes

{
  "error": {
    "code": "Request_UnsupportedQuery",
    "message": "Unsupported query.",
    "innerError": {
      "request-id": "UUID",
      "date": "2017-12-04T16:26:23"
    }
  }
}
</code>```

I even tried a filter like:

```<code>var filter = "accountEnabled eq true";
</code>```

But the same 400 error occurs.

Removing the filter results in a successful query.

Why is this filter resulting in an "Unsupported query"? From what I read in the Supported queries, filters, and paging options docs it sounds like these are valid queries.

答: The /users endpoint supports the $filter=startsWith() query parameter, but the /users/delta endpoint does not. The /delta endpoint has a number of query limitations, including this one.

Body:
I am using the .NET Microsoft Graph API to issue a query against my Azure AD as follows:

var userRequest = _graphClient.Users.Delta();
var filter = "startswith(displayName,'K')";
var users = await userRequest.Request().Filter(filter).GetAsync();

According to Fiddler traffic, this results in a HTTP GET that looks like:

GET /v1.0/users/microsoft.graph.delta?$filter=startswith(displayName,'K')

Response:

400 Bad Request
Content-Type: application/json
Preference-Applied: odata.track-changes

{
  "error": {
    "code": "Request_UnsupportedQuery",
    "message": "Unsupported query.",
    "innerError": {
      "request-id": "UUID",
      "date": "2017-12-04T16:26:23"
    }
  }
}

I even tried a filter like:

var filter = "accountEnabled eq true";

But the same 400 error occurs.

Removing the filter results in a successful query.

Why is this filter resulting in an "Unsupported query"? From what I read in the Supported queries, filters, and paging options docs it sounds like these are valid queries.


答:
To see the difference, overwrite the coolMethod in your subclass Moo:</p>

```<code>public String coolMethod() {
    return "Moo baby!";
}
</code>```

Title: What's the difference between this and super keywords in java?

Body:
To see the difference, overwrite the coolMethod in your subclass Moo:

public String coolMethod() {
    return "Moo baby!";
}


问:
UnitofWork [aspnetboilerplate] Transaction Management
I am currently working on implementing aspnetboilerplate's transaction management

Below is the method I am using to insert a order and products associated with the order

```<code>public class OrderController
{
    IOrderAppService _orderAppService;
    public OrderController(IOrderAppService orderAppService)
    {
        _orderAppService = orderAppService;
    }
    public void TestOrder()
    {
        _orderAppService.TestTransaction();
    }
}

public class OrderAppService : IOrderAppService
{
    //repositories are injected here

    public void TestTransaction()
    {
        //Created 'order' and 'products' here

        //Committing the created objects
        CommitOrderTransaction();

    }

    private void CommitOrderTransaction()
    {
        using (var unitOfWork = _unitOfWorkManager.Begin())
        {
            //Inserts the Order record
            CommitInsertOrderHeader(); // Order Header is saved in database by using SaveChanges() method

            //Inserts the Product records associated with OrderId
            CommitInsertOrderDetails();

            unitOfWork.Complete();
        }
    }
}
</code>```

As the aspnetboilerplate documentation tells that,
"if current unit of work is transactional, all changes in the transaction are rolled back if an exception occurs, even saved changes."

In my case when an exception occurs on inserting the OrderDetails, I would like the header record to be rolled back as well but I still have the Order header record in database.

答:
you don't need to handle transaction manually. ABP handles it for you! All application service methods are automatically set as UnitOfWork. It's an atomic operation. So if any exception occurs in the middle of transactions all the db operations are being rolled back.

further information check out https://aspnetboilerplate.com/Pages/Documents/Unit-Of-Work

Body:
I am currently working on implementing aspnetboilerplate's transaction management

Below is the method I am using to insert a order and products associated with the order

public class OrderController
{
    IOrderAppService _orderAppService;
    public OrderController(IOrderAppService orderAppService)
    {
        _orderAppService = orderAppService;
    }
    public void TestOrder()
    {
        _orderAppService.TestTransaction();
    }
}

public class OrderAppService : IOrderAppService
{
    //repositories are injected here

    public void TestTransaction()
    {
        //Created 'order' and 'products' here

        //Committing the created objects
        CommitOrderTransaction();

    }

    private void CommitOrderTransaction()
    {
        using (var unitOfWork = _unitOfWorkManager.Begin())
        {
            //Inserts the Order record
            CommitInsertOrderHeader(); // Order Header is saved in database by using SaveChanges() method

            //Inserts the Product records associated with OrderId
            CommitInsertOrderDetails();

            unitOfWork.Complete();
        }
    }
}

As the aspnetboilerplate documentation tells that,
"if current unit of work is transactional, all changes in the transaction are rolled back if an exception occurs, even saved changes."

In my case when an exception occurs on inserting the OrderDetails, I would like the header record to be rolled back as well but I still have the Order header record in database.


问:
Limit the number of hex digits with string formatting
I am displaying values in hex within my project of 4 characters as the value is 16bit.  In this instance the values are also signed and this is where my problem lies.

My code:

```<code>int16_t i = -4;
printf("i = %04x\n", i);
</code>```

Output:

```<code>i = fffffffc
</code>```

What I would like is the hex output to remain only 16-bit and not switch to a 32-bit representation.  Which it does for positive values.

Desired output:

```<code>i = fffc
</code>```

Is there a nice way of specifying the total number of hex digits, or locking the output to be the same as the minimum?  (Ideally without having to resort to manipulating the value as a string :) )

Oh, I also thought this might be a compiler thing, as I 1st experienced it within Visual C++, but testing with GCC gave the same results under Linux.

答:
Thanks to Pete Becker for explaining the root cause: The int16_t is being promoted to int which on my system is 32-bit, hence negative numbers having a 32-bit hex representation.  By casting i to uint16_t within the printf the sign is hidden from the int and formatting of the hex is as desired.

```<code>int16_t i = -4;
printf("i = %04x\n", (uint16_t) i);
</code>```

Output:

```<code>i = fffc
</code>```

Body:
I am displaying values in hex within my project of 4 characters as the value is 16bit.  In this instance the values are also signed and this is where my problem lies.

My code:

int16_t i = -4;
printf("i = %04x\n", i);

Output:

i = fffffffc

What I would like is the hex output to remain only 16-bit and not switch to a 32-bit representation.  Which it does for positive values.

Desired output:

i = fffc

Is there a nice way of specifying the total number of hex digits, or locking the output to be the same as the minimum?  (Ideally without having to resort to manipulating the value as a string :) )

Oh, I also thought this might be a compiler thing, as I 1st experienced it within Visual C++, but testing with GCC gave the same results under Linux.

Body:
Thanks to Pete Becker for explaining the root cause: The int16_t is being promoted to int which on my system is 32-bit, hence negative numbers having a 32-bit hex representation.  By casting i to uint16_t within the printf the sign is hidden from the int and formatting of the hex is as desired.

int16_t i = -4;
printf("i = %04x\n", (uint16_t) i);

Output:

i = fffc


问:
npm install -g mongodb not installing mongodb globally on OSX 10.8.4?
I know similar questions have been asked to death, but I can't find my particular situation. I'm just trying to install mongodb globally so that I may run it from any directory by typing mongod instead of being in the specific bin and typing ./mongod which works.  I get this error when trying:

```<code>Squirrels-MacBook-Air:Code Squirrel$ npm install -g mongodb
npm http GET https://registry.npmjs.org/mongodb
npm http 304 https://registry.npmjs.org/mongodb
npm ERR! Error: EACCES, mkdir '/usr/local/lib/node_modules/mongodb'
npm ERR!  { [Error: EACCES, mkdir '/usr/local/lib/node_modules/mongodb']
npm ERR!   errno: 3,
npm ERR!   code: 'EACCES',
npm ERR!   path: '/usr/local/lib/node_modules/mongodb',
npm ERR!   fstream_type: 'Directory',
npm ERR!   fstream_path: '/usr/local/lib/node_modules/mongodb',
npm ERR!   fstream_class: 'DirWriter',
npm ERR!   fstream_stack:
npm ERR!    [ '/usr/local/lib/node_modules/npm/node_modules/fstream/lib/dir-writer.js:36:23',
npm ERR!      '/usr/local/lib/node_modules/npm/node_modules/mkdirp/index.js:37:53',
npm ERR!      'Object.oncomplete (fs.js:107:15)' ] }
npm ERR!
npm ERR! Please try running this command again as root/Administrator.

npm ERR! System Darwin 12.4.1
npm ERR! command "node" "/usr/local/bin/npm" "install" "-g" "mongodb"
npm ERR! cwd /Users/Squirrel/Documents/Code
npm ERR! node -v v0.10.21
npm ERR! npm -v 1.3.11
npm ERR! path /usr/local/lib/node_modules/mongodb
npm ERR! fstream_path /usr/local/lib/node_modules/mongodb
npm ERR! fstream_type Directory
npm ERR! fstream_class DirWriter
npm ERR! code EACCES
npm ERR! errno 3
npm ERR! stack Error: EACCES, mkdir '/usr/local/lib/node_modules/mongodb'
npm ERR! fstream_stack /usr/local/lib/node_modules/npm/node_modules/fstream/lib/dir-writer.js:36:23
npm ERR! fstream_stack /usr/local/lib/node_modules/npm/node_modules/mkdirp/index.js:37:53
npm ERR! fstream_stack Object.oncomplete (fs.js:107:15)
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR!     /Users/Squirrel/Documents/Code/npm-debug.log
npm ERR! not ok code 0
</code>```

So I did sudo npm install mongodb -g:

```<code>Squirrels-MacBook-Air:~ Squirrel$ sudo npm install mongodb -g
Password:
npm http GET https://registry.npmjs.org/mongodb
npm http 304 https://registry.npmjs.org/mongodb
npm http GET https://registry.npmjs.org/bson/0.2.2
npm http GET https://registry.npmjs.org/kerberos/0.0.3
npm http 304 https://registry.npmjs.org/bson/0.2.2
npm http 304 https://registry.npmjs.org/kerberos/0.0.3

> kerberos@0.0.3 install /usr/local/lib/node_modules/mongodb/node_modules/kerberos
> (node-gyp rebuild 2> builderror.log) || (exit 0)

> bson@0.2.2 install /usr/local/lib/node_modules/mongodb/node_modules/bson
> (node-gyp rebuild 2> builderror.log) || (exit 0)

mongodb@1.3.19 /usr/local/lib/node_modules/mongodb
├── kerberos@0.0.3
└── bson@0.2.2
</code>```

But still it's not global.  I've also tried export PATH=/path/to/mongodb/bin:$PATH but that only works if I type it into terminal each session.  I've had it before so that I could simply type mongo or mongod from any directory, but I'm having trouble on my new computer.  Thank you.

I deleted the mongodb directory within the usr/local/lib/node_modules folder, and than I ran

```<code>$ sudo find / -name mongod
find: /dev/fd/3: Not a directory
find: /dev/fd/4: Not a directory
/Users/Squirrel/.npm/mongod
/Users/Squirrel/.Trash/mongodb-osx-x86_64-2.4.8/bin/mongod
/Users/Squirrel/.Trash/mongodb-osx-x86_64-2.4.8 6.43.25 PM/bin/mongod
/Users/Squirrel/.Trash/mongodb-osx-x86_64-2.4.8 6.57.12 PM/bin/mongod
/Users/Squirrel/.Trash/mongodb-osx-x86_64-2.4.8 7.18.50 PM/bin/mongod
/Users/Squirrel/Documents/Code/mongodb-osx-x86_64-2.4.8/bin/mongod
</code>```

Should i rm -r /Users/Squirrel/.npm/mongod ?

I also tried homebrew with no luck:

$ ruby -e "$(curl -fsSL https://raw.github.com/mxcl/homebrew/go)"
brew install mongodb

答: run ps -xa | grep mongod and you don't see a --dbpath which explicitly tells mongod to look at that parameter for the db location and you don't have a dbpath in your mongodb.conf, then the default location will be: /data/db/ and you should look there.</p>

Body:
I know similar questions have been asked to death, but I can't find my particular situation. I'm just trying to install mongodb globally so that I may run it from any directory by typing mongod instead of being in the specific bin and typing ./mongod which works.  I get this error when trying:

Squirrels-MacBook-Air:Code Squirrel$ npm install -g mongodb
npm http GET https://registry.npmjs.org/mongodb
npm http 304 https://registry.npmjs.org/mongodb
npm ERR! Error: EACCES, mkdir '/usr/local/lib/node_modules/mongodb'
npm ERR!  { [Error: EACCES, mkdir '/usr/local/lib/node_modules/mongodb']
npm ERR!   errno: 3,
npm ERR!   code: 'EACCES',
npm ERR!   path: '/usr/local/lib/node_modules/mongodb',
npm ERR!   fstream_type: 'Directory',
npm ERR!   fstream_path: '/usr/local/lib/node_modules/mongodb',
npm ERR!   fstream_class: 'DirWriter',
npm ERR!   fstream_stack:
npm ERR!    [ '/usr/local/lib/node_modules/npm/node_modules/fstream/lib/dir-writer.js:36:23',
npm ERR!      '/usr/local/lib/node_modules/npm/node_modules/mkdirp/index.js:37:53',
npm ERR!      'Object.oncomplete (fs.js:107:15)' ] }
npm ERR!
npm ERR! Please try running this command again as root/Administrator.

npm ERR! System Darwin 12.4.1
npm ERR! command "node" "/usr/local/bin/npm" "install" "-g" "mongodb"
npm ERR! cwd /Users/Squirrel/Documents/Code
npm ERR! node -v v0.10.21
npm ERR! npm -v 1.3.11
npm ERR! path /usr/local/lib/node_modules/mongodb
npm ERR! fstream_path /usr/local/lib/node_modules/mongodb
npm ERR! fstream_type Directory
npm ERR! fstream_class DirWriter
npm ERR! code EACCES
npm ERR! errno 3
npm ERR! stack Error: EACCES, mkdir '/usr/local/lib/node_modules/mongodb'
npm ERR! fstream_stack /usr/local/lib/node_modules/npm/node_modules/fstream/lib/dir-writer.js:36:23
npm ERR! fstream_stack /usr/local/lib/node_modules/npm/node_modules/mkdirp/index.js:37:53
npm ERR! fstream_stack Object.oncomplete (fs.js:107:15)
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR!     /Users/Squirrel/Documents/Code/npm-debug.log
npm ERR! not ok code 0

So I did sudo npm install mongodb -g:

Squirrels-MacBook-Air:~ Squirrel$ sudo npm install mongodb -g
Password:
npm http GET https://registry.npmjs.org/mongodb
npm http 304 https://registry.npmjs.org/mongodb
npm http GET https://registry.npmjs.org/bson/0.2.2
npm http GET https://registry.npmjs.org/kerberos/0.0.3
npm http 304 https://registry.npmjs.org/bson/0.2.2
npm http 304 https://registry.npmjs.org/kerberos/0.0.3

> kerberos@0.0.3 install /usr/local/lib/node_modules/mongodb/node_modules/kerberos
> (node-gyp rebuild 2> builderror.log) || (exit 0)

> bson@0.2.2 install /usr/local/lib/node_modules/mongodb/node_modules/bson
> (node-gyp rebuild 2> builderror.log) || (exit 0)

mongodb@1.3.19 /usr/local/lib/node_modules/mongodb
├── kerberos@0.0.3
└── bson@0.2.2

But still it's not global.  I've also tried export PATH=/path/to/mongodb/bin:$PATH but that only works if I type it into terminal each session.  I've had it before so that I could simply type mongo or mongod from any directory, but I'm having trouble on my new computer.  Thank you.

I deleted the mongodb directory within the usr/local/lib/node_modules folder, and than I ran

$ sudo find / -name mongod
find: /dev/fd/3: Not a directory
find: /dev/fd/4: Not a directory
/Users/Squirrel/.npm/mongod
/Users/Squirrel/.Trash/mongodb-osx-x86_64-2.4.8/bin/mongod
/Users/Squirrel/.Trash/mongodb-osx-x86_64-2.4.8 6.43.25 PM/bin/mongod
/Users/Squirrel/.Trash/mongodb-osx-x86_64-2.4.8 6.57.12 PM/bin/mongod
/Users/Squirrel/.Trash/mongodb-osx-x86_64-2.4.8 7.18.50 PM/bin/mongod
/Users/Squirrel/Documents/Code/mongodb-osx-x86_64-2.4.8/bin/mongod

Should i rm -r /Users/Squirrel/.npm/mongod ?

I also tried homebrew with no luck:

$ ruby -e "$(curl -fsSL https://raw.github.com/mxcl/homebrew/go)"
brew install mongodb


问:
GridSearch Using Spark Sklearn throws multiprocessing.pool.MaybeEncodingError:
I am trying to use Spark Sklearn to parallelize gridsearchcv operation using the spark sklearn module. I'm using an EMR cluster with the following configuration

Master Node - c4.4xlarge (30 G, 8 cores/ 16 vCPUs)
Slaves Nodes (3 numbers) - c4.8xlarge (60 G, 18 cores/ 36 vCPUs)

Edit: I can't paste the code, but it's fairly simple

```<code>from spark_sklearn import GridSearchCV as sp_GridSearchCV
grid_search = sp_GridSearchCV(sc, clf, parameters, n_jobs=-1, cv=cv, scoring='f1_macro',verbose=verbose)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, stratify=y)
grid_search.fit(X_train, y_train)
</code>```

More here [https://github.com/databricks/spark-sklearn]

I'm running the following spark-submit

```<code> spark-submit --executor-cores 7 --num-executors 4 --executor-memory 9G --master yarn --deploy-mode client <program_name.py>
</code>```

Size of the data: 6000 rows text data, that's it. It's not big enough.

but I keep running in this error below, which stems from the standard sklearn's gridsearch.py

Has anyone encountered this error before or would have an idea what am I doing wrong?

```<code>SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/mnt/yarn/usercache/hadoop/filecache/11/__spark_libs__7590692825730242151.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
    17/11/30 22:53:46 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 18282@ip-10-17-36-80
    17/11/30 22:53:46 INFO SignalUtils: Registered signal handler for TERM
    17/11/30 22:53:46 INFO SignalUtils: Registered signal handler for HUP
    17/11/30 22:53:46 INFO SignalUtils: Registered signal handler for INT
    17/11/30 22:53:47 INFO SecurityManager: Changing view acls to: yarn,hadoop
    17/11/30 22:53:47 INFO SecurityManager: Changing modify acls to: yarn,hadoop
    17/11/30 22:53:47 INFO SecurityManager: Changing view acls groups to:
    17/11/30 22:53:47 INFO SecurityManager: Changing modify acls groups to:
    17/11/30 22:53:47 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yarn, hadoop); groups with view permissions: Set(); users  with modify permissions: Set(yarn, hadoop); groups with modify permissions: Set()
    17/11/30 22:53:47 INFO TransportClientFactory: Successfully created connection to /10.17.36.61:36396 after 53 ms (0 ms spent in bootstraps)
    17/11/30 22:53:47 INFO SecurityManager: Changing view acls to: yarn,hadoop
    17/11/30 22:53:47 INFO SecurityManager: Changing modify acls to: yarn,hadoop
    17/11/30 22:53:47 INFO SecurityManager: Changing view acls groups to:
    17/11/30 22:53:47 INFO SecurityManager: Changing modify acls groups to:
    17/11/30 22:53:47 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yarn, hadoop); groups with view permissions: Set(); users  with modify permissions: Set(yarn, hadoop); groups with modify permissions: Set()
    17/11/30 22:53:47 INFO TransportClientFactory: Successfully created connection to /10.17.36.61:36396 after 0 ms (0 ms spent in bootstraps)
    17/11/30 22:53:47 INFO DiskBlockManager: Created local directory at /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/blockmgr-9390d584-49ab-407b-ab0d-c98f5cae1bfe
    17/11/30 22:53:47 INFO MemoryStore: MemoryStore started with capacity 7.5 GB
    17/11/30 22:53:47 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@10.17.36.61:36396
    17/11/30 22:53:47 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
    17/11/30 22:53:47 INFO Executor: Starting executor ID 1 on host ip-10-17-36-80.us-west-2.compute.internal
    17/11/30 22:53:48 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44955.
    17/11/30 22:53:48 INFO NettyBlockTransferService: Server created on ip-10-17-36-80.us-west-2.compute.internal:44955
    17/11/30 22:53:48 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
    17/11/30 22:53:48 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(1, ip-10-17-36-80.us-west-2.compute.internal, 44955, None)
    17/11/30 22:53:48 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(1, ip-10-17-36-80.us-west-2.compute.internal, 44955, None)
    17/11/30 22:53:48 INFO BlockManager: external shuffle service port = 7337
    17/11/30 22:53:48 INFO BlockManager: Registering executor with local external shuffle service.
    17/11/30 22:53:48 INFO TransportClientFactory: Successfully created connection to ip-10-17-36-80.us-west-2.compute.internal/10.17.36.80:7337 after 0 ms (0 ms spent in bootstraps)
    17/11/30 22:53:48 INFO BlockManager: Initialized BlockManager: BlockManagerId(1, ip-10-17-36-80.us-west-2.compute.internal, 44955, None)
    17/11/30 22:53:50 INFO CoarseGrainedExecutorBackend: Got assigned task 5
    17/11/30 22:53:50 INFO CoarseGrainedExecutorBackend: Got assigned task 14
    17/11/30 22:53:50 INFO CoarseGrainedExecutorBackend: Got assigned task 23
    17/11/30 22:53:50 INFO CoarseGrainedExecutorBackend: Got assigned task 32
    17/11/30 22:53:50 INFO CoarseGrainedExecutorBackend: Got assigned task 41
    17/11/30 22:53:50 INFO Executor: Running task 5.0 in stage 0.0 (TID 5)
    17/11/30 22:53:50 INFO Executor: Running task 23.0 in stage 0.0 (TID 23)
    17/11/30 22:53:50 INFO Executor: Running task 14.0 in stage 0.0 (TID 14)
    17/11/30 22:53:50 INFO Executor: Running task 32.0 in stage 0.0 (TID 32)
    17/11/30 22:53:50 INFO Executor: Running task 41.0 in stage 0.0 (TID 41)
    17/11/30 22:53:50 INFO Executor: Fetching spark://10.17.36.61:36396/files/main.py with timestamp 1512082429416
    17/11/30 22:53:50 INFO TransportClientFactory: Successfully created connection to /10.17.36.61:36396 after 1 ms (0 ms spent in bootstraps)
    17/11/30 22:53:50 INFO Utils: Fetching spark://10.17.36.61:36396/files/main.py to /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/spark-2016730f-a9a7-42f6-af43-e214e17a799c/fetchFileTemp5747466536720143705.tmp
    17/11/30 22:53:50 INFO Utils: Copying /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/spark-2016730f-a9a7-42f6-af43-e214e17a799c/-7964204541512082429416_cache to /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/container_1512081909079_0001_01_000002/./main.py
    17/11/30 22:53:50 INFO Executor: Fetching spark://10.17.36.61:36396/files/config.py with timestamp 1512082429407
    17/11/30 22:53:50 INFO Utils: Fetching spark://10.17.36.61:36396/files/config.py to /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/spark-2016730f-a9a7-42f6-af43-e214e17a799c/fetchFileTemp1456561651760694351.tmp
    17/11/30 22:53:50 INFO Utils: Copying /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/spark-2016730f-a9a7-42f6-af43-e214e17a799c/-13639922071512082429407_cache to /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/container_1512081909079_0001_01_000002/./config.py
    17/11/30 22:53:50 INFO Executor: Fetching spark://10.17.36.61:36396/files/helper.py with timestamp 1512082429375
    17/11/30 22:53:50 INFO Utils: Fetching spark://10.17.36.61:36396/files/helper.py to /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/spark-2016730f-a9a7-42f6-af43-e214e17a799c/fetchFileTemp3268157159972473452.tmp
    17/11/30 22:53:50 INFO Utils: Copying /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/spark-2016730f-a9a7-42f6-af43-e214e17a799c/7433209651512082429375_cache to /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/container_1512081909079_0001_01_000002/./helper.py
    17/11/30 22:53:50 INFO Executor: Fetching spark://10.17.36.61:36396/files/transformers.py with timestamp 1512082429412
    17/11/30 22:53:50 INFO Utils: Fetching spark://10.17.36.61:36396/files/transformers.py to /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/spark-2016730f-a9a7-42f6-af43-e214e17a799c/fetchFileTemp5944429394790398969.tmp
    17/11/30 22:53:50 INFO Utils: Copying /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/spark-2016730f-a9a7-42f6-af43-e214e17a799c/-3422273991512082429412_cache to /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/container_1512081909079_0001_01_000002/./transformers.py
    17/11/30 22:53:50 INFO TorrentBroadcast: Started reading broadcast variable 2
    17/11/30 22:53:50 INFO TransportClientFactory: Successfully created connection to /10.17.36.61:42897 after 1 ms (0 ms spent in bootstraps)
    17/11/30 22:53:50 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 9.9 KB, free 7.5 GB)
    17/11/30 22:53:50 INFO TorrentBroadcast: Reading broadcast variable 2 took 99 ms
    17/11/30 22:53:50 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 14.3 KB, free 7.5 GB)
    17/11/30 22:53:51 INFO TorrentBroadcast: Started reading broadcast variable 0
    17/11/30 22:53:51 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 226.1 KB, free 7.5 GB)
    17/11/30 22:53:51 INFO TorrentBroadcast: Reading broadcast variable 0 took 9 ms
    17/11/30 22:53:51 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 368.0 B, free 7.5 GB)
    17/11/30 22:53:51 INFO TorrentBroadcast: Started reading broadcast variable 1
    17/11/30 22:53:51 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 6.7 KB, free 7.5 GB)
    17/11/30 22:53:51 INFO TorrentBroadcast: Reading broadcast variable 1 took 6 ms
    17/11/30 22:53:51 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 368.0 B, free 7.5 GB)
    /usr/local/lib64/python3.5/site-packages/sklearn/linear_model/logistic.py:1228: UserWarning: 'n_jobs' > 1 does not have any effect when 'solver' is set to 'liblinear'. Got 'n_jobs' = -1.
      " = {}.".format(self.n_jobs))
    17/11/30 23:35:37 ERROR Executor: Exception in task 32.0 in stage 0.0 (TID 32)
    org.apache.spark.api.python.PythonException: Traceback (most recent call last):
      File "/mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/container_1512081909079_0001_01_000002/pyspark.zip/pyspark/worker.py", line 177, in main
        process()
      File "/mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/container_1512081909079_0001_01_000002/pyspark.zip/pyspark/worker.py", line 172, in process
        serializer.dump_stream(func(split_index, iterator), outfile)
      File "/mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/container_1512081909079_0001_01_000002/pyspark.zip/pyspark/serializers.py", line 268, in dump_stream
        vs = list(itertools.islice(iterator, batch))
      File "/usr/local/lib/python3.5/site-packages/spark_sklearn/grid_search.py", line 319, in fun
        return_parameters=True, error_score=error_score)
      File "/usr/local/lib64/python3.5/site-packages/sklearn/model_selection/_validation.py", line 437, in _fit_and_score
        estimator.fit(X_train, y_train, **fit_params)
      File "/usr/local/lib64/python3.5/site-packages/sklearn/pipeline.py", line 257, in fit
        Xt, fit_params = self._fit(X, y, **fit_params)
      File "/usr/local/lib64/python3.5/site-packages/sklearn/pipeline.py", line 222, in _fit
        **fit_params_steps[name])
      File "/usr/local/lib64/python3.5/site-packages/sklearn/externals/joblib/memory.py", line 362, in __call__
        return self.func(*args, **kwargs)
      File "/usr/local/lib64/python3.5/site-packages/sklearn/pipeline.py", line 589, in _fit_transform_one
        res = transformer.fit_transform(X, y, **fit_params)
      File "/usr/local/lib64/python3.5/site-packages/sklearn/pipeline.py", line 746, in fit_transform
        for name, trans, weight in self._iter())
      File "/usr/local/lib64/python3.5/site-packages/sklearn/externals/joblib/parallel.py", line 789, in __call__
        self.retrieve()
      File "/usr/local/lib64/python3.5/site-packages/sklearn/externals/joblib/parallel.py", line 699, in retrieve
        self._output.extend(job.get(timeout=self.timeout))
      File "/usr/lib64/python3.5/multiprocessing/pool.py", line 608, in get
        raise self._value
    multiprocessing.pool.MaybeEncodingError: Error sending result: '[(array([[ 0.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           ...,
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.]]), Pipeline(memory=None,
         steps=[('features', CustomFeatures()), ('tfidf', TfidfVectorizer(analyzer='word', binary=False, decode_error='strict',
            dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',
            lowercase=True, max_df=1.0, max_features=None, min_df=1,
            ngram_range=(1, 2), norm='l2', prepr...ac3c18>), ('kbest', SelectPercentile(percentile=100, score_func=<function chi2 at 0x7f38fbf23f28>))]))]'. Reason: 'MemoryError()'

        at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:193)
        at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:234)
        at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:152)
        at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
        at org.apache.spark.scheduler.Task.run(Task.scala:108)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
</code>```

答: Okay, the solution to the above problem was to reduce the dimensionality of the dataset. I used max_df and min_df in the TFIDF step to achieve this. This works perfectly fine now.</p>

Body:
I am trying to use Spark Sklearn to parallelize gridsearchcv operation using the spark sklearn module. I'm using an EMR cluster with the following configuration

Master Node - c4.4xlarge (30 G, 8 cores/ 16 vCPUs)
Slaves Nodes (3 numbers) - c4.8xlarge (60 G, 18 cores/ 36 vCPUs)

Edit: I can't paste the code, but it's fairly simple

from spark_sklearn import GridSearchCV as sp_GridSearchCV
grid_search = sp_GridSearchCV(sc, clf, parameters, n_jobs=-1, cv=cv, scoring='f1_macro',verbose=verbose)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, stratify=y)
grid_search.fit(X_train, y_train)

More here [https://github.com/databricks/spark-sklearn]

I'm running the following spark-submit

 spark-submit --executor-cores 7 --num-executors 4 --executor-memory 9G --master yarn --deploy-mode client <program_name.py>

Size of the data: 6000 rows text data, that's it. It's not big enough.

but I keep running in this error below, which stems from the standard sklearn's gridsearch.py

Has anyone encountered this error before or would have an idea what am I doing wrong?

SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/mnt/yarn/usercache/hadoop/filecache/11/__spark_libs__7590692825730242151.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
    17/11/30 22:53:46 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 18282@ip-10-17-36-80
    17/11/30 22:53:46 INFO SignalUtils: Registered signal handler for TERM
    17/11/30 22:53:46 INFO SignalUtils: Registered signal handler for HUP
    17/11/30 22:53:46 INFO SignalUtils: Registered signal handler for INT
    17/11/30 22:53:47 INFO SecurityManager: Changing view acls to: yarn,hadoop
    17/11/30 22:53:47 INFO SecurityManager: Changing modify acls to: yarn,hadoop
    17/11/30 22:53:47 INFO SecurityManager: Changing view acls groups to:
    17/11/30 22:53:47 INFO SecurityManager: Changing modify acls groups to:
    17/11/30 22:53:47 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yarn, hadoop); groups with view permissions: Set(); users  with modify permissions: Set(yarn, hadoop); groups with modify permissions: Set()
    17/11/30 22:53:47 INFO TransportClientFactory: Successfully created connection to /10.17.36.61:36396 after 53 ms (0 ms spent in bootstraps)
    17/11/30 22:53:47 INFO SecurityManager: Changing view acls to: yarn,hadoop
    17/11/30 22:53:47 INFO SecurityManager: Changing modify acls to: yarn,hadoop
    17/11/30 22:53:47 INFO SecurityManager: Changing view acls groups to:
    17/11/30 22:53:47 INFO SecurityManager: Changing modify acls groups to:
    17/11/30 22:53:47 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yarn, hadoop); groups with view permissions: Set(); users  with modify permissions: Set(yarn, hadoop); groups with modify permissions: Set()
    17/11/30 22:53:47 INFO TransportClientFactory: Successfully created connection to /10.17.36.61:36396 after 0 ms (0 ms spent in bootstraps)
    17/11/30 22:53:47 INFO DiskBlockManager: Created local directory at /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/blockmgr-9390d584-49ab-407b-ab0d-c98f5cae1bfe
    17/11/30 22:53:47 INFO MemoryStore: MemoryStore started with capacity 7.5 GB
    17/11/30 22:53:47 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@10.17.36.61:36396
    17/11/30 22:53:47 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
    17/11/30 22:53:47 INFO Executor: Starting executor ID 1 on host ip-10-17-36-80.us-west-2.compute.internal
    17/11/30 22:53:48 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44955.
    17/11/30 22:53:48 INFO NettyBlockTransferService: Server created on ip-10-17-36-80.us-west-2.compute.internal:44955
    17/11/30 22:53:48 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
    17/11/30 22:53:48 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(1, ip-10-17-36-80.us-west-2.compute.internal, 44955, None)
    17/11/30 22:53:48 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(1, ip-10-17-36-80.us-west-2.compute.internal, 44955, None)
    17/11/30 22:53:48 INFO BlockManager: external shuffle service port = 7337
    17/11/30 22:53:48 INFO BlockManager: Registering executor with local external shuffle service.
    17/11/30 22:53:48 INFO TransportClientFactory: Successfully created connection to ip-10-17-36-80.us-west-2.compute.internal/10.17.36.80:7337 after 0 ms (0 ms spent in bootstraps)
    17/11/30 22:53:48 INFO BlockManager: Initialized BlockManager: BlockManagerId(1, ip-10-17-36-80.us-west-2.compute.internal, 44955, None)
    17/11/30 22:53:50 INFO CoarseGrainedExecutorBackend: Got assigned task 5
    17/11/30 22:53:50 INFO CoarseGrainedExecutorBackend: Got assigned task 14
    17/11/30 22:53:50 INFO CoarseGrainedExecutorBackend: Got assigned task 23
    17/11/30 22:53:50 INFO CoarseGrainedExecutorBackend: Got assigned task 32
    17/11/30 22:53:50 INFO CoarseGrainedExecutorBackend: Got assigned task 41
    17/11/30 22:53:50 INFO Executor: Running task 5.0 in stage 0.0 (TID 5)
    17/11/30 22:53:50 INFO Executor: Running task 23.0 in stage 0.0 (TID 23)
    17/11/30 22:53:50 INFO Executor: Running task 14.0 in stage 0.0 (TID 14)
    17/11/30 22:53:50 INFO Executor: Running task 32.0 in stage 0.0 (TID 32)
    17/11/30 22:53:50 INFO Executor: Running task 41.0 in stage 0.0 (TID 41)
    17/11/30 22:53:50 INFO Executor: Fetching spark://10.17.36.61:36396/files/main.py with timestamp 1512082429416
    17/11/30 22:53:50 INFO TransportClientFactory: Successfully created connection to /10.17.36.61:36396 after 1 ms (0 ms spent in bootstraps)
    17/11/30 22:53:50 INFO Utils: Fetching spark://10.17.36.61:36396/files/main.py to /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/spark-2016730f-a9a7-42f6-af43-e214e17a799c/fetchFileTemp5747466536720143705.tmp
    17/11/30 22:53:50 INFO Utils: Copying /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/spark-2016730f-a9a7-42f6-af43-e214e17a799c/-7964204541512082429416_cache to /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/container_1512081909079_0001_01_000002/./main.py
    17/11/30 22:53:50 INFO Executor: Fetching spark://10.17.36.61:36396/files/config.py with timestamp 1512082429407
    17/11/30 22:53:50 INFO Utils: Fetching spark://10.17.36.61:36396/files/config.py to /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/spark-2016730f-a9a7-42f6-af43-e214e17a799c/fetchFileTemp1456561651760694351.tmp
    17/11/30 22:53:50 INFO Utils: Copying /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/spark-2016730f-a9a7-42f6-af43-e214e17a799c/-13639922071512082429407_cache to /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/container_1512081909079_0001_01_000002/./config.py
    17/11/30 22:53:50 INFO Executor: Fetching spark://10.17.36.61:36396/files/helper.py with timestamp 1512082429375
    17/11/30 22:53:50 INFO Utils: Fetching spark://10.17.36.61:36396/files/helper.py to /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/spark-2016730f-a9a7-42f6-af43-e214e17a799c/fetchFileTemp3268157159972473452.tmp
    17/11/30 22:53:50 INFO Utils: Copying /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/spark-2016730f-a9a7-42f6-af43-e214e17a799c/7433209651512082429375_cache to /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/container_1512081909079_0001_01_000002/./helper.py
    17/11/30 22:53:50 INFO Executor: Fetching spark://10.17.36.61:36396/files/transformers.py with timestamp 1512082429412
    17/11/30 22:53:50 INFO Utils: Fetching spark://10.17.36.61:36396/files/transformers.py to /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/spark-2016730f-a9a7-42f6-af43-e214e17a799c/fetchFileTemp5944429394790398969.tmp
    17/11/30 22:53:50 INFO Utils: Copying /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/spark-2016730f-a9a7-42f6-af43-e214e17a799c/-3422273991512082429412_cache to /mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/container_1512081909079_0001_01_000002/./transformers.py
    17/11/30 22:53:50 INFO TorrentBroadcast: Started reading broadcast variable 2
    17/11/30 22:53:50 INFO TransportClientFactory: Successfully created connection to /10.17.36.61:42897 after 1 ms (0 ms spent in bootstraps)
    17/11/30 22:53:50 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 9.9 KB, free 7.5 GB)
    17/11/30 22:53:50 INFO TorrentBroadcast: Reading broadcast variable 2 took 99 ms
    17/11/30 22:53:50 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 14.3 KB, free 7.5 GB)
    17/11/30 22:53:51 INFO TorrentBroadcast: Started reading broadcast variable 0
    17/11/30 22:53:51 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 226.1 KB, free 7.5 GB)
    17/11/30 22:53:51 INFO TorrentBroadcast: Reading broadcast variable 0 took 9 ms
    17/11/30 22:53:51 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 368.0 B, free 7.5 GB)
    17/11/30 22:53:51 INFO TorrentBroadcast: Started reading broadcast variable 1
    17/11/30 22:53:51 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 6.7 KB, free 7.5 GB)
    17/11/30 22:53:51 INFO TorrentBroadcast: Reading broadcast variable 1 took 6 ms
    17/11/30 22:53:51 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 368.0 B, free 7.5 GB)
    /usr/local/lib64/python3.5/site-packages/sklearn/linear_model/logistic.py:1228: UserWarning: 'n_jobs' > 1 does not have any effect when 'solver' is set to 'liblinear'. Got 'n_jobs' = -1.
      " = {}.".format(self.n_jobs))
    17/11/30 23:35:37 ERROR Executor: Exception in task 32.0 in stage 0.0 (TID 32)
    org.apache.spark.api.python.PythonException: Traceback (most recent call last):
      File "/mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/container_1512081909079_0001_01_000002/pyspark.zip/pyspark/worker.py", line 177, in main
        process()
      File "/mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/container_1512081909079_0001_01_000002/pyspark.zip/pyspark/worker.py", line 172, in process
        serializer.dump_stream(func(split_index, iterator), outfile)
      File "/mnt/yarn/usercache/hadoop/appcache/application_1512081909079_0001/container_1512081909079_0001_01_000002/pyspark.zip/pyspark/serializers.py", line 268, in dump_stream
        vs = list(itertools.islice(iterator, batch))
      File "/usr/local/lib/python3.5/site-packages/spark_sklearn/grid_search.py", line 319, in fun
        return_parameters=True, error_score=error_score)
      File "/usr/local/lib64/python3.5/site-packages/sklearn/model_selection/_validation.py", line 437, in _fit_and_score
        estimator.fit(X_train, y_train, **fit_params)
      File "/usr/local/lib64/python3.5/site-packages/sklearn/pipeline.py", line 257, in fit
        Xt, fit_params = self._fit(X, y, **fit_params)
      File "/usr/local/lib64/python3.5/site-packages/sklearn/pipeline.py", line 222, in _fit
        **fit_params_steps[name])
      File "/usr/local/lib64/python3.5/site-packages/sklearn/externals/joblib/memory.py", line 362, in __call__
        return self.func(*args, **kwargs)
      File "/usr/local/lib64/python3.5/site-packages/sklearn/pipeline.py", line 589, in _fit_transform_one
        res = transformer.fit_transform(X, y, **fit_params)
      File "/usr/local/lib64/python3.5/site-packages/sklearn/pipeline.py", line 746, in fit_transform
        for name, trans, weight in self._iter())
      File "/usr/local/lib64/python3.5/site-packages/sklearn/externals/joblib/parallel.py", line 789, in __call__
        self.retrieve()
      File "/usr/local/lib64/python3.5/site-packages/sklearn/externals/joblib/parallel.py", line 699, in retrieve
        self._output.extend(job.get(timeout=self.timeout))
      File "/usr/lib64/python3.5/multiprocessing/pool.py", line 608, in get
        raise self._value
    multiprocessing.pool.MaybeEncodingError: Error sending result: '[(array([[ 0.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           ...,
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.]]), Pipeline(memory=None,
         steps=[('features', CustomFeatures()), ('tfidf', TfidfVectorizer(analyzer='word', binary=False, decode_error='strict',
            dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',
            lowercase=True, max_df=1.0, max_features=None, min_df=1,
            ngram_range=(1, 2), norm='l2', prepr...ac3c18>), ('kbest', SelectPercentile(percentile=100, score_func=<function chi2 at 0x7f38fbf23f28>))]))]'. Reason: 'MemoryError()'

        at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:193)
        at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:234)
        at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:152)
        at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
        at org.apache.spark.scheduler.Task.run(Task.scala:108)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)


问:
Case-insensitive exact match with SQLAlchemy
How can I ensure that the = operator is always rendered case-insensitive? Are comparisions with the LOWER or the UPPER functions the best bet for performance? ILIKE seems to be very slow.

答:
If you need only case-insensitivity use upper or lower since like is not only about case-insensitivity

example of lower:

my_string = 'BarFoo'
session.query(Foo).filter(func.lower(Foo.bar) == my_string.lower()).all()
```

see some more info on like here how to execute LIKE query in sqlalchemy?

Body:
If you need only case-insensitivity use upper or lower since like is not only about case-insensitivity

example of lower:

my_string = 'BarFoo'
session.query(Foo).filter(func.lower(Foo.bar) == my_string.lower()).all()

see some more info on like here how to execute LIKE query in sqlalchemy?