问:
Loading flatfiles into a normalized MySQL database
What is the fastest way to load data from flatfiles into a MySQL database, and then create the relations between the tables via foreign keys?

For example... I have a flat file in the format:

```<code>[INDIVIDUAL]   [POP]  [MARKER]  [GENOTYPE]

"INDIVIDUAL1", "CEU", "rs55555","AA"
"INDIVIDUAL1", "CEU", "rs535454","GA"
"INDIVIDUAL1", "CEU", "rs555566","AT"
"INDIVIDUAL1", "CEU", "rs12345","TT"
...
"INDIVIDUAL2", "JPT", "rs55555","AT"
</code>```

Which I need to load into four tables:

```<code>IND (id,fk_pop,name)
POP (id,population)
MARKER (id,rsid)
GENOTYPE (id,fk_ind,fk_rsid,call)
</code>```

Specifically, how does one populate the foreign keys in a way that scales? The figures are in the range of 1000+ individuals, each with 1 million+ genotypes.

答:
There is a simpler way.

First, make sure you have a UNIQUE constraint on those columns that should have one (name, population, rsid).

Then use something like the following:

```<code> LOAD DATA INFILE 'data.txt' IGNORE INTO TABLE POP FIELDS TERMINATED BY ','
    ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES
    (@name, population, @rsid, @call);
 LOAD DATA INFILE 'data.txt' IGNORE INTO TABLE MARKER FIELDS TERMINATED BY ','
    ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES
    (@name, @population, rsid, @call);
 LOAD DATA INFILE 'data.txt' IGNORE INTO TABLE IND FIELDS TERMINATED BY ','
    ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES
    (name, @population, @rsid, @call)
    SET fk_pop = (SELECT id FROM POP WHERE population = @population);
 LOAD DATA INFILE 'data.txt' IGNORE INTO TABLE GENOTYPE FIELDS TERMINATED BY ','
    ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES
    (@name, @population, @rsid, call)
    SET fk_ind = (SELECT id FROM IND where name = @name),
    fk_rsid = (SELECT id FROM MARKER where rsid = @rsid);
</code>```

Note where the @ is used to indicate variables, rather than column names.  In the first 2 LOAD DATAs, these are just used to ignore data.  In the second 2, they are used to look up the foreign keys.

Might not be very fast, mind :).

Body:
What is the fastest way to load data from flatfiles into a MySQL database, and then create the relations between the tables via foreign keys?

For example... I have a flat file in the format:

[INDIVIDUAL]   [POP]  [MARKER]  [GENOTYPE]

"INDIVIDUAL1", "CEU", "rs55555","AA"
"INDIVIDUAL1", "CEU", "rs535454","GA"
"INDIVIDUAL1", "CEU", "rs555566","AT"
"INDIVIDUAL1", "CEU", "rs12345","TT"
...
"INDIVIDUAL2", "JPT", "rs55555","AT"

Which I need to load into four tables:

IND (id,fk_pop,name)
POP (id,population)
MARKER (id,rsid)
GENOTYPE (id,fk_ind,fk_rsid,call)

Specifically, how does one populate the foreign keys in a way that scales? The figures are in the range of 1000+ individuals, each with 1 million+ genotypes.

Body:
There is a simpler way.

First, make sure you have a UNIQUE constraint on those columns that should have one (name, population, rsid).

Then use something like the following:

 LOAD DATA INFILE 'data.txt' IGNORE INTO TABLE POP FIELDS TERMINATED BY ','
    ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES
    (@name, population, @rsid, @call);
 LOAD DATA INFILE 'data.txt' IGNORE INTO TABLE MARKER FIELDS TERMINATED BY ','
    ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES
    (@name, @population, rsid, @call);
 LOAD DATA INFILE 'data.txt' IGNORE INTO TABLE IND FIELDS TERMINATED BY ','
    ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES
    (name, @population, @rsid, @call)
    SET fk_pop = (SELECT id FROM POP WHERE population = @population);
 LOAD DATA INFILE 'data.txt' IGNORE INTO TABLE GENOTYPE FIELDS TERMINATED BY ','
    ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES
    (@name, @population, @rsid, call)
    SET fk_ind = (SELECT id FROM IND where name = @name),
    fk_rsid = (SELECT id FROM MARKER where rsid = @rsid);

Note where the @ is used to indicate variables, rather than column names.  In the first 2 LOAD DATAs, these are just used to ignore data.  In the second 2, they are used to look up the foreign keys.

Might not be very fast, mind :).

OwnerDisplayName: naughtybeagle


来源: stackexchange

问:
Progress Bar with axios
I have to display the upload status of the file using a Progress Bar. I am using axios to make http requests. I followed the example from their github page https://github.com/mzabriskie/axios/blob/master/examples/upload/index.html

My code looks like this:

```<code>this.store().then(() => {
    var form = new FormData();
        form.append('video', this.file);
        form.append('uid', this.uid);

        axios.post('/upload', form, {
            progress: (progressEvent) => {
                    if (progressEvent.lengthComputable) {
                       console.log(progressEvent.loaded + ' ' + progressEvent.total);
                       this.updateProgressBarValue(progressEvent);
                    }
           }
       })
});
</code>```

However, it is not executing the console.log(progressEvent.loaded + ' ' + progressEvent.total); at all nor is it calling this.updateProgressBarValue(progressEvent);

How can I solve this??

答: I found the answer. The name of the event is onUploadProgress and I was using progress

ContentLicense: CC BY-SA 4.0

Title: Progress Bar with axios

Body:
I have to display the upload status of the file using a Progress Bar. I am using axios to make http requests. I followed the example from their github page https://github.com/mzabriskie/axios/blob/master/examples/upload/index.html

My code looks like this:

this.store().then(() => {
    var form = new FormData();
        form.append('video', this.file);
        form.append('uid', this.uid);

        axios.post('/upload', form, {
            progress: (progressEvent) => {
                    if (progressEvent.lengthComputable) {
                       console.log(progressEvent.loaded + ' ' + progressEvent.total);
                       this.updateProgressBarValue(progressEvent);
                    }
           }
       })
});

However, it is not executing the console.log(progressEvent.loaded + ' ' + progressEvent.total); at all nor is it calling this.updateProgressBarValue(progressEvent);

How can I solve this??

ContentLicense: CC BY-SA 3.0


问:
C loop with two conditions - difference
Whats the difference between:

```<code>int i;
for( i=0;i<5 && i<3;i++)
{
    printf("stackoverflow.com");
}
</code>```

and

```<code>int i;
for( i=0;i<5 , i<3;i++)
{
    printf("stackoverflow.com");
}
</code>```

I mean use of && instead of ','

Regards

答:
In the second code block, only i < 3 is actually used to evaluate whether the loop should exit. The i < 5 expression is evaluated, but its value is discarded. See the comma operator for more info.

There is functionally no difference between your examples because i < 3 is the limiting expression and appears second. Both loops will exit when i reaches 3. However, if you switched the terms so that you have i < 3, i < 5, then the second kicks out when it reaches 5 because only the value of i < 5 is considered.

Title: C loop with two conditions - difference

Body:
Whats the difference between:

int i;
for( i=0;i<5 && i<3;i++)
{
    printf("stackoverflow.com");
}

and

int i;
for( i=0;i<5 , i<3;i++)
{
    printf("stackoverflow.com");
}

I mean use of && instead of ','

Regards


问:
Replace "\n" in EL
In a JSP file I wanto to replace newlines (\n) with
. I tried

```<code>${fn:replace(someString, '\n', '<br />')}
</code>```

But I get an error '\n' ecnountered, was expeting one of...

Which I guess it means the parser doesn't like something like this.

Is it possible to do something like this using EL?

答:
For me, it worked by first defining a variable newLine, either with

```<code><% pageContext.setAttribute("newLine", "\n"); %>
</code>```

Or

```<code><c:set var="newLine" value="<%= '\n' %>" />
</code>```

And then

```<code>${fn:replace(someString, newLine, '<br/>')}
</code>```

Title: Replace "\n" in EL

Body:
In a JSP file I wanto to replace newlines (\n) with
. I tried

${fn:replace(someString, '\n', '
')}

But I get an error '\n' ecnountered, was expeting one of...

Which I guess it means the parser doesn't like something like this.

Is it possible to do something like this using EL?

Body:
For me, it worked by first defining a variable newLine, either with

<% pageContext.setAttribute("newLine", "\n"); %>

Or

<c:set var="newLine" value="<%= '\n' %>" />

And then

${fn:replace(someString, newLine, '<br/>')}


问:
SQL replace a null xml value
I'm a beginner in sql, and I don't know how to find a solution to my problem.

I have a table with items, a column contains xml value. I want to check if a field exist and update it.

I find a way to modify an existing xml value, but when it's NULL it doesn't work

there is my code :

```<code>SELECT @xml=IDA_Xml,
   @isData = IDA_Xml.exist('(/Root/Data[1]/text())[1]')
FROM DATA
WHERE IDA_UId=@guid

If (@xml is NULL)
BEGIN
    -- Create a xml value with <Root></Root>
    -- It will be update later
    -- set isData to 0
END

IF (@isData = 0)
    UPDATE ItemData SET IDA_Xml.modify
    ('
        insert <Data Name = "Info">{sql:variable("@number")}</Data>
        into (/Root/*)[1]
    ')
    WHERE IDA_UId = @guid;
ELSE
    UPDATE ItemData SET IDA_Xml.modify
    ('
        replace value of (/Root/Data[1]/text())[1]
        with    sql:variable("@number")
    ')
    WHERE IDA_UId = @guid;
</code>```

答:
In this case the easiest approach in most cases is to throw away the existing element and insert it as new:</p>

```<code>DECLARE @mockup TABLE(ID INT IDENTITY, Descr VARCHAR(100),YourXml XML);
INSERT INTO @mockup VALUES
 ('element exists','<root><element>test</element><SomeOther>blah</SomeOther></root>')
,('element doesn''t exist','<root><SomeOther>blah</SomeOther></root>')
,('element exists, but empty','<root><element/><SomeOther>blah</SomeOther></root>');

--At the end, all should be `<element>new</element>`

UPDATE @mockup SET YourXml.modify('delete (/root/element)[1]');
UPDATE @mockup SET YourXml.modify('insert <element>new</element> as first into (/root)[1]');

SELECT * FROM @mockup;
</code>```

Title: SQL replace a null xml value

Body:
I'm a beginner in sql, and I don't know how to find a solution to my problem.

I have a table with items, a column contains xml value. I want to check if a field exist and update it.

I find a way to modify an existing xml value, but when it's NULL it doesn't work

there is my code :

SELECT @xml=IDA_Xml,
   @isData = IDA_Xml.exist('(/Root/Data[1]/text())[1]')
FROM DATA
WHERE IDA_UId=@guid

If (@xml is NULL)
BEGIN
    -- Create a xml value with <Root></Root>
    -- It will be update later
    -- set isData to 0
END

IF (@isData = 0)
    UPDATE ItemData SET IDA_Xml.modify
    ('
        insert <Data Name = "Info">{sql:variable("@number")}</Data>
        into (/Root/*)[1]
    ')
    WHERE IDA_UId = @guid;
ELSE
    UPDATE ItemData SET IDA_Xml.modify
    ('
        replace value of (/Root/Data[1]/text())[1]
        with    sql:variable("@number")
    ')
    WHERE IDA_UId = @guid;

Body:
In this case the easiest approach in most cases is to throw away the existing element and insert it as new:

DECLARE @mockup TABLE(ID INT IDENTITY, Descr VARCHAR(100),YourXml XML);
INSERT INTO @mockup VALUES
 ('element exists','<root><element>test</element><SomeOther>blah</SomeOther></root>')
,('element doesn''t exist','<root><SomeOther>blah</SomeOther></root>')
,('element exists, but empty','<root><element/><SomeOther>blah</SomeOther></root>');

--At the end, all should be `<element>new</element>`

UPDATE @mockup SET YourXml.modify('delete (/root/element)[1]');
UPDATE @mockup SET YourXml.modify('insert <element>new</element> as first into (/root)[1]');

SELECT * FROM @mockup;


问:
ADC for PIC18F47J53 code unexpected behavior when passing by value
I am implementing the ADC for PIC18F47J53 using MPLAB v3.51.

Why is the value of the variable CH in the main function not passed correctly to the function ADC_Read with the argument channel?

```<code>unsigned int ADC_Read(unsigned char channel);
void ADC_Initialize();

#pragma config ADCSEL = BIT12                  //12 bit conversion mode enabled

#define ANCON_AiBi4         ANCON0bits.PCFG1   // Analog Input channels

int main(int argc, char** argv)
{
    unsigned int j=0;
    unsigned char CH;

    ADC_Initialize();

    while (1)
    {
        CH = 1;
        j = ADC_Read(CH); //store the result of adc in j.
    }
    return (0);
}

void ADC_Initialize()
{
     ADCON0 = 0x01;   //00000001      ADC ON, AVss, AVdd
     ADCON1 = 0xBD;   //10111101      R justified, Normal op, 20TAD, FOSC/16
}

/* Select the analog channel */
unsigned int ADC_Read(unsigned char channel)
{
    ADCON0 &= 0xC3;                     //11000011    Clearing the Channel Selection Bits
    ADCON0 |= channel<<2;               //Setting the required Bits
    ADCON0bits.GO_DONE = 1;
    while(ADCON0bits.GO_DONE);
    return ((ADRESH * 256) + ADRESL);   //Returns Result
}
</code>```

Generated assembly code

```<code>!unsigned int ADC_Read(volatile unsigned char channel)
0xB0: MOVFF FSR2, POSTINC1
0xB2: NOP
0xB4: MOVFF FSR1, FSR2
0xB6: NOP
0xB8: SUBFSR 2, 0x2
!{
!    ADCON0 &= 0xC3;                     //11000011    Clearing the Channel Selection Bits
0xBA: MOVLW 0xC3
0xBC: ANDWF ADCON0, F, ACCESS
!    ADCON0 |= channel<<2;               //Setting the required Bits
0xBE: MOVF [0x0], W, ACCESS
0xC0: MULLW 0x4
0xC2: MOVF PROD, W, ACCESS
0xC4: IORWF ADCON0, F, ACCESS
!    ADCON0bits.GO_DONE = 1;
0xC6: BSF ADCON0, 1, ACCESS
!    while(ADCON0bits.GO_DONE);
0xC8: BTFSC ADCON0, 1, ACCESS
0xCA: BRA 0xC8
!    return ((ADRESH * 256) + ADRESL);   //Returns Result
0xCC: MOVF ADRESH, W, ACCESS
0xCE: MOVLB 0x0
0xD0: MOVWF 0xC, BANKED
0xD2: CLRF 0xD, BANKED
0xD4: MOVFF 0xC, 0xD
0xD6: NOP
0xD8: MOVLB 0x0
0xDA: CLRF 0xC, BANKED
0xDC: MOVF ADRES, W, ACCESS
0xDE: MOVLB 0x0
0xE0: ADDWF 0xC, W, BANKED
0xE2: MOVLB 0x0
0xE4: MOVWF __tmp_0, BANKED
0xE6: MOVLW 0x0
0xE8: MOVLB 0x0
0xEA: ADDWFC 0xD, W, BANKED
0xEC: MOVLB 0x0
0xEE: MOVWF 0xB, BANKED
0xF0: MOVFF __tmp_0, PROD
0xF2: NOP
0xF4: MOVFF 0xB, PRODH
0xF6: NOP
0xF8: BRA 0xFA
!}
0xFA: SUBFSR 1, 0x1
0xFC: MOVFF INDF1, FSR2
0xFE: NOP
0x100: RETURN 0
</code>```

Configuration Bits source code

```<code>// PIC18F47J53 Configuration Bit Settings

// 'C' source line config statements

#include <p18F47J53.h>

// CONFIG1L
#pragma config WDTEN = ON       // Watchdog Timer (Enabled)
#pragma config PLLDIV = 1       // PLL Prescaler Selection (No prescale (4 MHz oscillator input drives PLL directly))
#pragma config CFGPLLEN = OFF   // PLL Enable Configuration Bit (PLL Disabled)
#pragma config STVREN = ON      // Stack Overflow/Underflow Reset (Enabled)
#pragma config XINST = ON       // Extended Instruction Set (Enabled)

// CONFIG1H
#pragma config CPUDIV = OSC1    // CPU System Clock Postscaler (No CPU system clock divide)
#pragma config CP0 = OFF        // Code Protect (Program memory is not code-protected)

// CONFIG2L
#pragma config OSC = INTOSC     // Oscillator (INTOSC)
#pragma config SOSCSEL = HIGH   // T1OSC/SOSC Power Selection Bits (High Power T1OSC/SOSC circuit selected)
#pragma config CLKOEC = ON      // EC Clock Out Enable Bit  (CLKO output enabled on the RA6 pin)
#pragma config FCMEN = ON       // Fail-Safe Clock Monitor (Enabled)
#pragma config IESO = ON        // Internal External Oscillator Switch Over Mode (Enabled)

// CONFIG2H
#pragma config WDTPS = 32768    // Watchdog Postscaler (1:32768)

// CONFIG3L
#pragma config DSWDTOSC = INTOSCREF// DSWDT Clock Select (DSWDT uses INTRC)
#pragma config RTCOSC = T1OSCREF// RTCC Clock Select (RTCC uses T1OSC/T1CKI)
#pragma config DSBOREN = ON     // Deep Sleep BOR (Enabled)
#pragma config DSWDTEN = ON     // Deep Sleep Watchdog Timer (Enabled)
#pragma config DSWDTPS = G2     // Deep Sleep Watchdog Postscaler (1:2,147,483,648 (25.7 days))

// CONFIG3H
#pragma config IOL1WAY = ON     // IOLOCK One-Way Set Enable bit (The IOLOCK bit (PPSCON<0>) can be set once)
#pragma config ADCSEL = BIT12   // ADC 10 or 12 Bit Select (12 - Bit ADC Enabled)
#pragma config MSSP7B_EN = MSK7 // MSSP address masking (7 Bit address masking mode)

// CONFIG4L
#pragma config WPFP = PAGE_127  // Write/Erase Protect Page Start/End Location (Write Protect Program Flash Page 127)
#pragma config WPCFG = OFF      // Write/Erase Protect Configuration Region  (Configuration Words page not erase/write-protected)

// CONFIG4H
#pragma config WPDIS = OFF      // Write Protect Disable bit (WPFP<6:0>/WPEND region ignored)
#pragma config WPEND = PAGE_WPFP// Write/Erase Protect Region Select bit (valid when WPDIS = 0) (Pages WPFP<6:0> through Configuration Words erase/write protected)
#pragma config LS48MHZ = SYS48X8// Low Speed USB mode with 48 MHz system clock bit (System clock at 48 MHz USB CLKEN divide-by is set to 8)
</code>```

MPLAB variables view

ADCON0 and channel unexpected values

答: It was a stack error. However, it is working fine now after I enabled the “multi-bank stack model”. Thanks guys</p>

Body:
I am implementing the ADC for PIC18F47J53 using MPLAB v3.51.

Why is the value of the variable CH in the main function not passed correctly to the function ADC_Read with the argument channel?

unsigned int ADC_Read(unsigned char channel);
void ADC_Initialize();

#pragma config ADCSEL = BIT12                  //12 bit conversion mode enabled

#define ANCON_AiBi4         ANCON0bits.PCFG1   // Analog Input channels

int main(int argc, char** argv)
{
    unsigned int j=0;
    unsigned char CH;

    ADC_Initialize();

    while (1)
    {
        CH = 1;
        j = ADC_Read(CH); //store the result of adc in j.
    }
    return (0);
}

void ADC_Initialize()
{
     ADCON0 = 0x01;   //00000001      ADC ON, AVss, AVdd
     ADCON1 = 0xBD;   //10111101      R justified, Normal op, 20TAD, FOSC/16
}

/* Select the analog channel */
unsigned int ADC_Read(unsigned char channel)
{
    ADCON0 &= 0xC3;                     //11000011    Clearing the Channel Selection Bits
    ADCON0 |= channel<<2;               //Setting the required Bits
    ADCON0bits.GO_DONE = 1;
    while(ADCON0bits.GO_DONE);
    return ((ADRESH * 256) + ADRESL);   //Returns Result
}

Generated assembly code

!unsigned int ADC_Read(volatile unsigned char channel)
0xB0: MOVFF FSR2, POSTINC1
0xB2: NOP
0xB4: MOVFF FSR1, FSR2
0xB6: NOP
0xB8: SUBFSR 2, 0x2
!{
!    ADCON0 &= 0xC3;                     //11000011    Clearing the Channel Selection Bits
0xBA: MOVLW 0xC3
0xBC: ANDWF ADCON0, F, ACCESS
!    ADCON0 |= channel<<2;               //Setting the required Bits
0xBE: MOVF [0x0], W, ACCESS
0xC0: MULLW 0x4
0xC2: MOVF PROD, W, ACCESS
0xC4: IORWF ADCON0, F, ACCESS
!    ADCON0bits.GO_DONE = 1;
0xC6: BSF ADCON0, 1, ACCESS
!    while(ADCON0bits.GO_DONE);
0xC8: BTFSC ADCON0, 1, ACCESS
0xCA: BRA 0xC8
!    return ((ADRESH * 256) + ADRESL);   //Returns Result
0xCC: MOVF ADRESH, W, ACCESS
0xCE: MOVLB 0x0
0xD0: MOVWF 0xC, BANKED
0xD2: CLRF 0xD, BANKED
0xD4: MOVFF 0xC, 0xD
0xD6: NOP
0xD8: MOVLB 0x0
0xDA: CLRF 0xC, BANKED
0xDC: MOVF ADRES, W, ACCESS
0xDE: MOVLB 0x0
0xE0: ADDWF 0xC, W, BANKED
0xE2: MOVLB 0x0
0xE4: MOVWF __tmp_0, BANKED
0xE6: MOVLW 0x0
0xE8: MOVLB 0x0
0xEA: ADDWFC 0xD, W, BANKED
0xEC: MOVLB 0x0
0xEE: MOVWF 0xB, BANKED
0xF0: MOVFF __tmp_0, PROD
0xF2: NOP
0xF4: MOVFF 0xB, PRODH
0xF6: NOP
0xF8: BRA 0xFA
!}
0xFA: SUBFSR 1, 0x1
0xFC: MOVFF INDF1, FSR2
0xFE: NOP
0x100: RETURN 0

Configuration Bits source code

// PIC18F47J53 Configuration Bit Settings

// 'C' source line config statements

#include <p18F47J53.h>

// CONFIG1L
#pragma config WDTEN = ON       // Watchdog Timer (Enabled)
#pragma config PLLDIV = 1       // PLL Prescaler Selection (No prescale (4 MHz oscillator input drives PLL directly))
#pragma config CFGPLLEN = OFF   // PLL Enable Configuration Bit (PLL Disabled)
#pragma config STVREN = ON      // Stack Overflow/Underflow Reset (Enabled)
#pragma config XINST = ON       // Extended Instruction Set (Enabled)

// CONFIG1H
#pragma config CPUDIV = OSC1    // CPU System Clock Postscaler (No CPU system clock divide)
#pragma config CP0 = OFF        // Code Protect (Program memory is not code-protected)

// CONFIG2L
#pragma config OSC = INTOSC     // Oscillator (INTOSC)
#pragma config SOSCSEL = HIGH   // T1OSC/SOSC Power Selection Bits (High Power T1OSC/SOSC circuit selected)
#pragma config CLKOEC = ON      // EC Clock Out Enable Bit  (CLKO output enabled on the RA6 pin)
#pragma config FCMEN = ON       // Fail-Safe Clock Monitor (Enabled)
#pragma config IESO = ON        // Internal External Oscillator Switch Over Mode (Enabled)

// CONFIG2H
#pragma config WDTPS = 32768    // Watchdog Postscaler (1:32768)

// CONFIG3L
#pragma config DSWDTOSC = INTOSCREF// DSWDT Clock Select (DSWDT uses INTRC)
#pragma config RTCOSC = T1OSCREF// RTCC Clock Select (RTCC uses T1OSC/T1CKI)
#pragma config DSBOREN = ON     // Deep Sleep BOR (Enabled)
#pragma config DSWDTEN = ON     // Deep Sleep Watchdog Timer (Enabled)
#pragma config DSWDTPS = G2     // Deep Sleep Watchdog Postscaler (1:2,147,483,648 (25.7 days))

// CONFIG3H
#pragma config IOL1WAY = ON     // IOLOCK One-Way Set Enable bit (The IOLOCK bit (PPSCON<0>) can be set once)
#pragma config ADCSEL = BIT12   // ADC 10 or 12 Bit Select (12 - Bit ADC Enabled)
#pragma config MSSP7B_EN = MSK7 // MSSP address masking (7 Bit address masking mode)

// CONFIG4L
#pragma config WPFP = PAGE_127  // Write/Erase Protect Page Start/End Location (Write Protect Program Flash Page 127)
#pragma config WPCFG = OFF      // Write/Erase Protect Configuration Region  (Configuration Words page not erase/write-protected)

// CONFIG4H
#pragma config WPDIS = OFF      // Write Protect Disable bit (WPFP<6:0>/WPEND region ignored)
#pragma config WPEND = PAGE_WPFP// Write/Erase Protect Region Select bit (valid when WPDIS = 0) (Pages WPFP<6:0> through Configuration Words erase/write protected)
#pragma config LS48MHZ = SYS48X8// Low Speed USB mode with 48 MHz system clock bit (System clock at 48 MHz USB CLKEN divide-by is set to 8)

MPLAB variables view

ADCON0 and channel unexpected values


问:
Streaming Video Recommendations
I'm trying to find a viable option for streaming flash video for most browsers.

The setup we have no is a Server 2008/IIS server using ASP. Our site gets somewhere in the neighborhood of 1,200 hits a day, and the videos wouldn't likely be watched more than a hundred or so times a day, at most.

Still, that said we have about 20gb/~100 videos from various speaking engagements that are about an hour long or so, and our main goal is to make them streamed, so that users don't need to wait for progressive downloads to jump around in the video.

I've read a bit about http streaming, and if that's doable I'd like to hear a reliable method to use; otherwise what sort of server software could we purchase to use, preferably on the same server to stream our videos to users?

答:
Here is a short article on http streaming.

I have actually found the Adobe Flash Media Server a good and reasonably affordable solution.

You might also want to check out some third company streaming hosts - depending on the size of your company and the actual traffic you expect, the cost/benefit ratio of outsourcing can be better than when you're setting up and maintaining a server in-house.

ContentLicense: CC BY-SA 2.5

Title: Streaming Video Recommendations


问:
Randomize scan period for eventual beacon detection
I am trying to understand the statement "We recommend not reducing the scan period to be less than 1.1 seconds, since many beacons only transmit at a frequency of 1 Hz" used in android beacon library documentation.

Is it because we might miss the beacon packet as the scan time is too less?
My use case requires to keep the battery consumption in check. I wanted to ask if this particular scenario will work.

I have set background scan time of 20ms and background between scan time of 980ms.

This means that there is a scan every second. This configuration can increase responsiveness of the application. Also, as the scan period is low, it should save battery too.

1) Will this configuration be able to detect beacon?
2) If it misses due to low scan period, is it still possible to detect beacons in the next scan?
3) If the frequency of of transmission is 1Hz and if it misses onces, will it miss everytime as the total of both the background scan time and between scan time is 1 second?
4) If yes, will randomizing scan periods within a range help?

Thanks

答:
Consider the following factors:

If a beacon packet is sent over the air when a scan is active, you have a 90% chance of detecting it at close range.

If the scan is started or stopped in the middle of the transmission, the packet will not be detected.

Starting scanning is not instantaneous.  It takes time.  From the time you command scanning to start, you will miss packets until scanning is fully started.  This time this takes is device dependent, but it is probably in the 10s of milliseconds.

If you set a scan period shorter than the beacon transmission rate, the odds of detecting it are reduced approximately by the ratio of the scan period divided by the transmission period.

BLE advertisement packets are sent out irregularly.

All of the above factors mean that short scan intervals are bad for reliable detections.   If you want to save battery, you are better off leaving the scanner on for 1.1 seconds, then leaving it off for as long as you need to reach your battery targets.  Trying to make average detection times faster by reducing both the scan period and the between scan period will only make the average detection time slower, because you are losing useful scan time by the time wasted cycling on and off based on points 2 and 3.


问:
How can I merge 2 spotfire tables by a regex match?
I am working on a spotfire tool, and I am using a calculated column in my main data table to group data rows into 'families' through a regex match. For example, one row might have a 'name' of ABC1234xyz, so it would be part of the ABC family because it contains the string 'ABC'. Another rows could be something like AQRST31x2af, and belong to the QRST family. The main point is that the 'family' is decided by matching a substring in the the name, but that substring could be any length, and isn't necessarily the beginning of the name string.

Right now I am doing this by a large nested If statement with a calculated column. However, this is tedious for adding new families, and maintaining the current list of families. What I would like to do is create a table with 2 columns, the string match and the family name. Then, I would like to match from this table to determine family instead of the nested if. So, it might look like the below tables:

```<code>Match Table:

id_string | family
----------------------
ABC       | ABC
QRST      | QRST
SUP       | Super

Main Data Table:

name        | data        | family
---------------------------------------
ABC1234     | 1.02342     | ABC
ABC1215     | 1.23749     | ABC
AQRST31x2af | 1.04231     | QRST
BQRST32x2ac | 1.12312     | QRST
1903xSUP    | 1.51231     | Super
1204xSUP    | 1.68123     | Super
</code>```

If you have any suggestions, I would appreciate it.

Thanks.

答:
As @ksp585 mentioned, it doesn't seem like Spotfire can do exactly what I want, so I have come up with a solution using IronPython. Essentially, here is what I have done:

Created a table called FAMILIES, with the columns IDString and Family, which looks like this (using the same example strings above):

```<code>IDString  | Family
------------------------
ABC       | ABC
SUP       | Super
QRST      | QRST
</code>```

Created a table called NAMES, as a pivot off of my main data table, with the only column being NAME. This just creates a list of unique names (since the data table has many rows for each name):

```<code>NAME
------------------------
ABC1234
ABC1215
AQRST31x2af
BQRST32x2ac
...
</code>```

Created a Text Area with a button labeled Match Families, which calls an IronPython script. That script reads the NAMES table, and the FAMILIES table, compares each name to the IDString column with a regex, and associates each name with a family from the results. Any names that don't match a single IDString get the family name 'Other'. Then, it generates a new table called NAME_FAMILY_MAP, with the columns NAME and FAMILY.

With this new table, I can then add a column back to the original data table using a left outer join from NAME_FAMILY_MAP, matching on NAME. Because NAME_FAMILY_MAP is not directly linked to the NAMES table (generated by a button press), it does not create a circular dependency.

I can then add families to the FAMILIES table using another script, or by just replacing the FAMILIES table with an updated list. It's slightly more tedious than what I was hoping, but it works, so I'm happy.

Body:
I am working on a spotfire tool, and I am using a calculated column in my main data table to group data rows into 'families' through a regex match. For example, one row might have a 'name' of ABC1234xyz, so it would be part of the ABC family because it contains the string 'ABC'. Another rows could be something like AQRST31x2af, and belong to the QRST family. The main point is that the 'family' is decided by matching a substring in the the name, but that substring could be any length, and isn't necessarily the beginning of the name string.

Right now I am doing this by a large nested If statement with a calculated column. However, this is tedious for adding new families, and maintaining the current list of families. What I would like to do is create a table with 2 columns, the string match and the family name. Then, I would like to match from this table to determine family instead of the nested if. So, it might look like the below tables:

Match Table:

id_string | family
----------------------
ABC       | ABC
QRST      | QRST
SUP       | Super

Main Data Table:

name        | data        | family
---------------------------------------
ABC1234     | 1.02342     | ABC
ABC1215     | 1.23749     | ABC
AQRST31x2af | 1.04231     | QRST
BQRST32x2ac | 1.12312     | QRST
1903xSUP    | 1.51231     | Super
1204xSUP    | 1.68123     | Super

If you have any suggestions, I would appreciate it.

Thanks.

Body:
As @ksp585 mentioned, it doesn't seem like Spotfire can do exactly what I want, so I have come up with a solution using IronPython. Essentially, here is what I have done:

Created a table called FAMILIES, with the columns IDString and Family, which looks like this (using the same example strings above):

IDString  | Family
------------------------
ABC       | ABC
SUP       | Super
QRST      | QRST

Created a table called NAMES, as a pivot off of my main data table, with the only column being NAME. This just creates a list of unique names (since the data table has many rows for each name):

NAME
------------------------
ABC1234
ABC1215
AQRST31x2af
BQRST32x2ac
...

Created a Text Area with a button labeled Match Families, which calls an IronPython script. That script reads the NAMES table, and the FAMILIES table, compares each name to the IDString column with a regex, and associates each name with a family from the results. Any names that don't match a single IDString get the family name 'Other'. Then, it generates a new table called NAME_FAMILY_MAP, with the columns NAME and FAMILY.

With this new table, I can then add a column back to the original data table using a left outer join from NAME_FAMILY_MAP, matching on NAME. Because NAME_FAMILY_MAP is not directly linked to the NAMES table (generated by a button press), it does not create a circular dependency.

I can then add families to the FAMILIES table using another script, or by just replacing the FAMILIES table with an updated list. It's slightly more tedious than what I was hoping, but it works, so I'm happy.


问:
create n item long playlist-like list from multiple other lists
I'm trying to create a list with values out of three other lists (high, medium and low). The new list should evaluate from the probability_array from which list it should pick a random value. The new list should be playlist-like: before a randomly chosen value will be picked twice (from high, medium or low), all other values of the specific list (high, medium or low) must be in the newly generated list.

Any ideas, how I can achieve this. My code so far:

```<code>import numpy as np

array_length = 30

high = [(1, 2), 3, 4, 5, 6, (7, 8, 9)]
medium = [10, 11, (12, 13), 14]
low = [100, 101, 102, (103, 104)]

probability_array = np.random.choice(
    ['High', 'Medium', 'Low',],
    array_length,
    p=[4/7, 2/7, 1/7]
)
# i. e.
"""
['Low' 'High' 'High' 'High' 'High' 'High' 'Medium' 'Medium' 'High' 'Medium'
 'High' 'High' 'Medium' 'Medium' 'High' 'High' 'Medium' 'Medium' 'High'
 'High' 'High' 'High' 'High' 'Medium' 'Medium' 'High' 'High' 'High' 'Low'
 'High']
"""

# new list should look like:
"""
[102, (1, 2), 4, (7, 8, 9), 3, 6, 14, ...]
"""
</code>```

Thanks

答:
Try this:</p>

```<code>def make_list(high, medium, low, parr):
    _, (nhigh, nlow, nmed) = np.unique(parr, return_counts=True)
    # A check can be added to make sure labels are in the set ['High', 'Low', 'Medium']
    dl = {}
    dl['High'] = (nhigh // len(high) + 1) * np.random.permutation(np.asanyarray(high, dtype=np.object)).tolist()
    dl['Medium'] = (nmed // len(medium) + 1) * np.random.permutation(np.asanyarray(medium, dtype=np.object)).tolist()
    dl['Low'] = (nlow // len(low) + 1) * np.random.permutation(np.asanyarray(low, dtype=np.object)).tolist()
    play_list = []
    for p in parr:
        play_list.append(dl[p].pop())
    return play_list
</code>```

Body:
I'm trying to create a list with values out of three other lists (high, medium and low). The new list should evaluate from the probability_array from which list it should pick a random value. The new list should be playlist-like: before a randomly chosen value will be picked twice (from high, medium or low), all other values of the specific list (high, medium or low) must be in the newly generated list.

Any ideas, how I can achieve this. My code so far:

import numpy as np

array_length = 30

high = [(1, 2), 3, 4, 5, 6, (7, 8, 9)]
medium = [10, 11, (12, 13), 14]
low = [100, 101, 102, (103, 104)]

probability_array = np.random.choice(
    ['High', 'Medium', 'Low',],
    array_length,
    p=[4/7, 2/7, 1/7]
)
# i. e.
"""
['Low' 'High' 'High' 'High' 'High' 'High' 'Medium' 'Medium' 'High' 'Medium'
 'High' 'High' 'Medium' 'Medium' 'High' 'High' 'Medium' 'Medium' 'High'
 'High' 'High' 'High' 'High' 'Medium' 'Medium' 'High' 'High' 'High' 'Low'
 'High']
"""

# new list should look like:
"""
[102, (1, 2), 4, (7, 8, 9), 3, 6, 14, ...]
"""

Thanks

Body:
Try this:

def make_list(high, medium, low, parr):
    _, (nhigh, nlow, nmed) = np.unique(parr, return_counts=True)
    # A check can be added to make sure labels are in the set ['High', 'Low', 'Medium']
    dl = {}
    dl['High'] = (nhigh // len(high) + 1) * np.random.permutation(np.asanyarray(high, dtype=np.object)).tolist()
    dl['Medium'] = (nmed // len(medium) + 1) * np.random.permutation(np.asanyarray(medium, dtype=np.object)).tolist()
    dl['Low'] = (nlow // len(low) + 1) * np.random.permutation(np.asanyarray(low, dtype=np.object)).tolist()
    play_list = []
    for p in parr:
        play_list.append(dl[p].pop())
    return play_list


问:
How to interface a python program with a third party program over a known protocol?
I'm trying to interface with a program (GoGui - https://sourceforge.net/projects/gogui/) using the Go Text Protocol (GTP - http://www.lysator.liu.se/~gunnar/gtp/) which has its documentation here. (Link can also be found on the previous website.)

So I've written a bit of code to at least get GoGui to acknowledge the existence of my program:

```<code>import sys
import engine
import input_processing

for line in sys.stdin:
    if line == 'name\n':
        sys.stdout.write(' Muizz-Bot\n\n')
    if line == 'protocol_version\n':
        sys.stdout.write('2\n\n')
    if line == 'version\n':
        sys.stdout.write('')
</code>```

Now this doesn't seem unreasonable per se, but results in GoGui giving me the following error:

Which is of course a problem. So I figured that I made a mistake somewhere in my programming, but when I simply run the program through visual studio, everything works as expected:

This makes me think that the problem lies in interfacing the two applications, and maybe I should be looking at other functions than stdin and stdout. Does anyone know what may be going wrong here?

EDIT FOR COMMENTS: The code I'm currently working on for command parsing (in its entirety) looks like this:

```<code>import sys

commands = ['protocol_version', 'name', 'version', 'list_commands', 'known_command', 'quit', 'boardsize',
            'clear_board', 'komi', 'play', 'genmove']

pre_game_out = ['2','Muizz-Bot','']

# Define all output functions
def list_commands():
    out = '\n'.join(commands)
    return(out)
def known():
    return(True)
def quit():
    return(None)
def boardsize():
    return(None)
def clear_board():
    return(None)
def komi():
    return(None)
def play():
    return(None)
def genmove():
    return("A1")

# Create dictionary to point to all functions.
output = {'list_commands':list_commands, 'known_command':known, 'quit':quit, 'boardsize':boardsize,
            'clear_board':clear_board, 'komi':komi, 'play':play, 'genmove':genmove}

# Define the function that will pass the commands and write outputs.
def parse(line):
    if line.strip() in commands:
        i = commands.index(line.strip())
        if i<3:
            sys.stdout.write('= '+ pre_game_out[i]+'\n\n')
            sys.stdout.flush()
        else:
            sys.stdout.write('= ' + output[line.strip()]() + '\n\n')
            sys.stdout.flush()
</code>```

For pre processing:

```<code>def input(inp):
    # Remove control characters
    inp = inp.replace('\r', '')
    inp = inp.replace(' ', '')
    inp = inp.split('#', 1)[0]
    inp = inp.replace('\t', ' ')

    # Check if empty
    if inp.isspace() or inp==None or inp=='':
        return
    else:
        return(inp)
</code>```

答:
You're not flushing your response so nothing gets sent back to the caller (as the command is not big enough to trigger auto buffer flush). Also, strolling through the protocol document it clearly says that your response should be in the form of = response\n\n so even if you were flushing it probably still wouldn't work.

Try with something like:

```<code>import sys

for line in sys.stdin:
    if line.strip() == 'name':
        sys.stdout.write('= Muizz-Bot\n\n')
        sys.stdout.flush()
    elif line.strip() == 'protocol_version':
        sys.stdout.write('= 2\n\n')
        sys.stdout.flush()
    elif line.strip() == 'version':
        sys.stdout.write('=\n\n')
        sys.stdout.flush()
</code>```

You might want to create a simple function for parsing commands / responding back instead of repeating the code, tho. Also, this probably won't (fully) work either as the protocol document states that you need to implement quite a number of commands (6.1 Required Commands) but it should get you started.

UPDATE - Here's one way to make it more manageable and in line with the specs - you can create a function for each command so you can easily add/remove them as you please, for example:

```<code>def cmd_name(*args):
    return "Muizz-Bot"

def cmd_protocol_version(*args):
    return 2

def cmd_version(*args):
    return ""

def cmd_list_commands(*args):
    return " ".join(x[4:] for x in globals() if x[:4] == "cmd_")

def cmd_known_command(*args):
    commands = {x[4:] for x in globals() if x[:4] == "cmd_"}
    return "true" if args and args[0] in commands else "false"

# etc.
</code>```

Here all the command functions are prefixed with "cmd_" (and cmd_list_commands() and cmd_known_command() use that fact to check for the command functions in the global namespace) but you can also move them to a different module and then 'scan' the module instead. With such structure it's very easy to add a new command, for example to add the required quit command all you need is to define it:

```<code>def cmd_quit(*args):
    raise EOFError()  # we'll use EOFError to denote an exit state bellow
</code>```

Also, we'll deal bellow with the situation when a command needs to return an error - all you need to do from your functions is to raise ValueError("error response") and it will be sent back as an error.

Once you have your set of commands added as functions all you need is to parse the input command, call the right function with the right arguments and print back the response:

```<code>def call_command(command):
    command = "".join(x for x in command if 31 < ord(x) < 127 or x == "\t")  # 3.1.1
    command = command.strip()  # 3.1.4
    if not command:  # ... return if there's nothing to do
        return
    command = command.split()  # split to get the [id], cmd, [arg1, arg2, ...] structure
    try:  # try to convert to int the first slice to check for command ID
        command_id = int(command[0])
        command_args = command[2:] if len(command) > 2 else []  # args or an empty list
        command = command[1]  # command name
    except ValueError:  # failed, no command ID present
        command_id = ""  # set it to blank
        command_args = command[1:] if len(command) > 1 else []  # args or an empty list
        command = command[0]  # command name
    # now, lets try to call our command as cmd_<command name> function and get its response
    try:
        response = globals()["cmd_" + command](*command_args)
        if response != "":  # response not empty, prepend it with space as per 3.4
            response = " {}".format(response)
        sys.stdout.write("={}{}\n\n".format(command_id, response))
    except KeyError:  # unknown command, return standard error as per 3.6
        sys.stdout.write("?{} unknown command\n\n".format(command_id))
    except ValueError as e:  # the called function raised a ValueError
        sys.stdout.write("?{} {}\n\n".format(command_id, e))
    except EOFError:  # a special case when we need to quit
        sys.stdout.write("={}\n\n".format(command_id))
        sys.stdout.flush()
        sys.exit(0)
    sys.stdout.flush()  # flush the STDOUT
</code>```

Finally, all you need is to listen to your STDIN and forward the command lines to this function to do the heavy lifting. In that regard, I'd actually explicitly read line-by-line from your STDIN rather than trying to iterate over it as it's a safer approach so:

```<code>if __name__ == "__main__":  # make sure we're executing instead of importing this script
    while True:  # main loop
        try:
            line = sys.stdin.readline()  # read a line from STDIN
            if not line:  # reached the end of STDIN
                break  # exit the main loop
            call_command(line)  # call our command
        except Exception:  # too broad, but we don't care at this point as we're exiting
            break  # exit the main loop
</code>```

Of course, as I mentioned earlier, it might be a better idea to pack your commands in a separate module, but this should at least give you an idea how to do 'separation of concerns' so you worry about responding to your commands rather than on how they get called and how they respond back to the caller.

Body:
I'm trying to interface with a program (GoGui - https://sourceforge.net/projects/gogui/) using the Go Text Protocol (GTP - http://www.lysator.liu.se/~gunnar/gtp/) which has its documentation here. (Link can also be found on the previous website.)

So I've written a bit of code to at least get GoGui to acknowledge the existence of my program:

import sys
import engine
import input_processing

for line in sys.stdin:
    if line == 'name\n':
        sys.stdout.write(' Muizz-Bot\n\n')
    if line == 'protocol_version\n':
        sys.stdout.write('2\n\n')
    if line == 'version\n':
        sys.stdout.write('')

Now this doesn't seem unreasonable per se, but results in GoGui giving me the following error:

Which is of course a problem. So I figured that I made a mistake somewhere in my programming, but when I simply run the program through visual studio, everything works as expected:

This makes me think that the problem lies in interfacing the two applications, and maybe I should be looking at other functions than stdin and stdout. Does anyone know what may be going wrong here?

EDIT FOR COMMENTS: The code I'm currently working on for command parsing (in its entirety) looks like this:

import sys

commands = ['protocol_version', 'name', 'version', 'list_commands', 'known_command', 'quit', 'boardsize',
            'clear_board', 'komi', 'play', 'genmove']

pre_game_out = ['2','Muizz-Bot','']

# Define all output functions
def list_commands():
    out = '\n'.join(commands)
    return(out)
def known():
    return(True)
def quit():
    return(None)
def boardsize():
    return(None)
def clear_board():
    return(None)
def komi():
    return(None)
def play():
    return(None)
def genmove():
    return("A1")

# Create dictionary to point to all functions.
output = {'list_commands':list_commands, 'known_command':known, 'quit':quit, 'boardsize':boardsize,
            'clear_board':clear_board, 'komi':komi, 'play':play, 'genmove':genmove}

# Define the function that will pass the commands and write outputs.
def parse(line):
    if line.strip() in commands:
        i = commands.index(line.strip())
        if i<3:
            sys.stdout.write('= '+ pre_game_out[i]+'\n\n')
            sys.stdout.flush()
        else:
            sys.stdout.write('= ' + output[line.strip()]() + '\n\n')
            sys.stdout.flush()

For pre processing:

def input(inp):
    # Remove control characters
    inp = inp.replace('\r', '')
    inp = inp.replace(' ', '')
    inp = inp.split('#', 1)[0]
    inp = inp.replace('\t', ' ')

    # Check if empty
    if inp.isspace() or inp==None or inp=='':
        return
    else:
        return(inp)

Body:
You're not flushing your response so nothing gets sent back to the caller (as the command is not big enough to trigger auto buffer flush). Also, strolling through the protocol document it clearly says that your response should be in the form of = response\n\n so even if you were flushing it probably still wouldn't work.

Try with something like:

import sys

for line in sys.stdin:
    if line.strip() == 'name':
        sys.stdout.write('= Muizz-Bot\n\n')
        sys.stdout.flush()
    elif line.strip() == 'protocol_version':
        sys.stdout.write('= 2\n\n')
        sys.stdout.flush()
    elif line.strip() == 'version':
        sys.stdout.write('=\n\n')
        sys.stdout.flush()

You might want to create a simple function for parsing commands / responding back instead of repeating the code, tho. Also, this probably won't (fully) work either as the protocol document states that you need to implement quite a number of commands (6.1 Required Commands) but it should get you started.

UPDATE - Here's one way to make it more manageable and in line with the specs - you can create a function for each command so you can easily add/remove them as you please, for example:

def cmd_name(*args):
    return "Muizz-Bot"

def cmd_protocol_version(*args):
    return 2

def cmd_version(*args):
    return ""

def cmd_list_commands(*args):
    return " ".join(x[4:] for x in globals() if x[:4] == "cmd_")

def cmd_known_command(*args):
    commands = {x[4:] for x in globals() if x[:4] == "cmd_"}
    return "true" if args and args[0] in commands else "false"

# etc.

Here all the command functions are prefixed with "cmd_" (and cmd_list_commands() and cmd_known_command() use that fact to check for the command functions in the global namespace) but you can also move them to a different module and then 'scan' the module instead. With such structure it's very easy to add a new command, for example to add the required quit command all you need is to define it:

def cmd_quit(*args):
    raise EOFError()  # we'll use EOFError to denote an exit state bellow

Also, we'll deal bellow with the situation when a command needs to return an error - all you need to do from your functions is to raise ValueError("error response") and it will be sent back as an error.

Once you have your set of commands added as functions all you need is to parse the input command, call the right function with the right arguments and print back the response:

def call_command(command):
    command = "".join(x for x in command if 31 < ord(x) < 127 or x == "\t")  # 3.1.1
    command = command.strip()  # 3.1.4
    if not command:  # ... return if there's nothing to do
        return
    command = command.split()  # split to get the [id], cmd, [arg1, arg2, ...] structure
    try:  # try to convert to int the first slice to check for command ID
        command_id = int(command[0])
        command_args = command[2:] if len(command) > 2 else []  # args or an empty list
        command = command[1]  # command name
    except ValueError:  # failed, no command ID present
        command_id = ""  # set it to blank
        command_args = command[1:] if len(command) > 1 else []  # args or an empty list
        command = command[0]  # command name
    # now, lets try to call our command as cmd_<command name> function and get its response
    try:
        response = globals()["cmd_" + command](*command_args)
        if response != "":  # response not empty, prepend it with space as per 3.4
            response = " {}".format(response)
        sys.stdout.write("={}{}\n\n".format(command_id, response))
    except KeyError:  # unknown command, return standard error as per 3.6
        sys.stdout.write("?{} unknown command\n\n".format(command_id))
    except ValueError as e:  # the called function raised a ValueError
        sys.stdout.write("?{} {}\n\n".format(command_id, e))
    except EOFError:  # a special case when we need to quit
        sys.stdout.write("={}\n\n".format(command_id))
        sys.stdout.flush()
        sys.exit(0)
    sys.stdout.flush()  # flush the STDOUT

Finally, all you need is to listen to your STDIN and forward the command lines to this function to do the heavy lifting. In that regard, I'd actually explicitly read line-by-line from your STDIN rather than trying to iterate over it as it's a safer approach so:

if __name__ == "__main__":  # make sure we're executing instead of importing this script
    while True:  # main loop
        try:
            line = sys.stdin.readline()  # read a line from STDIN
            if not line:  # reached the end of STDIN
                break  # exit the main loop
            call_command(line)  # call our command
        except Exception:  # too broad, but we don't care at this point as we're exiting
            break  # exit the main loop

Of course, as I mentioned earlier, it might be a better idea to pack your commands in a separate module, but this should at least give you an idea how to do 'separation of concerns' so you worry about responding to your commands rather than on how they get called and how they respond back to the caller.


问:
Spark CSV : insert data into an existing table without changing a specific col
I want to update my database from a csv file publicated every week.
So far I've managed to insert a csv file into my database.

My problems, I will have two more colums that I don't want to touch. One will come from a different CSV File and the other from Solr_query.

So how can I ignore these two columns ?

Example(CSV File):

```<code>FirstName;Lastname;City
Peter;Johns;New York
</code>```

Example(Table1 the one I have):

```<code>FirsName-Lastname-City
</code>```

Example(Table2 The one I want)

```<code>FisrtName-Lastname-City-Age-SolrQuery
</code>```

My code:

```<code>DataFrame DF = sqlContext
                        .read()
                        .format("csv")
                        .option("delimiter", ";")
                        .option("header", "true")
                        //mv .option("inferschema", "true")
                        .load("file:///home/2.csv");

query = "select firstname, lastname, city from mytable"

DataFrame letsgo = sqlContext.sql(query);

letsgo.write().mode(SaveMode.Append).saveAsTable("mytable");
</code>```

It works fine when I have 3 columns but not I my table has 5 columns
What can I do?

答:
Additional columns with null value can be included in query:</p>

```<code>query = "select firstname, lastname, city, null as Age, null as SolrQuery from mytable"
</code>```

Body:
I want to update my database from a csv file publicated every week.
So far I've managed to insert a csv file into my database.

My problems, I will have two more colums that I don't want to touch. One will come from a different CSV File and the other from Solr_query.

So how can I ignore these two columns ?

Example(CSV File):

FirstName;Lastname;City
Peter;Johns;New York

Example(Table1 the one I have):

FirsName-Lastname-City

Example(Table2 The one I want)

FisrtName-Lastname-City-Age-SolrQuery

My code:

DataFrame DF = sqlContext
                        .read()
                        .format("csv")
                        .option("delimiter", ";")
                        .option("header", "true")
                        //mv .option("inferschema", "true")
                        .load("file:///home/2.csv");

query = "select firstname, lastname, city from mytable"

DataFrame letsgo = sqlContext.sql(query);

letsgo.write().mode(SaveMode.Append).saveAsTable("mytable");

It works fine when I have 3 columns but not I my table has 5 columns
What can I do?

Body:
Additional columns with null value can be included in query:

query = "select firstname, lastname, city, null as Age, null as SolrQuery from mytable"


问:
C beginner's problems with multiple boolean operators
I'm looking for a way to use multiple boolean operators in a single expression, but can't figure it out.
The code below is an attempt to translate the following into C:

pseudocode:

```<code>if
(100 = 100) AND (10 is either 10 or 100)
print "1"
else
print "0"
</code>```

C:

```<code>#include <stdio.h>
#include <stdbool.h>

int main(void)
{
    int num;
    num= 10;
    bool n;
    n = ((100==100) && (num == (10 || 100)));
    printf("%i", n);
}
</code>```

Obviously I'm doing something wrong.

答:
This expression

```<code>(a = b) and (c = (d or f)
</code>```

can be represented in C in an if statement like

```<code>#include <iso646.h>

//...

if ( ( a == b ) and ( c == d or c == f ) )
</code>```

or just like

```<code>if ( ( a == b ) && ( c == d || c == f ) )
</code>```

Or in an expression statement like

```<code>#include <iso646.h>
#include <stdbool.h>

//...

bool n = ( a == b ) and ( c == d or c == f );
</code>```

or

```<code>#include <stdbool.h>

//...

bool n = ( a == b ) && ( c == d || c == f );
</code>```

Body:
I'm looking for a way to use multiple boolean operators in a single expression, but can't figure it out.
The code below is an attempt to translate the following into C:

pseudocode:

if
(100 = 100) AND (10 is either 10 or 100)
print "1"
else
print "0"

C:

#include <stdio.h>
#include <stdbool.h>

int main(void)
{
    int num;
    num= 10;
    bool n;
    n = ((100==100) && (num == (10 || 100)));
    printf("%i", n);
}

Obviously I'm doing something wrong.

Body:
This expression

(a = b) and (c = (d or f)

can be represented in C in an if statement like

#include <iso646.h>

//...

if ( ( a == b ) and ( c == d or c == f ) )

or just like

if ( ( a == b ) && ( c == d || c == f ) )

Or in an expression statement like

#include <iso646.h>
#include <stdbool.h>

//...

bool n = ( a == b ) and ( c == d or c == f );

or

#include <stdbool.h>

//...

bool n = ( a == b ) && ( c == d || c == f );


问:
path works from root only
I read through all I could find on the topic, but nothing worked for me. I run PHP 5.3.29.

I have this PHP code, that works from root, but not from folders:

```<code>  // Fetch latest poster
  $base_url = 'posters';
  $newest_mtime = 0;
  if ($handle = opendir($base_url)) {
  while (false !== ($latestFile = readdir($handle))) {
      if (($latestFile != '.') && ($latestFile != '..')) {
         $mtime = filemtime("$base_url/$latestFile");
         if ($mtime > $newest_mtime) {
            $newest_mtime = $mtime;
            $show_file = "$base_url/$latestFile";
         }
      }
    }
  }
  // Show latest poster
  $show_name = basename(trim($show_file, ".jpg"));
  echo '
  <a href="/' . $show_file . '">
  <img src="' . $show_file . '" class="jpg" width="150" alt="latest poster" title="' . $show_name . '">
  </a>';
</code>```

I'd love to just add a slash to get root-relative paths, but that doesn't work in PHP.

This is the site with this issue:

http://flamencopeko.net (here there is no issue)

http://flamencopeko.net/disco_pages/sundays.php (this one has the issue)

答:
The reason you can only display the images on the root dir has nothing to do with your script. It has to do with the url's your placing in your <img.. tag, like below;

Root

```<code><img src='/posters/NAME_OF_IMG.jpg' ... />
</code>```

Which would resolve to something like: http://www.yourwebsite.com/posters/NAME_OF_IMG.jpg

Sub Dir

Which would resolve to something like: http://www.yourwebsite.com/disco_pages/posters/NAME_OF_IMG.jpg - WHICH DOESN't EXIST.

What you need to do is modify your code to generate the $show_file url like this:

```<code>$base_url = getcwd() . 'posters';
$newest_mtime = 0;
if ($handle = opendir($base_url)) {
while (false !== ($latestFile = readdir($handle))) {
    if (($latestFile != '.') && ($latestFile != '..')) {
       $mtime = filemtime("$base_url/$latestFile");
       if ($mtime > $newest_mtime) {
            $newest_mtime = $mtime;
            $show_file = "http://www.yourdomain.com/posters/" . $latestFile;
       }
    }
  }
}
</code>```

Alternatively, you could use $_SERVER['DOCUMENT_ROOT'] instead of getcwd() if required.

This should work for you from every directory you attempt. You have to understand what your code is doing to understand why it isn't working as you want.

You're traversing through your directories at file level to find the newest file, correct? Hence you need the absolute path of the file to check these aspects.

You're providing that absolute path to the <img ... tag which won't work. Your website is expecting the relative path (http://www.yourwebsite.com/posters/THE_IMAGE_NAME.jpg).

Title: path works from root only

Body:
I read through all I could find on the topic, but nothing worked for me. I run PHP 5.3.29.

I have this PHP code, that works from root, but not from folders:

  // Fetch latest poster
  $base_url = 'posters';
  $newest_mtime = 0;
  if ($handle = opendir($base_url)) {
  while (false !== ($latestFile = readdir($handle))) {
      if (($latestFile != '.') && ($latestFile != '..')) {
         $mtime = filemtime("$base_url/$latestFile");
         if ($mtime > $newest_mtime) {
            $newest_mtime = $mtime;
            $show_file = "$base_url/$latestFile";
         }
      }
    }
  }
  // Show latest poster
  $show_name = basename(trim($show_file, ".jpg"));
  echo '

  ';

I'd love to just add a slash to get root-relative paths, but that doesn't work in PHP.

This is the site with this issue:

http://flamencopeko.net (here there is no issue)

http://flamencopeko.net/disco_pages/sundays.php (this one has the issue)

Body:
The reason you can only display the images on the root dir has nothing to do with your script. It has to do with the url's your placing in your <img.. tag, like below;

Root

<img src='/posters/NAME_OF_IMG.jpg' ... />

Which would resolve to something like: http://www.yourwebsite.com/posters/NAME_OF_IMG.jpg

Sub Dir

Which would resolve to something like: http://www.yourwebsite.com/disco_pages/posters/NAME_OF_IMG.jpg - WHICH DOESN't EXIST.

What you need to do is modify your code to generate the $show_file url like this:

$base_url = getcwd() . 'posters';
$newest_mtime = 0;
if ($handle = opendir($base_url)) {
while (false !== ($latestFile = readdir($handle))) {
    if (($latestFile != '.') && ($latestFile != '..')) {
       $mtime = filemtime("$base_url/$latestFile");
       if ($mtime > $newest_mtime) {
            $newest_mtime = $mtime;
            $show_file = "http://www.yourdomain.com/posters/" . $latestFile;
       }
    }
  }
}

Alternatively, you could use $_SERVER['DOCUMENT_ROOT'] instead of getcwd() if required.

This should work for you from every directory you attempt. You have to understand what your code is doing to understand why it isn't working as you want.

You're traversing through your directories at file level to find the newest file, correct? Hence you need the absolute path of the file to check these aspects.

You're providing that absolute path to the <img ... tag which won't work. Your website is expecting the relative path (http://www.yourwebsite.com/posters/THE_IMAGE_NAME.jpg).


问:
Made an etch-a-sketch that works on JSFiddle but not on Github Pages
So I made a simple etch-a-sketch that seems to be working on Codepen:                      https://codepen.io/Chango-/pen/GEBpbx

```<code><head>
  <title>Etch-a-Sketch</title>
  <link rel="stylesheet" href="stylesheet.css"></link>
  <!-- Latest compiled and minified CSS -->
  <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">
  <link href="https://fonts.googleapis.com/css?family=Lato" rel="stylesheet">

</head>

<body>
  <h4>The Greatest Etch-A-Sketch in the Universe</h4>

  <center>
    <button class="newGrid btn btn-primary" type="button">New sketch!</button>
    <button class="randomColor btn btn-primary">Random Color</button>
  </center>

  <div id="container"></div>

  <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js"></script>
</body>
</code>```

I'm trying to deploy the static site on Github but the JQuery doesn't seem to be working even though the script is included in the HTML file. Not sure what's the problem

Link to Github repo: https://github.com/CGrijalva90/etch-a-sketch

Link to Github page: https://cgrijalva90.github.io/etch-a-sketch/

答:
Add the script tag to your html

```<code><script src="./EtchaSketch.js"></script>
</code>```

Also fix the css tag in your head with the right href

```<code><link rel="stylesheet" href="./EtchaSketch.css">
</code>```

Body:
So I made a simple etch-a-sketch that seems to be working on Codepen:                      https://codepen.io/Chango-/pen/GEBpbx

  <title>Etch-a-Sketch</title>
  <link rel="stylesheet" href="stylesheet.css"></link>

  <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">
  <link href="https://fonts.googleapis.com/css?family=Lato" rel="stylesheet">

The Greatest Etch-A-Sketch in the Universe

    <button class="newGrid btn btn-primary" type="button">New sketch!</button>
    <button class="randomColor btn btn-primary">Random Color</button>

I'm trying to deploy the static site on Github but the JQuery doesn't seem to be working even though the script is included in the HTML file. Not sure what's the problem

Link to Github repo: https://github.com/CGrijalva90/etch-a-sketch

Link to Github page: https://cgrijalva90.github.io/etch-a-sketch/

Body:
Add the script tag to your html

<script src="./EtchaSketch.js"></script>

Also fix the css tag in your head with the right href

<link rel="stylesheet" href="./EtchaSketch.css">


问:
object does not support indexing when overriding __getitem__
The Solution class inherits from object class, TypeError occurs: 'Solution' object does not support indexing? But if Solution class does not inherit from object, the code works find. My question is why Solution class does not support indexing when it inherits from object, even though it override getitem method.

```<code>class Solution(object):
    def findMin(self, nums):
        self.__getitem__ = lambda i: nums[i] <= nums[-1]
        return nums[bisect.bisect(self, False, 0, len(nums))]
</code>```

答:
You need to do it like something this:

```<code>import bisect

class Solution(object):
    __getitem__ = lambda self, i: self.nums[i] <= self.nums[-1]

    def findMin(self, nums):
        self.nums = nums
        return nums[bisect.bisect(self, False, 0, len(nums))]

sol = Solution()
sol.findMin([42, 1, 13])
</code>```

__getitem__ needs to be defined at the class, not instance, level in new-style classes (those whose base-class is object in Python 2, and all classes in Python 3—whether explicitly specified or not).

Body:
The Solution class inherits from object class, TypeError occurs: 'Solution' object does not support indexing? But if Solution class does not inherit from object, the code works find. My question is why Solution class does not support indexing when it inherits from object, even though it override getitem method.

class Solution(object):
    def findMin(self, nums):
        self.__getitem__ = lambda i: nums[i] <= nums[-1]
        return nums[bisect.bisect(self, False, 0, len(nums))]

Body:
You need to do it like something this:

import bisect

class Solution(object):
    __getitem__ = lambda self, i: self.nums[i] <= self.nums[-1]

    def findMin(self, nums):
        self.nums = nums
        return nums[bisect.bisect(self, False, 0, len(nums))]

sol = Solution()
sol.findMin([42, 1, 13])

__getitem__ needs to be defined at the class, not instance, level in new-style classes (those whose base-class is object in Python 2, and all classes in Python 3—whether explicitly specified or not).


问:
react-native function is not defined while using firebase
I'm very new to react and react-native world. I have made a new application which uses firebase that has a login screen, user can register with email and password. I wanted to check the user is already registered and redirect to another screen accordingly, as a first step I used constructor and onAuthStateChanged,  I wanted to set a state. But I always get function not defined, can anyone please help me.

```<code>constructor(props) {
        super(props);

        firebase.initializeApp(FireBaseConfig);

        this.state = {
            currentUser: null,
            emailAddress: '',
            password: '',
            loginError: {
                show: false,
                message: "Unknown error"
            }
        }

        this.saveCurrentUser = this.saveCurrentUser.bind(this);
        this.doLogin = this.doLogin.bind(this);

        firebase.auth().onAuthStateChanged(function (user) {
            saveCurrentUser(user);
        });
    }

    saveCurrentUser(user) {
        this.setState({
            currentUser: user
        });
    }
</code>```

Thanks

答:
saveCurrentUser is indeed not defined. You can access it under the this keyword, which is a reference to your class.
But since you are using a regular function callback on that firebase call, you gotta reference this before.. That's why I added

var saveCurrentUser = this.saveCurrentUser

```<code>constructor(props) {
    super(props);

    firebase.initializeApp(FireBaseConfig);

    this.state = {
        currentUser: null,
        emailAddress: '',
        password: '',
        loginError: {
            show: false,
            message: "Unknown error"
        }
    }

    this.saveCurrentUser = this.saveCurrentUser.bind(this);
    this.doLogin = this.doLogin.bind(this);

    var saveCurrentUser = this.saveCurrentUser

    firebase.auth().onAuthStateChanged(function (user) {
        saveCurrentUser(user);
    });
}

saveCurrentUser(user) {
    this.setState({
        currentUser: user
    });
}
</code>```

You could have used an arrow-function so the var declaration would become unnecessary.

Body:
I'm very new to react and react-native world. I have made a new application which uses firebase that has a login screen, user can register with email and password. I wanted to check the user is already registered and redirect to another screen accordingly, as a first step I used constructor and onAuthStateChanged,  I wanted to set a state. But I always get function not defined, can anyone please help me.

constructor(props) {
        super(props);

        firebase.initializeApp(FireBaseConfig);

        this.state = {
            currentUser: null,
            emailAddress: '',
            password: '',
            loginError: {
                show: false,
                message: "Unknown error"
            }
        }

        this.saveCurrentUser = this.saveCurrentUser.bind(this);
        this.doLogin = this.doLogin.bind(this);

        firebase.auth().onAuthStateChanged(function (user) {
            saveCurrentUser(user);
        });
    }

    saveCurrentUser(user) {
        this.setState({
            currentUser: user
        });
    }

Thanks

Body:
saveCurrentUser is indeed not defined. You can access it under the this keyword, which is a reference to your class.
But since you are using a regular function callback on that firebase call, you gotta reference this before.. That's why I added

var saveCurrentUser = this.saveCurrentUser

constructor(props) {
    super(props);

    firebase.initializeApp(FireBaseConfig);

    this.state = {
        currentUser: null,
        emailAddress: '',
        password: '',
        loginError: {
            show: false,
            message: "Unknown error"
        }
    }

    this.saveCurrentUser = this.saveCurrentUser.bind(this);
    this.doLogin = this.doLogin.bind(this);

    var saveCurrentUser = this.saveCurrentUser

    firebase.auth().onAuthStateChanged(function (user) {
        saveCurrentUser(user);
    });
}

saveCurrentUser(user) {
    this.setState({
        currentUser: user
    });
}

You could have used an arrow-function so the var declaration would become unnecessary.


问:
Markdown rendered with Redcarpet missing dashes for lists and new lines between sections
I have some markdown that I generated which looks like this in text form:

"### We're looking for someone with…\r\n\r\n- Significant Rails
  experience\r\n- Good communication skills
  (recommended)\r\n\r\n\r\n### You should be located near:\r\n\r\n-
  Berlin\r\n- San Francisco\r\n- Toronto\r\n"

In ApplicationHelper.rb, I have the following:

```<code>module ApplicationHelper
    def markdown(content)
      return '' if content.blank?

      @options = {
           autolink: true,
           space_after_headers: true,
           underline: true,
           link_attributes: {rel: 'nofollow', target: "_blank"}
       }

      markdown = Redcarpet::Markdown.new(Redcarpet::Render::HTML, @options)

      markdown.render(content).html_safe
    end
end
</code>```

No matter how many line breaks I add between each of the H3 sections (in this case it's 3 new lines), the two sections hug right up next to each other instead of respecting the 3 \ns in the text.

Also, there's no - or * character before each item in a list.

I looked on the Redcarpet docs and I don't see anything that I can enable to respect the new lines and the list items. Whitelisting every single html tag that markdown compiles to doesn't seem like the right solution.

Any help would be appreciated. Thanks!

答:
The output of Redcarpet is HTML. How it appears in the browser depends on how you have styled this HTML, and is more or less independent of the spacing in the source markdown.

The markdown you give produces the following HTML:

We're looking for someone with…

Significant Rails experience

Good communication skills (recommended)

You should be located near:

Berlin

San Francisco

Toronto

```

The exact same HTML is generated no matter how many newlines you have between the list and the header.

How much space appears when this is rendered depends on your CSS. For example you might want to add a margin-top or padding-top to the h3 elements to create more space.

The same is true for the list icons, you probably have list-style set to none, which is why nothing is appearing for the bullets.

Body:
I have some markdown that I generated which looks like this in text form:

"### We're looking for someone with…\r\n\r\n- Significant Rails
  experience\r\n- Good communication skills
  (recommended)\r\n\r\n\r\n### You should be located near:\r\n\r\n-
  Berlin\r\n- San Francisco\r\n- Toronto\r\n"

In ApplicationHelper.rb, I have the following:

module ApplicationHelper
    def markdown(content)
      return '' if content.blank?

      @options = {
           autolink: true,
           space_after_headers: true,
           underline: true,
           link_attributes: {rel: 'nofollow', target: "_blank"}
       }

      markdown = Redcarpet::Markdown.new(Redcarpet::Render::HTML, @options)

      markdown.render(content).html_safe
    end
end

No matter how many line breaks I add between each of the H3 sections (in this case it's 3 new lines), the two sections hug right up next to each other instead of respecting the 3 \ns in the text.

Also, there's no - or * character before each item in a list.

I looked on the Redcarpet docs and I don't see anything that I can enable to respect the new lines and the list items. Whitelisting every single html tag that markdown compiles to doesn't seem like the right solution.

Any help would be appreciated. Thanks!

Body:
The output of Redcarpet is HTML. How it appears in the browser depends on how you have styled this HTML, and is more or less independent of the spacing in the source markdown.

The markdown you give produces the following HTML:

<h3>We&#39;re looking for someone with…</h3>

<ul>
<li>Significant Rails experience</li>
<li>Good communication skills (recommended)</li>
</ul>

<h3>You should be located near:</h3>

<ul>
<li>Berlin</li>
<li>San Francisco</li>
<li>Toronto</li>
</ul>

The exact same HTML is generated no matter how many newlines you have between the list and the header.

How much space appears when this is rendered depends on your CSS. For example you might want to add a margin-top or padding-top to the h3 elements to create more space.

The same is true for the list icons, you probably have list-style set to none, which is why nothing is appearing for the bullets.


问:
Faster way to generate large list of vectors from permuted datasets [R]
Setup For the purposes of my simulation, I'm generating a list of B=2000 elements, with each element being the output of a permutation procedure in which I first permute the rows of a 200x8000 matrix and for each column, I calculate the Kolmogorov-Smirnov test statistic between the first and second 100 rows (you can think of the first 100 rows as data from one group and the second 100 rows as data from another group).

Question This process takes a very long time (about 30-40 minutes) to generate the list. Is there a much faster way? In the future, I'd like to increase B to a larger value.

Code

```<code>B=2000
n.row=200; n.col=8000
#Generate sample data
samp.dat = matrix(rnorm(n.row*n.col),nrow=n.row)

perm.KS.list = NULL
for (b in 1:B){
    #permute the rows
    perm.dat.tmp = samp.dat[sample(nrow(samp.dat)),]
    #Compute the permutation-based test statistics
    perm.KS.list[[b]]= apply(perm.dat.tmp,2,function(y) ks.test.stat(y[1:100],y[101:200]))
}

#Modified KS-test function (from base package)
ks.test.stat <- function(x,y){
  x <- x[!is.na(x)]
  n <- length(x)
  y <- y[!is.na(y)]
  n.x <- as.double(n)
  n.y <- length(y)
  w <- c(x, y)
  z <- cumsum(ifelse(order(w) <= n.x, 1/n.x, -1/n.y))
  z <- z[c(which(diff(sort(w)) != 0), n.x + n.y)] #exclude ties
  STATISTIC <- max(abs(z))
  return(STATISTIC)
}
</code>```

答:
The 1:B loop has several places to optimize, but I agree that the real consumer is that inner function.  Because you're simulating your well-behaved bootstrap samples, you can make two simplifying assumptions that the general base function can't:

There aren't missing values.  This obviates the is.na() adjustments

The two sides (ie, x & y) have the same number of elements, so you don't need to count them separately.  instead of splitting y in the loop, and them joining them back in the function (into w), just keep it together.  The balanced sides also permit simplifications like remove the ifelse() clause.  It produces a bunch of 0/1s, which are rescaled to -1/1s with integer arithmetic.

The function is reduced, which saves about 25% of the time.  I added integers, instead of doubles inside cumsum().

```<code>ks.test.stat.balanced <- function(w){
  n     <- as.integer(length(w) * .5)
  # z   <- cumsum(ifelse(order(w) <= n, 1L, -1L)) / n
  z     <- cumsum((order(w) <= n)*2L - 1L) / n
  # z   <- z[c(which(diff(sort(w)) != 0), n + n)] #exclude ties
  return( max(abs(z)) )
}
</code>```

Ties shouldn't occur often with your gaussian rng, and the diff(sort(.)) is very expensive.  If you're willing to remove that protection, the time is reduced by about 65%.

If you move the equation for z into abs(), it saves a little time over all those reps.  I kept it separate above, so it's easier to read.

edit in case of an unbalanced simulation I'd recommend you:

still keep out the is.na,

still pass w,

still keep as much as possible in integer, not numeric, but

now include arguments n1 & n2 for the two group sizes.

Also, experiment w/ precalculating 1/n before cumsum() to avoid a lot of expensive divisions.  Try to think of other math-y ways to extract calculations from an inner loop so it occurs less frequently.

Body:
Setup For the purposes of my simulation, I'm generating a list of B=2000 elements, with each element being the output of a permutation procedure in which I first permute the rows of a 200x8000 matrix and for each column, I calculate the Kolmogorov-Smirnov test statistic between the first and second 100 rows (you can think of the first 100 rows as data from one group and the second 100 rows as data from another group).

Question This process takes a very long time (about 30-40 minutes) to generate the list. Is there a much faster way? In the future, I'd like to increase B to a larger value.

Code

B=2000
n.row=200; n.col=8000
#Generate sample data
samp.dat = matrix(rnorm(n.row*n.col),nrow=n.row)

perm.KS.list = NULL
for (b in 1:B){
    #permute the rows
    perm.dat.tmp = samp.dat[sample(nrow(samp.dat)),]
    #Compute the permutation-based test statistics
    perm.KS.list[[b]]= apply(perm.dat.tmp,2,function(y) ks.test.stat(y[1:100],y[101:200]))
}

#Modified KS-test function (from base package)
ks.test.stat <- function(x,y){
  x <- x[!is.na(x)]
  n <- length(x)
  y <- y[!is.na(y)]
  n.x <- as.double(n)
  n.y <- length(y)
  w <- c(x, y)
  z <- cumsum(ifelse(order(w) <= n.x, 1/n.x, -1/n.y))
  z <- z[c(which(diff(sort(w)) != 0), n.x + n.y)] #exclude ties
  STATISTIC <- max(abs(z))
  return(STATISTIC)
}

Body:
The 1:B loop has several places to optimize, but I agree that the real consumer is that inner function.  Because you're simulating your well-behaved bootstrap samples, you can make two simplifying assumptions that the general base function can't:

There aren't missing values.  This obviates the is.na() adjustments

The two sides (ie, x & y) have the same number of elements, so you don't need to count them separately.  instead of splitting y in the loop, and them joining them back in the function (into w), just keep it together.  The balanced sides also permit simplifications like remove the ifelse() clause.  It produces a bunch of 0/1s, which are rescaled to -1/1s with integer arithmetic.

The function is reduced, which saves about 25% of the time.  I added integers, instead of doubles inside cumsum().

ks.test.stat.balanced <- function(w){
  n     <- as.integer(length(w) * .5)
  # z   <- cumsum(ifelse(order(w) <= n, 1L, -1L)) / n
  z     <- cumsum((order(w) <= n)*2L - 1L) / n
  # z   <- z[c(which(diff(sort(w)) != 0), n + n)] #exclude ties
  return( max(abs(z)) )
}

Ties shouldn't occur often with your gaussian rng, and the diff(sort(.)) is very expensive.  If you're willing to remove that protection, the time is reduced by about 65%.

If you move the equation for z into abs(), it saves a little time over all those reps.  I kept it separate above, so it's easier to read.

edit in case of an unbalanced simulation I'd recommend you:

still keep out the is.na,

still pass w,

still keep as much as possible in integer, not numeric, but

now include arguments n1 & n2 for the two group sizes.

Also, experiment w/ precalculating 1/n before cumsum() to avoid a lot of expensive divisions.  Try to think of other math-y ways to extract calculations from an inner loop so it occurs less frequently.


问:
C# Easy drag and drop way to create DataTables from SQL?
When dealing with really small apps and the sqlbulkcopy, I normally create datatables by either using a FILL on an empty datatable OR I just type out something like this

```<code>DataTable dtGrps = new DataTable();
dtGrps.Columns.Add("objectGuid", typeof(Guid));
dtGrps.Columns.Add("DN", typeof(string));
dtGrps.Columns.Add("CN", typeof(string));
dtGrps.Columns.Add("groupType", typeof(string));
dtGrps.Columns.Add("description", typeof(string));
dtGrps.Columns.Add("whenCreated", typeof(string));
dtGrps.Columns.Add("whenChanged", typeof(string));
</code>```

but it occurred to me that surly there is a built in way (non EF or Linq) to create all the code above by some drag in drop method. I mean I am using VS2017, surely MS has added this feature and I have just missed it is all.

So does this exist?

答:
This is a very rough implementation, just intended as a starting point because my goal here isn't to write a new library. It could use a lot of optimization. But you can use strongly typed objects, use reflection to generate a DataTable based on the objects, and then use SqlBulkCopy to insert that.

```<code>using System;
using System.Collections.Generic;
using System.Data;

namespace StronglyStypedSqlBulkCopy
{
    class Program
    {
        static void Main(string[] args)
        {
            List<Car> cars = GetSampleData();
            DataTable dataTable = ConvertToDataTable(cars);

            Console.WriteLine("Press any key to exit.");
            Console.ReadKey(true);
        }

        public static List<Car> GetSampleData()
        {
            return new List<Car> {
                new Car { Id = 1, Make = "Toyota", Model = "Tacoma", DateOfManufacture = DateTime.Now.AddDays(-1) },
                new Car { Id = 2, Make = "Ford", Model = "Raptor", DateOfManufacture = DateTime.Now.AddDays(-2) },
                new Car { Id = 3, Make = "Ram", Model = "1500", DateOfManufacture = DateTime.Now.AddDays(-3) }
            };
        }

        public static DataTable ConvertToDataTable<T>(IEnumerable<T> objects)
        {
            var properties = objects.GetType().GetGenericArguments()[0].GetProperties();

            var table = new DataTable();

            foreach (var property in properties)
            {
                var columnName = property.Name; //may want to get from attribute also
                //probably want to define an explicit mapping of .NET types to SQL types, and allow an attribute to specifically specify the SQL type
                table.Columns.Add(columnName, property.PropertyType);
            }

            //probably want to cache the mapping from above in a real implementation

            foreach (var obj in objects)
            {
                var row = table.NewRow();

                foreach (var property in properties)
                {
                    var columnName = property.Name; //may want to get from attribute also
                    var propertyValue = property.GetValue(obj);
                    row[columnName] = propertyValue;
                }

                table.Rows.Add(row);
            }

            return table;
        }
    }

    public class Car
    {
        public int Id { get; set; }

        public string Make { get; set; }

        public string Model { get; set; }

        public DateTime DateOfManufacture { get; set; }
    }
}
</code>```

Once you have a nice implementation of ConvertToDataTable, it's simply a matter of defining strongly typed classes, which are much easier to work with than raw DataTables.

Body:
When dealing with really small apps and the sqlbulkcopy, I normally create datatables by either using a FILL on an empty datatable OR I just type out something like this

DataTable dtGrps = new DataTable();
dtGrps.Columns.Add("objectGuid", typeof(Guid));
dtGrps.Columns.Add("DN", typeof(string));
dtGrps.Columns.Add("CN", typeof(string));
dtGrps.Columns.Add("groupType", typeof(string));
dtGrps.Columns.Add("description", typeof(string));
dtGrps.Columns.Add("whenCreated", typeof(string));
dtGrps.Columns.Add("whenChanged", typeof(string));

but it occurred to me that surly there is a built in way (non EF or Linq) to create all the code above by some drag in drop method. I mean I am using VS2017, surely MS has added this feature and I have just missed it is all.

So does this exist?

Body:
This is a very rough implementation, just intended as a starting point because my goal here isn't to write a new library. It could use a lot of optimization. But you can use strongly typed objects, use reflection to generate a DataTable based on the objects, and then use SqlBulkCopy to insert that.

using System;
using System.Collections.Generic;
using System.Data;

namespace StronglyStypedSqlBulkCopy
{
    class Program
    {
        static void Main(string[] args)
        {
            List<Car> cars = GetSampleData();
            DataTable dataTable = ConvertToDataTable(cars);

            Console.WriteLine("Press any key to exit.");
            Console.ReadKey(true);
        }

        public static List<Car> GetSampleData()
        {
            return new List<Car> {
                new Car { Id = 1, Make = "Toyota", Model = "Tacoma", DateOfManufacture = DateTime.Now.AddDays(-1) },
                new Car { Id = 2, Make = "Ford", Model = "Raptor", DateOfManufacture = DateTime.Now.AddDays(-2) },
                new Car { Id = 3, Make = "Ram", Model = "1500", DateOfManufacture = DateTime.Now.AddDays(-3) }
            };
        }

        public static DataTable ConvertToDataTable<T>(IEnumerable<T> objects)
        {
            var properties = objects.GetType().GetGenericArguments()[0].GetProperties();

            var table = new DataTable();

            foreach (var property in properties)
            {
                var columnName = property.Name; //may want to get from attribute also
                //probably want to define an explicit mapping of .NET types to SQL types, and allow an attribute to specifically specify the SQL type
                table.Columns.Add(columnName, property.PropertyType);
            }

            //probably want to cache the mapping from above in a real implementation

            foreach (var obj in objects)
            {
                var row = table.NewRow();

                foreach (var property in properties)
                {
                    var columnName = property.Name; //may want to get from attribute also
                    var propertyValue = property.GetValue(obj);
                    row[columnName] = propertyValue;
                }

                table.Rows.Add(row);
            }

            return table;
        }
    }

    public class Car
    {
        public int Id { get; set; }

        public string Make { get; set; }

        public string Model { get; set; }

        public DateTime DateOfManufacture { get; set; }
    }
}

Once you have a nice implementation of ConvertToDataTable, it's simply a matter of defining strongly typed classes, which are much easier to work with than raw DataTables.


问:
Building URL- Confusion on a couple of lines
I am trying to understand this class/method involved in a Udacity Android Development course, and I am confused on a couple lines. I was wondering how the String Param_QUERY = "q" works, and looking for any explanation. In addition, I was also confused about PARAM_SORT, and sortBy. Any explanation on these three variables and how they are used would be much appreciated. Thanks, and sorry for any strange formatting. </p>

```<code>public class NetworkUtils {

     final static String GITHUB_BASE_URL =
             "https://api.github.com/search/repositories";

     final static String PARAM_QUERY = "q";

     final static String PARAM_SORT = "sort";
     final static String sortBy = "stars";

/**
 * Builds the URL used to query Github.
 *
 * @param githubSearchQuery The keyword that will be queried for.
 * @return The URL to use to query the weather server.
 */
public static URL buildUrl(String githubSearchQuery) {

    Uri builtUri = Uri.parse(GITHUB_BASE_URL).buildUpon()
            .appendQueryParameter(PARAM_QUERY, githubSearchQuery)
            .appendQueryParameter(PARAM_SORT, sortBy)
            .build();

    URL url = null;
    try {
        url = new URL(builtUri.toString());
    } catch (MalformedURLException e) {
        e.printStackTrace();
    }

    return url;
}
</code>```

答:
it will make your url looks like this, the appendQueryParameter will add used params inside the url</p>

```<code>https://api.github.com/search/repositories?q=githubSearchQueryValue&sort=stars
</code>```

Body:
I am trying to understand this class/method involved in a Udacity Android Development course, and I am confused on a couple lines. I was wondering how the String Param_QUERY = "q" works, and looking for any explanation. In addition, I was also confused about PARAM_SORT, and sortBy. Any explanation on these three variables and how they are used would be much appreciated. Thanks, and sorry for any strange formatting.

public class NetworkUtils {

     final static String GITHUB_BASE_URL =
             "https://api.github.com/search/repositories";

     final static String PARAM_QUERY = "q";

     final static String PARAM_SORT = "sort";
     final static String sortBy = "stars";

/**
 * Builds the URL used to query Github.
 *
 * @param githubSearchQuery The keyword that will be queried for.
 * @return The URL to use to query the weather server.
 */
public static URL buildUrl(String githubSearchQuery) {

    Uri builtUri = Uri.parse(GITHUB_BASE_URL).buildUpon()
            .appendQueryParameter(PARAM_QUERY, githubSearchQuery)
            .appendQueryParameter(PARAM_SORT, sortBy)
            .build();

    URL url = null;
    try {
        url = new URL(builtUri.toString());
    } catch (MalformedURLException e) {
        e.printStackTrace();
    }

    return url;
}

Body:
it will make your url looks like this, the appendQueryParameter will add used params inside the url

https://api.github.com/search/repositories?q=githubSearchQueryValue&sort=stars


问:
Is there a better way to produce a membership matrix (one-hot array) for an array of cluster assignments in Python?
After running kmeans I can easily get an array with the assigned clusters for ever data point. Now I want to get a membership matrix (one-hot array) which has the different clusters as columns and indicates the cluster assignment by either 1 or 0 in the matrix for each data point.

My code is shown below and it works but I am wondering if there is a more elegant way to do the same.

```<code>km = KMeans(n_clusters=3).fit(data)
membership_matrix = np.stack([np.where(km.labels_ == 0, 1,0),
                              np.where(km.labels_ == 1, 1,0),
                              np.where(km.labels_ == 2, 1,0)]
                              axis = 1)
</code>```

答:
You are looking for LabelBinarizer. Give this code a try:

from sklearn import preprocessing
lb = preprocessing.LabelBinarizer()
membership_matrix = lb.fit_transform(km.labels_)
```

<p>In contrast to other solutions proposed here, this approach:  </p>

<ul>
<li>Generates a compact membership matrix when the labels are <strong>not consecutive numbers</strong>.</li>
<li>Is able to deal with <strong>categorical labels</strong>.</li>
</ul>

<p><strong>Sample run:</strong></p>

<pre class="lang-python prettyprint-override"><code>In [9]: lb.fit_transform([0, 1, 2, 0, 2, 2])
Out[9]:
array([[1, 0, 0],
       [0, 1, 0],
       [0, 0, 1],
       [1, 0, 0],
       [0, 0, 1],
       [0, 0, 1]])

In [10]: lb.fit_transform([0, 1, 9, 0, 9, 9])
Out[10]:
array([[1, 0, 0],
       [0, 1, 0],
       [0, 0, 1],
       [1, 0, 0],
       [0, 0, 1],
       [0, 0, 1]])

In [11]: lb.fit_transform(['first', 'second', 'third', 'first', 'third', 'third'])
Out[11]:
array([[1, 0, 0],
       [0, 1, 0],
       [0, 0, 1],
       [1, 0, 0],
       [0, 0, 1],
       [0, 0, 1]])
</code>```

Body:
After running kmeans I can easily get an array with the assigned clusters for ever data point. Now I want to get a membership matrix (one-hot array) which has the different clusters as columns and indicates the cluster assignment by either 1 or 0 in the matrix for each data point.

My code is shown below and it works but I am wondering if there is a more elegant way to do the same.

km = KMeans(n_clusters=3).fit(data)
membership_matrix = np.stack([np.where(km.labels_ == 0, 1,0),
                              np.where(km.labels_ == 1, 1,0),
                              np.where(km.labels_ == 2, 1,0)]
                              axis = 1)

Body:
You are looking for LabelBinarizer. Give this code a try:

from sklearn import preprocessing
lb = preprocessing.LabelBinarizer()
membership_matrix = lb.fit_transform(km.labels_)

In contrast to other solutions proposed here, this approach:

Generates a compact membership matrix when the labels are not consecutive numbers.

Is able to deal with categorical labels.

Sample run:

In [9]: lb.fit_transform([0, 1, 2, 0, 2, 2])
Out[9]:
array([[1, 0, 0],
       [0, 1, 0],
       [0, 0, 1],
       [1, 0, 0],
       [0, 0, 1],
       [0, 0, 1]])

In [10]: lb.fit_transform([0, 1, 9, 0, 9, 9])
Out[10]:
array([[1, 0, 0],
       [0, 1, 0],
       [0, 0, 1],
       [1, 0, 0],
       [0, 0, 1],
       [0, 0, 1]])

In [11]: lb.fit_transform(['first', 'second', 'third', 'first', 'third', 'third'])
Out[11]:
array([[1, 0, 0],
       [0, 1, 0],
       [0, 0, 1],
       [1, 0, 0],
       [0, 0, 1],
       [0, 0, 1]])


问:
Using .BY, .GRP or other methods to add a multicolumn aggregation with data.table
Say we have this toy data.table example:

```<code>temp <- data.table(V=c("A", "B", "C", "D","A"), GR=c(1,1,1,2,2))
"V" "GR"
 A   1
 B   1
 C   1
 D   2
 A   2
</code>```

I would like to generate all ordered combinations with combn within each subset defined by GR and create with it a new data.table and with a new column with the grouping factor.

For example, for GR=1 we have (A,B),(A,C),(B,C)

for GR=2 we have (D,A)

If I create the result manually it would be

```<code>cbind(V=c(1,1,1,2),rbind(t(combn(c("A", "B", "C"),2)),t(combn(c( "D","A"),2))))

1  A  B
1  A  C
1  B  C
2  D  A
</code>```

But I would like to do it with data.table easily instead.

This two option don't work:

```<code>temp[,cbind(rep(.GRP,.N),as.data.frame(t(combn(V,2)))),by=GR]
temp[,cbind(rep(.BY,.N),as.data.frame(t(combn(V,2)))),by=GR]
</code>```

This one work, but I don't understand why. I'm afraid it could copy the whole B vector as is instead of the proper value.

```<code>temp[,.(GR,as.list(as.data.frame((combn(V,2))))),by=GR]
</code>```

And I guess it should be a shorter way to write it.

答:
I have one solution but it seems long and complex too.

```<code>temp[,do.call(c, apply(t(combn(V,2)), 2, list)),by=GR]
</code>```

I've also found that combn is 10 times slower than some specialized packages such as iterpc or combinat

```<code>temp[,do.call(c, apply(combn2(V), 2, list)),by=GR]
</code>```

You must also first filter out any group having just one row because otherwise it would cause an error.

And this is my final version, much faster and needs much less memory:

```<code>temp[,.(from=rep(V,(.N-1):0),to=V[unlist(sapply(2:.N, seq, .N, simplify = T))]), by=GR]
</code>```

Body:
Say we have this toy data.table example:

temp <- data.table(V=c("A", "B", "C", "D","A"), GR=c(1,1,1,2,2))
"V" "GR"
 A   1
 B   1
 C   1
 D   2
 A   2

I would like to generate all ordered combinations with combn within each subset defined by GR and create with it a new data.table and with a new column with the grouping factor.

For example, for GR=1 we have (A,B),(A,C),(B,C)

for GR=2 we have (D,A)

If I create the result manually it would be

cbind(V=c(1,1,1,2),rbind(t(combn(c("A", "B", "C"),2)),t(combn(c( "D","A"),2))))

1  A  B
1  A  C
1  B  C
2  D  A

But I would like to do it with data.table easily instead.

This two option don't work:

temp[,cbind(rep(.GRP,.N),as.data.frame(t(combn(V,2)))),by=GR]
temp[,cbind(rep(.BY,.N),as.data.frame(t(combn(V,2)))),by=GR]

This one work, but I don't understand why. I'm afraid it could copy the whole B vector as is instead of the proper value.

temp[,.(GR,as.list(as.data.frame((combn(V,2))))),by=GR]

And I guess it should be a shorter way to write it.

Body:
I have one solution but it seems long and complex too.

temp[,do.call(c, apply(t(combn(V,2)), 2, list)),by=GR]

I've also found that combn is 10 times slower than some specialized packages such as iterpc or combinat

temp[,do.call(c, apply(combn2(V), 2, list)),by=GR]

You must also first filter out any group having just one row because otherwise it would cause an error.

And this is my final version, much faster and needs much less memory:

temp[,.(from=rep(V,(.N-1):0),to=V[unlist(sapply(2:.N, seq, .N, simplify = T))]), by=GR]