问:
JavaScript numbers to Words
I'm trying to convert numbers into english words, for example 1234 would become: "one thousand two hundred thirty four".

My Tactic goes like this:

Separate the digits to three and put them on Array (finlOutPut), from right to left.

Convert each group (each cell in the finlOutPut array) of three digits to a word (this what the triConvert function does). If all the three digits are zero convert them to "dontAddBigSuffix"

From Right to left, add thousand, million, billion, etc. If the finlOutPut cell equals "dontAddBigSufix" (because it was only zeroes), don't add the word and set the cell to " " (nothing).

It seems to work pretty well, but I've got some problems with numbers like 190000009, converted to: "one hundred ninety million". Somehow it "forgets" the last numbers when there are a few zeros.

What did I do wrong? Where is the bug? Why does it not work perfectly?

function update(){
    var bigNumArry = new Array('', ' thousand', ' million', ' billion', ' trillion', ' quadrillion', ' quintillion');

    var output = '';
    var numString =   document.getElementById('number').value;
    var finlOutPut = new Array();

    if (numString == '0') {
        document.getElementById('container').innerHTML = 'Zero';
        return;
    }

    if (numString == 0) {
        document.getElementById('container').innerHTML = 'messeg tell to enter numbers';
        return;
    }

    var i = numString.length;
    i = i - 1;

    //cut the number to grups of three digits and add them to the Arry
    while (numString.length > 3) {
        var triDig = new Array(3);
        triDig[2] = numString.charAt(numString.length - 1);
        triDig[1] = numString.charAt(numString.length - 2);
        triDig[0] = numString.charAt(numString.length - 3);

        var varToAdd = triDig[0] + triDig[1] + triDig[2];
        finlOutPut.push(varToAdd);
        i--;
        numString = numString.substring(0, numString.length - 3);
    }
    finlOutPut.push(numString);
    finlOutPut.reverse();

    //conver each grup of three digits to english word
    //if all digits are zero the triConvert
    //function return the string "dontAddBigSufix"
    for (j = 0; j < finlOutPut.length; j++) {
        finlOutPut[j] = triConvert(parseInt(finlOutPut[j]));
    }

    var bigScalCntr = 0; //this int mark the million billion trillion... Arry

    for (b = finlOutPut.length - 1; b >= 0; b--) {
        if (finlOutPut[b] != "dontAddBigSufix") {
            finlOutPut[b] = finlOutPut[b] + bigNumArry[bigScalCntr] + ' , ';
            bigScalCntr++;
        }
        else {
            //replace the string at finlOP[b] from "dontAddBigSufix" to empty String.
            finlOutPut[b] = ' ';
            bigScalCntr++; //advance the counter
        }
    }

        //convert The output Arry to , more printable string
        for(n = 0; n<finlOutPut.length; n++){
            output +=finlOutPut[n];
        }

    document.getElementById('container').innerHTML = output;//print the output
}

//simple function to convert from numbers to words from 1 to 999
function triConvert(num){
    var ones = new Array('', ' one', ' two', ' three', ' four', ' five', ' six', ' seven', ' eight', ' nine', ' ten', ' eleven', ' twelve', ' thirteen', ' fourteen', ' fifteen', ' sixteen', ' seventeen', ' eighteen', ' nineteen');
    var tens = new Array('', '', ' twenty', ' thirty', ' forty', ' fifty', ' sixty', ' seventy', ' eighty', ' ninety');
    var hundred = ' hundred';
    var output = '';
    var numString = num.toString();

    if (num == 0) {
        return 'dontAddBigSufix';
    }
    //the case of 10, 11, 12 ,13, .... 19
    if (num < 20) {
        output = ones[num];
        return output;
    }

    //100 and more
    if (numString.length == 3) {
        output = ones[parseInt(numString.charAt(0))] + hundred;
        output += tens[parseInt(numString.charAt(1))];
        output += ones[parseInt(numString.charAt(2))];
        return output;
    }

    output += tens[parseInt(numString.charAt(0))];
    output += ones[parseInt(numString.charAt(1))];

    return output;
}   ```
<pre class="snippet-code-html lang-html prettyprint-override"><code><input type="text"
    id="number"
    size="70"
    onkeyup="update();"
    /*this code prevent non numeric letters*/
    onkeydown="return (event.ctrlKey || event.altKey
                    || (47<event.keyCode && event.keyCode<58 && event.shiftKey==false)
                    || (95<event.keyCode && event.keyCode<106)
                    || (event.keyCode==8) || (event.keyCode==9)
                    || (event.keyCode>34 && event.keyCode<40)
                    || (event.keyCode==46) )"/>
                    <br/>
<div id="container">Here The Numbers Printed</div></code>```

答:
I am providing here my solution for converting numbers to the equivalent English words using the Single Loop String Triplets (SLST) Method which I have published and explained in detail here at Code Review with illustration graphics.

Simple Number to Words using a Single Loop String Triplets in JavaScript

The concept is simple and easily coded and is also very efficient and fast. The code is very short compared with other alternative methods described in this article.

The concept also permits the conversion of very vary large numbers as it does not rely on using the numeric/math function of the language and therefore avoids any limitations.

The Scale Array may be increased by adding more scale names if required after "Decillion".

A sample test function is provided below to generate numbers from 1 to 1099 as an example.

A full fancy version is available that caters for adding the word "and" and commas between the scales and numbers to align with the UK and US ways of spelling the numbers.

Hope this is useful.

/************************************************************/
function NumToWordsInt(NumIn) {
/************************************************************
* Convert Integer Numbers to English Words
* Using the Single Loop String Triplets Method
* @Param : {Number}  The number to be converted
*                    For large numbers use a string
* @Return: {String}  Wordified Number (Number in English Words)
* @Author: Mohsen Alyafei 10 July 2019
* @Notes : Call separately for whole and for fractional parts.
*          Scale Array may be increased by adding more scale
*          names if required after Decillion.
/************************************************************/

 if (NumIn==0) return "Zero";
 var  Ones  = ["", "One", "Two", "Three", "Four", "Five", "Six", "Seven", "Eight", "Nine", "Ten", "Eleven", "Twelve", "Thirteen", "Fourteen", "Fifteen", "Sixteen", "Seventeen", "Eighteen", "Nineteen"],
      Tens  = ["", "", "Twenty", "Thirty", "Forty", "Fifty", "Sixty", "Seventy", "Eighty", "Ninety"],
      Scale = ["", "Thousand", "Million", "Billion", "Trillion", "Quadrillion", "Quintillion", "Sextillion", "Septillion", "Octillion", "Nonillion", "Decillion"],
      N1, N2, Sep, j, i, h, Trplt, tns="", NumAll = "";
 NumIn += "";                                            // Make NumIn a String
//----------------- code starts -------------------
 NumIn = "0".repeat(NumIn.length * 2 % 3) + NumIn;       //Create shortest string triplets 0 padded
 j = 0;                                                  //Start with the highest triplet from LH
    for (i = NumIn.length / 3 - 1; i >= 0; i--) {        //Loop thru number of triplets from LH most
      Trplt = NumIn.substring(j, j + 3);                 //Get a triplet number starting from LH
      if (Trplt !="000") {                               //Skip empty trplets
        Sep = Trplt[2] !="0" ? "-":" ";                  //Dash only for 21 to 99
        N1 = Number(Trplt[0]);                           //Get Hundreds digit
        N2 = Number(Trplt.substr(1));                    //Get 2 lowest digits (00 to 99)
        tns = N2 > 19 ? Tens[Number(Trplt[1])] + Sep + Ones[Number(Trplt[2])] : Ones[N2];
        NumAll += ((h = N1>0 ? Ones[N1] + " Hundred": "") + " " + tns).trim() + " " + Scale[i]+ " ";
      }
      j += 3;                                            //Next lower triplets (move to RH)
    }
//----------------- code Ends --------------------
 return NumAll.trim();                                   //Return trimming excess spaces if any
}

// ----------------- test sample -----------------
console.log(NumToWordsInt(67123))
console.log(NumToWordsInt(120003123))
console.log(NumToWordsInt(123999))
console.log(NumToWordsInt(789123))
console.log(NumToWordsInt(100178912))
console.log(NumToWordsInt(777))
console.log(NumToWordsInt(999999999))
console.log(NumToWordsInt(45))```

Title: JavaScript numbers to Words

Body:
I'm trying to convert numbers into english words, for example 1234 would become: "one thousand two hundred thirty four".

My Tactic goes like this:

Separate the digits to three and put them on Array (finlOutPut), from right to left.

Convert each group (each cell in the finlOutPut array) of three digits to a word (this what the triConvert function does). If all the three digits are zero convert them to "dontAddBigSuffix"

From Right to left, add thousand, million, billion, etc. If the finlOutPut cell equals "dontAddBigSufix" (because it was only zeroes), don't add the word and set the cell to " " (nothing).

It seems to work pretty well, but I've got some problems with numbers like 190000009, converted to: "one hundred ninety million". Somehow it "forgets" the last numbers when there are a few zeros.

What did I do wrong? Where is the bug? Why does it not work perfectly?

function update(){
    var bigNumArry = new Array('', ' thousand', ' million', ' billion', ' trillion', ' quadrillion', ' quintillion');

    var output = '';
    var numString =   document.getElementById('number').value;
    var finlOutPut = new Array();

    if (numString == '0') {
        document.getElementById('container').innerHTML = 'Zero';
        return;
    }

    if (numString == 0) {
        document.getElementById('container').innerHTML = 'messeg tell to enter numbers';
        return;
    }

    var i = numString.length;
    i = i - 1;

    //cut the number to grups of three digits and add them to the Arry
    while (numString.length > 3) {
        var triDig = new Array(3);
        triDig[2] = numString.charAt(numString.length - 1);
        triDig[1] = numString.charAt(numString.length - 2);
        triDig[0] = numString.charAt(numString.length - 3);

        var varToAdd = triDig[0] + triDig[1] + triDig[2];
        finlOutPut.push(varToAdd);
        i--;
        numString = numString.substring(0, numString.length - 3);
    }
    finlOutPut.push(numString);
    finlOutPut.reverse();

    //conver each grup of three digits to english word
    //if all digits are zero the triConvert
    //function return the string "dontAddBigSufix"
    for (j = 0; j < finlOutPut.length; j++) {
        finlOutPut[j] = triConvert(parseInt(finlOutPut[j]));
    }

    var bigScalCntr = 0; //this int mark the million billion trillion... Arry

    for (b = finlOutPut.length - 1; b >= 0; b--) {
        if (finlOutPut[b] != "dontAddBigSufix") {
            finlOutPut[b] = finlOutPut[b] + bigNumArry[bigScalCntr] + ' , ';
            bigScalCntr++;
        }
        else {
            //replace the string at finlOP[b] from "dontAddBigSufix" to empty String.
            finlOutPut[b] = ' ';
            bigScalCntr++; //advance the counter
        }
    }

        //convert The output Arry to , more printable string
        for(n = 0; n<finlOutPut.length; n++){
            output +=finlOutPut[n];
        }

    document.getElementById('container').innerHTML = output;//print the output
}

//simple function to convert from numbers to words from 1 to 999
function triConvert(num){
    var ones = new Array('', ' one', ' two', ' three', ' four', ' five', ' six', ' seven', ' eight', ' nine', ' ten', ' eleven', ' twelve', ' thirteen', ' fourteen', ' fifteen', ' sixteen', ' seventeen', ' eighteen', ' nineteen');
    var tens = new Array('', '', ' twenty', ' thirty', ' forty', ' fifty', ' sixty', ' seventy', ' eighty', ' ninety');
    var hundred = ' hundred';
    var output = '';
    var numString = num.toString();

    if (num == 0) {
        return 'dontAddBigSufix';
    }
    //the case of 10, 11, 12 ,13, .... 19
    if (num < 20) {
        output = ones[num];
        return output;
    }

    //100 and more
    if (numString.length == 3) {
        output = ones[parseInt(numString.charAt(0))] + hundred;
        output += tens[parseInt(numString.charAt(1))];
        output += ones[parseInt(numString.charAt(2))];
        return output;
    }

    output += tens[parseInt(numString.charAt(0))];
    output += ones[parseInt(numString.charAt(1))];

    return output;
}

<input type="text"
    id="number"
    size="70"
    onkeyup="update();"
    /*this code prevent non numeric letters*/
    onkeydown="return (event.ctrlKey || event.altKey
                    || (47<event.keyCode && event.keyCode<58 && event.shiftKey==false)
                    || (95<event.keyCode && event.keyCode<106)
                    || (event.keyCode==8) || (event.keyCode==9)
                    || (event.keyCode>34 && event.keyCode<40)
                    || (event.keyCode==46) )"/>

Here The Numbers Printed


来源: stackexchange

问:
Getting the zoom ratio in ggsave with transparent background
I want to create some plots with transparent background so I can use them in a presentation without overlapping other elements. I did some research and found a way to create a transparent plot using the ggsave function. This worked well until I wanted to save a plot that have labels overlapping. What I want to do have is the zoomed version of the plot with transparent background. I couldn't find a solution for this problem. Can someone help me with this problem?

This is the plot with transparent background, but with overlapping labels:
Plot1

This is the zoomed version with the labels in the correct location: Plot2

I want to have plot 2 with transparent backgorund.

I used the following code getting plot 1:

```<code>branche <- dataset %>%
  filter(!is.na(branche)) %>%
  ggplot(aes(x= "", fill = branche)) +
  geom_bar(stat= "count", width = 1, color = "white") +
  geom_text(aes(label = scales::percent(..count.. / sum(..count..))),
            stat = "count", position = position_stack(vjust = 0.5), size = 10) +
  coord_polar("y", start = 0, direction = -1) +
  labs(fill ="") +
  scale_fill_manual(labels = c("Industri", "Tjenesteytelse", "Handel", "Bygg og anlegg"),
                    values = c("dodgerblue", "steelblue1", "seagreen", "goldenrod4")) +
  theme_void() +
  theme(legend.text = element_text(size = 20),
        plot.background = element_rect(fill = "transparent", color = NA))

branche

ggsave(branche, filename = "branche.png", bg = "transparent")
</code>```

Greetings

答:
Saving files in R is linked to the resolution and aspect ratio of your graphics device.  To try this out and see for yourself:

create your plot again and use ggsave() to save it as a .png file.  Make a note of the dimensions of the saved image (you should see a notice like Saving 8.24 x 5.07 in image).

resize the window of the plot.

type the same ggsave() command and note that the image dimensions are now different.

Interesting to note: text doesn't exactly size the same way that other graphics do, so when adjusting the width and height, your text will seem to stay pretty much the same but everything else will seem to grow/shrink.

So I would say that the fix is to try different aspect ratios/image dimensions.  Rather than resizing and saving, I would suggest you can pass different arguments to ggsave() through height= and width=.  So something like specifying:

```<code>ggsave(branche, filename = "branche1.png", bg = "transparent", width=5, height=8)
ggsave(branche, filename = "branche2.png", bg = "transparent", width=6, height=9)
... etc
</code>```

You should be able to get your transparent one to look right.  I can't help you much further without sharing the dataset and the code you also used to generate the non-transparent plot.  If the ggplot() code for the non-transparent plot is identical to the transparent one, then it's likely you had resized the window between the two and that's why you're seeing the different ratios.

ContentLicense: CC BY-SA 4.0

Body:
I want to create some plots with transparent background so I can use them in a presentation without overlapping other elements. I did some research and found a way to create a transparent plot using the ggsave function. This worked well until I wanted to save a plot that have labels overlapping. What I want to do have is the zoomed version of the plot with transparent background. I couldn't find a solution for this problem. Can someone help me with this problem?

This is the plot with transparent background, but with overlapping labels:
Plot1

This is the zoomed version with the labels in the correct location: Plot2

I want to have plot 2 with transparent backgorund.

I used the following code getting plot 1:

branche <- dataset %>%
  filter(!is.na(branche)) %>%
  ggplot(aes(x= "", fill = branche)) +
  geom_bar(stat= "count", width = 1, color = "white") +
  geom_text(aes(label = scales::percent(..count.. / sum(..count..))),
            stat = "count", position = position_stack(vjust = 0.5), size = 10) +
  coord_polar("y", start = 0, direction = -1) +
  labs(fill ="") +
  scale_fill_manual(labels = c("Industri", "Tjenesteytelse", "Handel", "Bygg og anlegg"),
                    values = c("dodgerblue", "steelblue1", "seagreen", "goldenrod4")) +
  theme_void() +
  theme(legend.text = element_text(size = 20),
        plot.background = element_rect(fill = "transparent", color = NA))

branche

ggsave(branche, filename = "branche.png", bg = "transparent")

Greetings

Body:
Saving files in R is linked to the resolution and aspect ratio of your graphics device.  To try this out and see for yourself:

create your plot again and use ggsave() to save it as a .png file.  Make a note of the dimensions of the saved image (you should see a notice like Saving 8.24 x 5.07 in image).

resize the window of the plot.

type the same ggsave() command and note that the image dimensions are now different.

Interesting to note: text doesn't exactly size the same way that other graphics do, so when adjusting the width and height, your text will seem to stay pretty much the same but everything else will seem to grow/shrink.

So I would say that the fix is to try different aspect ratios/image dimensions.  Rather than resizing and saving, I would suggest you can pass different arguments to ggsave() through height= and width=.  So something like specifying:

ggsave(branche, filename = "branche1.png", bg = "transparent", width=5, height=8)
ggsave(branche, filename = "branche2.png", bg = "transparent", width=6, height=9)
... etc

You should be able to get your transparent one to look right.  I can't help you much further without sharing the dataset and the code you also used to generate the non-transparent plot.  If the ggplot() code for the non-transparent plot is identical to the transparent one, then it's likely you had resized the window between the two and that's why you're seeing the different ratios.


问:
ProGuard for Android and GSON
I'm setting up ProGuard for my Android project. My project also uses GSON.

I've researched ProGuard configurations for compatibility with GSON and Android and came across this example offered by google-gson https://code.google.com/p/google-gson/source/browse/trunk/examples/android-proguard-example/proguard.cfg.

ProGuard config copied below:

```<code>##---------------Begin: proguard configuration common for all Android apps ----------
-optimizationpasses 5
-dontusemixedcaseclassnames
-dontskipnonpubliclibraryclasses
-dontskipnonpubliclibraryclassmembers
-dontpreverify
-verbose
-dump class_files.txt
-printseeds seeds.txt
-printusage unused.txt
-printmapping mapping.txt
-optimizations !code/simplification/arithmetic,!field/*,!class/merging/*

-allowaccessmodification
-keepattributes *Annotation*
-renamesourcefileattribute SourceFile
-keepattributes SourceFile,LineNumberTable
-repackageclasses ''

-keep public class * extends android.app.Activity
-keep public class * extends android.app.Application
-keep public class * extends android.app.Service
-keep public class * extends android.content.BroadcastReceiver
-keep public class * extends android.content.ContentProvider
-keep public class * extends android.app.backup.BackupAgentHelper
-keep public class * extends android.preference.Preference
-keep public class com.android.vending.licensing.ILicensingService
-dontnote com.android.vending.licensing.ILicensingService

# Explicitly preserve all serialization members. The Serializable interface
# is only a marker interface, so it wouldn't save them.
-keepclassmembers class * implements java.io.Serializable {
    static final long serialVersionUID;
    private static final java.io.ObjectStreamField[] serialPersistentFields;
    private void writeObject(java.io.ObjectOutputStream);
    private void readObject(java.io.ObjectInputStream);
    java.lang.Object writeReplace();
    java.lang.Object readResolve();
}

# Preserve all native method names and the names of their classes.
-keepclasseswithmembernames class * {
    native <methods>;
}

-keepclasseswithmembernames class * {
    public <init>(android.content.Context, android.util.AttributeSet);
}

-keepclasseswithmembernames class * {
    public <init>(android.content.Context, android.util.AttributeSet, int);
}

# Preserve static fields of inner classes of R classes that might be accessed
# through introspection.
-keepclassmembers class **.R$* {
  public static <fields>;
}

# Preserve the special static methods that are required in all enumeration classes.
-keepclassmembers enum * {
    public static **[] values();
    public static ** valueOf(java.lang.String);
}

-keep public class * {
    public protected *;
}

-keep class * implements android.os.Parcelable {
  public static final android.os.Parcelable$Creator *;
}
##---------------End: proguard configuration common for all Android apps ----------

##---------------Begin: proguard configuration for Gson  ----------
# Gson uses generic type information stored in a class file when working with fields. Proguard
# removes such information by default, so configure it to keep all of it.
-keepattributes Signature

# For using GSON @Expose annotation
-keepattributes *Annotation*

# Gson specific classes
-keep class sun.misc.Unsafe { *; }
#-keep class com.google.gson.stream.** { *; }

# Application classes that will be serialized/deserialized over Gson
-keep class com.google.gson.examples.android.model.** { *; }

##---------------End: proguard configuration for Gson  ----------
</code>```

Questions:

I see that this file has not been updated since 2011, is it still recommended for use? I ask because Android/GSON has changed quite a bit since then so I don't know how much of the above in unnecessary or incorrect.

If this is not recommended, is there a new recommended ProGuard configuration for GSON in Android?

答:
In my case, I just used GSON for deserializing JSON to an Object. So it was enough to add the following line to proguard file.</p>

```<code>-keep class your.data.object.modals.package.** { <fields>; }
</code>```

ContentLicense: CC BY-SA 3.0

Title: ProGuard for Android and GSON

Body:
I'm setting up ProGuard for my Android project. My project also uses GSON.

I've researched ProGuard configurations for compatibility with GSON and Android and came across this example offered by google-gson https://code.google.com/p/google-gson/source/browse/trunk/examples/android-proguard-example/proguard.cfg.

ProGuard config copied below:

##---------------Begin: proguard configuration common for all Android apps ----------
-optimizationpasses 5
-dontusemixedcaseclassnames
-dontskipnonpubliclibraryclasses
-dontskipnonpubliclibraryclassmembers
-dontpreverify
-verbose
-dump class_files.txt
-printseeds seeds.txt
-printusage unused.txt
-printmapping mapping.txt
-optimizations !code/simplification/arithmetic,!field/*,!class/merging/*

-allowaccessmodification
-keepattributes *Annotation*
-renamesourcefileattribute SourceFile
-keepattributes SourceFile,LineNumberTable
-repackageclasses ''

-keep public class * extends android.app.Activity
-keep public class * extends android.app.Application
-keep public class * extends android.app.Service
-keep public class * extends android.content.BroadcastReceiver
-keep public class * extends android.content.ContentProvider
-keep public class * extends android.app.backup.BackupAgentHelper
-keep public class * extends android.preference.Preference
-keep public class com.android.vending.licensing.ILicensingService
-dontnote com.android.vending.licensing.ILicensingService

# Explicitly preserve all serialization members. The Serializable interface
# is only a marker interface, so it wouldn't save them.
-keepclassmembers class * implements java.io.Serializable {
    static final long serialVersionUID;
    private static final java.io.ObjectStreamField[] serialPersistentFields;
    private void writeObject(java.io.ObjectOutputStream);
    private void readObject(java.io.ObjectInputStream);
    java.lang.Object writeReplace();
    java.lang.Object readResolve();
}

# Preserve all native method names and the names of their classes.
-keepclasseswithmembernames class * {
    native <methods>;
}

-keepclasseswithmembernames class * {
    public <init>(android.content.Context, android.util.AttributeSet);
}

-keepclasseswithmembernames class * {
    public <init>(android.content.Context, android.util.AttributeSet, int);
}

# Preserve static fields of inner classes of R classes that might be accessed
# through introspection.
-keepclassmembers class **.R$* {
  public static <fields>;
}

# Preserve the special static methods that are required in all enumeration classes.
-keepclassmembers enum * {
    public static **[] values();
    public static ** valueOf(java.lang.String);
}

-keep public class * {
    public protected *;
}

-keep class * implements android.os.Parcelable {
  public static final android.os.Parcelable$Creator *;
}
##---------------End: proguard configuration common for all Android apps ----------

##---------------Begin: proguard configuration for Gson  ----------
# Gson uses generic type information stored in a class file when working with fields. Proguard
# removes such information by default, so configure it to keep all of it.
-keepattributes Signature

# For using GSON @Expose annotation
-keepattributes *Annotation*

# Gson specific classes
-keep class sun.misc.Unsafe { *; }
#-keep class com.google.gson.stream.** { *; }

# Application classes that will be serialized/deserialized over Gson
-keep class com.google.gson.examples.android.model.** { *; }

##---------------End: proguard configuration for Gson  ----------

Questions:

I see that this file has not been updated since 2011, is it still recommended for use? I ask because Android/GSON has changed quite a bit since then so I don't know how much of the above in unnecessary or incorrect.

If this is not recommended, is there a new recommended ProGuard configuration for GSON in Android?

Body:
In my case, I just used GSON for deserializing JSON to an Object. So it was enough to add the following line to proguard file.

-keep class your.data.object.modals.package.** { <fields>; }


问:
ExtJS 7 Force Froala editor to fit parent component size
Is it possible to configure Froala editor so that it always matches the size of the parent component in height and width?

Currently it goes beyond the boundaries of the parent component as the amount of text increases, not matter what layout I am using.

I have checked the list of available options and examples, but did not find an appropriate settings. The only option I can use now is explicitly set height and width for a Ext.froala.Mixin.editor config.

I am using ExtJS 7.2, modern toolkit.

Fiddle

I expect a scroll inside a Froala editor component, but instead a Froala editor component exceed height of the parent component.

答:
Set overflow: auto; for flora wrapper:

```<code>    .fr-box.fr-basic.fr-top .fr-wrapper {
        overflow: auto;
    }
</code>```

Fiddle

Body:
Set overflow: auto; for flora wrapper:

    .fr-box.fr-basic.fr-top .fr-wrapper {
        overflow: auto;
    }

Fiddle


问:
How do I display this data in my columns?
I have a problem with VueJS. I have a set of a lot of data. I want it to appear in my columns by order. How do I insert the code properly? Thanks for help.

```<code>[
  {
    "sort": 0,
    "title": "My title",
    "description": "My description"
  },
  {
    "sort": 1,
    "title": "My Title1",
    "description": "My description"
  },
  {
    "sort": 2,
    "title": "My title2",
    "description": "My description"
  },
]
</code>```

I want it to go in here by order

```<code><div class="row pt-md-5 mt-md-5 mb-5 p-4">
  <div class="col-xl-6 mx-auto">
    <div class="card p-3">
      <h3>Want the title here</h3>
      <p>Description here </p>
    </div>
  </div>
  <div class="col-xl-6 mx-auto">
    <div class="card p-3">
      <h3>Want the title here</h3>
      <p>Description here </p>
    </div>
  </div>
</div>
</div>
</code>```

答:
With Vue you can use the v-for directive to iterate over your objects and display them how you want:

```<code>  <div
    v-for="(object, index) in array"
    :key="index"
    class="col-xl-6 mx-auto"
  >
    <div class="card p-3">
      <h3>{{ object.title }}</h3>
      <p>{{ object.description }}</p>
    </div>
  </div>
</code>```

In the v-for, object can be named however you like, array has to be the name of your array, index is just the index of each object within that array, then, inside the v-for, you can access each objects properties using templates, i.e. {{ object.title }}.

Body:
I have a problem with VueJS. I have a set of a lot of data. I want it to appear in my columns by order. How do I insert the code properly? Thanks for help.

[
  {
    "sort": 0,
    "title": "My title",
    "description": "My description"
  },
  {
    "sort": 1,
    "title": "My Title1",
    "description": "My description"
  },
  {
    "sort": 2,
    "title": "My title2",
    "description": "My description"
  },
]

I want it to go in here by order

Want the title here

Description here

Want the title here

Description here

Body:
With Vue you can use the v-for directive to iterate over your objects and display them how you want:

  <div
    v-for="(object, index) in array"
    :key="index"
    class="col-xl-6 mx-auto"
  >
    <div class="card p-3">
      <h3>{{ object.title }}</h3>
      <p>{{ object.description }}</p>
    </div>
  </div>

In the v-for, object can be named however you like, array has to be the name of your array, index is just the index of each object within that array, then, inside the v-for, you can access each objects properties using templates, i.e. {{ object.title }}.


问:
Access to a Struct inside an Union
I'm working on an external libraries that has the following code in C:

```<code>struct ndpi_flow_struct {

....

  union {
    struct ndpi_flow_tcp_struct tcp;
    struct ndpi_flow_udp_struct udp;
  } l4;

  struct ndpi_id_struct *server_id;
  /* HTTP host or DNS query */
  u_char host_server_name[256];

  struct {
    ndpi_http_method method;
    char *url, *content_type;
    u_int8_t  num_request_headers, num_response_headers;
    u_int8_t  request_version; /* 0=1.0 and 1=1.1. Create an enum for this? */
    u_char response_status_code[5]; /* 200, 404, etc. */
  } http;

  union {
    /* the only fields useful for nDPI and ntopng */
    struct {
      u_int8_t num_queries, num_answers, reply_code;
      u_int16_t query_type, query_class, rsp_type;
    } dns;

    struct {
      u_int8_t request_code;
      u_int8_t version;
    } ntp;

    struct {
      char client_certificate[48], server_certificate[48];
    } ssl;

    struct {
      char client_signature[48], server_signature[48];
    } ssh;

    struct {
      char answer[96];
    } mdns;

    struct {
      char version[96];
    } ubntac2;

    struct {
      /* Via HTTP User-Agent */
      u_char detected_os[32];
      /* Via HTTP X-Forwarded-For */
      u_char nat_ip[24];
    } http;

    struct {
      /* Bittorrent hash */
      u_char hash[20];
    } bittorrent;

    struct {
      char fingerprint[48];
      char class_ident[48];
    } dhcp;
  } protos;
</code>```

And when I do:

```<code>if(flow->ndpi_flow!=NULL && flow->ndpi_flow->detected_os!=NULL){
        snprintf(text[c], textSize, "%s", flow->ndpi_flow->detected_os);
</code>```

The compiler (gcc 4.9.2) said:

error: ‘struct ndpi_flow_struct’ has no member named ‘detected_os’

I read on the web that C11 should be read the value of a struct field inside an union in this way.
Thanks in advance!

答: detected_os is a member of http, which is a member of protos, so you need flow->ndpi_flow->protos.http.detected_os.

Title: Access to a Struct inside an Union

Body:
I'm working on an external libraries that has the following code in C:

struct ndpi_flow_struct {

....

  union {
    struct ndpi_flow_tcp_struct tcp;
    struct ndpi_flow_udp_struct udp;
  } l4;

  struct ndpi_id_struct *server_id;
  /* HTTP host or DNS query */
  u_char host_server_name[256];

  struct {
    ndpi_http_method method;
    char *url, *content_type;
    u_int8_t  num_request_headers, num_response_headers;
    u_int8_t  request_version; /* 0=1.0 and 1=1.1. Create an enum for this? */
    u_char response_status_code[5]; /* 200, 404, etc. */
  } http;

  union {
    /* the only fields useful for nDPI and ntopng */
    struct {
      u_int8_t num_queries, num_answers, reply_code;
      u_int16_t query_type, query_class, rsp_type;
    } dns;

    struct {
      u_int8_t request_code;
      u_int8_t version;
    } ntp;

    struct {
      char client_certificate[48], server_certificate[48];
    } ssl;

    struct {
      char client_signature[48], server_signature[48];
    } ssh;

    struct {
      char answer[96];
    } mdns;

    struct {
      char version[96];
    } ubntac2;

    struct {
      /* Via HTTP User-Agent */
      u_char detected_os[32];
      /* Via HTTP X-Forwarded-For */
      u_char nat_ip[24];
    } http;

    struct {
      /* Bittorrent hash */
      u_char hash[20];
    } bittorrent;

    struct {
      char fingerprint[48];
      char class_ident[48];
    } dhcp;
  } protos;

And when I do:

if(flow->ndpi_flow!=NULL && flow->ndpi_flow->detected_os!=NULL){
        snprintf(text[c], textSize, "%s", flow->ndpi_flow->detected_os);

The compiler (gcc 4.9.2) said:

error: ‘struct ndpi_flow_struct’ has no member named ‘detected_os’

I read on the web that C11 should be read the value of a struct field inside an union in this way.
Thanks in advance!


问:
How do I check if something is an elem of a qw?
According to the documentation b c d e> and qw|a b c d e| are similar, and furthermore they both make lists, which means they should be equivalent to ('a', 'b', 'c', 'd', 'e').

Why then am I not able to do this: '2' (elem) <1 2 3 4 5>?

2 (elem) <1 2 3 4 5> doesn't work either, but 'a' (elem) b c d e> DOES.

What's going on? Shouldn't I be able to check the existence of the number like anything else?

答:
If you do :

```<code><1 2 3 4 5>.raku.say
</code>```

You'll see that it creates a list of IntStr objects (which are objects that can be either a String or an Integer). When we put these in a set we get a Set of IntStr objects.

The problem is (elem) doesn't use smart-matching but equivalence and neither the Int 1 or the String "1" are equivalent to IntStr.new(1, "1"). They do both smart-match with the IntStr...

You probably want to use first to find the needle :

```<code>defined <1 2 3 4 5>.first( 2, :k )
</code>```

first( $match, :k ) returns the index of the first matching item and Nil if there's no match. So we test to see if it's defined and if so our item exists.

Or you can cast your list to String ot Integers using a map.

```<code>2 (elem) <1 2 3 4 5>.map( *.Int )
</code>```

Body:
According to the documentation  and qw|a b c d e| are similar, and furthermore they both make lists, which means they should be equivalent to ('a', 'b', 'c', 'd', 'e').

Why then am I not able to do this: '2' (elem) <1 2 3 4 5>?

2 (elem) <1 2 3 4 5> doesn't work either, but 'a' (elem)  DOES.

What's going on? Shouldn't I be able to check the existence of the number like anything else?

Body:
If you do :

<1 2 3 4 5>.raku.say

You'll see that it creates a list of IntStr objects (which are objects that can be either a String or an Integer). When we put these in a set we get a Set of IntStr objects.

The problem is (elem) doesn't use smart-matching but equivalence and neither the Int 1 or the String "1" are equivalent to IntStr.new(1, "1"). They do both smart-match with the IntStr...

You probably want to use first to find the needle :

defined <1 2 3 4 5>.first( 2, :k )

first( $match, :k ) returns the index of the first matching item and Nil if there's no match. So we test to see if it's defined and if so our item exists.

Or you can cast your list to String ot Integers using a map.

2 (elem) <1 2 3 4 5>.map( *.Int )


问:
Species distribution model to produce psuedo-absence data using the randomPoints() function in the Dismo package in R: Error Message
Aim:

My aim is to build a species distribution prediction model using the function randomPoints() in the Dismo package with the utlimate aim of generating pseudo-absence points and plotting them on a map. These points will be converted into a rastor file in order to extract meta-data (i.e sea surface salinity, chlorophyll levels etc) from MODIS files to act as important ecological predictors to determine how they affect the distribution of blue whales.

The idea is to plug both presence and pseudo absence data with associated meta-data values into general linear mixed (GLM’s), which will ultimately make my models more balanced and accurate.

Problem Outline:

I am attempting to follow this species distribution exercise to generate pseudo-absence points (desired output: see image 1) using the randomPoints() function. However, after running my R-code (see below), I am experiencing this R error message (see below). My r-code also produces a map with the GPS points plotted (image 2).

Error Message:

Error in (function (classes, fdef, mtable)  :
  unable to find an inherited method for function ‘nlayers’ for signature ‘"standardGeneric"’

I have tried to find the solution. However, I am relatively new in regards to at working with maps in R, and I am feeling really confused as to what the problem is here!

My data frame contains 918 rows and I would like to produce the same number of pseudo-absence points as presence points. Unfortunately, I cannot publish my data but I have provided a mini data frame to use as an example.

If anyone can help me, I would be deeply appreciative!

Many thanks in advance!

R-code:

```<code> ###Open Packages

    library("sp")
    library("rgdal")
    library("raster")
    library("maptools")
    library("rgdal")
    library("dismo")
    library("spatialEco")
    library("ggplot2")
    library("dplyr")
    library("maps")
    library("ggspatial")
    library("GADMTools")
    library("maps")

    ##Mini Dataframe
    Blue.whale_New <- data.frame(longitude = c(80.5, 80.5, 80.5, 80.5, 80.4, 80.4, 80.5, 80.5, 80.4),
                             latitude = c(5.84, 5.82, 5.85, 5.85, 5.89, 5.82, 5.82, 5.84, 5.83))

    ####World Bioclim Data + GADM Object

    ##Creating an array object with just longitude and latitude decimal coordinates
    ##Upload the maps
    ##Plotting the map of Sri Lanka
    ###GADM OBJECT

    dev.new()
    bioclim1.data <- getData('GADM', country='LKA', level=1)

    #####Worldclim raster layers

    bioclim.data <- getData(name = "worldclim",
                            var = "bio",
                            res = 2.5,
                            path = "./")

    ####Get bounding box of Sri Lanka shape file
    bb=bioclim1.data@bbox

    # Determine geographic extent of our data
    max.lat <- ceiling(max(Blue.whale$latitude))
    min.lat <- floor(min(Blue.whale$latitude))
    max.lon <- ceiling(max(Blue.whale$longitude))
    min.lon <- floor(min(Blue.whale$longitude))
    geographic.extent <- extent(x = c(min.lon, max.lon, min.lat, max.lat))

    #####Plot map
    dev.new()

    plot(bioclim1.data,
         xlim = c(min(c(min.lon,bb[1,1])), max(c(max.lon,bb[1,2]))),
         ylim = c(min(c(min.lat,bb[2,1])), max(c(max.lat,bb[2,2]))),
         axes = TRUE,
         col = "grey95")

    # Add the points for individual observation
    points(x = Blue.whale$longitude,
           y = Blue.whale$latitude,
           col = "olivedrab",
           pch = 15,
           cex = 0.50)

    ###Building a model and visualising results

    ##Crop bioclim data to geographic extent of blue whales GADM Map
    bioclim.data.blue.whale_1<-crop(x=bioclim1.data, y=geographic.extent)

    ##Crop bioclim data to geographic extent of blue whales World Clim
    bioclim.data.blue.whale_2<-crop(x = bioclim.data, y = geographic.extent)

    #Build distribution mode using the World Clim)

    bw.model <- bioclim(x = bioclim.data, p = Blue.whale_New)

    # Predict presence from model
    predict.presence <- dismo::predict(object = bw.model, x = bioclim.data, ext = geographic.extent)

    # Plot base map

    dev.new()

    plot(bioclim1.data,
         xlim = c(min(c(min.lon,bb[1,1])), max(c(max.lon,bb[1,2]))),
         ylim = c(min(c(min.lat,bb[2,1])), max(c(max.lat,bb[2,2]))),
         axes = TRUE,
         col = "grey95")

    # Add model probabilities
    plot(predict.presence, add = TRUE)

    # Redraw those country borders
    plot(bioclim1.data, add = TRUE, border = "grey5")

    # Add original observations
    points(Blue.whale_New$longitude,
           Blue.whale_New$latitude,
           col = "olivedrab", pch = 20, cex = 0.75)

    ##Psuedo Absence Points

    # Use the bioclim data files for sampling resolution
    bil.files <- list.files(path = "data/wc2-5",
                            pattern = "*.bil$",
                            full.names = TRUE)

    # Randomly sample points (same number as our observed points)

    ##Mask = provides resolution of sampling points
    ##n = number of random points
    ##ext = Spatially restricts sampling
    ##extf = expands sampling a little bit

    background <- randomPoints(mask = mask,
                               n = nrow(Blue.whale_New),
                               ext = geographic.extent,
                               extf = 1.25)

#Error in (function (classes, fdef, mtable)  :
  unable to find an inherited method for function ‘nlayers’ for signature ‘"standardGeneric"’
</code>```

Code I plan to run once I have solved my problem

```<code>Visualize the pseudo-absence points on a map:

# Plot the base map

dev.new()

plot(bioclim1.data,
     xlim = c(min(c(min.lon,bb[1,1])), max(c(max.lon,bb[1,2]))),
     ylim = c(min(c(min.lat,bb[2,1])), max(c(max.lat,bb[2,2]))),
     axes = TRUE,
     col = "grey95")

# Add the background points
points(background, col = "grey30", pch = 1, cex = 0.75)

# Add the observations
points(x = Blue.whale_New$longitude,
       y = Blue.whale_New$latitude,
       col = "olivedrab",
       pch = 20,
       cex = 0.75)

box()

# Arbitrarily assign group 1 as the testing data group
testing.group <- 1

# Create vector of group memberships
group.presence <- kfold(x = Blue.whale_New, k = 5) # kfold is in dismo package
</code>```

Image 1 (Desired Output)

Image 2:

答:
I think your problem is that you are not passing a proper mask (e.g. a raster layer) to the randomPoints function, but rather you are passing the mask function itself, which is a standardGeneric, hence the error message.

You can generate a mask from the predictor raster you want to use and then pass it to the randomPoints function

```<code>mask_r <- bioclim1.data[[1]] #just one raterLayer
mask_r[!is.na(mask_r)] <- 1 #set all non-NA values to 1
background <- randomPoints(mask = mask_r, #note the proper mask layer
                           n = nrow(Blue.whale_New),
                           ext = geographic.extent,
                           extf = 1.25)
</code>```

I haven't checked, but this solution should work.

Best,

Emilio

Body:
Aim:

My aim is to build a species distribution prediction model using the function randomPoints() in the Dismo package with the utlimate aim of generating pseudo-absence points and plotting them on a map. These points will be converted into a rastor file in order to extract meta-data (i.e sea surface salinity, chlorophyll levels etc) from MODIS files to act as important ecological predictors to determine how they affect the distribution of blue whales.

The idea is to plug both presence and pseudo absence data with associated meta-data values into general linear mixed (GLM’s), which will ultimately make my models more balanced and accurate.

Problem Outline:

I am attempting to follow this species distribution exercise to generate pseudo-absence points (desired output: see image 1) using the randomPoints() function. However, after running my R-code (see below), I am experiencing this R error message (see below). My r-code also produces a map with the GPS points plotted (image 2).

Error Message:

Error in (function (classes, fdef, mtable)  :
  unable to find an inherited method for function ‘nlayers’ for signature ‘"standardGeneric"’

I have tried to find the solution. However, I am relatively new in regards to at working with maps in R, and I am feeling really confused as to what the problem is here!

My data frame contains 918 rows and I would like to produce the same number of pseudo-absence points as presence points. Unfortunately, I cannot publish my data but I have provided a mini data frame to use as an example.

If anyone can help me, I would be deeply appreciative!

Many thanks in advance!

R-code:

 ###Open Packages

    library("sp")
    library("rgdal")
    library("raster")
    library("maptools")
    library("rgdal")
    library("dismo")
    library("spatialEco")
    library("ggplot2")
    library("dplyr")
    library("maps")
    library("ggspatial")
    library("GADMTools")
    library("maps")

    ##Mini Dataframe
    Blue.whale_New <- data.frame(longitude = c(80.5, 80.5, 80.5, 80.5, 80.4, 80.4, 80.5, 80.5, 80.4),
                             latitude = c(5.84, 5.82, 5.85, 5.85, 5.89, 5.82, 5.82, 5.84, 5.83))

    ####World Bioclim Data + GADM Object

    ##Creating an array object with just longitude and latitude decimal coordinates
    ##Upload the maps
    ##Plotting the map of Sri Lanka
    ###GADM OBJECT

    dev.new()
    bioclim1.data <- getData('GADM', country='LKA', level=1)

    #####Worldclim raster layers

    bioclim.data <- getData(name = "worldclim",
                            var = "bio",
                            res = 2.5,
                            path = "./")

    ####Get bounding box of Sri Lanka shape file
    bb=bioclim1.data@bbox

    # Determine geographic extent of our data
    max.lat <- ceiling(max(Blue.whale$latitude))
    min.lat <- floor(min(Blue.whale$latitude))
    max.lon <- ceiling(max(Blue.whale$longitude))
    min.lon <- floor(min(Blue.whale$longitude))
    geographic.extent <- extent(x = c(min.lon, max.lon, min.lat, max.lat))

    #####Plot map
    dev.new()

    plot(bioclim1.data,
         xlim = c(min(c(min.lon,bb[1,1])), max(c(max.lon,bb[1,2]))),
         ylim = c(min(c(min.lat,bb[2,1])), max(c(max.lat,bb[2,2]))),
         axes = TRUE,
         col = "grey95")

    # Add the points for individual observation
    points(x = Blue.whale$longitude,
           y = Blue.whale$latitude,
           col = "olivedrab",
           pch = 15,
           cex = 0.50)

    ###Building a model and visualising results

    ##Crop bioclim data to geographic extent of blue whales GADM Map
    bioclim.data.blue.whale_1<-crop(x=bioclim1.data, y=geographic.extent)

    ##Crop bioclim data to geographic extent of blue whales World Clim
    bioclim.data.blue.whale_2<-crop(x = bioclim.data, y = geographic.extent)

    #Build distribution mode using the World Clim)

    bw.model <- bioclim(x = bioclim.data, p = Blue.whale_New)

    # Predict presence from model
    predict.presence <- dismo::predict(object = bw.model, x = bioclim.data, ext = geographic.extent)

    # Plot base map

    dev.new()

    plot(bioclim1.data,
         xlim = c(min(c(min.lon,bb[1,1])), max(c(max.lon,bb[1,2]))),
         ylim = c(min(c(min.lat,bb[2,1])), max(c(max.lat,bb[2,2]))),
         axes = TRUE,
         col = "grey95")

    # Add model probabilities
    plot(predict.presence, add = TRUE)

    # Redraw those country borders
    plot(bioclim1.data, add = TRUE, border = "grey5")

    # Add original observations
    points(Blue.whale_New$longitude,
           Blue.whale_New$latitude,
           col = "olivedrab", pch = 20, cex = 0.75)

    ##Psuedo Absence Points

    # Use the bioclim data files for sampling resolution
    bil.files <- list.files(path = "data/wc2-5",
                            pattern = "*.bil$",
                            full.names = TRUE)

    # Randomly sample points (same number as our observed points)

    ##Mask = provides resolution of sampling points
    ##n = number of random points
    ##ext = Spatially restricts sampling
    ##extf = expands sampling a little bit

    background <- randomPoints(mask = mask,
                               n = nrow(Blue.whale_New),
                               ext = geographic.extent,
                               extf = 1.25)

#Error in (function (classes, fdef, mtable)  :
  unable to find an inherited method for function ‘nlayers’ for signature ‘"standardGeneric"’

Code I plan to run once I have solved my problem

Visualize the pseudo-absence points on a map:

# Plot the base map

dev.new()

plot(bioclim1.data,
     xlim = c(min(c(min.lon,bb[1,1])), max(c(max.lon,bb[1,2]))),
     ylim = c(min(c(min.lat,bb[2,1])), max(c(max.lat,bb[2,2]))),
     axes = TRUE,
     col = "grey95")

# Add the background points
points(background, col = "grey30", pch = 1, cex = 0.75)

# Add the observations
points(x = Blue.whale_New$longitude,
       y = Blue.whale_New$latitude,
       col = "olivedrab",
       pch = 20,
       cex = 0.75)

box()

# Arbitrarily assign group 1 as the testing data group
testing.group <- 1

# Create vector of group memberships
group.presence <- kfold(x = Blue.whale_New, k = 5) # kfold is in dismo package

Image 1 (Desired Output)

Image 2:

Body:
I think your problem is that you are not passing a proper mask (e.g. a raster layer) to the randomPoints function, but rather you are passing the mask function itself, which is a standardGeneric, hence the error message.

You can generate a mask from the predictor raster you want to use and then pass it to the randomPoints function

mask_r <- bioclim1.data[[1]] #just one raterLayer
mask_r[!is.na(mask_r)] <- 1 #set all non-NA values to 1
background <- randomPoints(mask = mask_r, #note the proper mask layer
                           n = nrow(Blue.whale_New),
                           ext = geographic.extent,
                           extf = 1.25)

I haven't checked, but this solution should work.

Best,

Emilio


问:
Partition structure for 2 kafka consumers on same topic
If I create

2 kafka consumer instances

passing same properties

subscribe on same topic

Will these 2 Consumer instances (at diff group Id), have similar partition structures, or could be different ?

i.e, if I do .assignment() will I get same result at both

My actual problem statement, where I will be using this validation

In my application, I am attaining offset of broker, at a particular state(This is being done through my 1st kafka consumer object).

Later, I am creating the 2nd kafka consumer object, and using this to iterate over the topic, seeking from the earlier offset attained.

(So if the supposition mentioned at question is false, my logic will fail)

答:
Let's say you have 10 partitions in the topic for which you are doing subscribe after creating the consumer object. When you create the first consumer object and start to perform the poll operation, All the 10 partitions will be assigned to this consumer object.

When you create the second consumer object, the consumer group co-ordinator would realize that some other consumer has joined in the group and hence a rebalance would trigger. Depending on the partition assignment strategy used, some of the partitions would be assigned to the second consumer. In the default case, 5 of the partition would be taken from the first consumer and assigned to the second consumer. Now each consumer will have 5 partitions each.

So the partition structure would change after creating the second consumer and after the poll operation from the second consumer.

EDIT1:
If you do the .assignment() on both the consumer, after you have started to consume from both the consumer. You will get the same result.

EDIT2:: If you have two different consumer group id, and if there is only 1 consumer in both the group, then yeah the partition structure would be the same.

If you have multiple consumers under a consumer group, but both of them have the same number of consumers in the consumer group(say 3 consumer in each consumer group) and the consumer partition strategy used is also the same, then the partition structure would be same.

If you have multiple numbers of consumers in both the group but different numbers in each consumer group(say first consumer group has 2 consumers and second consumer group has 3 consumers), then as you can guess the partition structure would be different.


问:
Reverse for '<WSGIRequest: POST '/signup/'> error on Signup page
I am trying to create a user account and all is well when I use the signup view and form to sign up on the signup HTML page below is some info but I get this error

NoReverseMatch at /signup/ Reverse for '' not found. '' is not a
  valid view function or pattern name. Request Method:  POST Request
  URL:  http://127.0.0.1:8000/signup/ Django Version:   2.1.7 Exception
  Type: NoReverseMatch Exception Value:  Reverse for '' not found. '' is not a
  valid view function or pattern name. Exception
  Location: C:\WINDOWS\system32\dev\pastebinclonedirectory\lib\site-packages\django\urls\resolvers.py
  in _reverse_with_prefix, line 622 Python
  Executable:   C:\WINDOWS\system32\dev\pastebinclonedirectory\Scripts\python.exe
  Python Version:   3.6.8 Python Path:

  ['C:\WINDOWS\system32\dev\pastebinclonedirectory\pastebin',
  'C:\WINDOWS\system32\dev\pastebinclonedirectory\Scripts\python36.zip',
  'C:\WINDOWS\system32\dev\pastebinclonedirectory\DLLs',
  'C:\WINDOWS\system32\dev\pastebinclonedirectory\lib',
  'C:\WINDOWS\system32\dev\pastebinclonedirectory\Scripts',
  'c:\users\aadeo\appdata\local\programs\python\python36\Lib',
  'c:\users\aadeo\appdata\local\programs\python\python36\DLLs',
  'C:\WINDOWS\system32\dev\pastebinclonedirectory',
  'C:\WINDOWS\system32\dev\pastebinclonedirectory\lib\site-packages']
  Server time:  Wed, 13 Mar 2019 06:46:57 +0000

view.py

```<code>def signup(request):
    if request.method == 'POST':
        form = UserRegisterForm(request.POST)
        if form.is_valid():
            form.save()
            username = form.cleaned_data.get('username')
            raw_password = form.cleaned_data.get('password1')
            user = authenticate(username=username, password=raw_password)
            login(request, user)
            return redirect(request,'app/index.html')
    else:
        form = UserRegisterForm()
    return render(request, 'app/signup.html', {'form': form})
</code>```

forms.py

```<code>from django import forms
from .models import Post
from django.contrib.auth.forms import UserCreationForm
from django.contrib.auth.models import User

class Userpostform(forms.ModelForm):
    class Meta:
        model = Post
        fields = ('title','content','private',)

class Visitorpostform(forms.ModelForm):
    class Meta:
        model = Post
        fields = ('title','content')
</code>```

urls.py in the apps folder

```<code>path('',views.home,name='home'),
path('accounts/', include('django.contrib.auth.urls'),name='login'),
path('signup/',views.signup,name='signup'),
path('accounts/profile/',views.profile,name='profile'),
path('logout',views.logout_view,name='logout'),
path('userpostnew', views.userpostnew, name='userpostnew'),
path('visitorpostnew', views.visitorpostnew, name='visitorpostnew'),
</code>```

答:
Remove the request argument inside the redirect function.

The redirect function is defined as: redirect(to, *args, permanent=False, **kwargs):

It has no request parameter.

Use the name or route of the path instead of the template name.

In your case: 'home' or '/' instead of 'app/index.html'

def signup(request):
    ...
    if form.is_valid():
       ...
       return redirect('home')
       # or return redirect ('/')
```

Body:
I am trying to create a user account and all is well when I use the signup view and form to sign up on the signup HTML page below is some info but I get this error

NoReverseMatch at /signup/ Reverse for '' not found. '' is not a
  valid view function or pattern name. Request Method:  POST Request
  URL:  http://127.0.0.1:8000/signup/ Django Version:   2.1.7 Exception
  Type: NoReverseMatch Exception Value:  Reverse for '' not found. '' is not a
  valid view function or pattern name. Exception
  Location: C:\WINDOWS\system32\dev\pastebinclonedirectory\lib\site-packages\django\urls\resolvers.py
  in _reverse_with_prefix, line 622 Python
  Executable:   C:\WINDOWS\system32\dev\pastebinclonedirectory\Scripts\python.exe
  Python Version:   3.6.8 Python Path:

  ['C:\WINDOWS\system32\dev\pastebinclonedirectory\pastebin',
  'C:\WINDOWS\system32\dev\pastebinclonedirectory\Scripts\python36.zip',
  'C:\WINDOWS\system32\dev\pastebinclonedirectory\DLLs',
  'C:\WINDOWS\system32\dev\pastebinclonedirectory\lib',
  'C:\WINDOWS\system32\dev\pastebinclonedirectory\Scripts',
  'c:\users\aadeo\appdata\local\programs\python\python36\Lib',
  'c:\users\aadeo\appdata\local\programs\python\python36\DLLs',
  'C:\WINDOWS\system32\dev\pastebinclonedirectory',
  'C:\WINDOWS\system32\dev\pastebinclonedirectory\lib\site-packages']
  Server time:  Wed, 13 Mar 2019 06:46:57 +0000

view.py

def signup(request):
    if request.method == 'POST':
        form = UserRegisterForm(request.POST)
        if form.is_valid():
            form.save()
            username = form.cleaned_data.get('username')
            raw_password = form.cleaned_data.get('password1')
            user = authenticate(username=username, password=raw_password)
            login(request, user)
            return redirect(request,'app/index.html')
    else:
        form = UserRegisterForm()
    return render(request, 'app/signup.html', {'form': form})

forms.py

from django import forms
from .models import Post
from django.contrib.auth.forms import UserCreationForm
from django.contrib.auth.models import User

class Userpostform(forms.ModelForm):
    class Meta:
        model = Post
        fields = ('title','content','private',)

class Visitorpostform(forms.ModelForm):
    class Meta:
        model = Post
        fields = ('title','content')

urls.py in the apps folder

path('',views.home,name='home'),
path('accounts/', include('django.contrib.auth.urls'),name='login'),
path('signup/',views.signup,name='signup'),
path('accounts/profile/',views.profile,name='profile'),
path('logout',views.logout_view,name='logout'),
path('userpostnew', views.userpostnew, name='userpostnew'),
path('visitorpostnew', views.visitorpostnew, name='visitorpostnew'),


问:
ASP - how to remove single quote from user input text
```<code>answer  = Request.Form("Text" & i)
</code>```

In a form a user inputs random text which is inserted into a database. Currently if the user puts in single quotes it creates an error. How do i remove just single quotes' from the users answer?

答:
Use

```<code>answer.Replace("\'", "");
</code>```

The above will replace the Single quote.

Body:
answer  = Request.Form("Text" & i)

In a form a user inputs random text which is inserted into a database. Currently if the user puts in single quotes it creates an error. How do i remove just single quotes' from the users answer?

Body:
Use

answer.Replace("\'", "");

The above will replace the Single quote.


问:
Partition by getting the previous date in the same partition
I have the following table with output including the result of my sql query that is not giving me the output I would like to see (I am doing this is in pyspark, so the hive version of sql):

Here is the query I am running (I have tried a bunch of variations of this with range/row and different preceding/following unbounded etc none of which seem to give me the output I want):

```<code>select id, date_1, acct_num, date_2, txn_am,
min(date_1) OVER(PARTITION BY acct_num
ORDER BY date_1 ASC RANGE BETWEEN 2 PRECEDING AND CURRENT ROW) as lower_bound_timestamp
from abc.def
order by acct_num, date_2
</code>```

This is what I want the output to look (in this query I am just creating the lower_bound_timestamp column). If you notice the key thing I want is the lower_bound_timestamp
not to be necessarily the value of date_1 in the preceding row, I want it to be the
preceding value of the date_1  (it is ordered by date asc) because if there are
say 5 rows with the same date followed by 3 rows with the different next date, I want
those 3 next rows to have a lower_bound_timestamp of THAT previous date_1 only however my
queries the many variants I try none of them accomplish it:

```<code>id        date_1     acct_num   date_2      txn_am        lower_bound_timestamp

123       2019-06-26   123      2019-06-10   10           2019-06-26
354       2019-06-26   123      2019-06-15   10           2019-06-26
357       2019-06-28   123      2019-06-15   10           2019-06-26
359       2019-06-30   123      2019-06-17   10           2019-06-28
360       2019-07-05   123      2019-06-18   10           2019-06-30
361       2019-07-05   123      2019-06-20   10           2019-06-30
</code>```

答:
You can use range(), but with numbers:

```<code>select id, date_1, acct_num, date_2, txn_am,
       min(date_1) over (partition by acct_num
                         order by datediff(date_1, '2000-01-01')
                         ranged between unbounded preceding and 1 preceding
                        ) as prev_date
from abc.def
order by acct_num, date_2
</code>```

Body:
I have the following table with output including the result of my sql query that is not giving me the output I would like to see (I am doing this is in pyspark, so the hive version of sql):

Here is the query I am running (I have tried a bunch of variations of this with range/row and different preceding/following unbounded etc none of which seem to give me the output I want):

select id, date_1, acct_num, date_2, txn_am,
min(date_1) OVER(PARTITION BY acct_num
ORDER BY date_1 ASC RANGE BETWEEN 2 PRECEDING AND CURRENT ROW) as lower_bound_timestamp
from abc.def
order by acct_num, date_2

This is what I want the output to look (in this query I am just creating the lower_bound_timestamp column). If you notice the key thing I want is the lower_bound_timestamp
not to be necessarily the value of date_1 in the preceding row, I want it to be the
preceding value of the date_1  (it is ordered by date asc) because if there are
say 5 rows with the same date followed by 3 rows with the different next date, I want
those 3 next rows to have a lower_bound_timestamp of THAT previous date_1 only however my
queries the many variants I try none of them accomplish it:

id        date_1     acct_num   date_2      txn_am        lower_bound_timestamp

123       2019-06-26   123      2019-06-10   10           2019-06-26
354       2019-06-26   123      2019-06-15   10           2019-06-26
357       2019-06-28   123      2019-06-15   10           2019-06-26
359       2019-06-30   123      2019-06-17   10           2019-06-28
360       2019-07-05   123      2019-06-18   10           2019-06-30
361       2019-07-05   123      2019-06-20   10           2019-06-30

Body:
You can use range(), but with numbers:

select id, date_1, acct_num, date_2, txn_am,
       min(date_1) over (partition by acct_num
                         order by datediff(date_1, '2000-01-01')
                         ranged between unbounded preceding and 1 preceding
                        ) as prev_date
from abc.def
order by acct_num, date_2


问:
Given a, b, c and alpha find x
I was doing image processing to determine the distance between two points in a picture. it involves a fair amount of geometry. One of the problems which I tried to solve using basic geometry but failed to arrive at a solution is the following. I have transformed the question into mathematical terms so that a wider audience could answer it.

The sides a, b, c, and the angle alpha are given. The length x is to be found
Using sine and cosine laws I`ve found:
Using Cosine Law and,
Using Sine Law

where beta is the angle opposite the side b

答:
This is not a trivial problem, and maybe it should have been asked Math.SE.

But here is my take:

Considering the triangle abx

```<code>b^2 = x^2 + a^2 - 2*a*x*cos(β)        #1
</code>```

And the triangle a1cx

```<code>c^2 = x^2 + a1^2 -2*a1*x*cos(β)       #2
sin(α)/α1 = sin(β)/c                  #3
</code>```

Three non-linear equations to be solved for x, a1 and β.

Subtract #2 from #1 to eliminate x^2 (with some simplifications)

```<code>b^2 - c^2  = -2*x *(a-a1)*cos(β)+a^2 -a1^2        #4
</code>```

use #3 to eliminate β in terms of a1 in #4

```<code>b^2 - c^2 = -2*x*(a-a1)*sqrt(1 - c^2/a1^2*sin(α)^2)+a^2-a1^2  #5
</code>```

Now subtract (a/a1)*#2 from #1 to eliminate a1^2

```<code>b^2 - a*c^2/a1 = -(a-a1)*(x^2-a*a1)/a1            #6
</code>```

Equations #5 and #6 are two non-linear equations to be solved for x and a1.

From #5 we have x in terms of a1 with

```<code>x = a1*(a^2-a1^2-b^2+c^2)/(2*(a-a1)*sqrt(a1^2-c^2*sin(α)^2))    #7
</code>```

Unfortunately using the above in #6 results in a sixth order polynomial to be solved for a1.

It can only be solved numerically at this point. If a1 is found, then #7 also gives us x.

```<code>0 = 4*a^2*c^2*g^2
+ a1*(4*a*g^2*(a^2-b^2-c^2))
+ a1^2*(a^4-2a^2(b^2+c^2+4g^2)+b^4+2b^2(2g^2-c^2)+c^4)
+ a1^3*(-4a(a^2-b^2-c^2-g^2))
+ a1^4*(2(3a^2-b^2-c^2))
+ a1^5*(-4*a)
+ a1^6
</code>```

where g = c*sin(α)

Title: Given a, b, c and alpha find x

Body:
This is not a trivial problem, and maybe it should have been asked Math.SE.

But here is my take:

Considering the triangle abx

b^2 = x^2 + a^2 - 2*a*x*cos(β)        #1

And the triangle a1cx

c^2 = x^2 + a1^2 -2*a1*x*cos(β)       #2
sin(α)/α1 = sin(β)/c                  #3

Three non-linear equations to be solved for x, a1 and β.

Subtract #2 from #1 to eliminate x^2 (with some simplifications)

b^2 - c^2  = -2*x *(a-a1)*cos(β)+a^2 -a1^2        #4

use #3 to eliminate β in terms of a1 in #4

b^2 - c^2 = -2*x*(a-a1)*sqrt(1 - c^2/a1^2*sin(α)^2)+a^2-a1^2  #5

Now subtract (a/a1)*#2 from #1 to eliminate a1^2

b^2 - a*c^2/a1 = -(a-a1)*(x^2-a*a1)/a1            #6

Equations #5 and #6 are two non-linear equations to be solved for x and a1.

From #5 we have x in terms of a1 with

x = a1*(a^2-a1^2-b^2+c^2)/(2*(a-a1)*sqrt(a1^2-c^2*sin(α)^2))    #7

Unfortunately using the above in #6 results in a sixth order polynomial to be solved for a1.

It can only be solved numerically at this point. If a1 is found, then #7 also gives us x.

0 = 4*a^2*c^2*g^2
+ a1*(4*a*g^2*(a^2-b^2-c^2))
+ a1^2*(a^4-2a^2(b^2+c^2+4g^2)+b^4+2b^2(2g^2-c^2)+c^4)
+ a1^3*(-4a(a^2-b^2-c^2-g^2))
+ a1^4*(2(3a^2-b^2-c^2))
+ a1^5*(-4*a)
+ a1^6

where g = c*sin(α)


问:
Google Cloud App Engine - Django App using PostGreSQL loses SQL connection
I have been using PostGreSQL via Google Cloud App Engine for my Django Application. Lately, I have started seeing following issue :

```<code>...
File "/env/lib/python3.7/site-packages/django/utils/asyncio.py", line 26, in inner      return func(*args, **kwargs)
File "/env/lib/python3.7/site-packages/django/db/backends/base/base.py", line 220, in ensure_connection      self.connect()
File "/env/lib/python3.7/site-packages/django/db/utils.py", line 90, in __exit__      raise dj_exc_value.with_traceback(traceback) from exc_value
File "/env/lib/python3.7/site-packages/django/db/backends/base/base.py", line 220, in ensure_connection      self.connect()
File "/env/lib/python3.7/site-packages/django/utils/asyncio.py", line 26, in inner      return func(*args, **kwargs)
File "/env/lib/python3.7/site-packages/django/db/backends/base/base.py", line 197, in connect      self.connection = self.get_new_connection(conn_params)
File "/env/lib/python3.7/site-packages/django/utils/asyncio.py", line 26, in inner      return func(*args, **kwargs)
File "/env/lib/python3.7/site-packages/django/db/backends/postgresql/base.py", line 185, in get_new_connection      connection = Database.connect(**conn_params)
File "/env/lib/python3.7/site-packages/psycopg2/__init__.py", line 126, in connect      conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: FATAL:  remaining connection slots are reserved for non-replication superuser connections
</code>```

My database settings are as follow :

```<code>if os.getenv('GAE_APPLICATION', None):
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.postgresql_psycopg2',
            'HOST': '/cloudsql/MYAPP:europe-west2:MYAPP',
            'NAME': 'MYNAME',
            'USER': 'MYUSER',
            'PASSWORD': access_secret_version("MYSECRET_SQL_PASSWORD"),
            'PORT': '5432',
        }
    }
</code>```

What does this actually mean ?

答:
This basically means that you are reaching the concurrent connection limit to your instance and need to either increase the limit or close some connections that are not needed.

You can increase the connection limit by using the max_connections flag to configure it, however, it is recommended to set a limit that keeps the instance stable. You can also change your instance machine type whith higher memory as it offers a higher default concurrent connection limit.

Another solution would be to use db connection pooling with pgbouncer, as Django framework does not offer database connection pooling. In addition, setting CONN_MAX_AGE to zero would help in some cases, as it allows closing the database connection at the end of each request.

Body:
I have been using PostGreSQL via Google Cloud App Engine for my Django Application. Lately, I have started seeing following issue :

...
File "/env/lib/python3.7/site-packages/django/utils/asyncio.py", line 26, in inner      return func(*args, **kwargs)
File "/env/lib/python3.7/site-packages/django/db/backends/base/base.py", line 220, in ensure_connection      self.connect()
File "/env/lib/python3.7/site-packages/django/db/utils.py", line 90, in __exit__      raise dj_exc_value.with_traceback(traceback) from exc_value
File "/env/lib/python3.7/site-packages/django/db/backends/base/base.py", line 220, in ensure_connection      self.connect()
File "/env/lib/python3.7/site-packages/django/utils/asyncio.py", line 26, in inner      return func(*args, **kwargs)
File "/env/lib/python3.7/site-packages/django/db/backends/base/base.py", line 197, in connect      self.connection = self.get_new_connection(conn_params)
File "/env/lib/python3.7/site-packages/django/utils/asyncio.py", line 26, in inner      return func(*args, **kwargs)
File "/env/lib/python3.7/site-packages/django/db/backends/postgresql/base.py", line 185, in get_new_connection      connection = Database.connect(**conn_params)
File "/env/lib/python3.7/site-packages/psycopg2/__init__.py", line 126, in connect      conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: FATAL:  remaining connection slots are reserved for non-replication superuser connections

My database settings are as follow :

if os.getenv('GAE_APPLICATION', None):
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.postgresql_psycopg2',
            'HOST': '/cloudsql/MYAPP:europe-west2:MYAPP',
            'NAME': 'MYNAME',
            'USER': 'MYUSER',
            'PASSWORD': access_secret_version("MYSECRET_SQL_PASSWORD"),
            'PORT': '5432',
        }
    }

What does this actually mean ?


问:
Angular Material Datepicker - Incorrect date passed
I am pulling my hair out with this, and I know it has been answered a million times before.  But I cannot seem to find a solution to the problem.

I am using Angular material datepicker with a custom date provider to format the date.

This is what it looks like:

```<code>import { NativeDateAdapter } from ‘@angular/material’;
import { MatDateFormats } from ‘@angular/material/core’;
export class AppDateAdapter extends NativeDateAdapter {
  format(date: Date, displayFormat: Object): string {
    if (displayFormat === ‘input’) {
      let day: string = date.getDate().toString();
      day = +day < 10 ? ‘0’ + day : day;
      let month: string = (date.getMonth() + 1).toString();
      month = +month < 10 ? ‘0’ + month : month;
      let year = date.getFullYear();
      return `${day}-${month}-${year}`;
    }
    return date.toDateString();
  }
}
export const APP_DATE_FORMATS: MatDateFormats = {
  parse: {
    dateInput: { month: ‘short’, year: ‘numeric’, day: ‘numeric’ },
  },
  display: {
    dateInput: ‘input’,
    monthYearLabel: { year: ‘numeric’, month: ‘numeric’ },
    dateA11yLabel: { year: ‘numeric’, month: ‘long’, day: ‘numeric’
    },
    monthYearA11yLabel: { year: ‘numeric’, month: ‘long’ },
  }
};
</code>```

Now, the issue is, when I pass 02 Jan 2019, the date is displayed fine. But when submitting the data and inspecting the payload, the date is passed as 2019-02-01T22:00:00.000Z.  Now I know this has to do with UTC, but I cannot seem to fix the issue.  When using no provider at all, or extending MomentDateAdapter, the date is passed fine as 2019-02-01T00:00:00.000Z

My real issue is that when I create another provider and extend MomentDateAdapter instead of NativeDateAdapter, everything is perfect.  BUT, I cannot format the date for the MomentDateAdapter.

I have even tried returning the date formatted as moment().utc() but to no avail.

答:
I've had a similar issue, but I use MomentDateAdapter and when I receive the date in my component I use moment to format instead of formating in the adapter and it seems to work.
That problems seems to me that is caused by the timezone and using moment().utc() will not give you your current timezone</p>

Body:
I am pulling my hair out with this, and I know it has been answered a million times before.  But I cannot seem to find a solution to the problem.

I am using Angular material datepicker with a custom date provider to format the date.

This is what it looks like:

import { NativeDateAdapter } from ‘@angular/material’;
import { MatDateFormats } from ‘@angular/material/core’;
export class AppDateAdapter extends NativeDateAdapter {
  format(date: Date, displayFormat: Object): string {
    if (displayFormat === ‘input’) {
      let day: string = date.getDate().toString();
      day = +day < 10 ? ‘0’ + day : day;
      let month: string = (date.getMonth() + 1).toString();
      month = +month < 10 ? ‘0’ + month : month;
      let year = date.getFullYear();
      return `${day}-${month}-${year}`;
    }
    return date.toDateString();
  }
}
export const APP_DATE_FORMATS: MatDateFormats = {
  parse: {
    dateInput: { month: ‘short’, year: ‘numeric’, day: ‘numeric’ },
  },
  display: {
    dateInput: ‘input’,
    monthYearLabel: { year: ‘numeric’, month: ‘numeric’ },
    dateA11yLabel: { year: ‘numeric’, month: ‘long’, day: ‘numeric’
    },
    monthYearA11yLabel: { year: ‘numeric’, month: ‘long’ },
  }
};

Now, the issue is, when I pass 02 Jan 2019, the date is displayed fine. But when submitting the data and inspecting the payload, the date is passed as 2019-02-01T22:00:00.000Z.  Now I know this has to do with UTC, but I cannot seem to fix the issue.  When using no provider at all, or extending MomentDateAdapter, the date is passed fine as 2019-02-01T00:00:00.000Z

My real issue is that when I create another provider and extend MomentDateAdapter instead of NativeDateAdapter, everything is perfect.  BUT, I cannot format the date for the MomentDateAdapter.

I have even tried returning the date formatted as moment().utc() but to no avail.


问:
In R, calculate a dataframe as efficifiently and as fast as possible from thousands of external files
I am building a Shiny application in which a large ggplot2 fortified dataframe needs to be calculated over and over again, using a large amount of external source files. I am searching for the fastest and most efficient way to do this. In the following paragraph I will delve a little bit more into the subject and the code I have so far and also provide the input data to enable your kind assistance.

I am using the Helsinki Region Travel Time Matrix 2018, a dataset provided by Digital Geography Lab, a research group in the University of Helsinki. This data uses a generalised map of Helsinki capital region, in 250 x 250 meter cells (in my code grid_f), to calculate travel times between all cells in the map (grid ids are called YKR_ID, n=13231) by public transport, private car, bicycle and by foot. The calculations are stored in delimited .txt files, one text file for all the travel times to a specific cell id. The data is available for download at this website, under "Download the data". NB, the unzipped data is 13.8 GB in size.

Here is a selection from a text file in the dataset:

```<code>from_id;to_id;walk_t;walk_d;bike_s_t;bike_f_t;bike_d;pt_r_tt;pt_r_t;pt_r_d;pt_m_tt;pt_m_t;pt_m_d;car_r_t;car_r_d;car_m_t;car_m_d;car_sl_t
5785640;5785640;0;0;-1;-1;-1;0;0;0;0;0;0;-1;0;-1;0;-1
5785641;5785640;48;3353;51;32;11590;48;48;3353;48;48;3353;22;985;21;985;16
5785642;5785640;50;3471;51;32;11590;50;50;3471;50;50;3471;22;12167;21;12167;16
5785643;5785640;54;3764;41;26;9333;54;54;3764;54;54;3764;22;10372;21;10370;16
5787544;5785640;38;2658;10;7;1758;38;38;2658;38;38;2658;7;2183;7;2183;6
</code>```

My interest is to visualise (with ggplot2) this 250x250m Helsinki region map for one travel mode, the private car, using any of the possible 13231 cell ids, repeatedly if the user wants. Because of this it is important that the dataframe fetch is as fast and efficient as possible. For this question, let's concentrate on the fetching and processing of the data from the external files and use only one specific id value.

In a nutshell, After I have produced a ggplot2::fortify() version of the 250 x 250 meter grid spatial dataset grid_f,

I need to scan through all the 13231 Travel Time Matrix 2018 text files

Pick only the relevant columns (from_id, to_id, car_r_t, car_m_t, car_sl_t) in each file

Pick the relevant row using from_id (in this case, origin_id <- "5985086") in each file

Join the the resulting row to the fortified spatial data grid_f

My code is as follows:

# Libraries
library(ggplot2)
library(dplyr)
library(rgdal)
library(data.table)
library(sf)
library(sp)

# File paths. ttm_path is the folder which contains the unchanged Travel
# Time Matrix 2018 data from the research group's home page
ttm_path <- "HelsinkiTravelTimeMatrix2018"
gridpath <- "MetropAccess_YKR_grid_EurefFIN.shp"

#### Import grid cells
# use this CRS information throughout the app
app_crs <- sp::CRS("+init=epsg:3067")

# Read grid shapefile and transform
grid_f <- rgdal::readOGR(gridpath, stringsAsFactors = TRUE) %>%
  sp::spTransform(., app_crs) %>%
  # preserve grid dataframe data in the fortify
  {dplyr::left_join(ggplot2::fortify(.),
                    as.data.frame(.) %>%
                      dplyr::mutate(id = as.character(dplyr::row_number() - 1)))} %>%
  dplyr::select(-c(x, y))
```

<p>The code above this point is meant to run only once. The code below, more or less, would be run over and over with different <code>origin_id</code>s.</p>

```#### Fetch TTM18 data
origin_id <- "5985086"
origin_id_num <- as.numeric(origin_id)

# column positions of columns from_id, to_id, car_r_t, car_m_t, car_sl_t
col_range <- c(1, 2, 14, 16, 18)

# grid_f as data.table version
dt_grid <- as.data.table(grid_f)

# Get filepaths of all of the TTM18 data. Remove metadata textfile filepath.
all_files <- list.files(path = ttm_path,
                        pattern = ".txt$",
                        recursive = TRUE,
                        full.names = TRUE)
all_files <- all_files[-length(all_files)]

# lapply function
TTM18_fetch <- function(x, col_range, origin_id) {
  res <- fread(x, select = col_range)
  res <- subset(res, from_id == origin_id)
  return(res)
}

# The part of the code that needs to be fast and efficient
result <-
  lapply(all_files, FUN = TTM18_fetch, col_range, origin_id_num) %>%
  data.table::rbindlist(., fill = TRUE) %>%
  data.table::merge.data.table(dt_grid, ., by.x = "YKR_ID", by.y = "to_id")
```

The dataframe result should have 66155 rows of 12 variables, five rows for each 250x250 meter grid cell. The columns are YKR_ID, long, lat, order, hole, piece, id, group, from_id, car_r_t, car_m_t, car_sl_t.

My current lapply() and data.table::fread() solution takes about 2-3 minutes to complete. I think this is already a good achievement, but I can't help and think there are better and faster ways to complete this. So far, I have tried these alternatives to what I now have:

A conventional for loop: that was obviously a slow solution

I tried to teach myself more about vectorised functions in R, but that did not lead anywhere. Used this link

Tried to dabble with with() unsuccessfully using this SO question, inspired by this SO question

Looked into package parallel but ended up not utilising that because of the Windows environment I am using

Tried to find alternative ways to solve this with apply() and sapply() but nothing noteworthy came out of that.

As to why I didn't do all this to the data before ggplot2::fortify, I simply found it troublesome to work with a SpatialPolygonsDataFrame.

Thank you for your time.

答:
Whenver I’m trying to figure out how to improve the performance of my R
functions, I generally use the following approach. First, I look for any
function calls that may be unesscesary or identify places where multiple
function calls can be simplified into one. Then, I look for places in my
code that are incurring the greatest time penalty by benchmarking each
part separately. This can easily be done using the microbenchmark
package.

For example, we can ask if we get better performance with or without
piping (e.g. %>%).

# hint... piping is always slower
library(magrittr)
library(microbenchmark)
microbenchmark(
  pipe = iris %>% subset(Species=='setosa'),
  no_pipe = subset(iris, Species=='setosa'),
  times = 200)
```

```Unit: microseconds
    expr     min      lq     mean   median       uq      max neval cld
    pipe 157.518 196.739 308.1328 229.6775 312.6565 2473.582   200   b
 no_pipe  84.894 116.386 145.4039 126.1950 139.4100  612.492   200  a
```

<p>Here, we find that removing subseting a <code>data.frame</code> without piping
takes nearly half the time to execute!</p>

<p>Next, I determine the <strong><em>net time penalty</em></strong> for each place I
benchmarked by multipling the execution time by total number of times it
needs to be executed. For the areas with the greatest net time penalty,
I try to replace it with faster functions and/or try reduce the total
number of times it needs to be executed.</p>

<h1><strong>TLDR</strong></h1>

<p>In your case, you can speed things up by using the <code>fst</code> package
although you would need to convert your csv files to fst files.</p>

<pre class="lang-r prettyprint-override"><code># before
TTM18_fetch <- function(x, col_range, origin_id) {
  res <- data.table::fread(x, select = col_range)
  res <- subset(res, from_id == origin_id)
  return(res)
}

# after (NB x needs to be a fst file)
col_range <- c('from_id', 'to_id', 'car_r_t', 'car_m_t', 'car_sl_t')

TTM18_fetch <- function(x, col_range, origin_id) {
  res <- fst::read_fst(path = x,
                       columns = col_range,
                       as.data.table = TRUE)[from_id==origin_id]
  return(res)
}
</code>```

To convert your csv files to fst

library(data.table)
library(fst)
ttm_path <- 'REPLACE THIS'
new_ttm_path <- 'REPLACE THIS'

# Get filepaths of all of the TTM18 data. Remove metadata textfile filepath.
all_files <- list.files(path = ttm_path,
                        pattern = ".txt$",
                        recursive = TRUE,
                        full.names = TRUE)

all_files <- all_files[-grepl('[Mm]eta', all_files)]

# creating new file paths and names for fst files
file_names <- list.files(path = ttm_path,
                        pattern = ".txt$",
                        recursive = TRUE)
file_names <-  file_names[-grepl('[Mm]eta', file_names)]

file_names <- gsub(pattern = '.csv$',
                   replacement = '.fst',
                   x =file_names)

file_names <- file.path(new_ttm_path, file_names)

# csv to fst conversion

require(progress) # this will help you create track of things
pb <- progress_bar$new(
  format = " :what [:bar] :percent eta: :eta",
  clear = FALSE, total = length(file_names), width = 60)

# an index file to store from_id file locations
from_id_paths <- data.table(from_id = numeric(),
                            file_path = character())

for(i in seq_along(file_names)){

  pb$tick(tokens = list(what = 'reading'))
  tmp <- data.table::fread(all_files[i], key = 'from_id')

  pb$update(tokens = list(what = 'writing'))
  fst::write_fst(tmp,
                 compress = 50,  # less compressed files read faster
                 path = file_names[i] )

  pb$update(tokens = list(what = 'indexing'))
  from_id_paths <- rbind(from_id_paths,
                         data.table(from_id = unique(tmp$from_id),
                                    file_path = file_names[i]))

}

setkey(from_id_paths, from_id)
write_fst(from_id_paths,
          path =  file.path('new_ttm_path', 'from_id_index.fst'),
          compress = 0)
```

<h2>This would be the replacement</h2>

<pre class="lang-r prettyprint-override"><code>library(fst)
library(data.table)
new_ttm_path <- 'REPLACE THIS'

#### Fetch TTM18 data
origin_id <- "5985086"
origin_id_num <- as.numeric(origin_id)

# column positions of columns from_id, to_id, car_r_t, car_m_t, car_sl_t
col_range <- c('from_id', 'to_id', 'car_r_t', 'car_m_t', 'car_sl_t')

# grid_f as data.table version
dt_grid <- as.data.table(grid_f)

nescessary_files <- read_fst(path = file.path(new_ttm_path,
                                              'from_id_index.fst'),
                             as.data.table = TRUE
                             )[from_id==origin_id,file_path]

TTM18_fetch <- function(x, col_range, origin_id) {
  res <- fst::read_fst(path = x,
                       columns = col_range,
                       as.data.table = TRUE)[from_id==origin_id]
  return(res)
}

result <-  rbindlist(lapply(nescessary_files, FUN = TTM18_fetch, col_range,  origin_id_num),
                     fill = TRUE)
result <- data.table::merge.data.table(dt_grid, result, by.x = "YKR_ID", by.y = "to_id")
</code>```

Body:
I am building a Shiny application in which a large ggplot2 fortified dataframe needs to be calculated over and over again, using a large amount of external source files. I am searching for the fastest and most efficient way to do this. In the following paragraph I will delve a little bit more into the subject and the code I have so far and also provide the input data to enable your kind assistance.

I am using the Helsinki Region Travel Time Matrix 2018, a dataset provided by Digital Geography Lab, a research group in the University of Helsinki. This data uses a generalised map of Helsinki capital region, in 250 x 250 meter cells (in my code grid_f), to calculate travel times between all cells in the map (grid ids are called YKR_ID, n=13231) by public transport, private car, bicycle and by foot. The calculations are stored in delimited .txt files, one text file for all the travel times to a specific cell id. The data is available for download at this website, under "Download the data". NB, the unzipped data is 13.8 GB in size.

Here is a selection from a text file in the dataset:

from_id;to_id;walk_t;walk_d;bike_s_t;bike_f_t;bike_d;pt_r_tt;pt_r_t;pt_r_d;pt_m_tt;pt_m_t;pt_m_d;car_r_t;car_r_d;car_m_t;car_m_d;car_sl_t
5785640;5785640;0;0;-1;-1;-1;0;0;0;0;0;0;-1;0;-1;0;-1
5785641;5785640;48;3353;51;32;11590;48;48;3353;48;48;3353;22;985;21;985;16
5785642;5785640;50;3471;51;32;11590;50;50;3471;50;50;3471;22;12167;21;12167;16
5785643;5785640;54;3764;41;26;9333;54;54;3764;54;54;3764;22;10372;21;10370;16
5787544;5785640;38;2658;10;7;1758;38;38;2658;38;38;2658;7;2183;7;2183;6

My interest is to visualise (with ggplot2) this 250x250m Helsinki region map for one travel mode, the private car, using any of the possible 13231 cell ids, repeatedly if the user wants. Because of this it is important that the dataframe fetch is as fast and efficient as possible. For this question, let's concentrate on the fetching and processing of the data from the external files and use only one specific id value.

In a nutshell, After I have produced a ggplot2::fortify() version of the 250 x 250 meter grid spatial dataset grid_f,

I need to scan through all the 13231 Travel Time Matrix 2018 text files

Pick only the relevant columns (from_id, to_id, car_r_t, car_m_t, car_sl_t) in each file

Pick the relevant row using from_id (in this case, origin_id <- "5985086") in each file

Join the the resulting row to the fortified spatial data grid_f

My code is as follows:

# Libraries
library(ggplot2)
library(dplyr)
library(rgdal)
library(data.table)
library(sf)
library(sp)

# File paths. ttm_path is the folder which contains the unchanged Travel
# Time Matrix 2018 data from the research group's home page
ttm_path <- "HelsinkiTravelTimeMatrix2018"
gridpath <- "MetropAccess_YKR_grid_EurefFIN.shp"

#### Import grid cells
# use this CRS information throughout the app
app_crs <- sp::CRS("+init=epsg:3067")

# Read grid shapefile and transform
grid_f <- rgdal::readOGR(gridpath, stringsAsFactors = TRUE) %>%
  sp::spTransform(., app_crs) %>%
  # preserve grid dataframe data in the fortify
  {dplyr::left_join(ggplot2::fortify(.),
                    as.data.frame(.) %>%
                      dplyr::mutate(id = as.character(dplyr::row_number() - 1)))} %>%
  dplyr::select(-c(x, y))

The code above this point is meant to run only once. The code below, more or less, would be run over and over with different origin_ids.

#### Fetch TTM18 data
origin_id <- "5985086"
origin_id_num <- as.numeric(origin_id)

# column positions of columns from_id, to_id, car_r_t, car_m_t, car_sl_t
col_range <- c(1, 2, 14, 16, 18)

# grid_f as data.table version
dt_grid <- as.data.table(grid_f)

# Get filepaths of all of the TTM18 data. Remove metadata textfile filepath.
all_files <- list.files(path = ttm_path,
                        pattern = ".txt$",
                        recursive = TRUE,
                        full.names = TRUE)
all_files <- all_files[-length(all_files)]

# lapply function
TTM18_fetch <- function(x, col_range, origin_id) {
  res <- fread(x, select = col_range)
  res <- subset(res, from_id == origin_id)
  return(res)
}

# The part of the code that needs to be fast and efficient
result <-
  lapply(all_files, FUN = TTM18_fetch, col_range, origin_id_num) %>%
  data.table::rbindlist(., fill = TRUE) %>%
  data.table::merge.data.table(dt_grid, ., by.x = "YKR_ID", by.y = "to_id")

The dataframe result should have 66155 rows of 12 variables, five rows for each 250x250 meter grid cell. The columns are YKR_ID, long, lat, order, hole, piece, id, group, from_id, car_r_t, car_m_t, car_sl_t.

My current lapply() and data.table::fread() solution takes about 2-3 minutes to complete. I think this is already a good achievement, but I can't help and think there are better and faster ways to complete this. So far, I have tried these alternatives to what I now have:

A conventional for loop: that was obviously a slow solution

I tried to teach myself more about vectorised functions in R, but that did not lead anywhere. Used this link

Tried to dabble with with() unsuccessfully using this SO question, inspired by this SO question

Looked into package parallel but ended up not utilising that because of the Windows environment I am using

Tried to find alternative ways to solve this with apply() and sapply() but nothing noteworthy came out of that.

As to why I didn't do all this to the data before ggplot2::fortify, I simply found it troublesome to work with a SpatialPolygonsDataFrame.

Thank you for your time.

Body:
Whenver I’m trying to figure out how to improve the performance of my R
functions, I generally use the following approach. First, I look for any
function calls that may be unesscesary or identify places where multiple
function calls can be simplified into one. Then, I look for places in my
code that are incurring the greatest time penalty by benchmarking each
part separately. This can easily be done using the microbenchmark
package.

For example, we can ask if we get better performance with or without
piping (e.g. %>%).

# hint... piping is always slower
library(magrittr)
library(microbenchmark)
microbenchmark(
  pipe = iris %>% subset(Species=='setosa'),
  no_pipe = subset(iris, Species=='setosa'),
  times = 200)

Unit: microseconds
    expr     min      lq     mean   median       uq      max neval cld
    pipe 157.518 196.739 308.1328 229.6775 312.6565 2473.582   200   b
 no_pipe  84.894 116.386 145.4039 126.1950 139.4100  612.492   200  a

Here, we find that removing subseting a data.frame without piping
takes nearly half the time to execute!

Next, I determine the net time penalty for each place I
benchmarked by multipling the execution time by total number of times it
needs to be executed. For the areas with the greatest net time penalty,
I try to replace it with faster functions and/or try reduce the total
number of times it needs to be executed.

TLDR

In your case, you can speed things up by using the fst package
although you would need to convert your csv files to fst files.

# before
TTM18_fetch <- function(x, col_range, origin_id) {
  res <- data.table::fread(x, select = col_range)
  res <- subset(res, from_id == origin_id)
  return(res)
}

# after (NB x needs to be a fst file)
col_range <- c('from_id', 'to_id', 'car_r_t', 'car_m_t', 'car_sl_t')

TTM18_fetch <- function(x, col_range, origin_id) {
  res <- fst::read_fst(path = x,
                       columns = col_range,
                       as.data.table = TRUE)[from_id==origin_id]
  return(res)
}

To convert your csv files to fst

library(data.table)
library(fst)
ttm_path <- 'REPLACE THIS'
new_ttm_path <- 'REPLACE THIS'

# Get filepaths of all of the TTM18 data. Remove metadata textfile filepath.
all_files <- list.files(path = ttm_path,
                        pattern = ".txt$",
                        recursive = TRUE,
                        full.names = TRUE)

all_files <- all_files[-grepl('[Mm]eta', all_files)]

# creating new file paths and names for fst files
file_names <- list.files(path = ttm_path,
                        pattern = ".txt$",
                        recursive = TRUE)
file_names <-  file_names[-grepl('[Mm]eta', file_names)]

file_names <- gsub(pattern = '.csv$',
                   replacement = '.fst',
                   x =file_names)

file_names <- file.path(new_ttm_path, file_names)

# csv to fst conversion

require(progress) # this will help you create track of things
pb <- progress_bar$new(
  format = " :what [:bar] :percent eta: :eta",
  clear = FALSE, total = length(file_names), width = 60)

# an index file to store from_id file locations
from_id_paths <- data.table(from_id = numeric(),
                            file_path = character())

for(i in seq_along(file_names)){

  pb$tick(tokens = list(what = 'reading'))
  tmp <- data.table::fread(all_files[i], key = 'from_id')

  pb$update(tokens = list(what = 'writing'))
  fst::write_fst(tmp,
                 compress = 50,  # less compressed files read faster
                 path = file_names[i] )

  pb$update(tokens = list(what = 'indexing'))
  from_id_paths <- rbind(from_id_paths,
                         data.table(from_id = unique(tmp$from_id),
                                    file_path = file_names[i]))

}

setkey(from_id_paths, from_id)
write_fst(from_id_paths,
          path =  file.path('new_ttm_path', 'from_id_index.fst'),
          compress = 0)

This would be the replacement

library(fst)
library(data.table)
new_ttm_path <- 'REPLACE THIS'

#### Fetch TTM18 data
origin_id <- "5985086"
origin_id_num <- as.numeric(origin_id)

# column positions of columns from_id, to_id, car_r_t, car_m_t, car_sl_t
col_range <- c('from_id', 'to_id', 'car_r_t', 'car_m_t', 'car_sl_t')

# grid_f as data.table version
dt_grid <- as.data.table(grid_f)

nescessary_files <- read_fst(path = file.path(new_ttm_path,
                                              'from_id_index.fst'),
                             as.data.table = TRUE
                             )[from_id==origin_id,file_path]

TTM18_fetch <- function(x, col_range, origin_id) {
  res <- fst::read_fst(path = x,
                       columns = col_range,
                       as.data.table = TRUE)[from_id==origin_id]
  return(res)
}

result <-  rbindlist(lapply(nescessary_files, FUN = TTM18_fetch, col_range,  origin_id_num),
                     fill = TRUE)
result <- data.table::merge.data.table(dt_grid, result, by.x = "YKR_ID", by.y = "to_id")