问:
How to make tree pattern fit constraints but so algorithm is generic instead of hardcoded?
Building off this question, the next question is, how to build an algorithm which will take as input an index/integer, and output the path to the appropriate node in the tree. The examples of how the trees might be structured are as follows, but I might be wrong. Ideally there would be a pattern which all of them follow so we can have an equation to map index to path.

```<code>base-1
  a

base-2
  a
  b

base-3
  a
  b
  c
  null

base-4
  a
  b
  c
  d

base-5
  a
  b
  c
  tree
    d
    e

base-6
  a
  b
  c
  d
  e
  f
  null
  null

base-7
  a
  b
  c
  d
  e
  f
  g
  null

base-8
  a
  b
  c
  d
  e
  f
  g
  h

base-9
  a
  b
  c
  d
  e
  f
  g
  tree
    h
    i

base-10
  a
  b
  c
  d
  e
  f
  g
  tree
    h
    i
    j
    null

base-11
  a
  b
  c
  d
  e
  f
  g
  tree
    h
    i
    j
    k

base-12
  a
  b
  c
  d
  e
  f
  tree
    g
    h
    i
    j
  tree
    k
    l

base-13
  a
  b
  c
  d
  e
  f
  g
  tree
    h
    i
    j
    k
    l
    m
    null
    null

base-14
  a
  b
  c
  d
  e
  f
  g
  tree
    h
    i
    j
    k
    l
    m
    n
    null

base-15
  a
  b
  c
  d
  e
  f
  g
  tree
    h
    i
    j
    k
    l
    m
    n
    o

base-16
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  o
  p

base-17
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  o
  tree
    p
    q

base-18
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  o
  tree
    p
    q
    r
    null

base-19
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  o
  tree
    p
    q
    r
    s

base-20
  a
  b
  c
  d
  e
  f
  tree
    g
    h
    i
    j
    k
    l
    m
    n
  tree
    o
    p
    q
    r
    s
    t
    null
    null

base-21
  a
  b
  c
  d
  e
  f
  tree
    g
    h
    i
    j
    k
    l
    m
    n
  tree
    o
    p
    q
    r
    s
    t
    u
    null

base-22
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  o
  tree
    p
    q
    r
    s
    t
    u
    v
    null

base-23
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  o
  tree
    p
    q
    r
    s
    t
    u
    v
    w

base-24
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  tree
    o
    p
    q
    r
    s
    t
    u
    v
  tree
    w
    x

base-25
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  tree
    o
    p
    q
    r
    s
    t
    u
    v
  tree
    w
    x
    y
    null

base-26
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  tree
    o
    p
    q
    r
    s
    t
    u
    v
  tree
    w
    x
    y
    z

base-27
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  tree
    n
    o
    p
    q
    r
    s
    t
    u
  tree
    v
    w
    x
    y
  tree
    z
    aa

base-28
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  tree
    o
    p
    q
    r
    s
    t
    u
    v
  tree
    w
    x
    y
    z
    aa
    ab
    null
    null

base-29
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  o
  tree
    p
    q
    r
    s
    t
    u
    v
    w
    x
    y
    z
    aa
    ab
    ac
    null
    null

base-30
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  o
  p
  q
  r
  s
  t
  u
  v
  w
  x
  y
  z
  aa
  ab
  ac
  ad
  null
  null

base-31
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  o
  p
  q
  r
  s
  t
  u
  v
  w
  x
  y
  z
  aa
  ab
  ac
  ad
  ae
  null

base-32
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  o
  p
  q
  r
  s
  t
  u
  v
  w
  x
  y
  z
  aa
  ab
  ac
  ad
  ae
  af
</code>```

There can only be 32 possible items in each level, and max 2 levels. Each tree can only have power-of-2 number of elements. In some cases I put null because it requires less nodes or less traversal than adding a new tree. If there is not a consistent pattern to it, you can create an appropriate similar pattern if no precise pattern is found. Ideally there is a pattern so an equation can be used to generate the path from an index.

Some other things to note:

Always start the list with the top-most levels filled out as much as possible.

Allow up to 2 null values on 8 or more.

My attempt is pretty hardcoded still.

```<code>function getPathFromIndex(size, index) {
  if (size < 5) {
    return [index]
  }

  if (size === 5) {
    if (index > 2) {
      return [2, index - 3]
    } else {
      return [index]
    }
  }

  if (size < 9) {
    return [index]
  }

  if (size < 12) {
    if (index > 6) {
      return [6, index - 7]
    } else {
      return [index]
    }
  }

  // continue hardcoding.
}
</code>```

Is there a way of accomplishing a similar goal (with the power of 2 constraint, and only 1 level of nesting), yet make the algorithm less hardcoded? Can you restructure the trees in such a way to make that possible?

Some hints:

How many trees does it need to be divided into?

Given that number, how to automatically chunk the array into the tree?

答:
This is going to be a long road...

As I also could not find a mathematical formula to solve this, I went for a data-driven solution: a lookup table that gives me the shape of the (potentially) nested array for each individual size (between 1 and 128).

A shape can be defined by a few numbers:

The number of top-level, non-null values

The number of non-null values in the first sub array if there is one

The number of non-null values in the second sub array if there is one

The number of non-null values in the third sub array if there is one

The number of non-null values in the fourth sub array if there is one

The potential null padding can be inferred from this information, as we know that a power of 2 must be reached.

An example:

Size = 102

This can be represented with the following numbers:

28, 30, 30, and 14

This means an array of that size will look like this:

[
    0,
    1,
    2,
    3,
    ...,
    27,
    [28, 29,..., 57, null, null],
    [58, 59,..., 87, null, null],
    [88, 89,..., 101, null, null],
    null
]
```
<p>Note that there are <code>null</code> values involved to reach powers of 2. The top level has (including the final <code>null</code>) 32 elements. The first two inner sub arrays have a total size of 32 (including the padding <code>null</code> values), and the third one has 16 elements (also including the padding).</p>
<h3>Generating the shapes</h3>
<p>To avoid having to determine shapes manually for each of the 128 array sizes, I wrote a brute force function to make a valid shape choice for each of these sizes. This I just used to fix these shapes, and is not part of the final solution:</p>
<pre class="lang-js prettyprint-override"><code>function shape(n) { // Returns number of atomic entries, followed by data-size(s) of subarrays
    const greatestPowerOf2 = (n) =>
        n >= 32 ? 32 : n >= 16 ? 16 : n >= 8 ? 8 : n >= 4 ? 4 : n >= 2 ? 2 : 1;

    let p = greatestPowerOf2(n+2);
    if (p >= n) {
        // The only cases where there are no subarrays
        return [n];
    }
    // Try with one subarray
    for (let sub = 2; sub < n && sub <= 32; sub *= 2) {
        let maxInnerNulls = sub == 2 ? 0 : sub == 4 ? 1 : 2;
        let top = n - sub + 1;
        p = greatestPowerOf2(top+2+maxInnerNulls);
        if (p >= top) {
            let nulls = p - top;
            let innerNulls = Math.min(maxInnerNulls, nulls);
            nulls -= innerNulls;
            return [p - 1 - nulls, sub - innerNulls];
        }
    }
    // Try with two subarrays
    for (let sub1 = 2; sub1 < n && sub1 <= 32; sub1 *= 2) {
        let maxInnerNulls1 = sub1 == 2 ? 0 : sub1 == 4 ? 1 : 2;
        for (let sub2 = 2; sub2 <= sub1; sub2 *= 2) {
            let top = n - sub1 - sub2 + 2;
            if (top < 0) break;
            let maxInnerNulls2 = sub2 == 2 ? 0 : sub2 == 4 ? 1 : 2;
            p = greatestPowerOf2(top+2+maxInnerNulls1+maxInnerNulls2);
            if (p >= top) {
                let nulls = p - top;
                let innerNulls1 = Math.min(maxInnerNulls1, nulls);
                nulls -= innerNulls1;
                let innerNulls2 = Math.min(maxInnerNulls2, nulls);
                nulls -= innerNulls2;
                return [p - 2 - nulls, sub1 - innerNulls1, sub2 - innerNulls2];
            }
        }
    }
    // Try with three subarrays
    for (let sub1 = 2; sub1 < n && sub1 <= 32; sub1 *= 2) {
        let maxInnerNulls1 = sub1 == 2 ? 0 : sub1 == 4 ? 1 : 2;
        for (let sub2 = 2; sub2 <= sub1; sub2 *= 2) {
            let maxInnerNulls2 = sub2 == 2 ? 0 : sub2 == 4 ? 1 : 2;
            for (let sub3 = 2; sub3 <= sub2; sub3 *= 2) {
                let top = n - sub1 - sub2 - sub3 + 3;
                if (top < 0) break;
                let maxInnerNulls3 = sub3 == 2 ? 0 : sub3 == 4 ? 1 : 2;
                p = greatestPowerOf2(top+2+maxInnerNulls1+maxInnerNulls2+maxInnerNulls3);
                if (p >= top) {
                    let nulls = p - top;
                    let innerNulls1 = Math.min(maxInnerNulls1, nulls);
                    nulls -= innerNulls1;
                    let innerNulls2 = Math.min(maxInnerNulls2, nulls);
                    nulls -= innerNulls2;
                    let innerNulls3 = Math.min(maxInnerNulls3, nulls);
                    nulls -= innerNulls3;
                    return [p - 3 - nulls, sub1 - innerNulls1, sub2 - innerNulls2, sub3 - innerNulls3];
                }
            }
        }
    }
    // Try with four subarrays
    for (let sub1 = 2; sub1 < n && sub1 <= 32; sub1 *= 2) {
        let maxInnerNulls1 = sub1 == 2 ? 0 : sub1 == 4 ? 1 : 2;
        for (let sub2 = 2; sub2 <= sub1; sub2 *= 2) {
            let maxInnerNulls2 = sub2 == 2 ? 0 : sub2 == 4 ? 1 : 2;
            for (let sub3 = 2; sub3 <= sub2; sub3 *= 2) {
                let maxInnerNulls3 = sub3 == 2 ? 0 : sub3 == 4 ? 1 : 2;
                for (let sub4 = 2; sub4 <= sub3; sub4 *= 2) {
                    let top = n - sub1 - sub2 - sub3 - sub4 + 4;
                    if (top < 0) break;
                    let maxInnerNulls4 = sub4 == 2 ? 0 : sub4 == 4 ? 1 : 2;
                    p = greatestPowerOf2(top+2+maxInnerNulls1+maxInnerNulls2+maxInnerNulls3+maxInnerNulls4);
                    if (p >= top) {
                        let nulls = p - top;
                        let innerNulls1 = Math.min(maxInnerNulls1, nulls);
                        nulls -= innerNulls1;
                        let innerNulls2 = Math.min(maxInnerNulls2, nulls);
                        nulls -= innerNulls2;
                        let innerNulls3 = Math.min(maxInnerNulls3, nulls);
                        nulls -= innerNulls3;
                        let innerNulls4 = Math.min(maxInnerNulls4, nulls);
                        nulls -= innerNulls4;
                        return [p - 4 - nulls, sub1 - innerNulls1, sub2 - innerNulls2, sub3 - innerNulls3, sub4 - innerNulls4];
                    }
                }
            }
        }
    }
}
</code>```

I admit this code is not elegant, with lots of code repetition, but it serves the intended purpose: it generates a shape (the set of numbers explained above) for any given array size.

Encoding the shapes in less space

The numbers for the sub arrays cannot be just any number. They must be either a power of 2, or one less (when one padding null is assumed) or two less (when two null values are assumed). So the possible numbers are in this set:

[1, 2, 3, 4, 6, 7, 8, 14, 15, 16, 30, 31, 32]
```
<p>The first number (which represents the count of values at the top level), can also be in that set, but can also be in the range 27 - 29. This is because the sub arrays that follow <em>and</em> potential <code>null</code> padding also count for reaching the power of 2 at the top level. This means that there are exactly 16 possible numbers in the first position of the shape "encoding". We could compress this encoding by mapping these numbers to 4-bit values (giving 16 possibilities). As it turned out there are never more than 4 subarrays needed, we will need 20 bits for encoding a shape.</p>
<p>Now we should determine what these 20-bit numbers are for 128 shapes and collect them in an array that can serve as lookup table.</p>
<p>Here is the function to encode numbers into that 20-bit encoding:</p>
<pre class="lang-js prettyprint-override"><code>function encode(shapeNumbers) {
    let code = 0;
    for (let i = shapeNumbers.length - 1; i >= 0; i--) {
        code = code*16 + [1,2,3,4,6,7,8,14,15,16,27,28,29,30,31,32].indexOf(shapeNumbers[i]);
    }
    return code;
}
</code>```

I collected the codes with this function:

function collectCodes() {
    let codes = [null];
    for (let n = 1; n <= 128; n++) {
        let shapeNumbers = shape(n);
        let code = encode(shapeNumbers);
        codes.push(code);
    }
    return codes;
}
```
<p><div class="snippet" data-lang="js" data-hide="true" data-console="true" data-babel="false">
<div class="snippet-code snippet-currently-hidden">
<pre class="snippet-code-js lang-js prettyprint-override"><code>function shape(n) { // Returns number of atomic entries, followed by data-size(s) of subarrays
    const greatestPowerOf2 = (n) =>
        n >= 32 ? 32 : n >= 16 ? 16 : n >= 8 ? 8 : n >= 4 ? 4 : n >= 2 ? 2 : 1;

    let p = greatestPowerOf2(n+2);
    if (p >= n) {
        // The only cases where there are no subarrays
        return [n];
    }
    // Try with one subarray
    for (let sub = 2; sub < n && sub <= 32; sub *= 2) {
        let maxInnerNulls = sub == 2 ? 0 : sub == 4 ? 1 : 2;
        let top = n - sub + 1;
        p = greatestPowerOf2(top+2+maxInnerNulls);
        if (p >= top && p <= 32) {
            let nulls = p - top;
            let innerNulls = Math.min(maxInnerNulls, nulls);
            nulls -= innerNulls;
            return [p - 1 - nulls, sub - innerNulls];
        }
    }
    // Try with two subarrays
    for (let sub1 = 2; sub1 < n && sub1 <= 32; sub1 *= 2) {
        let maxInnerNulls1 = sub1 == 2 ? 0 : sub1 == 4 ? 1 : 2;
        for (let sub2 = 2; sub2 <= sub1; sub2 *= 2) {
            let top = n - sub1 - sub2 + 2;
            if (top < 0) break;
            let maxInnerNulls2 = sub2 == 2 ? 0 : sub2 == 4 ? 1 : 2;
            p = greatestPowerOf2(top+2+maxInnerNulls1+maxInnerNulls2);
            if (p >= top && p <= 32) {
                let nulls = p - top;
                let innerNulls1 = Math.min(maxInnerNulls1, nulls);
                nulls -= innerNulls1;
                let innerNulls2 = Math.min(maxInnerNulls2, nulls);
                nulls -= innerNulls2;
                return [p - 2 - nulls, sub1 - innerNulls1, sub2 - innerNulls2];
            }
        }
    }
    // Try with three subarrays
    for (let sub1 = 2; sub1 < n && sub1 <= 32; sub1 *= 2) {
        let maxInnerNulls1 = sub1 == 2 ? 0 : sub1 == 4 ? 1 : 2;
        for (let sub2 = 2; sub2 <= sub1; sub2 *= 2) {
            let maxInnerNulls2 = sub2 == 2 ? 0 : sub2 == 4 ? 1 : 2;
            for (let sub3 = 2; sub3 <= sub2; sub3 *= 2) {
                let top = n - sub1 - sub2 - sub3 + 3;
                if (top < 0) break;
                let maxInnerNulls3 = sub3 == 2 ? 0 : sub3 == 4 ? 1 : 2;
                p = greatestPowerOf2(top+2+maxInnerNulls1+maxInnerNulls2+maxInnerNulls3);
                if (p >= top && p <= 32) {
                    let nulls = p - top;
                    let innerNulls1 = Math.min(maxInnerNulls1, nulls);
                    nulls -= innerNulls1;
                    let innerNulls2 = Math.min(maxInnerNulls2, nulls);
                    nulls -= innerNulls2;
                    let innerNulls3 = Math.min(maxInnerNulls3, nulls);
                    nulls -= innerNulls3;
                    return [p - 3 - nulls, sub1 - innerNulls1, sub2 - innerNulls2, sub3 - innerNulls3];
                }
            }
        }
    }
    // Try with four subarrays
    for (let sub1 = 2; sub1 < n && sub1 <= 32; sub1 *= 2) {
        let maxInnerNulls1 = sub1 == 2 ? 0 : sub1 == 4 ? 1 : 2;
        for (let sub2 = 2; sub2 <= sub1; sub2 *= 2) {
            let maxInnerNulls2 = sub2 == 2 ? 0 : sub2 == 4 ? 1 : 2;
            for (let sub3 = 2; sub3 <= sub2; sub3 *= 2) {
                let maxInnerNulls3 = sub3 == 2 ? 0 : sub3 == 4 ? 1 : 2;
                for (let sub4 = 2; sub4 <= sub3; sub4 *= 2) {
                    let top = n - sub1 - sub2 - sub3 - sub4 + 4;
                    if (top < 0) break;
                    let maxInnerNulls4 = sub4 == 2 ? 0 : sub4 == 4 ? 1 : 2;
                    p = greatestPowerOf2(top+2+maxInnerNulls1+maxInnerNulls2+maxInnerNulls3+maxInnerNulls4);
                    if (p >= top && p <= 32) {
                        let nulls = p - top;
                        let innerNulls1 = Math.min(maxInnerNulls1, nulls);
                        nulls -= innerNulls1;
                        let innerNulls2 = Math.min(maxInnerNulls2, nulls);
                        nulls -= innerNulls2;
                        let innerNulls3 = Math.min(maxInnerNulls3, nulls);
                        nulls -= innerNulls3;
                        let innerNulls4 = Math.min(maxInnerNulls4, nulls);
                        nulls -= innerNulls4;
                        return [p - 4 - nulls, sub1 - innerNulls1, sub2 - innerNulls2, sub3 - innerNulls3, sub4 - innerNulls4];
                    }
                }
            }
        }
    }
}

function encode(shapeNumbers) {
    let code = 0;
    for (let i = shapeNumbers.length - 1; i >= 0; i--) {
        code = code*16 + [1,2,3,4,6,7,8,14,15,16,27,28,29,30,31,32].indexOf(shapeNumbers[i]);
    }
    return code;
}

function collectCodes() {
    let codes = [null];
    for (let n = 1; n <= 128; n++) {
        let shapeNumbers = shape(n);
        let code = encode(shapeNumbers);
        codes.push(code);
    }
    return codes;
}

console.log(JSON.stringify(collectCodes()));</code>```

This gave this result:

[null,0,1,2,3,18,4,5,6,21,37,53,68,69,7,8,9,24,40,56,71,72,88,104,359,855,871,111,119,120,13,14,15,30,46,62,77,78,94,110,365,861,877,124,125,126,142,158,413,909,925,1405,1661,1677,1693,5788,1915,1916,1917,220,221,222,238,254,509,1005,1021,1501,1757,1773,1789,30588,2011,2012,2013,2269,2525,2541,2557,6652,14828,14844,26844,27100,27116,27132,30683,30684,3547,3548,3549,3805,4061,4077,4093,8188,16364,16380,28380,28636,28652,28668,32219,32220,36316,40412,40668,40924,40940,40956,106491,237547,237563,433883,434139,434155,434171,56794,56795,56796,60892,64988,65244,65500,65516,65532,131067,262123,262139]
```
<h2>The solution code</h2>
<p>Now that we have this, we can throw away the above JavaScript functions. This array has the info we need to reproduce a shape or to translate an index into a path.</p>
<p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
<div class="snippet-code">
<pre class="snippet-code-js lang-js prettyprint-override"><code>const codes = [null,0,1,2,3,18,4,5,6,21,37,53,68,69,7,8,9,24,40,56,71,72,88,104,359,855,871,111,119,120,13,14,15,30,46,62,77,78,94,110,365,861,877,124,125,126,142,158,413,909,925,1405,1661,1677,1693,5788,1915,1916,1917,220,221,222,238,254,509,1005,1021,1501,1757,1773,1789,30588,2011,2012,2013,2269,2525,2541,2557,6652,14828,14844,26844,27100,27116,27132,30683,30684,3547,3548,3549,3805,4061,4077,4093,8188,16364,16380,28380,28636,28652,28668,32219,32220,36316,40412,40668,40924,40940,40956,106491,237547,237563,433883,434139,434155,434171,56794,56795,56796,60892,64988,65244,65500,65516,65532,131067,262123,262139];

const codeMap = [1,2,3,4,6,7,8,14,15,16,27,28,29,30,31,32];

function getPath(size, i) {
    let code = codes[size];
    let limit = codeMap[code % 16];
    if (i < limit) return [i];
    for (let sub = limit; code; sub++) {
        i -= limit;
        code >>= 4;
        limit = codeMap[code % 16];
        if (i < limit) return [sub, i];
    }
}

// Demo with size 28
let size = 28;
for (let i = 0; i < size; i++) {
    console.log(i, JSON.stringify(getPath(size, i)));
}</code>```

Body:
Building off this question, the next question is, how to build an algorithm which will take as input an index/integer, and output the path to the appropriate node in the tree. The examples of how the trees might be structured are as follows, but I might be wrong. Ideally there would be a pattern which all of them follow so we can have an equation to map index to path.

base-1
  a

base-2
  a
  b

base-3
  a
  b
  c
  null

base-4
  a
  b
  c
  d

base-5
  a
  b
  c
  tree
    d
    e

base-6
  a
  b
  c
  d
  e
  f
  null
  null

base-7
  a
  b
  c
  d
  e
  f
  g
  null

base-8
  a
  b
  c
  d
  e
  f
  g
  h

base-9
  a
  b
  c
  d
  e
  f
  g
  tree
    h
    i

base-10
  a
  b
  c
  d
  e
  f
  g
  tree
    h
    i
    j
    null

base-11
  a
  b
  c
  d
  e
  f
  g
  tree
    h
    i
    j
    k

base-12
  a
  b
  c
  d
  e
  f
  tree
    g
    h
    i
    j
  tree
    k
    l

base-13
  a
  b
  c
  d
  e
  f
  g
  tree
    h
    i
    j
    k
    l
    m
    null
    null

base-14
  a
  b
  c
  d
  e
  f
  g
  tree
    h
    i
    j
    k
    l
    m
    n
    null

base-15
  a
  b
  c
  d
  e
  f
  g
  tree
    h
    i
    j
    k
    l
    m
    n
    o

base-16
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  o
  p

base-17
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  o
  tree
    p
    q

base-18
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  o
  tree
    p
    q
    r
    null

base-19
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  o
  tree
    p
    q
    r
    s

base-20
  a
  b
  c
  d
  e
  f
  tree
    g
    h
    i
    j
    k
    l
    m
    n
  tree
    o
    p
    q
    r
    s
    t
    null
    null

base-21
  a
  b
  c
  d
  e
  f
  tree
    g
    h
    i
    j
    k
    l
    m
    n
  tree
    o
    p
    q
    r
    s
    t
    u
    null

base-22
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  o
  tree
    p
    q
    r
    s
    t
    u
    v
    null

base-23
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  o
  tree
    p
    q
    r
    s
    t
    u
    v
    w

base-24
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  tree
    o
    p
    q
    r
    s
    t
    u
    v
  tree
    w
    x

base-25
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  tree
    o
    p
    q
    r
    s
    t
    u
    v
  tree
    w
    x
    y
    null

base-26
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  tree
    o
    p
    q
    r
    s
    t
    u
    v
  tree
    w
    x
    y
    z

base-27
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  tree
    n
    o
    p
    q
    r
    s
    t
    u
  tree
    v
    w
    x
    y
  tree
    z
    aa

base-28
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  tree
    o
    p
    q
    r
    s
    t
    u
    v
  tree
    w
    x
    y
    z
    aa
    ab
    null
    null

base-29
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  o
  tree
    p
    q
    r
    s
    t
    u
    v
    w
    x
    y
    z
    aa
    ab
    ac
    null
    null

base-30
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  o
  p
  q
  r
  s
  t
  u
  v
  w
  x
  y
  z
  aa
  ab
  ac
  ad
  null
  null

base-31
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  o
  p
  q
  r
  s
  t
  u
  v
  w
  x
  y
  z
  aa
  ab
  ac
  ad
  ae
  null

base-32
  a
  b
  c
  d
  e
  f
  g
  h
  i
  j
  k
  l
  m
  n
  o
  p
  q
  r
  s
  t
  u
  v
  w
  x
  y
  z
  aa
  ab
  ac
  ad
  ae
  af

There can only be 32 possible items in each level, and max 2 levels. Each tree can only have power-of-2 number of elements. In some cases I put null because it requires less nodes or less traversal than adding a new tree. If there is not a consistent pattern to it, you can create an appropriate similar pattern if no precise pattern is found. Ideally there is a pattern so an equation can be used to generate the path from an index.

Some other things to note:

Always start the list with the top-most levels filled out as much as possible.

Allow up to 2 null values on 8 or more.

My attempt is pretty hardcoded still.

function getPathFromIndex(size, index) {
  if (size < 5) {
    return [index]
  }

  if (size === 5) {
    if (index > 2) {
      return [2, index - 3]
    } else {
      return [index]
    }
  }

  if (size < 9) {
    return [index]
  }

  if (size < 12) {
    if (index > 6) {
      return [6, index - 7]
    } else {
      return [index]
    }
  }

  // continue hardcoding.
}

Is there a way of accomplishing a similar goal (with the power of 2 constraint, and only 1 level of nesting), yet make the algorithm less hardcoded? Can you restructure the trees in such a way to make that possible?

Some hints:

How many trees does it need to be divided into?

Given that number, how to automatically chunk the array into the tree?

Body:
This is going to be a long road...

As I also could not find a mathematical formula to solve this, I went for a data-driven solution: a lookup table that gives me the shape of the (potentially) nested array for each individual size (between 1 and 128).

A shape can be defined by a few numbers:

The number of top-level, non-null values

The number of non-null values in the first sub array if there is one

The number of non-null values in the second sub array if there is one

The number of non-null values in the third sub array if there is one

The number of non-null values in the fourth sub array if there is one

The potential null padding can be inferred from this information, as we know that a power of 2 must be reached.

An example:

Size = 102

This can be represented with the following numbers:

28, 30, 30, and 14

This means an array of that size will look like this:

[
    0,
    1,
    2,
    3,
    ...,
    27,
    [28, 29,..., 57, null, null],
    [58, 59,..., 87, null, null],
    [88, 89,..., 101, null, null],
    null
]

Note that there are null values involved to reach powers of 2. The top level has (including the final null) 32 elements. The first two inner sub arrays have a total size of 32 (including the padding null values), and the third one has 16 elements (also including the padding).

Generating the shapes

To avoid having to determine shapes manually for each of the 128 array sizes, I wrote a brute force function to make a valid shape choice for each of these sizes. This I just used to fix these shapes, and is not part of the final solution:

function shape(n) { // Returns number of atomic entries, followed by data-size(s) of subarrays
    const greatestPowerOf2 = (n) =>
        n >= 32 ? 32 : n >= 16 ? 16 : n >= 8 ? 8 : n >= 4 ? 4 : n >= 2 ? 2 : 1;

    let p = greatestPowerOf2(n+2);
    if (p >= n) {
        // The only cases where there are no subarrays
        return [n];
    }
    // Try with one subarray
    for (let sub = 2; sub < n && sub <= 32; sub *= 2) {
        let maxInnerNulls = sub == 2 ? 0 : sub == 4 ? 1 : 2;
        let top = n - sub + 1;
        p = greatestPowerOf2(top+2+maxInnerNulls);
        if (p >= top) {
            let nulls = p - top;
            let innerNulls = Math.min(maxInnerNulls, nulls);
            nulls -= innerNulls;
            return [p - 1 - nulls, sub - innerNulls];
        }
    }
    // Try with two subarrays
    for (let sub1 = 2; sub1 < n && sub1 <= 32; sub1 *= 2) {
        let maxInnerNulls1 = sub1 == 2 ? 0 : sub1 == 4 ? 1 : 2;
        for (let sub2 = 2; sub2 <= sub1; sub2 *= 2) {
            let top = n - sub1 - sub2 + 2;
            if (top < 0) break;
            let maxInnerNulls2 = sub2 == 2 ? 0 : sub2 == 4 ? 1 : 2;
            p = greatestPowerOf2(top+2+maxInnerNulls1+maxInnerNulls2);
            if (p >= top) {
                let nulls = p - top;
                let innerNulls1 = Math.min(maxInnerNulls1, nulls);
                nulls -= innerNulls1;
                let innerNulls2 = Math.min(maxInnerNulls2, nulls);
                nulls -= innerNulls2;
                return [p - 2 - nulls, sub1 - innerNulls1, sub2 - innerNulls2];
            }
        }
    }
    // Try with three subarrays
    for (let sub1 = 2; sub1 < n && sub1 <= 32; sub1 *= 2) {
        let maxInnerNulls1 = sub1 == 2 ? 0 : sub1 == 4 ? 1 : 2;
        for (let sub2 = 2; sub2 <= sub1; sub2 *= 2) {
            let maxInnerNulls2 = sub2 == 2 ? 0 : sub2 == 4 ? 1 : 2;
            for (let sub3 = 2; sub3 <= sub2; sub3 *= 2) {
                let top = n - sub1 - sub2 - sub3 + 3;
                if (top < 0) break;
                let maxInnerNulls3 = sub3 == 2 ? 0 : sub3 == 4 ? 1 : 2;
                p = greatestPowerOf2(top+2+maxInnerNulls1+maxInnerNulls2+maxInnerNulls3);
                if (p >= top) {
                    let nulls = p - top;
                    let innerNulls1 = Math.min(maxInnerNulls1, nulls);
                    nulls -= innerNulls1;
                    let innerNulls2 = Math.min(maxInnerNulls2, nulls);
                    nulls -= innerNulls2;
                    let innerNulls3 = Math.min(maxInnerNulls3, nulls);
                    nulls -= innerNulls3;
                    return [p - 3 - nulls, sub1 - innerNulls1, sub2 - innerNulls2, sub3 - innerNulls3];
                }
            }
        }
    }
    // Try with four subarrays
    for (let sub1 = 2; sub1 < n && sub1 <= 32; sub1 *= 2) {
        let maxInnerNulls1 = sub1 == 2 ? 0 : sub1 == 4 ? 1 : 2;
        for (let sub2 = 2; sub2 <= sub1; sub2 *= 2) {
            let maxInnerNulls2 = sub2 == 2 ? 0 : sub2 == 4 ? 1 : 2;
            for (let sub3 = 2; sub3 <= sub2; sub3 *= 2) {
                let maxInnerNulls3 = sub3 == 2 ? 0 : sub3 == 4 ? 1 : 2;
                for (let sub4 = 2; sub4 <= sub3; sub4 *= 2) {
                    let top = n - sub1 - sub2 - sub3 - sub4 + 4;
                    if (top < 0) break;
                    let maxInnerNulls4 = sub4 == 2 ? 0 : sub4 == 4 ? 1 : 2;
                    p = greatestPowerOf2(top+2+maxInnerNulls1+maxInnerNulls2+maxInnerNulls3+maxInnerNulls4);
                    if (p >= top) {
                        let nulls = p - top;
                        let innerNulls1 = Math.min(maxInnerNulls1, nulls);
                        nulls -= innerNulls1;
                        let innerNulls2 = Math.min(maxInnerNulls2, nulls);
                        nulls -= innerNulls2;
                        let innerNulls3 = Math.min(maxInnerNulls3, nulls);
                        nulls -= innerNulls3;
                        let innerNulls4 = Math.min(maxInnerNulls4, nulls);
                        nulls -= innerNulls4;
                        return [p - 4 - nulls, sub1 - innerNulls1, sub2 - innerNulls2, sub3 - innerNulls3, sub4 - innerNulls4];
                    }
                }
            }
        }
    }
}

I admit this code is not elegant, with lots of code repetition, but it serves the intended purpose: it generates a shape (the set of numbers explained above) for any given array size.

Encoding the shapes in less space

The numbers for the sub arrays cannot be just any number. They must be either a power of 2, or one less (when one padding null is assumed) or two less (when two null values are assumed). So the possible numbers are in this set:

[1, 2, 3, 4, 6, 7, 8, 14, 15, 16, 30, 31, 32]

The first number (which represents the count of values at the top level), can also be in that set, but can also be in the range 27 - 29. This is because the sub arrays that follow and potential null padding also count for reaching the power of 2 at the top level. This means that there are exactly 16 possible numbers in the first position of the shape "encoding". We could compress this encoding by mapping these numbers to 4-bit values (giving 16 possibilities). As it turned out there are never more than 4 subarrays needed, we will need 20 bits for encoding a shape.

Now we should determine what these 20-bit numbers are for 128 shapes and collect them in an array that can serve as lookup table.

Here is the function to encode numbers into that 20-bit encoding:

function encode(shapeNumbers) {
    let code = 0;
    for (let i = shapeNumbers.length - 1; i >= 0; i--) {
        code = code*16 + [1,2,3,4,6,7,8,14,15,16,27,28,29,30,31,32].indexOf(shapeNumbers[i]);
    }
    return code;
}

I collected the codes with this function:

function collectCodes() {
    let codes = [null];
    for (let n = 1; n <= 128; n++) {
        let shapeNumbers = shape(n);
        let code = encode(shapeNumbers);
        codes.push(code);
    }
    return codes;
}

function shape(n) { // Returns number of atomic entries, followed by data-size(s) of subarrays
    const greatestPowerOf2 = (n) =>
        n >= 32 ? 32 : n >= 16 ? 16 : n >= 8 ? 8 : n >= 4 ? 4 : n >= 2 ? 2 : 1;

    let p = greatestPowerOf2(n+2);
    if (p >= n) {
        // The only cases where there are no subarrays
        return [n];
    }
    // Try with one subarray
    for (let sub = 2; sub < n && sub <= 32; sub *= 2) {
        let maxInnerNulls = sub == 2 ? 0 : sub == 4 ? 1 : 2;
        let top = n - sub + 1;
        p = greatestPowerOf2(top+2+maxInnerNulls);
        if (p >= top && p <= 32) {
            let nulls = p - top;
            let innerNulls = Math.min(maxInnerNulls, nulls);
            nulls -= innerNulls;
            return [p - 1 - nulls, sub - innerNulls];
        }
    }
    // Try with two subarrays
    for (let sub1 = 2; sub1 < n && sub1 <= 32; sub1 *= 2) {
        let maxInnerNulls1 = sub1 == 2 ? 0 : sub1 == 4 ? 1 : 2;
        for (let sub2 = 2; sub2 <= sub1; sub2 *= 2) {
            let top = n - sub1 - sub2 + 2;
            if (top < 0) break;
            let maxInnerNulls2 = sub2 == 2 ? 0 : sub2 == 4 ? 1 : 2;
            p = greatestPowerOf2(top+2+maxInnerNulls1+maxInnerNulls2);
            if (p >= top && p <= 32) {
                let nulls = p - top;
                let innerNulls1 = Math.min(maxInnerNulls1, nulls);
                nulls -= innerNulls1;
                let innerNulls2 = Math.min(maxInnerNulls2, nulls);
                nulls -= innerNulls2;
                return [p - 2 - nulls, sub1 - innerNulls1, sub2 - innerNulls2];
            }
        }
    }
    // Try with three subarrays
    for (let sub1 = 2; sub1 < n && sub1 <= 32; sub1 *= 2) {
        let maxInnerNulls1 = sub1 == 2 ? 0 : sub1 == 4 ? 1 : 2;
        for (let sub2 = 2; sub2 <= sub1; sub2 *= 2) {
            let maxInnerNulls2 = sub2 == 2 ? 0 : sub2 == 4 ? 1 : 2;
            for (let sub3 = 2; sub3 <= sub2; sub3 *= 2) {
                let top = n - sub1 - sub2 - sub3 + 3;
                if (top < 0) break;
                let maxInnerNulls3 = sub3 == 2 ? 0 : sub3 == 4 ? 1 : 2;
                p = greatestPowerOf2(top+2+maxInnerNulls1+maxInnerNulls2+maxInnerNulls3);
                if (p >= top && p <= 32) {
                    let nulls = p - top;
                    let innerNulls1 = Math.min(maxInnerNulls1, nulls);
                    nulls -= innerNulls1;
                    let innerNulls2 = Math.min(maxInnerNulls2, nulls);
                    nulls -= innerNulls2;
                    let innerNulls3 = Math.min(maxInnerNulls3, nulls);
                    nulls -= innerNulls3;
                    return [p - 3 - nulls, sub1 - innerNulls1, sub2 - innerNulls2, sub3 - innerNulls3];
                }
            }
        }
    }
    // Try with four subarrays
    for (let sub1 = 2; sub1 < n && sub1 <= 32; sub1 *= 2) {
        let maxInnerNulls1 = sub1 == 2 ? 0 : sub1 == 4 ? 1 : 2;
        for (let sub2 = 2; sub2 <= sub1; sub2 *= 2) {
            let maxInnerNulls2 = sub2 == 2 ? 0 : sub2 == 4 ? 1 : 2;
            for (let sub3 = 2; sub3 <= sub2; sub3 *= 2) {
                let maxInnerNulls3 = sub3 == 2 ? 0 : sub3 == 4 ? 1 : 2;
                for (let sub4 = 2; sub4 <= sub3; sub4 *= 2) {
                    let top = n - sub1 - sub2 - sub3 - sub4 + 4;
                    if (top < 0) break;
                    let maxInnerNulls4 = sub4 == 2 ? 0 : sub4 == 4 ? 1 : 2;
                    p = greatestPowerOf2(top+2+maxInnerNulls1+maxInnerNulls2+maxInnerNulls3+maxInnerNulls4);
                    if (p >= top && p <= 32) {
                        let nulls = p - top;
                        let innerNulls1 = Math.min(maxInnerNulls1, nulls);
                        nulls -= innerNulls1;
                        let innerNulls2 = Math.min(maxInnerNulls2, nulls);
                        nulls -= innerNulls2;
                        let innerNulls3 = Math.min(maxInnerNulls3, nulls);
                        nulls -= innerNulls3;
                        let innerNulls4 = Math.min(maxInnerNulls4, nulls);
                        nulls -= innerNulls4;
                        return [p - 4 - nulls, sub1 - innerNulls1, sub2 - innerNulls2, sub3 - innerNulls3, sub4 - innerNulls4];
                    }
                }
            }
        }
    }
}

function encode(shapeNumbers) {
    let code = 0;
    for (let i = shapeNumbers.length - 1; i >= 0; i--) {
        code = code*16 + [1,2,3,4,6,7,8,14,15,16,27,28,29,30,31,32].indexOf(shapeNumbers[i]);
    }
    return code;
}

function collectCodes() {
    let codes = [null];
    for (let n = 1; n <= 128; n++) {
        let shapeNumbers = shape(n);
        let code = encode(shapeNumbers);
        codes.push(code);
    }
    return codes;
}

console.log(JSON.stringify(collectCodes()));

This gave this result:

[null,0,1,2,3,18,4,5,6,21,37,53,68,69,7,8,9,24,40,56,71,72,88,104,359,855,871,111,119,120,13,14,15,30,46,62,77,78,94,110,365,861,877,124,125,126,142,158,413,909,925,1405,1661,1677,1693,5788,1915,1916,1917,220,221,222,238,254,509,1005,1021,1501,1757,1773,1789,30588,2011,2012,2013,2269,2525,2541,2557,6652,14828,14844,26844,27100,27116,27132,30683,30684,3547,3548,3549,3805,4061,4077,4093,8188,16364,16380,28380,28636,28652,28668,32219,32220,36316,40412,40668,40924,40940,40956,106491,237547,237563,433883,434139,434155,434171,56794,56795,56796,60892,64988,65244,65500,65516,65532,131067,262123,262139]

The solution code

Now that we have this, we can throw away the above JavaScript functions. This array has the info we need to reproduce a shape or to translate an index into a path.

const codes = [null,0,1,2,3,18,4,5,6,21,37,53,68,69,7,8,9,24,40,56,71,72,88,104,359,855,871,111,119,120,13,14,15,30,46,62,77,78,94,110,365,861,877,124,125,126,142,158,413,909,925,1405,1661,1677,1693,5788,1915,1916,1917,220,221,222,238,254,509,1005,1021,1501,1757,1773,1789,30588,2011,2012,2013,2269,2525,2541,2557,6652,14828,14844,26844,27100,27116,27132,30683,30684,3547,3548,3549,3805,4061,4077,4093,8188,16364,16380,28380,28636,28652,28668,32219,32220,36316,40412,40668,40924,40940,40956,106491,237547,237563,433883,434139,434155,434171,56794,56795,56796,60892,64988,65244,65500,65516,65532,131067,262123,262139];

const codeMap = [1,2,3,4,6,7,8,14,15,16,27,28,29,30,31,32];

function getPath(size, i) {
    let code = codes[size];
    let limit = codeMap[code % 16];
    if (i < limit) return [i];
    for (let sub = limit; code; sub++) {
        i -= limit;
        code >>= 4;
        limit = codeMap[code % 16];
        if (i < limit) return [sub, i];
    }
}

// Demo with size 28
let size = 28;
for (let i = 0; i < size; i++) {
    console.log(i, JSON.stringify(getPath(size, i)));
}


来源: stackexchange

问:
DM user by ID c#
I want to DM a user by ID using c#, Here's what I've tried

```<code>        var u = "user-id";
        string msg = "your message";
        await UserExtensions.SendMessageAsync(u, msg);
</code>```

It returns with:

```<code>cannot convert from 'string' to 'Discord.IUser'
</code>```

I've searched and I couldn't find away to turn a string to Discord.IUser.

答:
You need to get the user as IUser, to get the user as IUser you can use _client.GetUserAsync('UserID'):

ulong u = 0; // Insert the user ID here
string msg = "your message";

// Get the user with the ID from your DiscordSocketClient
var user = _client.GetUserAsync(u).Result;

await UserExtensions.SendMessageAsync(user, msg);
```

ContentLicense: CC BY-SA 4.0

Title: DM user by ID c#

Body:
I want to DM a user by ID using c#, Here's what I've tried

        var u = "user-id";
        string msg = "your message";
        await UserExtensions.SendMessageAsync(u, msg);

It returns with:

cannot convert from 'string' to 'Discord.IUser'

I've searched and I couldn't find away to turn a string to Discord.IUser.


问:
Sending data with asp to stored procedures
How would I send data from ASP.NET to a stored procedure?

I want to be able to take data from my form and then send it to my ASP.NET (I know how to do this), then I want to use that information with the ASP.NET in a stored procedure that will update a row in my table.

答:
Below is an sample code to connect an stored procedure using ADO.NET

```<code>static string connectionString = ConfigurationSettings.AppSettings["SqlConnectionString"].ToString();
SqlConnection dbConnection = new SqlConnection();
dbConnection.ConnectionString = connectionString;
SqlCommand dbCommand = new SqlCommand("SpUpdateUser", dbConnection);
dbCommand.CommandType = CommandType.StoredProcedure;
dbCommand.Parameters.Add("@UserName", SqlDbType.VarChar).Value = userName;
dbCommand.Parameters.Add("@Age", SqlDbType.VarChar).Value = age;
dbConnection.Open();
dbCommand.ExecuteNonQuery();
dbConnection.Close();
</code>```

Here username and age are passed to the stored procedure SpUpdateUser, which expects two input arguments @UserName and @Age.

ContentLicense: CC BY-SA 3.0

Body:
Below is an sample code to connect an stored procedure using ADO.NET

static string connectionString = ConfigurationSettings.AppSettings["SqlConnectionString"].ToString();
SqlConnection dbConnection = new SqlConnection();
dbConnection.ConnectionString = connectionString;
SqlCommand dbCommand = new SqlCommand("SpUpdateUser", dbConnection);
dbCommand.CommandType = CommandType.StoredProcedure;
dbCommand.Parameters.Add("@UserName", SqlDbType.VarChar).Value = userName;
dbCommand.Parameters.Add("@Age", SqlDbType.VarChar).Value = age;
dbConnection.Open();
dbCommand.ExecuteNonQuery();
dbConnection.Close();

Here username and age are passed to the stored procedure SpUpdateUser, which expects two input arguments @UserName and @Age.


问:
Using arecord and aplay to transfer audio over a network
I have established a TCP connection between a client and a host. The idea is that the host will request an audio file from the client. At this point, the client will record what is coming through the mic and then send it back to the host.

Using arecord for the client and aplay for the host, how would I go about doing this?
I have spent the best part of 24 hours trying to work this out but just cannot do so!

Really hope someone can help and if any additional information is needed, just let me know.

Thanks

答:
TCP is difficult to make work correctly as you really need to use the UDP protocol for real-time transmissions.  TCP will always exhibit lag, and that lag will get worse over time. The reason is because as soon as a packet is dropped, TCP will try to auto recover the packet, thus introducing more lag delay in real-time transmissions.  On the other hand, UDP will ignore dropped packets, keeping your real-time interface current.  On Wi-Fi, dropped packets are common, so this issue is very noticeable.

GStreamer is pretty easy to get working over UDP.  I'm sure there's a way to get gstreamer to work with arecord and aplay, but there's really no need as gstreamer supports ALSA directly.   Example:

Microphone on client (192.168.44.10)  Transmitting to remote host (192.168.44.1):

```<code> $ gst-launch-1.0 alsasrc device=hw:2,0 ! audioconvert ! audioresample ! alawenc ! rtppcmapay ! udpsink host=192.168.44.1 port=5002
</code>```

Server host (192.168.44.1) {receiver}  Speakers:

```<code>$ gst-launch-1.0 udpsrc port=5002 caps=application/x-rtp ! rtppcmadepay ! alawdec ! audioconvert ! audioresample ! alsasink device=hw:1,0
</code>```

Use "aplay -l" to determine the hardware device number of your sound card(s).  For instance this is the output on my server host, which has the speakers attached:

```<code>$ aplay -l
**** List of PLAYBACK Hardware Devices ****
card 0: Headphones [bcm2835 Headphones], device 0: bcm2835 Headphones [bcm2835 Headphones]
  Subdevices: 8/8
  Subdevice #0: subdevice #0
  Subdevice #1: subdevice #1
  Subdevice #2: subdevice #2
  Subdevice #3: subdevice #3
  Subdevice #4: subdevice #4
  Subdevice #5: subdevice #5
  Subdevice #6: subdevice #6
  Subdevice #7: subdevice #7
card 1: Device [Generic USB Audio Device], device 0: USB Audio [USB Audio]
  Subdevices: 0/1
  Subdevice #0: subdevice #0
</code>```

Body:
TCP is difficult to make work correctly as you really need to use the UDP protocol for real-time transmissions.  TCP will always exhibit lag, and that lag will get worse over time. The reason is because as soon as a packet is dropped, TCP will try to auto recover the packet, thus introducing more lag delay in real-time transmissions.  On the other hand, UDP will ignore dropped packets, keeping your real-time interface current.  On Wi-Fi, dropped packets are common, so this issue is very noticeable.

GStreamer is pretty easy to get working over UDP.  I'm sure there's a way to get gstreamer to work with arecord and aplay, but there's really no need as gstreamer supports ALSA directly.   Example:

Microphone on client (192.168.44.10)  Transmitting to remote host (192.168.44.1):

 $ gst-launch-1.0 alsasrc device=hw:2,0 ! audioconvert ! audioresample ! alawenc ! rtppcmapay ! udpsink host=192.168.44.1 port=5002

Server host (192.168.44.1) {receiver}  Speakers:

$ gst-launch-1.0 udpsrc port=5002 caps=application/x-rtp ! rtppcmadepay ! alawdec ! audioconvert ! audioresample ! alsasink device=hw:1,0

Use "aplay -l" to determine the hardware device number of your sound card(s).  For instance this is the output on my server host, which has the speakers attached:

$ aplay -l
**** List of PLAYBACK Hardware Devices ****
card 0: Headphones [bcm2835 Headphones], device 0: bcm2835 Headphones [bcm2835 Headphones]
  Subdevices: 8/8
  Subdevice #0: subdevice #0
  Subdevice #1: subdevice #1
  Subdevice #2: subdevice #2
  Subdevice #3: subdevice #3
  Subdevice #4: subdevice #4
  Subdevice #5: subdevice #5
  Subdevice #6: subdevice #6
  Subdevice #7: subdevice #7
card 1: Device [Generic USB Audio Device], device 0: USB Audio [USB Audio]
  Subdevices: 0/1
  Subdevice #0: subdevice #0


问:
Android Studio/Emulator on macOS with ARM CPU M1
I started Android Studio 4.1 in macOS Big Sure 11.0.1 with new ARM cpu M1

After installing Rosetta: Yes, it builds !

But the Emulator has problems, it shows me CPU does not support VT-x

Even ARM images have this issue

I used the most recent, a very old api25 (Google has to do here something) but it's increadible slow

How to make the Emulator work and fast ?

答: In the emulator window (when all emulator tabs are closed) click on "Update" link and update developer tools.</p>


问:
Get all requests urls of a page in Python
Is there anyway to get all requested urls of a page? For example, when I visit a site, browsers make multiple network requests to for example, html, css and other files from the server that are necessary to display the page. I want to get all those assets urls using Python. Basically, I want to get all the urls like from the screenshot below. Can anyone point me in the right direction?

PS: I want to do this with scripting to automate the task. I know that I can accomplish this kind of task using Wireshark which I cannot automate.

答:
You can use gtk + webkit2. Check this gist for a full working example.</p>
```<code>import gi
gi.require_version('Gtk', '3.0')
gi.require_version('WebKit2', '4.0')
from gi.repository import Gtk, WebKit2

def on_resource(webview, resource, request):
    print(request.get_uri())
    return True

def load(url):
    window = Gtk.Window()
    window.set_default_size(800, 600)
    window.connect("destroy", Gtk.main_quit)

    webview = WebKit2.WebView()
    webview.connect("resource-load-started", on_resource)
    webview.load_uri(url)

    window.add(webview)
    window.show_all()

    Gtk.main()
</code>```

Body:
You can use gtk + webkit2. Check this gist for a full working example.

import gi
gi.require_version('Gtk', '3.0')
gi.require_version('WebKit2', '4.0')
from gi.repository import Gtk, WebKit2

def on_resource(webview, resource, request):
    print(request.get_uri())
    return True

def load(url):
    window = Gtk.Window()
    window.set_default_size(800, 600)
    window.connect("destroy", Gtk.main_quit)

    webview = WebKit2.WebView()
    webview.connect("resource-load-started", on_resource)
    webview.load_uri(url)

    window.add(webview)
    window.show_all()

    Gtk.main()


问:
Problem with passing data from one structore to another using pointers
I have a problem with addressing of data

I'm trying to pass all the data from one structure to another in one go using pointers

When I do this way:

```<code>#define ADDRESS_SPACE 8

struct dma_engine
{
    int (*Address)[ADDRESS_SPACE] = {nullptr};
};

struct data_engine
{
    int data[ADDRESS_SPACE] = {0x10,0x14,0x18,0x1B,0x20,0x24,0x28,0x2B};
};

int main(int argc, char* argv[])
{
    dma_engine  Dma_0;
    data_engine Data_0;

    Dma_0.Address = &Data_0.data;

    for(int i = 0;i <ADDRESS_SPACE;i++)
    {
        printf("*Dma_0.Address[%d] ---> 0x%x\n",i,*Dma_0.Address[i]);
    }
    printf("\n");

    return 0;
}
</code>```

I receive only first correct element of the data[ADDRESS_SPACE] array:

```<code>*Dma_0.Address[0] ---> 0x10
*Dma_0.Address[1] ---> 0x96408360
*Dma_0.Address[2] ---> 0x77a39b80
*Dma_0.Address[3] ---> 0xb6072bc0
*Dma_0.Address[4] ---> 0x0
*Dma_0.Address[5] ---> 0x0
*Dma_0.Address[6] ---> 0x96408368
*Dma_0.Address[7] ---> 0x0
</code>```

Is there a way to pass all the correct data in one go rather than handle every element of an array individually?

Many Thanks

答:
Assuming you truly desire a strongly-typed pointer-to-array as your Address member, the code you have is not dereferencing the target array correctly during your print loop. Specifically, this:

```<code>*Dma_0.Address[i]
</code>```

That member is a pointer to array-of-8-int. The syntax applies both * and [i] in the same expression against the same operand, which is not what you want. [] has higher precedence than * in C when applied in this context.

Therefore you may think that expression means:

dereference the pointer-to-array using * to get the base array

apply [i] against that array to acquire the desired i'th element.

But in actuality what you're asking is:

find the i'th array-of-8-int, starting at the specified address.

use * to acquire the first element of that array

Solution

Assuming you want to retain all of your existing type usage, the solution is to simply force the evaluation of * first by encasing it in something with higher priority than [] . Expression parenthesis has the highest priority in the language, and will deliver what you desire.

Therefore,

```<code>(*Dma_0.Address)[i]
</code>```

which says: evaluate *Dma_0.Address before applying [i] to the result. This one change in your program produces the following output*:

```<code>*Dma_0.Address[0] ---> 0x10
*Dma_0.Address[1] ---> 0x14
*Dma_0.Address[2] ---> 0x18
*Dma_0.Address[3] ---> 0x1b
*Dma_0.Address[4] ---> 0x20
*Dma_0.Address[5] ---> 0x24
*Dma_0.Address[6] ---> 0x28
*Dma_0.Address[7] ---> 0x2b
</code>```

which I suspect is what you're really after.

* I did not bother changing any of the printf messaging, opting only to change the problematic argument, when I tested this, in case that wasn't obvious

Body:
I have a problem with addressing of data

I'm trying to pass all the data from one structure to another in one go using pointers

When I do this way:

#define ADDRESS_SPACE 8

struct dma_engine
{
    int (*Address)[ADDRESS_SPACE] = {nullptr};
};

struct data_engine
{
    int data[ADDRESS_SPACE] = {0x10,0x14,0x18,0x1B,0x20,0x24,0x28,0x2B};
};

int main(int argc, char* argv[])
{
    dma_engine  Dma_0;
    data_engine Data_0;

    Dma_0.Address = &Data_0.data;

    for(int i = 0;i <ADDRESS_SPACE;i++)
    {
        printf("*Dma_0.Address[%d] ---> 0x%x\n",i,*Dma_0.Address[i]);
    }
    printf("\n");

    return 0;
}

I receive only first correct element of the data[ADDRESS_SPACE] array:

*Dma_0.Address[0] ---> 0x10
*Dma_0.Address[1] ---> 0x96408360
*Dma_0.Address[2] ---> 0x77a39b80
*Dma_0.Address[3] ---> 0xb6072bc0
*Dma_0.Address[4] ---> 0x0
*Dma_0.Address[5] ---> 0x0
*Dma_0.Address[6] ---> 0x96408368
*Dma_0.Address[7] ---> 0x0

Is there a way to pass all the correct data in one go rather than handle every element of an array individually?

Many Thanks

Body:
Assuming you truly desire a strongly-typed pointer-to-array as your Address member, the code you have is not dereferencing the target array correctly during your print loop. Specifically, this:

*Dma_0.Address[i]

That member is a pointer to array-of-8-int. The syntax applies both * and [i] in the same expression against the same operand, which is not what you want. [] has higher precedence than * in C when applied in this context.

Therefore you may think that expression means:

dereference the pointer-to-array using * to get the base array

apply [i] against that array to acquire the desired i'th element.

But in actuality what you're asking is:

find the i'th array-of-8-int, starting at the specified address.

use * to acquire the first element of that array

Solution

Assuming you want to retain all of your existing type usage, the solution is to simply force the evaluation of * first by encasing it in something with higher priority than [] . Expression parenthesis has the highest priority in the language, and will deliver what you desire.

Therefore,

(*Dma_0.Address)[i]

which says: evaluate *Dma_0.Address before applying [i] to the result. This one change in your program produces the following output*:

*Dma_0.Address[0] ---> 0x10
*Dma_0.Address[1] ---> 0x14
*Dma_0.Address[2] ---> 0x18
*Dma_0.Address[3] ---> 0x1b
*Dma_0.Address[4] ---> 0x20
*Dma_0.Address[5] ---> 0x24
*Dma_0.Address[6] ---> 0x28
*Dma_0.Address[7] ---> 0x2b

which I suspect is what you're really after.

* I did not bother changing any of the printf messaging, opting only to change the problematic argument, when I tested this, in case that wasn't obvious


问:
Scrape Text and save File with Bold Text Intact?
I am very new to Python and webscraping. I have tried to search for an answer, but cannot find it. It might be because I don't know the terminology to ask the right question.

I am trying to web scrape using python - beautiful soup in order to extract the English transliterations of verb tables from a website (https://www.pealim.com/dict/28-lavo/) that conjugates modern Hebrew verbs. I am then trying to save the text to a txt file. The sticking point is I am trying to get the bold formatting tag to remain intact during the scraping/saving to file, because they are important to know where the stress falls in the word.

Here is an example of what I am getting:
ba'im

And here is what I would like:
ba'im

I'm including an image because when I post the HTML code, it's automatically rendering it:

What I'm looking to do

By looking around the forums, I have come up with code gets me close to what I need, but I cannot figure out how to get the bold tags in there as well.

```<code>import requests
from bs4 import BeautifulSoup as bs

#load webpage content
r = requests.get("https://www.pealim.com/dict/28-lavo/")

#Convert to a soup object
soup = bs(r.content)

#Find the transliterations from the verb tables with the stress bolded
mine = [element.text for element in soup.find_all("div", "transcription")]

#Save to file
with open("lavo.txt", "w") as output:
    for i in mine:
        output.write('%s\n' % i)
</code>```

答:
You can use .contents property, cast it to string and join it. For example:

import requests
from bs4 import BeautifulSoup as bs

# load webpage content
r = requests.get("https://www.pealim.com/dict/28-lavo/")

# Convert to a soup object
soup = bs(r.content, "html.parser")

# Find the transliterations from the verb tables with the stress bolded
mine = [
    "".join(map(str, element.contents))
    for element in soup.find_all("div", "transcription")
]

with open("lavo.txt", "w") as output:
    for i in mine:
        output.write("%s\n" % i)
```
<p>Saves <code>lavo.txt</code>:</p>
<pre class="lang-none prettyprint-override"><code>b<b>a</b>
ba'<b>a</b>
ba'<b>i</b>m
ba'<b>o</b>t
b<b>a</b>ti
b<b>a</b>nu
b<b>a</b>ta
b<b>a</b>t
bat<b>e</b>m

...
</code>```

Body:
I am very new to Python and webscraping. I have tried to search for an answer, but cannot find it. It might be because I don't know the terminology to ask the right question.

I am trying to web scrape using python - beautiful soup in order to extract the English transliterations of verb tables from a website (https://www.pealim.com/dict/28-lavo/) that conjugates modern Hebrew verbs. I am then trying to save the text to a txt file. The sticking point is I am trying to get the bold formatting tag to remain intact during the scraping/saving to file, because they are important to know where the stress falls in the word.

Here is an example of what I am getting:
ba'im

And here is what I would like:
ba'im

I'm including an image because when I post the HTML code, it's automatically rendering it:

What I'm looking to do

By looking around the forums, I have come up with code gets me close to what I need, but I cannot figure out how to get the bold tags in there as well.

import requests
from bs4 import BeautifulSoup as bs

#load webpage content
r = requests.get("https://www.pealim.com/dict/28-lavo/")

#Convert to a soup object
soup = bs(r.content)

#Find the transliterations from the verb tables with the stress bolded
mine = [element.text for element in soup.find_all("div", "transcription")]

#Save to file
with open("lavo.txt", "w") as output:
    for i in mine:
        output.write('%s\n' % i)

Body:
You can use .contents property, cast it to string and join it. For example:

import requests
from bs4 import BeautifulSoup as bs

# load webpage content
r = requests.get("https://www.pealim.com/dict/28-lavo/")

# Convert to a soup object
soup = bs(r.content, "html.parser")

# Find the transliterations from the verb tables with the stress bolded
mine = [
    "".join(map(str, element.contents))
    for element in soup.find_all("div", "transcription")
]

with open("lavo.txt", "w") as output:
    for i in mine:
        output.write("%s\n" % i)

Saves lavo.txt:

b<b>a</b>
ba'<b>a</b>
ba'<b>i</b>m
ba'<b>o</b>t
b<b>a</b>ti
b<b>a</b>nu
b<b>a</b>ta
b<b>a</b>t
bat<b>e</b>m

...


问:
Is there a limit to how large an array can be inside of a function in cython?
This code compiles and runs just fine:

```<code>cdef enum:
    SIZE = 1000000
cdef int ar[SIZE]
cdef int i

for i in range(SIZE):
    ar[i] = i
print(ar[5])
</code>```

but this code:

```<code>cdef enum:
    SIZE = 1000000

def test():
    cdef int ar[SIZE]
    cdef int i

    for i in range(SIZE):
        ar[i] = i
    print(ar[5])

test()
</code>```

crashes the python kernel (I'm running this with jupyter magic).

Is there some limit to how large arrays can in inside of functions? If there is, is there a way to remove that limit?

答: In the first case, the array is a global-variable statically-defined plain C array which are allocated on the heap. In the second case, the local-variable array is allocated on the stack. The thing is the stack has a fixed maximum size (generally something like 2 MiB but this can change a lot on different platforms). If you try to write something allocated beyond the limit of the stack you get a crash or a nice stack overflow error if you are lucky. Note that function calls and local variables temporary take some space in the stack. While the size of the stack can be resized, the best solution is to use dynamic allocation for relatively big arrays or the ones with a variable size. Hopefully, this is possible in Cython using malloc and free (see the documentation) but you should be careful not to forget the free (nor to free the array twice). An alternative solution is to create a Numpy array and then work with memory views (this is more expensive but prevent any possible leak).

Body:
This code compiles and runs just fine:

cdef enum:
    SIZE = 1000000
cdef int ar[SIZE]
cdef int i

for i in range(SIZE):
    ar[i] = i
print(ar[5])

but this code:

cdef enum:
    SIZE = 1000000

def test():
    cdef int ar[SIZE]
    cdef int i

    for i in range(SIZE):
        ar[i] = i
    print(ar[5])

test()

crashes the python kernel (I'm running this with jupyter magic).

Is there some limit to how large arrays can in inside of functions? If there is, is there a way to remove that limit?


问:
Tools for scheduler debugging in Linux
I have an embedded linux system containing two threads that must run in real time (or soft real time). When using SCHED_OTHER, I noted a lot of jitter but the two threads always executed within their allocated time.

I have applied the RT patch with PREEMPT_RT enabled, and running those two threads with SCHED_FIFO (with a high thread priority of ~80) leads to a lot less jitter, it's overall a lot better, except once and a while both threads miss their deadline (instead of executing every 10 ms or so, they may not get schedule for almost a second!).

I wanted to ask which tool is best when debugging linux scheduling (under RT) on an embedded Linux OS. ftrace came to mind, but I don't know if it is the best and/or only tool. My goal is to find out why the two threads don't get scheduled for an extensive amount of time once in a while.

UPDATE: I've been running ftrace today with wakeup_rt. wakeup_rt as a tracer didn't get the job done: the max latency it recorded was 5ms when my thread can run up to 1000ms late. Maybe it is not a scheduler issue? What other tracer in ftrace would you recommend please?

答:
Try using rt-app which is used by scheduler developers.

You might also want to use SCHED_DEADLINE instead of SCHED_FIFO to reduce your jitter.

Title: Tools for scheduler debugging in Linux


问:
Can we use numpy array as input to perform Tfidfvectorizer() on text data, inside of make_column_transform()?
I am trying to perform multiple column transformations using OneHotEncoder() and TfidfVectorizer() on my training data which is a numpy array. I am trying to use make_column_transformer() to perform all transformations at once. X_train is my input data.

Input Data

```<code>print(X_train.shape)
>>> (75117, 6)
</code>```

Sample instance

```<code>print(X_train[5,:])
>>> ['electrical_contractor_license-electrical_contractor_license-general_contractor_license-refrigeration_contractor_lic.'
 'brennan_heating_company_inc' 'instal new electr boiler'
 'single_family_/_duplex' 0.0 0]
</code>```

Column Transformation code

```<code>column_trans = make_column_transformer(
    (OneHotEncoder(sparse=False, handle_unknown='ignore'), [0, 1, 3]),
    (TfidfVectorizer(min_df=1, stop_words='english', lowercase=False), [2]),
     remainder='passthrough')

z = column_trans.fit_transform(X_train)
</code>```

Using the above code, OneHotEncoder() works fine on columns (0, 1, 3) but when I add TfidfVectorizer() for column 2 it throws the following error.

TypeError: cannot use a string pattern on a bytes-like object

Full Error:

```<code>---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1167-68498e1c856a> in <module>
      4      remainder='passthrough')
      5
----> 6 z = column_trans.fit_transform(X_train)
      7 print(z[0,:].shape)
      8 print(z[0,:])

/opt/anaconda3/lib/python3.7/site-packages/sklearn/compose/_column_transformer.py in fit_transform(self, X, y)
    516         self._validate_remainder(X)
    517
--> 518         result = self._fit_transform(X, y, _fit_transform_one)
    519
    520         if not result:

/opt/anaconda3/lib/python3.7/site-packages/sklearn/compose/_column_transformer.py in _fit_transform(self, X, y, func, fitted)
    455                     message=self._log_message(name, idx, len(transformers)))
    456                 for idx, (name, trans, column, weight) in enumerate(
--> 457                         self._iter(fitted=fitted, replace_strings=True), 1))
    458         except ValueError as e:
    459             if "Expected 2D array, got 1D array instead" in str(e):

/opt/anaconda3/lib/python3.7/site-packages/joblib/parallel.py in __call__(self, iterable)
   1005                 self._iterating = self._original_iterator is not None
   1006
-> 1007             while self.dispatch_one_batch(iterator):
   1008                 pass
   1009

/opt/anaconda3/lib/python3.7/site-packages/joblib/parallel.py in dispatch_one_batch(self, iterator)
    833                 return False
    834             else:
--> 835                 self._dispatch(tasks)
    836                 return True
    837

/opt/anaconda3/lib/python3.7/site-packages/joblib/parallel.py in _dispatch(self, batch)
    752         with self._lock:
    753             job_idx = len(self._jobs)
--> 754             job = self._backend.apply_async(batch, callback=cb)
    755             # A job can complete so quickly than its callback is
    756             # called before we get here, causing self._jobs to

/opt/anaconda3/lib/python3.7/site-packages/joblib/_parallel_backends.py in apply_async(self, func, callback)
    207     def apply_async(self, func, callback=None):
    208         """Schedule a func to be run"""
--> 209         result = ImmediateResult(func)
    210         if callback:
    211             callback(result)

/opt/anaconda3/lib/python3.7/site-packages/joblib/_parallel_backends.py in __init__(self, batch)
    588         # Don't delay the application, to avoid keeping the input
    589         # arguments in memory
--> 590         self.results = batch()
    591
    592     def get(self):

/opt/anaconda3/lib/python3.7/site-packages/joblib/parallel.py in __call__(self)
    254         with parallel_backend(self._backend, n_jobs=self._n_jobs):
    255             return [func(*args, **kwargs)
--> 256                     for func, args, kwargs in self.items]
    257
    258     def __len__(self):

/opt/anaconda3/lib/python3.7/site-packages/joblib/parallel.py in <listcomp>(.0)
    254         with parallel_backend(self._backend, n_jobs=self._n_jobs):
    255             return [func(*args, **kwargs)
--> 256                     for func, args, kwargs in self.items]
    257
    258     def __len__(self):

/opt/anaconda3/lib/python3.7/site-packages/sklearn/pipeline.py in _fit_transform_one(transformer, X, y, weight, message_clsname, message, **fit_params)
    726     with _print_elapsed_time(message_clsname, message):
    727         if hasattr(transformer, 'fit_transform'):
--> 728             res = transformer.fit_transform(X, y, **fit_params)
    729         else:
    730             res = transformer.fit(X, y, **fit_params).transform(X)

/opt/anaconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in fit_transform(self, raw_documents, y)
   1857         """
   1858         self._check_params()
-> 1859         X = super().fit_transform(raw_documents)
   1860         self._tfidf.fit(X)
   1861         # X is already a transformed view of raw_documents so

/opt/anaconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in fit_transform(self, raw_documents, y)
   1218
   1219         vocabulary, X = self._count_vocab(raw_documents,
-> 1220                                           self.fixed_vocabulary_)
   1221
   1222         if self.binary:

/opt/anaconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in _count_vocab(self, raw_documents, fixed_vocab)
   1129         for doc in raw_documents:
   1130             feature_counter = {}
-> 1131             for feature in analyze(doc):
   1132                 try:
   1133                     feature_idx = vocabulary[feature]

/opt/anaconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in _analyze(doc, analyzer, tokenizer, ngrams, preprocessor, decoder, stop_words)
    103             doc = preprocessor(doc)
    104         if tokenizer is not None:
--> 105             doc = tokenizer(doc)
    106         if ngrams is not None:
    107             if stop_words is not None:

TypeError: cannot use a string pattern on a bytes-like object
</code>```

It does work when I use it outside of make_column_transformer(), but the reason why I am using make_column_transformer() and not separately, is because, if I do One hot encoding first and then tfidf, then its quite possible that the number of features generated by one hot encoder might vary, so hard coding the column index for tfidf might not be a good idea.

```<code>tf = TfidfVectorizer(min_df=1, stop_words='english')
n = tf.fit_transform(X_train[:,2])
n.toarray()

>>> array([[0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       ...,
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.]])
</code>```

答:
I had the same problem. The solution is to replace:

```<code>  (TfidfVectorizer(min_df=1, stop_words='english', lowercase=False), [2]),
</code>```

by

```<code>  (TfidfVectorizer(min_df=1, stop_words='english', lowercase=False), 2),
</code>```

Otherwise the TfidfVectorizer, upon calling fit, receives 2D data (of shape (#documents, 1)) which it can't handle.

It is explained in ColumnTransformer documentation:

columns: str, array-like of str, int, array-like of int, array-like of bool, slice or callable

Indexes the data on its second axis. Integers are interpreted as positional columns, while strings can reference DataFrame columns by name. A scalar string or int should be used where transformer expects X to be a 1d array-like (vector), otherwise a 2d array will be passed to the transformer. [...]

Body:
I am trying to perform multiple column transformations using OneHotEncoder() and TfidfVectorizer() on my training data which is a numpy array. I am trying to use make_column_transformer() to perform all transformations at once. X_train is my input data.

Input Data

print(X_train.shape)
>>> (75117, 6)

Sample instance

print(X_train[5,:])
>>> ['electrical_contractor_license-electrical_contractor_license-general_contractor_license-refrigeration_contractor_lic.'
 'brennan_heating_company_inc' 'instal new electr boiler'
 'single_family_/_duplex' 0.0 0]

Column Transformation code

column_trans = make_column_transformer(
    (OneHotEncoder(sparse=False, handle_unknown='ignore'), [0, 1, 3]),
    (TfidfVectorizer(min_df=1, stop_words='english', lowercase=False), [2]),
     remainder='passthrough')

z = column_trans.fit_transform(X_train)

Using the above code, OneHotEncoder() works fine on columns (0, 1, 3) but when I add TfidfVectorizer() for column 2 it throws the following error.

TypeError: cannot use a string pattern on a bytes-like object

Full Error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1167-68498e1c856a> in <module>
      4      remainder='passthrough')
      5
----> 6 z = column_trans.fit_transform(X_train)
      7 print(z[0,:].shape)
      8 print(z[0,:])

/opt/anaconda3/lib/python3.7/site-packages/sklearn/compose/_column_transformer.py in fit_transform(self, X, y)
    516         self._validate_remainder(X)
    517
--> 518         result = self._fit_transform(X, y, _fit_transform_one)
    519
    520         if not result:

/opt/anaconda3/lib/python3.7/site-packages/sklearn/compose/_column_transformer.py in _fit_transform(self, X, y, func, fitted)
    455                     message=self._log_message(name, idx, len(transformers)))
    456                 for idx, (name, trans, column, weight) in enumerate(
--> 457                         self._iter(fitted=fitted, replace_strings=True), 1))
    458         except ValueError as e:
    459             if "Expected 2D array, got 1D array instead" in str(e):

/opt/anaconda3/lib/python3.7/site-packages/joblib/parallel.py in __call__(self, iterable)
   1005                 self._iterating = self._original_iterator is not None
   1006
-> 1007             while self.dispatch_one_batch(iterator):
   1008                 pass
   1009

/opt/anaconda3/lib/python3.7/site-packages/joblib/parallel.py in dispatch_one_batch(self, iterator)
    833                 return False
    834             else:
--> 835                 self._dispatch(tasks)
    836                 return True
    837

/opt/anaconda3/lib/python3.7/site-packages/joblib/parallel.py in _dispatch(self, batch)
    752         with self._lock:
    753             job_idx = len(self._jobs)
--> 754             job = self._backend.apply_async(batch, callback=cb)
    755             # A job can complete so quickly than its callback is
    756             # called before we get here, causing self._jobs to

/opt/anaconda3/lib/python3.7/site-packages/joblib/_parallel_backends.py in apply_async(self, func, callback)
    207     def apply_async(self, func, callback=None):
    208         """Schedule a func to be run"""
--> 209         result = ImmediateResult(func)
    210         if callback:
    211             callback(result)

/opt/anaconda3/lib/python3.7/site-packages/joblib/_parallel_backends.py in __init__(self, batch)
    588         # Don't delay the application, to avoid keeping the input
    589         # arguments in memory
--> 590         self.results = batch()
    591
    592     def get(self):

/opt/anaconda3/lib/python3.7/site-packages/joblib/parallel.py in __call__(self)
    254         with parallel_backend(self._backend, n_jobs=self._n_jobs):
    255             return [func(*args, **kwargs)
--> 256                     for func, args, kwargs in self.items]
    257
    258     def __len__(self):

/opt/anaconda3/lib/python3.7/site-packages/joblib/parallel.py in <listcomp>(.0)
    254         with parallel_backend(self._backend, n_jobs=self._n_jobs):
    255             return [func(*args, **kwargs)
--> 256                     for func, args, kwargs in self.items]
    257
    258     def __len__(self):

/opt/anaconda3/lib/python3.7/site-packages/sklearn/pipeline.py in _fit_transform_one(transformer, X, y, weight, message_clsname, message, **fit_params)
    726     with _print_elapsed_time(message_clsname, message):
    727         if hasattr(transformer, 'fit_transform'):
--> 728             res = transformer.fit_transform(X, y, **fit_params)
    729         else:
    730             res = transformer.fit(X, y, **fit_params).transform(X)

/opt/anaconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in fit_transform(self, raw_documents, y)
   1857         """
   1858         self._check_params()
-> 1859         X = super().fit_transform(raw_documents)
   1860         self._tfidf.fit(X)
   1861         # X is already a transformed view of raw_documents so

/opt/anaconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in fit_transform(self, raw_documents, y)
   1218
   1219         vocabulary, X = self._count_vocab(raw_documents,
-> 1220                                           self.fixed_vocabulary_)
   1221
   1222         if self.binary:

/opt/anaconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in _count_vocab(self, raw_documents, fixed_vocab)
   1129         for doc in raw_documents:
   1130             feature_counter = {}
-> 1131             for feature in analyze(doc):
   1132                 try:
   1133                     feature_idx = vocabulary[feature]

/opt/anaconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in _analyze(doc, analyzer, tokenizer, ngrams, preprocessor, decoder, stop_words)
    103             doc = preprocessor(doc)
    104         if tokenizer is not None:
--> 105             doc = tokenizer(doc)
    106         if ngrams is not None:
    107             if stop_words is not None:

TypeError: cannot use a string pattern on a bytes-like object

It does work when I use it outside of make_column_transformer(), but the reason why I am using make_column_transformer() and not separately, is because, if I do One hot encoding first and then tfidf, then its quite possible that the number of features generated by one hot encoder might vary, so hard coding the column index for tfidf might not be a good idea.

tf = TfidfVectorizer(min_df=1, stop_words='english')
n = tf.fit_transform(X_train[:,2])
n.toarray()

>>> array([[0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       ...,
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.]])

Body:
I had the same problem. The solution is to replace:

  (TfidfVectorizer(min_df=1, stop_words='english', lowercase=False), [2]),

by

  (TfidfVectorizer(min_df=1, stop_words='english', lowercase=False), 2),

Otherwise the TfidfVectorizer, upon calling fit, receives 2D data (of shape (#documents, 1)) which it can't handle.

It is explained in ColumnTransformer documentation:

columns: str, array-like of str, int, array-like of int, array-like of bool, slice or callable

Indexes the data on its second axis. Integers are interpreted as positional columns, while strings can reference DataFrame columns by name. A scalar string or int should be used where transformer expects X to be a 1d array-like (vector), otherwise a 2d array will be passed to the transformer. [...]


问:
How do I obtain the result of a boolean XPath expression (e.g. 'false')?
I would like to have Java generate a '0' in case an XPath expression evaluates to 'false'.

I have this Java code:

```<code>//Read the input XML document
private SAXBuilder parser = new SAXBuilder();
    private Document characters;
    private XPath pProbs;
    private List<Attribute> probs;
    private Double[] dprobs;
    private String pathToSourceXml;
    private String content1, content2, content3, content4, content5;

    characters = parser.build(pathToSourceXml);
    pProbs = XPath.newInstance("/n-grams-sorted/n-gram[contains(.,"+content1+") or contains(.,"+content2+") or contains(.,"+content3+") or contains(.,"+content4+") or contains(.,"+content5+")]/@probability");
    probs = (List<Attribute>) pProbs.selectNodes(characters);
    ...

//Return all the values of the @probability attibrutes
public Double[] getProbs(String pathToSourceXml) {
    this.pathToSourceXml = pathToSourceXml;

    List<Double> theDoubles = new ArrayList();
    dprobs = new Double[5];
    for (int i=0; i<probs.size(); i++) {
        theDoubles.add(Double.parseDouble(probs.get(i).getValue()));
        dprobs[i] = theDoubles.get(i);
    }
    return dprobs;
}
</code>```

The problem here is that this code:

```<code>pProbs = XPath.newInstance("/n-grams-sorted/n-gram[contains(.,"+content1+") or contains(.,"+content2+") or contains(.,"+content3+") or contains(.,"+content4+") or contains(.,"+content5+")]/@probability");
</code>```

only returns 4 elements because 'content1' returns 'false'. There are no n-gram nodes whose content contains the string '$ $ $'. However, the rest of the content nodes evaluates to 'true'.

If one of these contains() expressions evaluates to 'false', I must draw a '0' in my Xaml code. A '0' must appear on the screen, such as:

```<code>'# # #' = 0.0015
'? ? ?' = 0.0047
'k i d' = 0.0012
'$ $ $' = 0
</code>```

I can't fetch this '0'. I don't know how to say: "return a zero in case that there are no nodes that contain '$ $ $' as content.

Any ideas on how to do this?

Thank you.

答:
Here is the answer:

```<code>public class XPathCharacters {
    private SAXBuilder parser = new SAXBuilder();
    private Document characters;
    private XPath pProbs, pContent1, pContent2, pContent3, pContent4, pContent5;
    private List<Attribute> probs1, probs2, probs3, probs4, probs5;
    private List<List<Attribute>> probs;
    private List<String> noNodes;
    private Double[] dprobs;
    private String pathToSourceXml;
    private String content1, content2, content3, content4, content5;
    private int noNode;

    public XPathCharacters(){

    }

    public XPathCharacters(String path, String content1,String content2,String content3,String content4,String content5){
        setPathToSourceXml(path);
        this.content1 = content1;
        this.content2 = content2;
        this.content3 = content3;
        this.content4 = content4;
        this.content5 = content5;
        noNode = 0;
        initiate();
    }

    public void initiate() {
        try {
        //Evaluate each XPath expression seperately
            characters = parser.build(pathToSourceXml);
            pContent1 = XPath.newInstance("/n-grams-sorted/n-gram[contains(.,"+content1+")]/@probability");
            pContent2 = XPath.newInstance("/n-grams-sorted/n-gram[contains(.,"+content2+")]/@probability");
            pContent3 = XPath.newInstance("/n-grams-sorted/n-gram[contains(.,"+content3+")]/@probability");
            pContent4 = XPath.newInstance("/n-grams-sorted/n-gram[contains(.,"+content4+")]/@probability");
            pContent5 = XPath.newInstance("/n-grams-sorted/n-gram[contains(.,"+content5+")]/@probability");

            //Convert the result of the above XPath expressions to nodes
            probs1 = (List<Attribute>) pContent1.selectNodes(characters);
            probs2 = (List<Attribute>) pContent2.selectNodes(characters);
            probs3 = (List<Attribute>) pContent3.selectNodes(characters);
            probs4 = (List<Attribute>) pContent4.selectNodes(characters);
            probs5 = (List<Attribute>) pContent5.selectNodes(characters);

        } catch (JDOMException jdome) {
            System.out.println("Error at XPathInformationgain.initiate(): JDOMException: " + jdome.getMessage());
        } catch (IOException ioe) {
            System.out.println("Error at XPathInformationgain.initiate(): IOException: " + ioe.getMessage());
        }
    }

    private void setPathToSourceXml(String path) {
        this.pathToSourceXml = path;
    }

    public Double[] getProbs(String pathToSourceXml) {
        this.pathToSourceXml = pathToSourceXml;

        probs = new ArrayList();
        probs.add(probs1);
        probs.add(probs2);
        probs.add(probs3);
        probs.add(probs4);
        probs.add(probs5);

        List<Double> theDoubles = new ArrayList();
        dprobs = new Double[5];
        noNodes = new ArrayList();
        int j=0;
        for (int i=0; i<5; i++) {
            if (probs.get(i).size() > 0) {
                System.out.println("i: "+i);
                System.out.println("j: "+j);
               //Add the value of all these nodes to an Array so you can work with them
               theDoubles.add(Double.parseDouble(probs.get(i).get(0).getValue()));
                dprobs[j] = theDoubles.get(j);
            }
            else {
                //If one of the values happens to be a zero-length array, add 0.0 to the array of values
                theDoubles.add(0.0);
            }
        }
        return dprobs;
    }

    public List<String> getNoNodes() {
        return noNodes;
    }
}
</code>```

(This code is not working for what I want to accomplish outside of this question, but it works for this question because I have a way of returning a '0' in java, when the XPath expression yields zero-length).

The solution is this part really:

```<code>        else {
            theDoubles.add(0.0);
        }
</code>```

Body:
I would like to have Java generate a '0' in case an XPath expression evaluates to 'false'.

I have this Java code:

//Read the input XML document
private SAXBuilder parser = new SAXBuilder();
    private Document characters;
    private XPath pProbs;
    private List<Attribute> probs;
    private Double[] dprobs;
    private String pathToSourceXml;
    private String content1, content2, content3, content4, content5;

    characters = parser.build(pathToSourceXml);
    pProbs = XPath.newInstance("/n-grams-sorted/n-gram[contains(.,"+content1+") or contains(.,"+content2+") or contains(.,"+content3+") or contains(.,"+content4+") or contains(.,"+content5+")]/@probability");
    probs = (List<Attribute>) pProbs.selectNodes(characters);
    ...

//Return all the values of the @probability attibrutes
public Double[] getProbs(String pathToSourceXml) {
    this.pathToSourceXml = pathToSourceXml;

    List<Double> theDoubles = new ArrayList();
    dprobs = new Double[5];
    for (int i=0; i<probs.size(); i++) {
        theDoubles.add(Double.parseDouble(probs.get(i).getValue()));
        dprobs[i] = theDoubles.get(i);
    }
    return dprobs;
}

The problem here is that this code:

pProbs = XPath.newInstance("/n-grams-sorted/n-gram[contains(.,"+content1+") or contains(.,"+content2+") or contains(.,"+content3+") or contains(.,"+content4+") or contains(.,"+content5+")]/@probability");

only returns 4 elements because 'content1' returns 'false'. There are no n-gram nodes whose content contains the string '$ $ $'. However, the rest of the content nodes evaluates to 'true'.

If one of these contains() expressions evaluates to 'false', I must draw a '0' in my Xaml code. A '0' must appear on the screen, such as:

'# # #' = 0.0015
'? ? ?' = 0.0047
'k i d' = 0.0012
'$ $ $' = 0

I can't fetch this '0'. I don't know how to say: "return a zero in case that there are no nodes that contain '$ $ $' as content.

Any ideas on how to do this?

Thank you.

Body:
Here is the answer:

public class XPathCharacters {
    private SAXBuilder parser = new SAXBuilder();
    private Document characters;
    private XPath pProbs, pContent1, pContent2, pContent3, pContent4, pContent5;
    private List<Attribute> probs1, probs2, probs3, probs4, probs5;
    private List<List<Attribute>> probs;
    private List<String> noNodes;
    private Double[] dprobs;
    private String pathToSourceXml;
    private String content1, content2, content3, content4, content5;
    private int noNode;

    public XPathCharacters(){

    }

    public XPathCharacters(String path, String content1,String content2,String content3,String content4,String content5){
        setPathToSourceXml(path);
        this.content1 = content1;
        this.content2 = content2;
        this.content3 = content3;
        this.content4 = content4;
        this.content5 = content5;
        noNode = 0;
        initiate();
    }

    public void initiate() {
        try {
        //Evaluate each XPath expression seperately
            characters = parser.build(pathToSourceXml);
            pContent1 = XPath.newInstance("/n-grams-sorted/n-gram[contains(.,"+content1+")]/@probability");
            pContent2 = XPath.newInstance("/n-grams-sorted/n-gram[contains(.,"+content2+")]/@probability");
            pContent3 = XPath.newInstance("/n-grams-sorted/n-gram[contains(.,"+content3+")]/@probability");
            pContent4 = XPath.newInstance("/n-grams-sorted/n-gram[contains(.,"+content4+")]/@probability");
            pContent5 = XPath.newInstance("/n-grams-sorted/n-gram[contains(.,"+content5+")]/@probability");

            //Convert the result of the above XPath expressions to nodes
            probs1 = (List<Attribute>) pContent1.selectNodes(characters);
            probs2 = (List<Attribute>) pContent2.selectNodes(characters);
            probs3 = (List<Attribute>) pContent3.selectNodes(characters);
            probs4 = (List<Attribute>) pContent4.selectNodes(characters);
            probs5 = (List<Attribute>) pContent5.selectNodes(characters);

        } catch (JDOMException jdome) {
            System.out.println("Error at XPathInformationgain.initiate(): JDOMException: " + jdome.getMessage());
        } catch (IOException ioe) {
            System.out.println("Error at XPathInformationgain.initiate(): IOException: " + ioe.getMessage());
        }
    }

    private void setPathToSourceXml(String path) {
        this.pathToSourceXml = path;
    }

    public Double[] getProbs(String pathToSourceXml) {
        this.pathToSourceXml = pathToSourceXml;

        probs = new ArrayList();
        probs.add(probs1);
        probs.add(probs2);
        probs.add(probs3);
        probs.add(probs4);
        probs.add(probs5);

        List<Double> theDoubles = new ArrayList();
        dprobs = new Double[5];
        noNodes = new ArrayList();
        int j=0;
        for (int i=0; i<5; i++) {
            if (probs.get(i).size() > 0) {
                System.out.println("i: "+i);
                System.out.println("j: "+j);
               //Add the value of all these nodes to an Array so you can work with them
               theDoubles.add(Double.parseDouble(probs.get(i).get(0).getValue()));
                dprobs[j] = theDoubles.get(j);
            }
            else {
                //If one of the values happens to be a zero-length array, add 0.0 to the array of values
                theDoubles.add(0.0);
            }
        }
        return dprobs;
    }

    public List<String> getNoNodes() {
        return noNodes;
    }
}

(This code is not working for what I want to accomplish outside of this question, but it works for this question because I have a way of returning a '0' in java, when the XPath expression yields zero-length).

The solution is this part really:

        else {
            theDoubles.add(0.0);
        }


问:
What is the use case for dynamic component loading?
I am having a little trouble understanding what the use case is for dynamic component loading. The explanation from the ng docs states:

An application might need to load new components at runtime.

Could you not handle this scenario with an ngSwtich that then loads a child-component?
Or is the use case for components that are not even bundled with the app that can be loaded dynamically at runtime? So in this case the component is run through the Angular compiler at runtime and compiles the component dynamically, then injecting the compiled component into the running application, or?

Aside from the technical side of this, when would this be necessary? Don't you normally have dynamic data feeding a component rather than having dynamic components inserted into a dom node? Maybe I am just lacking creativity, but I can not see when dynamically loading a component would be necessary.

答:
Component "loading" (as the term is used on that page) refers to the creation of a component instance within an identified area of the DOM (the ViewContainer).

Component loading is static if the decision to create the component is made at build time (by mentioning the component in a template to be compiled). Component loading is dynamic if the decision to create the component is made at runtime (by telling a ViewContainer to create a particular component).

Dynamic component loading is used by structural directives such as ngIf, ngSwitch, ngFor, router-outlet, ..., to conditionally select the type or number of components to create, as well as manage their lifetime.

In describing how these components manage that feat, the documentation empowers us to write our own structural directives should the default directives prove inadequate for our needs.

For instance, imagine router-outlet had not been invented yet. You could work around this by having one huge ngSwitch instead, but you probably agree that router-outlet is more convenient to use. Having seen the need for such specialized components, the angular team opted to give us the means to write them.

PS: I agree that dynamic component "loading" is a misleading phrase, because one naturally thinks about loading code from the server, which is a wholly different beast. Of course, it is possible to combine dynamic component creation with dynamic code loading, as the lazy routes feature of the router demonstrates, but it quite possible, and normal, to use dynamic components without dynamic code loading.


问:
How to use Image component in Next.js with unknown width and height
Since version 10, Next.js comes with the built-in Image component which provides the feature of image optimization and resizing image responsively. I like it a lot and I've been using it across my website for images with a fixed size. According to the official documentation, width and height are required props unless it's for layout=fill.

Now, I would like to use Image component even when I don't know width and height ahead of time. I have blog resources that come from a CMS where the exact size of the image is unknown.  In this case, is there any way I could use Image component?

Any help would be appreciated.

答:
I had the same issue too: I wanted to use Image from next/image in a masonry ImageList from MUI...

Anyway, this is the trick I used to get the size of my image and so to handle the size of the container:

    import React, { FC, useState } from 'react';
    import styled from 'styled-components';
    import Image from 'next/image';

    interface Props {
      src: string;
    }
    export const MasonryItem: FC<Props> = ({ src }) => {
      const [paddingTop, setPaddingTop] = useState('0');

      return (
        <Container style={{ paddingTop }}>
          <Image
            src={src}
            layout="fill"
            objectFit="contain"
            onLoad={({ target }) => {
              const { naturalWidth, naturalHeight } = target as HTMLImageElement;
              setPaddingTop(`calc(100% / (${naturalWidth} / ${naturalHeight})`);
            }}
          />
        </Container>
      );
    };

    const Container = styled.div`
      position: relative;
    `;
```

The idea: get size of image when it's loaded, calc the ratio and use it to make the container fill the require space with padding-top. This way no matter your image size the container will be fit to it.

Remark, I didn't use width or height on the container because I didn't need it with my project, but maybe you will have to set one of both for your project.

As I'm still beginning with React & NextJS & typescript, maybe the solution can be prettier, if someone have an idea to improve it I'll be pleased to read it!

Body:
I had the same issue too: I wanted to use Image from next/image in a masonry ImageList from MUI...

Anyway, this is the trick I used to get the size of my image and so to handle the size of the container:

    import React, { FC, useState } from 'react';
    import styled from 'styled-components';
    import Image from 'next/image';

    interface Props {
      src: string;
    }
    export const MasonryItem: FC<Props> = ({ src }) => {
      const [paddingTop, setPaddingTop] = useState('0');

      return (
        <Container style={{ paddingTop }}>
          <Image
            src={src}
            layout="fill"
            objectFit="contain"
            onLoad={({ target }) => {
              const { naturalWidth, naturalHeight } = target as HTMLImageElement;
              setPaddingTop(`calc(100% / (${naturalWidth} / ${naturalHeight})`);
            }}
          />
        </Container>
      );
    };

    const Container = styled.div`
      position: relative;
    `;

The idea: get size of image when it's loaded, calc the ratio and use it to make the container fill the require space with padding-top. This way no matter your image size the container will be fit to it.

Remark, I didn't use width or height on the container because I didn't need it with my project, but maybe you will have to set one of both for your project.

As I'm still beginning with React & NextJS & typescript, maybe the solution can be prettier, if someone have an idea to improve it I'll be pleased to read it!