问:
How to add audio to video with node fluent-ffmpeg?
I am trying to add audio on top video using ffmpeg

```<code>ffmpeg -i sample.mp4 -i sample.mp3 -c copy -map 0:v:0 -map 1:a:0 output.mp4
</code>```

Above code works in my terminal but I would like to use this via ffmpeg node wrapper fluent-ffmpeg

This code is not working and it also not throwing any error

```<code>ffmpeg()
    .input('DATA/sample.mp4')
    .input('DATA/sample.mp3')
    .videoCodec('copy')
    .outputOptions(['-map 0:v:0', '-map 1:a:0'])
    // .outputOption('-map 1:a:0')
    .save('node.mp4')
</code>```

答:
MergeAdd and mergeToFile actually have misleading names. They are used to concatenate videos into a single video file.

You should use addInput to add your audio file and then use saveToFile to save the merged output.

Body:
I am trying to add audio on top video using ffmpeg

ffmpeg -i sample.mp4 -i sample.mp3 -c copy -map 0:v:0 -map 1:a:0 output.mp4

Above code works in my terminal but I would like to use this via ffmpeg node wrapper fluent-ffmpeg

This code is not working and it also not throwing any error

ffmpeg()
    .input('DATA/sample.mp4')
    .input('DATA/sample.mp3')
    .videoCodec('copy')
    .outputOptions(['-map 0:v:0', '-map 1:a:0'])
    // .outputOption('-map 1:a:0')
    .save('node.mp4')


来源: stackexchange

问:
SQL Query Help for Chart Data
Here is our goal:

We want to select each month and the revenue gained for each month, for a particular user.

Here are our tables

product, purchase, user, months

user 1..* product, product 1..* purchase

product has a column 'user_id' and purchase has a column 'product_id'

months is just a table that contains each month as a string.  Currently, we are using this to do some left joins as you can see below.

```<code>SELECT months.name, IFNULL(sum(purchase.price), 0) as revenue
FROM months
    LEFT JOIN purchase
        ON DATE_FORMAT(purchase.purchase_date, '%M') = months.name
        AND DATE_FORMAT(purchase_date, '%Y') = DATE_FORMAT(CURRENT_DATE, '%Y')
        AND purchase.status = 2
GROUP BY name
ORDER BY months.id ASC;
</code>```

Which works great for ALL of the users and ALL of the purchases made this month (Purchase status of 2 means complete).  The next part is how do I filter this based on a user id? The table 'product' has the user_id we're looking for, and table purchase has a 'product_id'.  Everytime I try something it either does nothing or it removes all of the null values, which we don't want.

ATTEMPT @NathanialWools
The where clause will remove all of the rows with a revenue of zero (or null, because of the IFNULL statement).  This is what I tried:

```<code>SELECT name, IFNULL(sum(purchase.price), 0) as revenue
FROM months
    LEFT JOIN purchase
        ON DATE_FORMAT(purchase.purchase_date, '%M') = months.name
        AND DATE_FORMAT(purchase_date, '%Y') = DATE_FORMAT(CURRENT_DATE, '%Y')
        AND purchase.status = 2
    LEFT JOIN product
        ON product.id = purchase.product_id
WHERE product.user_id = 1
GROUP BY name
ORDER BY months.id ASC;
</code>```

答:
An inner-select Seems to do the trick.  The left join on the purchase alone was not enough filtering.  Here is the solution that works for me:

```<code>SELECT m.name, IFNULL(sum(b.price), 0) as revenue
FROM months as m
LEFT JOIN (SELECT price, purchase_date, status
           FROM purchase, product
           WHERE purchase.product_id = product.id
           AND product.user_id = 1) as b
    ON DATE_FORMAT(b.purchase_date, '%M') = m.name
    AND DATE_FORMAT(b.purchase_date, '%Y') = DATE_FORMAT(CURRENT_TIMESTAMP, '%Y')
    AND status = 2
GROUP BY m.name
ORDER BY m.id ASC
</code>```

Where you would obviously change the '1' based on which user you wanted to look at.

Thanks everyone for pitching in, made debugging faster.

ContentLicense: CC BY-SA 3.0

Title: SQL Query Help for Chart Data

Body:
Here is our goal:

We want to select each month and the revenue gained for each month, for a particular user.

Here are our tables

product, purchase, user, months

user 1..* product, product 1..* purchase

product has a column 'user_id' and purchase has a column 'product_id'

months is just a table that contains each month as a string.  Currently, we are using this to do some left joins as you can see below.

SELECT months.name, IFNULL(sum(purchase.price), 0) as revenue
FROM months
    LEFT JOIN purchase
        ON DATE_FORMAT(purchase.purchase_date, '%M') = months.name
        AND DATE_FORMAT(purchase_date, '%Y') = DATE_FORMAT(CURRENT_DATE, '%Y')
        AND purchase.status = 2
GROUP BY name
ORDER BY months.id ASC;

Which works great for ALL of the users and ALL of the purchases made this month (Purchase status of 2 means complete).  The next part is how do I filter this based on a user id? The table 'product' has the user_id we're looking for, and table purchase has a 'product_id'.  Everytime I try something it either does nothing or it removes all of the null values, which we don't want.

ATTEMPT @NathanialWools
The where clause will remove all of the rows with a revenue of zero (or null, because of the IFNULL statement).  This is what I tried:

SELECT name, IFNULL(sum(purchase.price), 0) as revenue
FROM months
    LEFT JOIN purchase
        ON DATE_FORMAT(purchase.purchase_date, '%M') = months.name
        AND DATE_FORMAT(purchase_date, '%Y') = DATE_FORMAT(CURRENT_DATE, '%Y')
        AND purchase.status = 2
    LEFT JOIN product
        ON product.id = purchase.product_id
WHERE product.user_id = 1
GROUP BY name
ORDER BY months.id ASC;

Body:
An inner-select Seems to do the trick.  The left join on the purchase alone was not enough filtering.  Here is the solution that works for me:

SELECT m.name, IFNULL(sum(b.price), 0) as revenue
FROM months as m
LEFT JOIN (SELECT price, purchase_date, status
           FROM purchase, product
           WHERE purchase.product_id = product.id
           AND product.user_id = 1) as b
    ON DATE_FORMAT(b.purchase_date, '%M') = m.name
    AND DATE_FORMAT(b.purchase_date, '%Y') = DATE_FORMAT(CURRENT_TIMESTAMP, '%Y')
    AND status = 2
GROUP BY m.name
ORDER BY m.id ASC

Where you would obviously change the '1' based on which user you wanted to look at.

Thanks everyone for pitching in, made debugging faster.


问:
How can I add custom metadata at runtime to document(s) created from a template
I am using composite templates to create an envelope. Every template will only have one document. I need to individually process every document that comes back signed. In order to do that I need to add metadata to each document that is being sent. How can I do that while also using composite templates before I send the envelope?
I know all about envelope metadata and custom fields but what if I need it more specific to documents and I'm not working with the actual documents? Here I'm working with the templates that contain the documents.
Here is the c# code I have so far</p>
```<code>List<CompositeTemplate> compositeTemplates = new List<CompositeTemplate>();
Recipients recipientsServerTemplate = new Recipients();
List<Signer> signers = new List<Signer>();
        List<CarbonCopy> carbonCopies = new List<CarbonCopy>();

        Signer signer1 = new Signer();
        signer1.Email = signerEmail;
        signer1.Name = signerName;
        signer1.RoleName = "signer";
        signer1.RecipientId = "1";
        signer1.Tabs = tabs;
        signers.Add(signer1);

        CarbonCopy cc1 = new CarbonCopy();
        cc1.Email = ccEmail;
        cc1.Name = ccName;
        cc1.RoleName = "cc";
        cc1.RecipientId = "2";
        carbonCopies.Add(cc1);

        recipientsServerTemplate.Signers = signers;
        recipientsServerTemplate.CarbonCopies = carbonCopies;

        int i = 1;
        foreach (string templateId in templateIds)
        {
            //add custom fields //this is per envelope. I need it more specific in my case
            TextCustomField textcustomField = new TextCustomField
            {
                Name = "MyCustomField" + i.ToString(),
                Required = "false",
                Show = "false",
                Value =  "653022"
            };
            CustomFields cf = new CustomFields
            {
                TextCustomFields = new List<TextCustomField> { textcustomField }
            };

            List<ServerTemplate> ServerTemplates = new List<ServerTemplate>();
            List<InlineTemplate> InlineTemplates = new List<InlineTemplate>();
            CompositeTemplate CT = new CompositeTemplate
            {
                CompositeTemplateId = i.ToString()
            };

            ServerTemplate ST = new ServerTemplate
            {
                Sequence = i.ToString(),
                TemplateId = templateId
            };

            InlineTemplate IT = new InlineTemplate
            {
                Recipients = recipientsServerTemplate,
                Sequence = (i+1).ToString(),
                CustomFields = cf//this is for the whole envelope
            };

            InlineTemplates.Add(IT);
            ServerTemplates.Add(ST);
            CT.ServerTemplates = ServerTemplates;
            CT.InlineTemplates = InlineTemplates;
            compositeTemplates.Add(CT);

            i++;
        }

        EnvelopeDefinition env = new EnvelopeDefinition
        {
            Status = "sent",
            CompositeTemplates = compositeTemplates
        };
</code>```

答:
I suggest adding metadata at the envelope level about the envelope's documents. I don't think you can add document-level metadata when using composite templates.

Added

Templates are usually created via the DocuSign web app. But the web app does not support document-level metadata.

When you use the API to create the envelope, you could create add additional fields to the documents. You could make those additional fields only readable to your company recipients and store document metadata there.

But I would not recommend that. Better to store the document-specific data at the envelope level. For example, include the document-specific metadata as a JSON string in the envelope-level metadata:

```<code>...
"compositeTemplates": [
  {
    "serverTemplates": [
      {
        "sequence": "1",
        "templateId": "61973947-7d29-490d-9bac-5432154321"
      }
    ],
    "inlineTemplates": [
      {
        "sequence": "2",
        "customFields": {
          "textCustomFields": [
            {
              "name": "Main document",
              "show": "false",
              "value": "{\"value\":34500, \"office\": \"SF\"}"
            }
          ]
        },
</code>```

ContentLicense: CC BY-SA 4.0

Body:
I am using composite templates to create an envelope. Every template will only have one document. I need to individually process every document that comes back signed. In order to do that I need to add metadata to each document that is being sent. How can I do that while also using composite templates before I send the envelope?
I know all about envelope metadata and custom fields but what if I need it more specific to documents and I'm not working with the actual documents? Here I'm working with the templates that contain the documents.
Here is the c# code I have so far

List<CompositeTemplate> compositeTemplates = new List<CompositeTemplate>();
Recipients recipientsServerTemplate = new Recipients();
List<Signer> signers = new List<Signer>();
        List<CarbonCopy> carbonCopies = new List<CarbonCopy>();

        Signer signer1 = new Signer();
        signer1.Email = signerEmail;
        signer1.Name = signerName;
        signer1.RoleName = "signer";
        signer1.RecipientId = "1";
        signer1.Tabs = tabs;
        signers.Add(signer1);

        CarbonCopy cc1 = new CarbonCopy();
        cc1.Email = ccEmail;
        cc1.Name = ccName;
        cc1.RoleName = "cc";
        cc1.RecipientId = "2";
        carbonCopies.Add(cc1);

        recipientsServerTemplate.Signers = signers;
        recipientsServerTemplate.CarbonCopies = carbonCopies;

        int i = 1;
        foreach (string templateId in templateIds)
        {
            //add custom fields //this is per envelope. I need it more specific in my case
            TextCustomField textcustomField = new TextCustomField
            {
                Name = "MyCustomField" + i.ToString(),
                Required = "false",
                Show = "false",
                Value =  "653022"
            };
            CustomFields cf = new CustomFields
            {
                TextCustomFields = new List<TextCustomField> { textcustomField }
            };

            List<ServerTemplate> ServerTemplates = new List<ServerTemplate>();
            List<InlineTemplate> InlineTemplates = new List<InlineTemplate>();
            CompositeTemplate CT = new CompositeTemplate
            {
                CompositeTemplateId = i.ToString()
            };

            ServerTemplate ST = new ServerTemplate
            {
                Sequence = i.ToString(),
                TemplateId = templateId
            };

            InlineTemplate IT = new InlineTemplate
            {
                Recipients = recipientsServerTemplate,
                Sequence = (i+1).ToString(),
                CustomFields = cf//this is for the whole envelope
            };

            InlineTemplates.Add(IT);
            ServerTemplates.Add(ST);
            CT.ServerTemplates = ServerTemplates;
            CT.InlineTemplates = InlineTemplates;
            compositeTemplates.Add(CT);

            i++;
        }

        EnvelopeDefinition env = new EnvelopeDefinition
        {
            Status = "sent",
            CompositeTemplates = compositeTemplates
        };

Body:
I suggest adding metadata at the envelope level about the envelope's documents. I don't think you can add document-level metadata when using composite templates.

Added

Templates are usually created via the DocuSign web app. But the web app does not support document-level metadata.

When you use the API to create the envelope, you could create add additional fields to the documents. You could make those additional fields only readable to your company recipients and store document metadata there.

But I would not recommend that. Better to store the document-specific data at the envelope level. For example, include the document-specific metadata as a JSON string in the envelope-level metadata:

...
"compositeTemplates": [
  {
    "serverTemplates": [
      {
        "sequence": "1",
        "templateId": "61973947-7d29-490d-9bac-5432154321"
      }
    ],
    "inlineTemplates": [
      {
        "sequence": "2",
        "customFields": {
          "textCustomFields": [
            {
              "name": "Main document",
              "show": "false",
              "value": "{\"value\":34500, \"office\": \"SF\"}"
            }
          ]
        },


问:
Hot Chocolate + Mongo: No additional call when I ask only for referene id
Let's assume that I have the following documents in mongo:

Product:

...

productTemplateId

...

Product Template:

id

...

I mapped product to the following schema:

```<code>product {
 ...
 productTemplate {
  id,
  ...
 }
 ...
}
</code>```

I would like to avoid to make additional call when I ask only for product { productTemplate {id}} because this information is in the product entity. Is it possible?

I would like to get from mongo only product with productTemplateId instead of whole document. It is significant because I have about 100 req/s. Is it possible to ask mongo only for expected fields from document?

This is part of my code:

```<code>public class ProductType : ObjectType<ProductDocument>
{
    protected override void Configure(IObjectTypeDescriptor<ProductDocument> descriptor)
    {
        descriptor.BindFieldsExplicitly();
        descriptor.Name("Product");
        descriptor.Field("productTemplate")
            .Type<ProductTemplateType>()
            .ResolveWith<ProductTemplatesResolver>(r => r.GetProductTemplate(default!, default!, default));
    }
}

public class ProductTemplatesResolver
{
    public Task<ProductTemplateDocument> GetProductTemplate([Parent] ProductDocument product,
        ProductTemplatesBatchDataLoader loader,
        CancellationToken cancellationToken)
    {
        return loader.LoadAsync(product.ProductTemplateId, cancellationToken);
    }
}

public class ProductTemplatesBatchDataLoader : BatchDataLoader<int, ProductTemplateDocument>
{
    private readonly ProductsDbContext _dbContext;

    public ProductTemplatesBatchDataLoader(IBatchScheduler batchScheduler, ProductsDbContext dbContext, DataLoaderOptions? options = null) : base(batchScheduler, options)
    {
        _dbContext = dbContext;
    }

    protected override async Task<IReadOnlyDictionary<int, ProductTemplateDocument>> LoadBatchAsync(IReadOnlyList<int> keys, CancellationToken cancellationToken)
    {
        var pts = await _dbContext.ProductTemplates
            .AsQueryable()
            .Where(x => keys.Contains(x.Id))
            .ToListAsync(cancellationToken);

        return pts.ToDictionary(x => x.Id, x => x);
    }
}
</code>```

Thanks

答:
You can get the selected field over IResolverContext GetSelections

public class ProductTemplatesResolver
{
    public Task<ProductTemplateDocument> GetProductTemplate(
        [Parent] ProductDocument product,
        IResolverContext context,
        ProductTemplatesBatchDataLoader loader,
        CancellationToken cancellationToken)
    {
        var selectedFields = context.GetSelections((ObjectType)context.Selection.Field.Type);
        if (selectedFields.Count == 1 && selectedFields[0].Field.Name == "id")
        {
            return new ProductDocument(product.TemplateId);
        }
        return loader.LoadAsync(product.ProductTemplateId, cancellationToken);
    }
}
```

Body:
Let's assume that I have the following documents in mongo:

Product:

...

productTemplateId

...

Product Template:

id

...

I mapped product to the following schema:

product {
 ...
 productTemplate {
  id,
  ...
 }
 ...
}

I would like to avoid to make additional call when I ask only for product { productTemplate {id}} because this information is in the product entity. Is it possible?

I would like to get from mongo only product with productTemplateId instead of whole document. It is significant because I have about 100 req/s. Is it possible to ask mongo only for expected fields from document?

This is part of my code:

public class ProductType : ObjectType<ProductDocument>
{
    protected override void Configure(IObjectTypeDescriptor<ProductDocument> descriptor)
    {
        descriptor.BindFieldsExplicitly();
        descriptor.Name("Product");
        descriptor.Field("productTemplate")
            .Type<ProductTemplateType>()
            .ResolveWith<ProductTemplatesResolver>(r => r.GetProductTemplate(default!, default!, default));
    }
}

public class ProductTemplatesResolver
{
    public Task<ProductTemplateDocument> GetProductTemplate([Parent] ProductDocument product,
        ProductTemplatesBatchDataLoader loader,
        CancellationToken cancellationToken)
    {
        return loader.LoadAsync(product.ProductTemplateId, cancellationToken);
    }
}

public class ProductTemplatesBatchDataLoader : BatchDataLoader<int, ProductTemplateDocument>
{
    private readonly ProductsDbContext _dbContext;

    public ProductTemplatesBatchDataLoader(IBatchScheduler batchScheduler, ProductsDbContext dbContext, DataLoaderOptions? options = null) : base(batchScheduler, options)
    {
        _dbContext = dbContext;
    }

    protected override async Task<IReadOnlyDictionary<int, ProductTemplateDocument>> LoadBatchAsync(IReadOnlyList<int> keys, CancellationToken cancellationToken)
    {
        var pts = await _dbContext.ProductTemplates
            .AsQueryable()
            .Where(x => keys.Contains(x.Id))
            .ToListAsync(cancellationToken);

        return pts.ToDictionary(x => x.Id, x => x);
    }
}

Thanks


问:
Turn one dataframe into several dfs and add them as CSVs to zip archive (without saving files locally)
I have a data frame which I read in from a locally saved CSV file.
I then want to loop over said file and create several CSV files based on a string in one column.

Lastly, I want to add all those files to a zip file, but without saving them locally. I just want one zip archive including all the different CSV files.

All my attempts using the io or zipfile modules only resulted in one zip file with one CSV file in it (pretty much with what I started with)

Any help would be much appreciated!
Here is my code so far, which works but saves all CSV files just to my hard drive.

```<code>import pandas as pd
from zipfile import ZipFile

df = pd.read_csv("myCSV.csv")
channelsList = df["Turn one column to list"].values.tolist()
channelsList = list(set(channelsList)) #delete duplicates from list

for channel in channelsList:
    newDf = df.loc[df['Something to match'] == channel]

    newDf.to_csv(f"{channel}.csv") # saves csv files to disk
</code>```

答:
DataFrame.to_csv() can write to any file-like object, and ZipFile.writestr() can accept a string (or bytes), so it is possible to avoid writing the CSV files to disk using io.StringIO.  See the example code below.

Note: If the channel is simply stored in a single column of your input data, then the more idiomatic (and more efficient) way to iterate over the partitions of your data is to use groupby().

```<code>from io import StringIO
from zipfile import ZipFile

import numpy as np
import pandas as pd

# Example data
df = pd.DataFrame(np.random.random((100,3)), columns=[*'xyz'])
df['channel'] = np.random.randint(5, size=len(df))

with ZipFile('/tmp/output.zip', 'w') as zf:
    for channel, channel_df in df.groupby('channel'):
        s = StringIO()
        channel_df.to_csv(s, index=False, header=True)
        zf.writestr(f"{channel}.csv", s.getvalue())
</code>```

Body:
I have a data frame which I read in from a locally saved CSV file.
I then want to loop over said file and create several CSV files based on a string in one column.

Lastly, I want to add all those files to a zip file, but without saving them locally. I just want one zip archive including all the different CSV files.

All my attempts using the io or zipfile modules only resulted in one zip file with one CSV file in it (pretty much with what I started with)

Any help would be much appreciated!
Here is my code so far, which works but saves all CSV files just to my hard drive.

import pandas as pd
from zipfile import ZipFile

df = pd.read_csv("myCSV.csv")
channelsList = df["Turn one column to list"].values.tolist()
channelsList = list(set(channelsList)) #delete duplicates from list

for channel in channelsList:
    newDf = df.loc[df['Something to match'] == channel]

    newDf.to_csv(f"{channel}.csv") # saves csv files to disk

Body:
DataFrame.to_csv() can write to any file-like object, and ZipFile.writestr() can accept a string (or bytes), so it is possible to avoid writing the CSV files to disk using io.StringIO.  See the example code below.

Note: If the channel is simply stored in a single column of your input data, then the more idiomatic (and more efficient) way to iterate over the partitions of your data is to use groupby().

from io import StringIO
from zipfile import ZipFile

import numpy as np
import pandas as pd

# Example data
df = pd.DataFrame(np.random.random((100,3)), columns=[*'xyz'])
df['channel'] = np.random.randint(5, size=len(df))

with ZipFile('/tmp/output.zip', 'w') as zf:
    for channel, channel_df in df.groupby('channel'):
        s = StringIO()
        channel_df.to_csv(s, index=False, header=True)
        zf.writestr(f"{channel}.csv", s.getvalue())


问:
How do I account for daylight savings time when retrieving current time in PHP?
When getting the time via PHP script (time()), during Daylight Savings Time it seems to return a time value that is one hour off. Is there any way to get the correct time, accounting for Daylight Savings Time?

答: time() gets the OS time in Unix Timestamp format. You could correct this with PHP but imho this should be something to consider fixing on OS level.</p>


问:
Access Intune configuration settings in a React Native application?
My company is doing Mobile Device Management with Microsoft Intune. We've successfully deployed an internal iOS app (using the Apple Developer Enterprise Program).

With Intunes' configuration settings we're trying to make each user's individual email available to the mobile app.
https://learn.microsoft.com/en-us/mem/intune/apps/app-configuration-policies-use-ios

How do you normally access these types of settings in an app? I found this library but I'd need to eject from Expo which is not ideal for me:
https://github.com/robinpowered/react-native-mdm

答:
As the MDM has native dependencies, You'll not be able to make it with Expo. Expo projects are written only in JavaScript and don't support packages that contain Objective-C or Java (Native code/dependencies).

Expo provides an advanced SDK called ExpoKit for when you absolutely need to use custom native code. However, there are some cases where developers need native capabilities outside of what Expo offers directly. The most common situation is when a project requires a specific Native Module (like MDM) that is not supported by React Native Core or the Expo SDK. You'll have to detach the Expo project to create Xcode and Android Studio projects that contain ExpoKit. This step will generate android and ios project directories. Then you would add custom Objective-C or Java the same way as with any other Xcode or Android Studio project.

The Expo docs warn about some of the downsides of writing custom native code and discourage most of our developers from taking this route, as Expo's motive, almost everything you need to do is better accomplished in a cross-platform way with JS. Writing in JS enables you to best take advantage of code aster deployment and benefit from ongoing updates and support from Expo. You should only do this if you have a particular demand from native code which Expo won’t do a good job supporting, such as (for example) specialized CPU-intensive video processing that must happen locally on the device, Custom native libraries.

Here are only two options to support the MDM, either eject the project or create react-native-cli project and migrate your project into newly created one.


问:
Login failed for user SA, when connecting to SQL Server Docker container, deployed in Kubernetes
I'm following this tutorial on Microservices
https://www.youtube.com/watch?v=DgVjEo3OGBI

At some point, I deploy a SQL Server image in Kubernetes, using this Yaml file:

```<code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: mssql-depl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mssql
  template:
    metadata:
      labels:
        app: mssql
    spec:
      containers:
        - name: mssql
          image: mcr.microsoft.com/mssql/server:2019-latest
          ports:
            - containerPort: 1433
          env:
          - name: MSSQL_PID
            value: "Express"
          - name: ACCEPT_EULA
            value: "Y"
          - name: SA_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mssql5
                key: SA_PASSWORD
          volumeMounts:
          - mountPath: /var/opt/mssql/data
            name: mssqldb
      volumes:
      - name: mssqldb
        persistentVolumeClaim:
          claimName: mssql-claim
---
apiVersion: v1
kind: Service
metadata:
  name: mssql-clusterip-srv
spec:
  type: ClusterIP
  selector:
    app: mssql
  ports:
  - name: mssql
    protocol: TCP
    port: 1433
    targetPort: 1433
---
apiVersion: v1
kind: Service
metadata:
  name: mssql-loadbalancer
spec:
  type: LoadBalancer
  selector:
    app: mssql
  ports:
  - protocol: TCP
    port: 1433
    targetPort: 1433
</code>```

And the PVC.yaml:

```<code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mssql-claim
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 200Mi
</code>```

I then create a secret in Kubernets, holding the password for the SQL Server sa account, and this secret is then used in the Yaml file.

```<code>kubectl create secret generic mssql5 --from-literal=SA_PASSWORD="MyC0m9l&xPassw0rd"
</code>```

It should then be possible to connect directly to the SQL Server container on localhost port 1433, using the sa account and the password.

However, I get the "Login failed for user SA" error when trying to connect. I've tried everything, including changing SA_PASSWORD to MSSQL_SA_PASSWORD, changing the complexity of the password, enabling login for the sa user in  SQL Server, which was disabled before and googled like I've never googled before. TCP/IP setting is enabled in SQL Server configuration manager. I cannot get access. Can anyone shed some light on this issue?

答:
TL;DR The solution to this particular problem was that https://stackoverflow.com/users/52045/soeren had an instance of SQL Server running on their computer that intercepted port 1433 requests before hitting K8s. Disabling the SQL Server instance on computer solved the problem and port 1433 traffic routed to K8s load balancer.

Original solution follows...

I have copy/pasted your YAML verbatim above into files deployment.yaml and pvc.yaml respectively and then run these commands.

```<code>kubectl create -f pvc.yaml
kubectl create secret generic mssql5 --from-literal=SA_PASSWORD="MyC0m9l&xPassw0rd"
kubectl create -f deployment.yaml
</code>```

I can connect via localhost and create database.

Conclusion is that your YAML looks fine.

If I then delete/create the deployment again...

```<code>kubectl delete -f deployment.yaml
kubectl create-f deployment.yaml
</code>```

..the SSMS connection still works and the Test database persists.

The only change I would personally make is from - mountPath: /var/opt/mssql/data to - mountPath: /var/opt/mssql/ in deployment.yaml so that your entire mssql path is persisted.

As you are including the /data folder this means that the logs and secrets folders are still within the POD and not the persisted volume. This might explain the ldf log file permissions issue you mentioned above?

So, next I investigated the persisted volume kubectl describe pv and on my system it is showing the following:

```<code>Type:          HostPath (bare host directory volume)
    Path:          /var/lib/k8s-pvs/mssql-claim/pvc-42f91115-efdb-4cea-90d8-204fd592f6b4
</code>```

I'm guessing you are using hostPath, which will store the files on the containing node/host. I'm using Docker Kubernetes as opposed to minikube. A single node cluster.

If you shell into the node (as opposed to the POD) as root thus:

```<code>docker run -it --rm --privileged --pid=host justincormack/nsenter1
</code>```

You can cd /var/lib/k8s-pvs/mssql-claim/pvc-42f91115-efdb-4cea-90d8-204fd592f6b4 to see the mounted volume for your persisted volume claim.

You should see something like:

```<code>/ # cd /var/lib/k8s-pvs/mssql-claim/pvc-42f91115-efdb-4cea-90d8-204fd592f6b4
/var/lib/k8s-pvs/mssql-claim/pvc-42f91115-efdb-4cea-90d8-204fd592f6b4 # ls -l
total 89348
-rw-r-----    1 10001    root           256 Feb 14 17:33 Entropy.bin
-rw-r-----    1 10001    root       8388608 Feb 14 17:36 Test.mdf
-rw-r-----    1 10001    root       8388608 Feb 14 17:55 Test_log.ldf
-rw-r-----    1 10001    root       4653056 Feb 14 17:54 master.mdf
-rw-r-----    1 10001    root       2097152 Feb 14 18:00 mastlog.ldf
-rw-r-----    1 10001    root       8388608 Feb 14 17:49 model.mdf
-rw-r-----    1 10001    root      14090240 Feb 14 17:49 model_msdbdata.mdf
-rw-r-----    1 10001    root        524288 Feb 14 17:49 model_msdblog.ldf
-rw-r-----    1 10001    root        524288 Feb 14 17:49 model_replicatedmaster.ldf
-rw-r-----    1 10001    root       4653056 Feb 14 17:49 model_replicatedmaster.mdf
-rw-r-----    1 10001    root       8388608 Feb 14 17:49 modellog.ldf
-rw-r-----    1 10001    root      14090240 Feb 14 17:36 msdbdata.mdf
-rw-r-----    1 10001    root        524288 Feb 14 17:49 msdblog.ldf
-rw-r-----    1 10001    root       8388608 Feb 14 17:49 tempdb.mdf
-rw-r-----    1 10001    root       8388608 Feb 14 17:55 templog.ldf
</code>```

If you had mapped mssql only above you will see data, logs and secrets subdirectories.

Obviously, replace your PVC with your name.

So again, in conclusion your YAML looks fine. I therefore suspect that you have got something lingering around from previous attempts.

The only other issue that has cropped up with me is that sometimes it's better to use single quotes with PowerShell and double quotes with cmd prompt.

Have you checked your pods are running OK and that the config is being read OK? (replace pod id with yours)

```<code>kubectl get pods
kubectl describe pod mssql-depl-7777487cf7-w9gj5
</code>```

Beyond that...I'm sure it's something to do with what's been left behind from previous testing that needs to be cleaned up.

Original answer below for completeness...

Have you tried clearing everything down (using a namespace makes this easier).

I have noticed that if you're using a persisted volume claim the master.mdf may hold an old password (SELECT * FROM master.dbo.syslogins) as the master.mdf isn't recreated fresh as it would be with a non persisted volume.

As an experiment, when I had everything working, I deleted the deployment but left the persisted volume claim.

I deleted the SA_PASSWORD secret and recreated it with a different password and then applied the deployment again.

I was able to connect to SQL from SSMS on 1433 using the old password even though the new POD contained the new password in the environment variable SA_PASSWORD.

In conclusion, I would try to clear your persisted volume claim and start again so that the master/msdb databases are recreated.

It's worth a try?

Incidentally, NodePort is all you need and not ClusterIP and LoadBalancer for your service (this obviously depends on your cluster environment. I presumed you're using minikube or local Docker?)

Body:
I'm following this tutorial on Microservices
https://www.youtube.com/watch?v=DgVjEo3OGBI

At some point, I deploy a SQL Server image in Kubernetes, using this Yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mssql-depl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mssql
  template:
    metadata:
      labels:
        app: mssql
    spec:
      containers:
        - name: mssql
          image: mcr.microsoft.com/mssql/server:2019-latest
          ports:
            - containerPort: 1433
          env:
          - name: MSSQL_PID
            value: "Express"
          - name: ACCEPT_EULA
            value: "Y"
          - name: SA_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mssql5
                key: SA_PASSWORD
          volumeMounts:
          - mountPath: /var/opt/mssql/data
            name: mssqldb
      volumes:
      - name: mssqldb
        persistentVolumeClaim:
          claimName: mssql-claim
---
apiVersion: v1
kind: Service
metadata:
  name: mssql-clusterip-srv
spec:
  type: ClusterIP
  selector:
    app: mssql
  ports:
  - name: mssql
    protocol: TCP
    port: 1433
    targetPort: 1433
---
apiVersion: v1
kind: Service
metadata:
  name: mssql-loadbalancer
spec:
  type: LoadBalancer
  selector:
    app: mssql
  ports:
  - protocol: TCP
    port: 1433
    targetPort: 1433

And the PVC.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mssql-claim
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 200Mi

I then create a secret in Kubernets, holding the password for the SQL Server sa account, and this secret is then used in the Yaml file.

kubectl create secret generic mssql5 --from-literal=SA_PASSWORD="MyC0m9l&xPassw0rd"

It should then be possible to connect directly to the SQL Server container on localhost port 1433, using the sa account and the password.

However, I get the "Login failed for user SA" error when trying to connect. I've tried everything, including changing SA_PASSWORD to MSSQL_SA_PASSWORD, changing the complexity of the password, enabling login for the sa user in  SQL Server, which was disabled before and googled like I've never googled before. TCP/IP setting is enabled in SQL Server configuration manager. I cannot get access. Can anyone shed some light on this issue?

Body:
TL;DR The solution to this particular problem was that https://stackoverflow.com/users/52045/soeren had an instance of SQL Server running on their computer that intercepted port 1433 requests before hitting K8s. Disabling the SQL Server instance on computer solved the problem and port 1433 traffic routed to K8s load balancer.

Original solution follows...

I have copy/pasted your YAML verbatim above into files deployment.yaml and pvc.yaml respectively and then run these commands.

kubectl create -f pvc.yaml
kubectl create secret generic mssql5 --from-literal=SA_PASSWORD="MyC0m9l&xPassw0rd"
kubectl create -f deployment.yaml

I can connect via localhost and create database.

Conclusion is that your YAML looks fine.

If I then delete/create the deployment again...

kubectl delete -f deployment.yaml
kubectl create-f deployment.yaml

..the SSMS connection still works and the Test database persists.

The only change I would personally make is from - mountPath: /var/opt/mssql/data to - mountPath: /var/opt/mssql/ in deployment.yaml so that your entire mssql path is persisted.

As you are including the /data folder this means that the logs and secrets folders are still within the POD and not the persisted volume. This might explain the ldf log file permissions issue you mentioned above?

So, next I investigated the persisted volume kubectl describe pv and on my system it is showing the following:

Type:          HostPath (bare host directory volume)
    Path:          /var/lib/k8s-pvs/mssql-claim/pvc-42f91115-efdb-4cea-90d8-204fd592f6b4

I'm guessing you are using hostPath, which will store the files on the containing node/host. I'm using Docker Kubernetes as opposed to minikube. A single node cluster.

If you shell into the node (as opposed to the POD) as root thus:

docker run -it --rm --privileged --pid=host justincormack/nsenter1

You can cd /var/lib/k8s-pvs/mssql-claim/pvc-42f91115-efdb-4cea-90d8-204fd592f6b4 to see the mounted volume for your persisted volume claim.

You should see something like:

/ # cd /var/lib/k8s-pvs/mssql-claim/pvc-42f91115-efdb-4cea-90d8-204fd592f6b4
/var/lib/k8s-pvs/mssql-claim/pvc-42f91115-efdb-4cea-90d8-204fd592f6b4 # ls -l
total 89348
-rw-r-----    1 10001    root           256 Feb 14 17:33 Entropy.bin
-rw-r-----    1 10001    root       8388608 Feb 14 17:36 Test.mdf
-rw-r-----    1 10001    root       8388608 Feb 14 17:55 Test_log.ldf
-rw-r-----    1 10001    root       4653056 Feb 14 17:54 master.mdf
-rw-r-----    1 10001    root       2097152 Feb 14 18:00 mastlog.ldf
-rw-r-----    1 10001    root       8388608 Feb 14 17:49 model.mdf
-rw-r-----    1 10001    root      14090240 Feb 14 17:49 model_msdbdata.mdf
-rw-r-----    1 10001    root        524288 Feb 14 17:49 model_msdblog.ldf
-rw-r-----    1 10001    root        524288 Feb 14 17:49 model_replicatedmaster.ldf
-rw-r-----    1 10001    root       4653056 Feb 14 17:49 model_replicatedmaster.mdf
-rw-r-----    1 10001    root       8388608 Feb 14 17:49 modellog.ldf
-rw-r-----    1 10001    root      14090240 Feb 14 17:36 msdbdata.mdf
-rw-r-----    1 10001    root        524288 Feb 14 17:49 msdblog.ldf
-rw-r-----    1 10001    root       8388608 Feb 14 17:49 tempdb.mdf
-rw-r-----    1 10001    root       8388608 Feb 14 17:55 templog.ldf

If you had mapped mssql only above you will see data, logs and secrets subdirectories.

Obviously, replace your PVC with your name.

So again, in conclusion your YAML looks fine. I therefore suspect that you have got something lingering around from previous attempts.

The only other issue that has cropped up with me is that sometimes it's better to use single quotes with PowerShell and double quotes with cmd prompt.

Have you checked your pods are running OK and that the config is being read OK? (replace pod id with yours)

kubectl get pods
kubectl describe pod mssql-depl-7777487cf7-w9gj5

Beyond that...I'm sure it's something to do with what's been left behind from previous testing that needs to be cleaned up.

Original answer below for completeness...

Have you tried clearing everything down (using a namespace makes this easier).

I have noticed that if you're using a persisted volume claim the master.mdf may hold an old password (SELECT * FROM master.dbo.syslogins) as the master.mdf isn't recreated fresh as it would be with a non persisted volume.

As an experiment, when I had everything working, I deleted the deployment but left the persisted volume claim.

I deleted the SA_PASSWORD secret and recreated it with a different password and then applied the deployment again.

I was able to connect to SQL from SSMS on 1433 using the old password even though the new POD contained the new password in the environment variable SA_PASSWORD.

In conclusion, I would try to clear your persisted volume claim and start again so that the master/msdb databases are recreated.

It's worth a try?

Incidentally, NodePort is all you need and not ClusterIP and LoadBalancer for your service (this obviously depends on your cluster environment. I presumed you're using minikube or local Docker?)


问:
code to generate radio button in existing wp theme
Good evening friends and neighbors,

I am currently utilizing MAMP v6.6 to customize and build wordpress themes locally on my
2017 iMac (Retina ~ 27 inch) running macOS Monterey 12.2

Also using:

Apache Netbeans 12.6

MySQL Workbench 8.0

Xdebug (once I figure out how to install)

( can’t quite figure out how to install PECL - the line commands i found on the web are not doing it - in order to install Xdebug, but am working on that )

I’ve added a third ‘menu layout’ option to an existing premium wordpress theme (VEGA by themepunch)…. menu center,  with the menu centered underneath the logo by adding some PHP code to the ‘template-menu.php’, ‘functions.php’ and ‘menu.lib.php’ files.

However, this is not generating a radio button for that option in the ‘theme settings’ panel (located in the lefthand column Wordpress dashboard menu) under the ‘Header’ tab (see reference jpeg )

Please see reference images attached for more info.

Here is the html source code in the 'theme settings' accessed via the wordpress dashboard left side menu:

```<code><!---  html source code for ‘menu layouts’ in menu settings under header tab --->

<div class="rm_input rm_select">

    <label for="pp_menu_layout">Menu layouts</label>
        <br/>

    <div style="margin-top:5px;float:left;width:300px">

        <div style="float:left;margin:0 20px 20px 0">

        <input style="float:left;" id="pp_menu_layout" name="pp_menu_layout" type="radio"
        checked="checked"           value="1"/>

            <div style="float:left;width:90%;"><img src="http://localhost:8888/wordpress/wp-content/themes/vega/functions/images/menu1.png"/>
            </div>

        </div>

        <div style="float:left;margin:0 20px 20px 0">

        <input style="float:left;" id="pp_menu_layout" name="pp_menu_layout" type="radio"
                    value="2"/>

            <div style="float:left;width:90%;">

                <img src="http://localhost:8888/wordpress/wp-content/themes/vega/functions/images/menu2.png"/>

            </div>

        </div>

    <small></small>

    <div class="clearfix">

        </div>

    </div>

</div>
</code>```

It’s probably something fairly easy, however, if anyone can give some guidance or speculation on what code to add in what file in order to generate the radio button for my third ‘menu layout option’, that would be greatly appreciated.

答:
Hello friends and neighbors,

I was able to figure this answer out by meticulously going through each file within the VEGA premium wordpress theme. When I opened the admin.lib.php file, I realized that it correlated directly with the 'Theme Settings' menu panel.
So I scrolled down til I got to the MENU SETTINGS-->Menu Layouts section of PHP code. There I just copied the value '2' menu option in the array and pasted that underneath on the next line. Then added a number 3 in place of the number 2 in the pasted code.

```<code>    "desc" => "",
    "id" => SHORTNAME."_menu_layout",
    "type" => "radio",
    "options" => array(
        '1' => '<div style="float:left;width:90%;"><img src="'.get_template_directory_uri().'/functions/images/menu1.png"/></div>',
        '2' => '<div style="float:left;width:90%;"><img src="'.get_template_directory_uri().'/functions/images/menu2.png"/></div>',
        '3' => '<div style="float:left;width:90%;"><img src="'.get_template_directory_uri().'/functions/images/menu3.png"/></div>',
    ),
</code>```

I had already created a thumbnail jpeg for menu layout number three (menu3.png) and placed it into the /functions/images directory.

This generated a new radio button and thumbnail image for my custom menu layout in the 'Theme Settings'-->'Header Tab' menu.

Body:
Good evening friends and neighbors,

I am currently utilizing MAMP v6.6 to customize and build wordpress themes locally on my
2017 iMac (Retina ~ 27 inch) running macOS Monterey 12.2

Also using:

Apache Netbeans 12.6

MySQL Workbench 8.0

Xdebug (once I figure out how to install)

( can’t quite figure out how to install PECL - the line commands i found on the web are not doing it - in order to install Xdebug, but am working on that )

I’ve added a third ‘menu layout’ option to an existing premium wordpress theme (VEGA by themepunch)…. menu center,  with the menu centered underneath the logo by adding some PHP code to the ‘template-menu.php’, ‘functions.php’ and ‘menu.lib.php’ files.

However, this is not generating a radio button for that option in the ‘theme settings’ panel (located in the lefthand column Wordpress dashboard menu) under the ‘Header’ tab (see reference jpeg )

Please see reference images attached for more info.

Here is the html source code in the 'theme settings' accessed via the wordpress dashboard left side menu:

    Menu layouts

        <input style="float:left;" id="pp_menu_layout" name="pp_menu_layout" type="radio"
        checked="checked"           value="1"/>

        <input style="float:left;" id="pp_menu_layout" name="pp_menu_layout" type="radio"
                    value="2"/>

It’s probably something fairly easy, however, if anyone can give some guidance or speculation on what code to add in what file in order to generate the radio button for my third ‘menu layout option’, that would be greatly appreciated.

Body:
Hello friends and neighbors,

I was able to figure this answer out by meticulously going through each file within the VEGA premium wordpress theme. When I opened the admin.lib.php file, I realized that it correlated directly with the 'Theme Settings' menu panel.
So I scrolled down til I got to the MENU SETTINGS-->Menu Layouts section of PHP code. There I just copied the value '2' menu option in the array and pasted that underneath on the next line. Then added a number 3 in place of the number 2 in the pasted code.

    "desc" => "",
    "id" => SHORTNAME."_menu_layout",
    "type" => "radio",
    "options" => array(
        '1' => '<div style="float:left;width:90%;"><img src="'.get_template_directory_uri().'/functions/images/menu1.png"/></div>',
        '2' => '<div style="float:left;width:90%;"><img src="'.get_template_directory_uri().'/functions/images/menu2.png"/></div>',
        '3' => '<div style="float:left;width:90%;"><img src="'.get_template_directory_uri().'/functions/images/menu3.png"/></div>',
    ),

I had already created a thumbnail jpeg for menu layout number three (menu3.png) and placed it into the /functions/images directory.

This generated a new radio button and thumbnail image for my custom menu layout in the 'Theme Settings'-->'Header Tab' menu.


问:
Simple PHP Class Question regarding methods
I have a simple question regarding PHP Classes.

Multiple times I have seen other class-frameworks etc use method calls like.

```<code>$post->data->text();
</code>```

I like this functionality, rather than just doing something like this.

```<code>$post->dataReturnAsText();
</code>```

But i'm not quite sure how they created this functionality to have perhaps a 'sub-method'? Hope someone can point me in the right direction....

答:
The example you provide has nothing special:

```<code><?php

class Post{
    public $data;
}

class Data{
    public function text(){
    }
}

$post = new Post;
$post->data = new Data;
$post->data->text();
</code>```

However, you've probably found it in the context of method chaining (very popular in JavaScript libraries):

```<code><?php

class Foo{
    public function doThis(){
        return $this;
    }

    public function doThat(){
        return $this;
    }
}

$foo = new Foo;
$foo->doThis()->doThat()->doThis();
</code>```

Body:
I have a simple question regarding PHP Classes.

Multiple times I have seen other class-frameworks etc use method calls like.

$post->data->text();

I like this functionality, rather than just doing something like this.

$post->dataReturnAsText();

But i'm not quite sure how they created this functionality to have perhaps a 'sub-method'? Hope someone can point me in the right direction....

Body:
The example you provide has nothing special:

<?php

class Post{
    public $data;
}

class Data{
    public function text(){
    }
}

$post = new Post;
$post->data = new Data;
$post->data->text();

However, you've probably found it in the context of method chaining (very popular in JavaScript libraries):

<?php

class Foo{
    public function doThis(){
        return $this;
    }

    public function doThat(){
        return $this;
    }
}

$foo = new Foo;
$foo->doThis()->doThat()->doThis();


问:
Is there any disadvantage of marking a variable or constant as private in terms of performance or something?
Is there any disadvantage in terms if performance or something by marking a variable as a private on a class or struct on Swift or view on SwiftUI?

I know about the encapsulation advantage you get when the variable is private, preventing access from other code outside the scope. My concern is about performance.

答:
I can say (Flutter experience) it won't slow down your code or app. Its just usable from this one file... but you mentioned that earlier in your question.

But no this won't make your app slower.


问:
Is there a clang analogue of gcc's `__attribute__((hot))`
I wish to mark a function for more aggressive optimizations. The function is not called often, so PGO won't help, but I still want it as low latency as possible.

gcc has attribute hot for this. Is there some clang equivalent I'm missing?

BTW clang does accept __attribute__((hot)), but AFAICT it has zero impact (and is undocumented). Probably for source compatibility with gcc.

答:
(Writing an answer instead of deleting, in case it is of use to others)

So turns out that since December 2020 Clang does support __attribute__((hot)), they just didn't document it. I nudged them.  When tested in a newer clang version I see a difference in binaries.

BTW they did support __attribute__((cold)) long before that. And as I suspected, the clang front end did accept hot for sake of gcc compatibility.


问:
How can I load and merge several .txt files in a memory efficient way in python?
I am trying to read several (>1000) .txt files (on average approx. 700 MB, delimited, header-less CSV, without commas or other separator) and merge them into one pandas dataframe (to next run an analysis on the entire dataset).

I am running this through SSH on a HPC server, on which I requested 50GB RAM, 1 node, 1 task-per-node (that was all just a wild guess, as I have never done this before).

So far, my idea was this:

```<code>li = []

for filename in all_files:
    df = pd.read_csv(filename, sep=None, header=0, engine='python')
    li.append(df)

df = pd.concat(li, axis=0, ignore_index=True)
</code>```

but after a few hours and having loaded the approx. 360th file the process gets killed and I get the error message:

numpy.core._exceptions.MemoryError: Unable to allocate 1.11 GiB for an array with shape (10, 14921599) and data type float64

Do you have any idea how to load and merge the data more memory efficient? (I assume just requesting more RAM still does not get me through the entire set of .txt files!?)

Also, I would like to save the resulting dataframe in a 'memory-efficient' way afterwards, do you know the best way/format (csv?) to do that?

Any help would be much appreciated!

答:
Q : "How can I ... Any help would be much appreciated!"

A :

best follow the laws of the ECONOMY-of-COMPUTING :

Your briefly sketched problem has, out of question, immense "setup"-costs, having unspecified amount of some useful work to be computed on an unspecified HPC-ecosystem.

Even without hardware & rental details ( devil is always hidden in detail(s) & one can easily pay hilarious amounts of money for trying to make a (hiddenly) "shared"-platform deliver any improved computing performance - many startups have experienced this on voucher-sponsored promises, the more if an overall computing strategy was poorly designed )

I cannot resist not to quote the so called 1st Etore's Law of Evolution of Systems' Dynamics :

If we open a can of worms,
the only way how to put them back
is to use a bigger can

Closing our eyes not to see the accumulating inefficiencies is the worst sin of sins, as devastatingly exponential growths of all of costs, time & resources, incl. energy-consumption are common to meet on such, often many-levels stacked-inefficiencies' complex systems

ELEMENTARY RULES-of-THUMB ... how much we pay in [TIME]

Sorry if these were known to you beforehand, just trying to build some common ground, as a platform to lay further argumentation rock-solid on. More details are here and this is only a needed beginning, as more problems will definitely come from any real-world O( Mx * Ny * ... )-scaling related issues in further modelling.

```<code>                0.1 ns - CPU NOP - a DO-NOTHING instruction
                0.5 ns - CPU L1 dCACHE reference           (1st introduced in late 80-ies )
                1   ns - speed-of-light (a photon) travel a 1 ft (30.5cm) distance -- will stay, throughout any foreseeable future :o)
              3~4   ns - CPU L2  CACHE reference           (2020/Q1)
                7   ns - CPU L2  CACHE reference
               19   ns - CPU L3  CACHE reference           (2020/Q1 considered slow on 28c Skylake)
______________________on_CPU______________________________________________________________________________________
               71   ns - CPU cross-QPI/NUMA best  case on XEON E5-46*
              100   ns - own DDR MEMORY reference
              135   ns - CPU cross-QPI/NUMA best  case on XEON E7-*
              325   ns - CPU cross-QPI/NUMA worst case on XEON E5-46*
            2,500   ns - Read  10 kB sequentially from  MEMORY------ HPC-node
           25,000   ns - Read 100 kB sequentially from  MEMORY------ HPC-node
          250,000   ns - Read   1 MB sequentially from  MEMORY------ HPC-node
        2,500,000   ns - Read  10 MB sequentially from  MEMORY------ HPC-node
       25,000,000   ns - Read 100 MB sequentially from  MEMORY------ HPC-node (abstracted from shared physical RAM-I/O-channels)
      250,000,000   ns - Read   1 GB sequentially from  MEMORY------ HPC-node (abstracted from shared physical RAM-I/O-channels)
    2,500,000,000   ns - Read  10 GB sequentially from  MEMORY------ HPC-node (abstracted from shared physical RAM-I/O-channels)
   25,000,000,000   ns - Read 100 GB sequentially from  MEMORY------ HPC-node (abstracted from shared physical RAM-I/O-channels)
_____________________________________________________________________________own_CPU/DDR__________________________
 |   |   |   |   |
 |   |   |   | ns|
 |   |   | us|
 |   | ms|
 |  s|
h|
</code>```
```<code>          500,000   ns - Round trip within a same DataCenter ------- HPC-node / HPC-storage latency on each access
       20,000,000   ns - Send   2 MB over 1 Gbps  NETWORK
      200,000,000   ns - Send  20 MB over 1 Gbps  NETWORK
    2,000,000,000   ns - Send 200 MB over 1 Gbps  NETWORK
   20,000,000,000   ns - Send   2 GB over 1 Gbps  NETWORK
  200,000,000,000   ns - Send  20 GB over 1 Gbps  NETWORK
2,000,000,000,000   ns - Send 200 GB over 1 Gbps  NETWORK
____________________________________________________________________________via_LAN_______________________________
      150,000,000   ns - Send a NETWORK packet CA -> Netherlands
____________________________________________________________________________via_WAN_______________________________
       10,000,000   ns - DISK seek spent to start file-I/O on spinning disks on any next piece of data seek/read
       30,000,000   ns - DISK   1 MB sequential READ from a DISK
      300,000,000   ns - DISK  10 MB sequential READ from a DISK
    3,000,000,000   ns - DISK 100 MB sequential READ from a DISK
   30,000,000,000   ns - DISK   1 GB sequential READ from a DISK
  300,000,000,000   ns - DISK  10 GB sequential READ from a DISK
3,000,000,000,000   ns - DISK 100 GB sequential READ from a DISK
______________________on_DISK_______________________________________________own_DISK______________________________
 |   |   |   |   |
 |   |   |   | ns|
 |   |   | us|
 |   | ms|
 |  s|
h|
</code>```

Given these elements, the end-to-end computing strategy may and shall be improved.

AS-WAS STATE
... where the crash prevented any computing at all

A naive figure shows more than thousands words

```<code>localhost          |
:       file-I/O ~ 25+GB SLOWEST/EXPENSIVE
:         1st time 25+GB file-I/O-s
:                  |
:                  | RAM       |
:                  |           |
+------+           |IOIOIOIOIOI|
|.CSV 0|           |IOIOIOIOIOI|
|+------+          |IOIOIOIOIOI|
||.CSV 1|          |IOIOIOIOIOI|
||+------+         |IOIOIOIOIOI|-> local ssh()-encrypt+encapsulate-process
|||.CSV 2|         |IOIOIOIOIOI|               25+GB of .CSV
+||      |         |IOIOIOIOIOI|~~~~~~~|
 ||      |         |IOIOIOIOIOI|~~~~~~~|
 +|      |         |           |~~~~~~~|
  |      |         |           |~~~~~~~|
  +------+         |           |~~~~~~~|
  ...              |           |~~~~~~~|-> LAN  SLOW
   ...             |                   |   WAN  SLOWER
    ...            |                   |   transfer of 30+GB to "HPC" ( ssh()-decryption & file-I/O storage-costs omited for clarity )
    +------+       |                   |               |                   30+GB           file-I/O ~ 25+GB SLOWEST/EXPENSIVE
    |.CSV 9|       |                   |~~~~~~~~~~~~~~~|                                     2nd time 25+GB file-I/O-s
    |+------+      |                   |~~~~~~~~~~~~~~~|
    ||.CSV 9|      |                   |~~~~~~~~~~~~~~~|
    ||+------+     |                   |~~~~~~~~~~~~~~~|
    |||.CSV 9|     |                   |~~~~~~~~~~~~~~~|
    +||     9|     |                   |~~~~~~~~~~~~~~~|
     ||     9|     |                   |~~~~~~~~~~~~~~~|
     +|      |     |                   |~~~~~~~~~~~~~~~|
      |      |     |                   |~~~~~~~~~~~~~~~|-> file-I/O into python
      +------+     |                                   |   all .CSV file to RAM ~ 25+GB SLOWEST/EXPENSIVE
                   |                                   |***|                        3rd time 25+GB file-I/O-s
                   |                                   |   RAM .CSV to df CPU work
                   |                                   |***|           df to LIST new RAM-allocation + list.append( df )-costs
                   |                                   |***|                    + 25+GB
                   |                                   |***|
many hours         |                                   |***|
  [SERIAL] flow ...|                                   |***|
/\/\/\/\/\/\/\/\/\/|\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\|***|/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
                   |                       crashed     |***|
                   |                    on about       |***|
                   |                       360-th file |***|
                   |                                   |***|->RAM ~ 50~GB with a LIST of all 25+GB dataframes held in LIST
                   |                                       |  CPU +mem-I/O costs LIST to new 25+GB dataframe RAM-allocation & DATA-processing
                   |                                       |~~~~~| mem-I/O RAM|              :: GB
                   |                                       |~~~~~| mem-I/O    |RAM flow of ~ 50+GB over only 2/3/? mem-I/O HW-channels
                   |                                       |~~~~~|                                      only if "HPC"
                   |                                       |~~~~~|                                           is *NOT* a "shared"-rental of cloud HW,
                   |                                       |~~~~~|                                                    remarketed as an "HPC"-illusion
                   |                                       |~~~~~|
                   |                                             :::::::::::::?
                   |                                             :::::::::::::?
                   |                                             :::::::::::::?
                   |                                            <...some amount of some usefull work --"HPC"-processing the ~ 25+GB dataframe...>
                   |                                            <...some amount of some usefull work                                         ...>
                   |                                            <...some amount of some usefull work         the more                        ...>
                   |                                            <...some amount of some usefull work         the better                      ...>
                   |                                            <...some amount of some usefull work             as                          ...>
                   |                                            <...some amount of some usefull work             it                          ...>
                   |                                            <...some amount of some usefull work             dissolves to AWFULLY        ...>
                   |                                            <...some amount of some usefull work                          HIGH           ...>
                   |                                            <...some amount of some usefull work                          SETUP COSTS    ...>
                   |                                            <...some amount of some usefull work                                         ...>
                   |                                            <...some amount of some usefull work --"HPC"-processing the ~ 25+GB dataframe...>
                   |                                             :::::::::::::?
                   |                                             :::::::::::::?
                   |                                             :::::::::::::?
                   |                                                          |-> file-I/O ~ 25+GB SLOWEST/EXPENSIVE
                   |                                                          |~~~~~|          4th time 25+GB file-I/O-s
                   |                                                          |~~~~~|
                   |                                                          |~~~~~|->file left on remote storage (?)
                   |                                                                |
                   |                                                               O?R
                   |                                                                |
                   |                                                                |-> file-I/O ~ 25+GB SLOWEST/EXPENSIVE
                   |                                                                |~~~~~|          5th time 25+GB file-I/O-s
                   |                                                                |~~~~~|
                   |                                                                |~~~~~|
                   |                                                                |~~~~~|
                   |                                                                |~~~~~|-> RAM / CPU ssh()-encrypt+encapsulate-process
                   |                                                                      |????????????|      25+GB of results for repatriation
                   |                                                                      |????????????|                        on localhost
                   |                                                                      |????????????|
                   |                                                                      |????????????|
                   |                                                                      |????????????|-> LAN  SLOW
                   |                                                                                |   WAN  SLOWER
                   |                                                                                |   transfer of 30+GB from "HPC" ( ssh()-decryption & file-I/O storage-costs omited for clarity )
                   |                                                                                |               |                     30+GB           file-I/O ~ 25+GB SLOWEST/EXPENSIVE
                   |                                                                                |~~~~~~~~~~~~~~~|                                       6th time 25+GB file-I/O-s
                   |                                                                                |~~~~~~~~~~~~~~~|
                   |                                                                                |~~~~~~~~~~~~~~~|
                   |                                                                                |~~~~~~~~~~~~~~~|
                   |                                                                                |~~~~~~~~~~~~~~~|
 SUCCESS ?         |                                                                                |~~~~~~~~~~~~~~~|-> file transferred back and stored on localhost storage
    after          |
          how many |
          failed   |
          attempts |
   having          |
          how high |
          recurring|
          costs    |
          for any  |
              next |
              model|
          recompute|
          step(s)  |
                   |
                   |
All                |
that               |
( at what overall  |
    [TIME]-domain  |
   & "HPC"-rental  |
           costs ) |

</code>```

Tips :

review and reduce, where possible, expensive data-items representation ( avoid using int64, where 8-bits are enough, packed bitmaps can help a lot )

precompute on localhost all items, that could be precomputed ( avoiding repetitive steps )

join the such "reduced" CSV-files, using a trivial O/S command, into a single input

compress all data before transports ( a few orders of magnitude saved )

prefer to code your computing using such algorithms' formulation, that can stream-process items along the data-flow, i.e. not waiting to load all in-RAM to next compute an average or similar trivial on-the-fly stream-computable values ( like .mean(), .sum(), .min(), .max() or even .rsi(), .EMA(), .TEMA(), .BollingerBands() ... and many more alike ) - stream-computing formulated algorithms reduce both the RAM-allocations, can be & shall be pre-computed (once) & minimise the [SERIAL]-one-after-another processing pipeline latency )

if indeed in a need to use pandas and fighting on overall physical RAM-ceilings, may try smart numpy-tools instead, where all the array syntax & methods remain the same, yet it can, by-design, work without moving all data at once from disk into physical RAM ( using this was my life-saving trick since ever, the more when using many-model simulations & HyperParameterSPACE optimisations on a few tens of GB data on 32-bit hardware )

For more details on going into the direction of RAM-protecting memory-mapped np.ndarray processing, with all smart numpy-vectorised and all other high performance-tuned tricks, read more details in this :

```<code>>> print( np.memmap.__doc__ )
Create a memory-map to an array stored in a *binary* file on disk.

   Memory-mapped files are used for accessing small segments of large files
   on disk, without reading the entire file into memory.  NumPy's
   memmap's are array-like objects. (...)
</code>```

Body:
I am trying to read several (>1000) .txt files (on average approx. 700 MB, delimited, header-less CSV, without commas or other separator) and merge them into one pandas dataframe (to next run an analysis on the entire dataset).

I am running this through SSH on a HPC server, on which I requested 50GB RAM, 1 node, 1 task-per-node (that was all just a wild guess, as I have never done this before).

So far, my idea was this:

li = []

for filename in all_files:
    df = pd.read_csv(filename, sep=None, header=0, engine='python')
    li.append(df)

df = pd.concat(li, axis=0, ignore_index=True)

but after a few hours and having loaded the approx. 360th file the process gets killed and I get the error message:

numpy.core._exceptions.MemoryError: Unable to allocate 1.11 GiB for an array with shape (10, 14921599) and data type float64

Do you have any idea how to load and merge the data more memory efficient? (I assume just requesting more RAM still does not get me through the entire set of .txt files!?)

Also, I would like to save the resulting dataframe in a 'memory-efficient' way afterwards, do you know the best way/format (csv?) to do that?

Any help would be much appreciated!

Body:
Q : "How can I ... Any help would be much appreciated!"

A :

best follow the laws of the ECONOMY-of-COMPUTING :

Your briefly sketched problem has, out of question, immense "setup"-costs, having unspecified amount of some useful work to be computed on an unspecified HPC-ecosystem.

Even without hardware & rental details ( devil is always hidden in detail(s) & one can easily pay hilarious amounts of money for trying to make a (hiddenly) "shared"-platform deliver any improved computing performance - many startups have experienced this on voucher-sponsored promises, the more if an overall computing strategy was poorly designed )

I cannot resist not to quote the so called 1st Etore's Law of Evolution of Systems' Dynamics :

If we open a can of worms,
the only way how to put them back
is to use a bigger can

Closing our eyes not to see the accumulating inefficiencies is the worst sin of sins, as devastatingly exponential growths of all of costs, time & resources, incl. energy-consumption are common to meet on such, often many-levels stacked-inefficiencies' complex systems

ELEMENTARY RULES-of-THUMB ... how much we pay in [TIME]

Sorry if these were known to you beforehand, just trying to build some common ground, as a platform to lay further argumentation rock-solid on. More details are here and this is only a needed beginning, as more problems will definitely come from any real-world O( Mx * Ny * ... )-scaling related issues in further modelling.

                0.1 ns - CPU NOP - a DO-NOTHING instruction
                0.5 ns - CPU L1 dCACHE reference           (1st introduced in late 80-ies )
                1   ns - speed-of-light (a photon) travel a 1 ft (30.5cm) distance -- will stay, throughout any foreseeable future :o)
              3~4   ns - CPU L2  CACHE reference           (2020/Q1)
                7   ns - CPU L2  CACHE reference
               19   ns - CPU L3  CACHE reference           (2020/Q1 considered slow on 28c Skylake)
______________________on_CPU______________________________________________________________________________________
               71   ns - CPU cross-QPI/NUMA best  case on XEON E5-46*
              100   ns - own DDR MEMORY reference
              135   ns - CPU cross-QPI/NUMA best  case on XEON E7-*
              325   ns - CPU cross-QPI/NUMA worst case on XEON E5-46*
            2,500   ns - Read  10 kB sequentially from  MEMORY------ HPC-node
           25,000   ns - Read 100 kB sequentially from  MEMORY------ HPC-node
          250,000   ns - Read   1 MB sequentially from  MEMORY------ HPC-node
        2,500,000   ns - Read  10 MB sequentially from  MEMORY------ HPC-node
       25,000,000   ns - Read 100 MB sequentially from  MEMORY------ HPC-node (abstracted from shared physical RAM-I/O-channels)
      250,000,000   ns - Read   1 GB sequentially from  MEMORY------ HPC-node (abstracted from shared physical RAM-I/O-channels)
    2,500,000,000   ns - Read  10 GB sequentially from  MEMORY------ HPC-node (abstracted from shared physical RAM-I/O-channels)
   25,000,000,000   ns - Read 100 GB sequentially from  MEMORY------ HPC-node (abstracted from shared physical RAM-I/O-channels)
_____________________________________________________________________________own_CPU/DDR__________________________
 |   |   |   |   |
 |   |   |   | ns|
 |   |   | us|
 |   | ms|
 |  s|
h|

          500,000   ns - Round trip within a same DataCenter ------- HPC-node / HPC-storage latency on each access
       20,000,000   ns - Send   2 MB over 1 Gbps  NETWORK
      200,000,000   ns - Send  20 MB over 1 Gbps  NETWORK
    2,000,000,000   ns - Send 200 MB over 1 Gbps  NETWORK
   20,000,000,000   ns - Send   2 GB over 1 Gbps  NETWORK
  200,000,000,000   ns - Send  20 GB over 1 Gbps  NETWORK
2,000,000,000,000   ns - Send 200 GB over 1 Gbps  NETWORK
____________________________________________________________________________via_LAN_______________________________
      150,000,000   ns - Send a NETWORK packet CA -> Netherlands
____________________________________________________________________________via_WAN_______________________________
       10,000,000   ns - DISK seek spent to start file-I/O on spinning disks on any next piece of data seek/read
       30,000,000   ns - DISK   1 MB sequential READ from a DISK
      300,000,000   ns - DISK  10 MB sequential READ from a DISK
    3,000,000,000   ns - DISK 100 MB sequential READ from a DISK
   30,000,000,000   ns - DISK   1 GB sequential READ from a DISK
  300,000,000,000   ns - DISK  10 GB sequential READ from a DISK
3,000,000,000,000   ns - DISK 100 GB sequential READ from a DISK
______________________on_DISK_______________________________________________own_DISK______________________________
 |   |   |   |   |
 |   |   |   | ns|
 |   |   | us|
 |   | ms|
 |  s|
h|

Given these elements, the end-to-end computing strategy may and shall be improved.

AS-WAS STATE
... where the crash prevented any computing at all

A naive figure shows more than thousands words

localhost          |
:       file-I/O ~ 25+GB SLOWEST/EXPENSIVE
:         1st time 25+GB file-I/O-s
:                  |
:                  | RAM       |
:                  |           |
+------+           |IOIOIOIOIOI|
|.CSV 0|           |IOIOIOIOIOI|
|+------+          |IOIOIOIOIOI|
||.CSV 1|          |IOIOIOIOIOI|
||+------+         |IOIOIOIOIOI|-> local ssh()-encrypt+encapsulate-process
|||.CSV 2|         |IOIOIOIOIOI|               25+GB of .CSV
+||      |         |IOIOIOIOIOI|~~~~~~~|
 ||      |         |IOIOIOIOIOI|~~~~~~~|
 +|      |         |           |~~~~~~~|
  |      |         |           |~~~~~~~|
  +------+         |           |~~~~~~~|
  ...              |           |~~~~~~~|-> LAN  SLOW
   ...             |                   |   WAN  SLOWER
    ...            |                   |   transfer of 30+GB to "HPC" ( ssh()-decryption & file-I/O storage-costs omited for clarity )
    +------+       |                   |               |                   30+GB           file-I/O ~ 25+GB SLOWEST/EXPENSIVE
    |.CSV 9|       |                   |~~~~~~~~~~~~~~~|                                     2nd time 25+GB file-I/O-s
    |+------+      |                   |~~~~~~~~~~~~~~~|
    ||.CSV 9|      |                   |~~~~~~~~~~~~~~~|
    ||+------+     |                   |~~~~~~~~~~~~~~~|
    |||.CSV 9|     |                   |~~~~~~~~~~~~~~~|
    +||     9|     |                   |~~~~~~~~~~~~~~~|
     ||     9|     |                   |~~~~~~~~~~~~~~~|
     +|      |     |                   |~~~~~~~~~~~~~~~|
      |      |     |                   |~~~~~~~~~~~~~~~|-> file-I/O into python
      +------+     |                                   |   all .CSV file to RAM ~ 25+GB SLOWEST/EXPENSIVE
                   |                                   |***|                        3rd time 25+GB file-I/O-s
                   |                                   |   RAM .CSV to df CPU work
                   |                                   |***|           df to LIST new RAM-allocation + list.append( df )-costs
                   |                                   |***|                    + 25+GB
                   |                                   |***|
many hours         |                                   |***|
  [SERIAL] flow ...|                                   |***|
/\/\/\/\/\/\/\/\/\/|\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\|***|/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
                   |                       crashed     |***|
                   |                    on about       |***|
                   |                       360-th file |***|
                   |                                   |***|->RAM ~ 50~GB with a LIST of all 25+GB dataframes held in LIST
                   |                                       |  CPU +mem-I/O costs LIST to new 25+GB dataframe RAM-allocation & DATA-processing
                   |                                       |~~~~~| mem-I/O RAM|              :: GB
                   |                                       |~~~~~| mem-I/O    |RAM flow of ~ 50+GB over only 2/3/? mem-I/O HW-channels
                   |                                       |~~~~~|                                      only if "HPC"
                   |                                       |~~~~~|                                           is *NOT* a "shared"-rental of cloud HW,
                   |                                       |~~~~~|                                                    remarketed as an "HPC"-illusion
                   |                                       |~~~~~|
                   |                                             :::::::::::::?
                   |                                             :::::::::::::?
                   |                                             :::::::::::::?
                   |                                            <...some amount of some usefull work --"HPC"-processing the ~ 25+GB dataframe...>
                   |                                            <...some amount of some usefull work                                         ...>
                   |                                            <...some amount of some usefull work         the more                        ...>
                   |                                            <...some amount of some usefull work         the better                      ...>
                   |                                            <...some amount of some usefull work             as                          ...>
                   |                                            <...some amount of some usefull work             it                          ...>
                   |                                            <...some amount of some usefull work             dissolves to AWFULLY        ...>
                   |                                            <...some amount of some usefull work                          HIGH           ...>
                   |                                            <...some amount of some usefull work                          SETUP COSTS    ...>
                   |                                            <...some amount of some usefull work                                         ...>
                   |                                            <...some amount of some usefull work --"HPC"-processing the ~ 25+GB dataframe...>
                   |                                             :::::::::::::?
                   |                                             :::::::::::::?
                   |                                             :::::::::::::?
                   |                                                          |-> file-I/O ~ 25+GB SLOWEST/EXPENSIVE
                   |                                                          |~~~~~|          4th time 25+GB file-I/O-s
                   |                                                          |~~~~~|
                   |                                                          |~~~~~|->file left on remote storage (?)
                   |                                                                |
                   |                                                               O?R
                   |                                                                |
                   |                                                                |-> file-I/O ~ 25+GB SLOWEST/EXPENSIVE
                   |                                                                |~~~~~|          5th time 25+GB file-I/O-s
                   |                                                                |~~~~~|
                   |                                                                |~~~~~|
                   |                                                                |~~~~~|
                   |                                                                |~~~~~|-> RAM / CPU ssh()-encrypt+encapsulate-process
                   |                                                                      |????????????|      25+GB of results for repatriation
                   |                                                                      |????????????|                        on localhost
                   |                                                                      |????????????|
                   |                                                                      |????????????|
                   |                                                                      |????????????|-> LAN  SLOW
                   |                                                                                |   WAN  SLOWER
                   |                                                                                |   transfer of 30+GB from "HPC" ( ssh()-decryption & file-I/O storage-costs omited for clarity )
                   |                                                                                |               |                     30+GB           file-I/O ~ 25+GB SLOWEST/EXPENSIVE
                   |                                                                                |~~~~~~~~~~~~~~~|                                       6th time 25+GB file-I/O-s
                   |                                                                                |~~~~~~~~~~~~~~~|
                   |                                                                                |~~~~~~~~~~~~~~~|
                   |                                                                                |~~~~~~~~~~~~~~~|
                   |                                                                                |~~~~~~~~~~~~~~~|
 SUCCESS ?         |                                                                                |~~~~~~~~~~~~~~~|-> file transferred back and stored on localhost storage
    after          |
          how many |
          failed   |
          attempts |
   having          |
          how high |
          recurring|
          costs    |
          for any  |
              next |
              model|
          recompute|
          step(s)  |
                   |
                   |
All                |
that               |
( at what overall  |
    [TIME]-domain  |
   & "HPC"-rental  |
           costs ) |

Tips :

review and reduce, where possible, expensive data-items representation ( avoid using int64, where 8-bits are enough, packed bitmaps can help a lot )

precompute on localhost all items, that could be precomputed ( avoiding repetitive steps )

join the such "reduced" CSV-files, using a trivial O/S command, into a single input

compress all data before transports ( a few orders of magnitude saved )

prefer to code your computing using such algorithms' formulation, that can stream-process items along the data-flow, i.e. not waiting to load all in-RAM to next compute an average or similar trivial on-the-fly stream-computable values ( like .mean(), .sum(), .min(), .max() or even .rsi(), .EMA(), .TEMA(), .BollingerBands() ... and many more alike ) - stream-computing formulated algorithms reduce both the RAM-allocations, can be & shall be pre-computed (once) & minimise the [SERIAL]-one-after-another processing pipeline latency )

if indeed in a need to use pandas and fighting on overall physical RAM-ceilings, may try smart numpy-tools instead, where all the array syntax & methods remain the same, yet it can, by-design, work without moving all data at once from disk into physical RAM ( using this was my life-saving trick since ever, the more when using many-model simulations & HyperParameterSPACE optimisations on a few tens of GB data on 32-bit hardware )

For more details on going into the direction of RAM-protecting memory-mapped np.ndarray processing, with all smart numpy-vectorised and all other high performance-tuned tricks, read more details in this :

>> print( np.memmap.__doc__ )
Create a memory-map to an array stored in a *binary* file on disk.

   Memory-mapped files are used for accessing small segments of large files
   on disk, without reading the entire file into memory.  NumPy's
   memmap's are array-like objects. (...)


问:
How to center in available space, but not use infinite height in Flutter?
Pre-conditions:

We can use a LayoutBuilder to obtain the height available for ListView

We don't know the height of Widget A or Widget B.

How can Widget B be centered in the available space in the left situation while still not causing Flutter to use infinite height for the space to center Widget B within?

Some basic code for the situation:

LayoutBuilder(builder: (context, constraints) {
  return ListView(
    children: [
      Container(
        constraints: BoxConstraints(minHeight: constraints.maxHeight),
        child: Column(
          children: [
            WidgetA(),
            WidgetB(),
          ],
        ),
      ),
    ]
  );
}),
```
<p>I would like to wrap WidgetB in Expanded but only up to the total height of <code>constraints.maxHeight</code> for WidgetA + WidgetB to not get the infinite height exception.</p>
<pre class="lang-dart prettyprint-override"><code>LayoutBuilder(builder: (context, constraints) {
  return ListView(
    children: [
      Container(
        constraints: BoxConstraints(minHeight: constraints.maxHeight),
        child: Column(
          children: [
            WidgetA(),
            Expanded(WidgetB()), // Infinite height exception
          ],
        ),
      ),
    ]
  );
}),
</code>```

答:
The probably easiest solution is to use slivers with CustomScrollView. There is a sliver class SliverFillRemaining which is a perfect match to the problem in this question.

CustomScrollView(
  slivers: [
    SliverToBoxAdapter(
      child: WidgetA(),
    ),
    SliverFillRemaining(
      hasScrollBody: false,
      child: Center(
        child: WidgetB(),
      ),
    ),
  ],
)
```

Based on this gist (dart pad) I got suggested on FlutterDev Discord.

Body:
Pre-conditions:

We can use a LayoutBuilder to obtain the height available for ListView

We don't know the height of Widget A or Widget B.

How can Widget B be centered in the available space in the left situation while still not causing Flutter to use infinite height for the space to center Widget B within?

Some basic code for the situation:

LayoutBuilder(builder: (context, constraints) {
  return ListView(
    children: [
      Container(
        constraints: BoxConstraints(minHeight: constraints.maxHeight),
        child: Column(
          children: [
            WidgetA(),
            WidgetB(),
          ],
        ),
      ),
    ]
  );
}),

I would like to wrap WidgetB in Expanded but only up to the total height of constraints.maxHeight for WidgetA + WidgetB to not get the infinite height exception.

LayoutBuilder(builder: (context, constraints) {
  return ListView(
    children: [
      Container(
        constraints: BoxConstraints(minHeight: constraints.maxHeight),
        child: Column(
          children: [
            WidgetA(),
            Expanded(WidgetB()), // Infinite height exception
          ],
        ),
      ),
    ]
  );
}),

Body:
The probably easiest solution is to use slivers with CustomScrollView. There is a sliver class SliverFillRemaining which is a perfect match to the problem in this question.

CustomScrollView(
  slivers: [
    SliverToBoxAdapter(
      child: WidgetA(),
    ),
    SliverFillRemaining(
      hasScrollBody: false,
      child: Center(
        child: WidgetB(),
      ),
    ),
  ],
)

Based on this gist (dart pad) I got suggested on FlutterDev Discord.


问:
How to replace em dash in a dataframe?
I am trying to remove the 'em dash' from a pandas DataFrame.
I have tried a bunch of different suggestions. But none of them are working.

This is what I have got

```<code># -*- coding: utf-8 -*-

my_str = 'word—word'
# my_str.replace(b'\xe2\x80\x94'.decode('utf-8'), '--')
my_str.replace('—'.decode('utf8'), 'xx')
#my_str.replace('\u2014', 'x')

print(my_str)
</code>```

I am trying to simply replace the em dash '—' with something else.

I am always getting an error.

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 9: ordinal not in range(128)

am stuck on this.  any help appreciated. thanks

Links I have used:

link one

link two

===========

Solution thanks to @Corralien. I am adding a few more points to solve this problem, since it was driving me bonkers.

First check the unicode character you are dealing with using

```<code>print(ord(u'—'))
</code>```

then look it up here

unicode lookup

find the python source code for it. in my case it was returned as 8212 which is in python

```<code>u"\u2014"
</code>```

in python2.7 you need to start the string with 'u' since only from version 3 onwards python interprets every string as unicode by default. so for python 3 use

```<code>"\u2014"
</code>```

fyi lastly the utf-8 hex representation works as well

```<code>.str.replace('\xe2\x80\x94', '+')
</code>```

which corresponds to the utf-8 hex representation

```<code>0xE2 0x80 0x94
</code>```

i hope this helps future visitors, at least i will be able to look this up next time before wasting hours debugging.

PS. in order for str.replace to work you need to use it on a single column only.  str.replace doesnt work for multicolumn dataframes.
so i used it as such:

```<code>df['column'] = df['column'].str.replace(u'\u2014', '+')
</code>```

replacing the column in question with a column where the emdash is replaced with a +.

答:
Use the unicode representation of 'Em dash'

```<code>>>> my_str.replace('\u2014', '*')
'word*word'
</code>```

From a dataframe:

```<code>>>> df['col'].str.replace('\u2014', '*')
0    word*word
Name: col, dtype: object
</code>```

Title: How to replace em dash in a dataframe?

Body:
I am trying to remove the 'em dash' from a pandas DataFrame.
I have tried a bunch of different suggestions. But none of them are working.

This is what I have got

# -*- coding: utf-8 -*-

my_str = 'word—word'
# my_str.replace(b'\xe2\x80\x94'.decode('utf-8'), '--')
my_str.replace('—'.decode('utf8'), 'xx')
#my_str.replace('\u2014', 'x')

print(my_str)

I am trying to simply replace the em dash '—' with something else.

I am always getting an error.

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 9: ordinal not in range(128)

am stuck on this.  any help appreciated. thanks

Links I have used:

link one

link two

===========

Solution thanks to @Corralien. I am adding a few more points to solve this problem, since it was driving me bonkers.

First check the unicode character you are dealing with using

print(ord(u'—'))

then look it up here

unicode lookup

find the python source code for it. in my case it was returned as 8212 which is in python

u"\u2014"

in python2.7 you need to start the string with 'u' since only from version 3 onwards python interprets every string as unicode by default. so for python 3 use

"\u2014"

fyi lastly the utf-8 hex representation works as well

.str.replace('\xe2\x80\x94', '+')

which corresponds to the utf-8 hex representation

0xE2 0x80 0x94

i hope this helps future visitors, at least i will be able to look this up next time before wasting hours debugging.

PS. in order for str.replace to work you need to use it on a single column only.  str.replace doesnt work for multicolumn dataframes.
so i used it as such:

df['column'] = df['column'].str.replace(u'\u2014', '+')

replacing the column in question with a column where the emdash is replaced with a +.

Body:
Use the unicode representation of 'Em dash'

>>> my_str.replace('\u2014', '*')
'word*word'

From a dataframe:

>>> df['col'].str.replace('\u2014', '*')
0    word*word
Name: col, dtype: object