问:
React limits the number of renders to prevent an infinite loop Error
Below is my code

```<code>function App() {
    const [isImportant, setIsImportant] = useState("Yes")

    function handleClick() {
        setIsImportant("No")
    }

    return (
        <div className="state">
            <h1 className="state--title">Is state important to know?</h1>
            <div className="state--value" onClick ={setIsImportant("No")} >
              {isImportant}
            </div>
        </div>
    )
}
</code>```

However I get the error which says 'Uncaught Error: Too many re-renders. React limits the number of renders to prevent an infinite loop.'

If someone can explain why the error is happening it would be grateful!!

答:
You are invoking the setIsImportant in onClick directly

Everytime render happens setIsImportant("No") will be fired and
setState causes the component to re-render and then re-rendering  will
fire setIsImportant("No") again and so on.. which leads to infinite
re-render.

You are setting the state again and again and it never stops.

Solution

To prevent this you need to pass a function to onClick handler

```<code>onClick ={() => setIsImportant("No")}
</code>```

Body:
Below is my code

function App() {
    const [isImportant, setIsImportant] = useState("Yes")

    function handleClick() {
        setIsImportant("No")
    }

    return (

Is state important to know?

              {isImportant}

    )
}

However I get the error which says 'Uncaught Error: Too many re-renders. React limits the number of renders to prevent an infinite loop.'

If someone can explain why the error is happening it would be grateful!!

Body:
You are invoking the setIsImportant in onClick directly

Everytime render happens setIsImportant("No") will be fired and
setState causes the component to re-render and then re-rendering  will
fire setIsImportant("No") again and so on.. which leads to infinite
re-render.

You are setting the state again and again and it never stops.

Solution

To prevent this you need to pass a function to onClick handler

onClick ={() => setIsImportant("No")}


来源: stackexchange

问:
Finding previous Jenkins job that had a specific param value, but without using `hudson.model.Hudson`

Goal:

The end-goal is to find the latest build (not necessarily the previous build!) that had a parameter MY_PARAM value equal to the MY_PARAM value of the current build.

The reason for this is to mimic the "Back to normal" operation of the Slack Plugin behavior on Freestyle Jobs, but making it more specific to check the previous build with a particular param set.

For example, let's say we have this build history:

```<code>Build #300 - MY_PARAM=foo - SUCCESS
Build #301 - MY_PARAM=bar - SUCCESS
Build #302 - MY_PARAM=foo - SUCCESS
Build #303 - MY_PARAM=foo - FAILED
Build #304 - MY_PARAM=bar - FAILED
Build #305 - MY_PARAM=abc - SUCCESS
Build #306 - MY_PARAM=foo - SUCCESS
</code>```

With the above job history, if we launch a new job (#307) with MY_PARAM=bar, I need to scan the most recent job with MY_PARAM=bar and check if it was successful or not (which in this case would have been build #304).

So, I wrote up some script (it works fine) to go in the success{} block of Pipeline post{} section:

```<code>script {
  def inst = hudson.model.Hudson.instance
  def project = null

  Boolean recovery = false
  Integer currentBuildNumber = env.BUILD_NUMBER

  inst.getItems(org.jenkinsci.plugins.workflow.job.WorkflowJob).each {
    if (it.displayName.equals(env.JOB_NAME)) {
      project = it
    }
  }

  def builds = project.getBuilds()

  for (build in builds) {
    Integer buildNumber = build.getNumber()

    // We don't care about the current build
    if (buildNumber == currentBuildNumber) {
      continue
    }

    // Get any params
    def parameters = build.actions.find{ it instanceof ParametersAction }?.parameters
    // Find the param we care about
    def myParam = parameters?.find{ it.name == 'MY_PARAM' }?.value

    if (myParam == params.MY_PARAM) {
      // Mark recovery = true if the previous build with this param wasn't a success
      if (build.result.name != 'SUCCESS') {
        recovery = true
      }
      break
    }
  }

  // Causes java.io.NotSerializableException if not cleared
  builds = project = inst = null

  if (recovery == true) {
    // Does some stuff....
  }
</code>```

So, this works fine! However, it seems like I may be overcomplicating and/or doing this wrong based on warnings I now see in settings.

Problems/Questions

I had to approve many signatures to get this to work. Now my Jenkins settings is glaring red with "unsafe" script approval for staticMethod hudson.model.Hudson getInstance which doesn't feel right. I tried the "assuming permission check" but it didn't work. Should I worry about this? The Jenkins instance is access controlled.

Is there an overall workaround to get the job/param data I need without using hudson.model.Hudson getInstance and without making the solution more complex (ex: w/o using shared libraries)?

Is there overall a different/better way to meet my original goal? All I really need is the build history and the params + build result of each build.

答:
You could iterate builds of current project without accessing Hudson/Jenkins instance

```<code>def build = currentBuild.previousBuild()
while( build ) {
   ...do smth here
   def build = build.previousBuild()
}
</code>```

https://javadoc.jenkins.io/plugin/workflow-support/org/jenkinsci/plugins/workflow/support/steps/build/RunWrapper.html#getPreviousBuild--

ContentLicense: CC BY-SA 4.0

Body:
Goal:

The end-goal is to find the latest build (not necessarily the previous build!) that had a parameter MY_PARAM value equal to the MY_PARAM value of the current build.

The reason for this is to mimic the "Back to normal" operation of the Slack Plugin behavior on Freestyle Jobs, but making it more specific to check the previous build with a particular param set.

For example, let's say we have this build history:

Build #300 - MY_PARAM=foo - SUCCESS
Build #301 - MY_PARAM=bar - SUCCESS
Build #302 - MY_PARAM=foo - SUCCESS
Build #303 - MY_PARAM=foo - FAILED
Build #304 - MY_PARAM=bar - FAILED
Build #305 - MY_PARAM=abc - SUCCESS
Build #306 - MY_PARAM=foo - SUCCESS

With the above job history, if we launch a new job (#307) with MY_PARAM=bar, I need to scan the most recent job with MY_PARAM=bar and check if it was successful or not (which in this case would have been build #304).

So, I wrote up some script (it works fine) to go in the success{} block of Pipeline post{} section:

script {
  def inst = hudson.model.Hudson.instance
  def project = null

  Boolean recovery = false
  Integer currentBuildNumber = env.BUILD_NUMBER

  inst.getItems(org.jenkinsci.plugins.workflow.job.WorkflowJob).each {
    if (it.displayName.equals(env.JOB_NAME)) {
      project = it
    }
  }

  def builds = project.getBuilds()

  for (build in builds) {
    Integer buildNumber = build.getNumber()

    // We don't care about the current build
    if (buildNumber == currentBuildNumber) {
      continue
    }

    // Get any params
    def parameters = build.actions.find{ it instanceof ParametersAction }?.parameters
    // Find the param we care about
    def myParam = parameters?.find{ it.name == 'MY_PARAM' }?.value

    if (myParam == params.MY_PARAM) {
      // Mark recovery = true if the previous build with this param wasn't a success
      if (build.result.name != 'SUCCESS') {
        recovery = true
      }
      break
    }
  }

  // Causes java.io.NotSerializableException if not cleared
  builds = project = inst = null

  if (recovery == true) {
    // Does some stuff....
  }

So, this works fine! However, it seems like I may be overcomplicating and/or doing this wrong based on warnings I now see in settings.

Problems/Questions

I had to approve many signatures to get this to work. Now my Jenkins settings is glaring red with "unsafe" script approval for staticMethod hudson.model.Hudson getInstance which doesn't feel right. I tried the "assuming permission check" but it didn't work. Should I worry about this? The Jenkins instance is access controlled.

Is there an overall workaround to get the job/param data I need without using hudson.model.Hudson getInstance and without making the solution more complex (ex: w/o using shared libraries)?

Is there overall a different/better way to meet my original goal? All I really need is the build history and the params + build result of each build.

Body:
You could iterate builds of current project without accessing Hudson/Jenkins instance

def build = currentBuild.previousBuild()
while( build ) {
   ...do smth here
   def build = build.previousBuild()
}

https://javadoc.jenkins.io/plugin/workflow-support/org/jenkinsci/plugins/workflow/support/steps/build/RunWrapper.html#getPreviousBuild--


问:
Python 3.0 Windows/COM
How to access a COM object from a python file using python 3.0.

And, yes, I know that not a lot of people are using Python 3.0.  Switching back to 2.6 is a huge hassle for me, so I don't want to unless I absolutely have to.

I appreciate your time, and any assistance!

答:
Install pywin32 and then create the object using it's progid:

```<code>import win32com.client
object = win32com.client.Dispatch("Outlook.Application")
</code>```

See also the Python and COM tutorial.

EDIT: Hmm... looks like they may not have a python 3.0 version yet.

ContentLicense: CC BY-SA 2.5

Title: Python 3.0 Windows/COM

Body:
Install pywin32 and then create the object using it's progid:

import win32com.client
object = win32com.client.Dispatch("Outlook.Application")

See also the Python and COM tutorial.

EDIT: Hmm... looks like they may not have a python 3.0 version yet.

OwnerDisplayName: jeffamaphone


问:
R data.frame to SQL - preserving ordered factors
I am just starting to use MySQL to handle data that is currently in R dataframe objects.  I was hoping for a simple round-trip to and from SQL that would recreate an R dataframe exactly:

```<code>library("compare",pos=2)
library("RMySQL",pos=2)
conR <- dbConnect(MySQL(),
             user = '...',
             password = '...',
             host = '...',
             dbname='r2014')
a3 <- data.frame(x=5:1,y=letters[1:5],z=ordered(c("NEVER","ALWAYS","NEVER","SOMETIMES","NEVER"),levels=c("NEVER","SOMETIMES","ALWAYS")))
a3
dbWriteTable(conn = conR, name = 'a3', value = a3)
a4 <- dbReadTable(conn = conR, name = 'a3')
compare(a3,a4)$detailedResult
a3$z
a4$z
</code>```

the result shows that factors end up as strings (columns y and z), and that the ordering information for ordered factors is lost (column z):

```<code>> a3
  x y         z
1 5 a     NEVER
2 4 b    ALWAYS
3 3 c     NEVER
4 2 d SOMETIMES
5 1 e     NEVER
> compare(a3,a4)$detailedResult
    x     y     z
  TRUE FALSE FALSE
> a3$z
[1] NEVER     ALWAYS    NEVER     SOMETIMES NEVER
Levels: NEVER < SOMETIMES < ALWAYS
> a4$z
[1] "NEVER"     "ALWAYS"    "NEVER"     "SOMETIMES" "NEVER"
> a3$y
[1] a b c d e
Levels: a b c d e
> a4$y
[1] "a" "b" "c" "d" "e"
</code>```

Is there some way to specify the information in the ordered factors in the creation of the table a3 in the database?

答:
I would change the code to:

```<code>dbWriteTable(conn = conR, name = 'a3', value = a3, row.names=TRUE)
a4 <- dbReadTable(conn = conR, name = 'a3', row.names=TRUE)
</code>```

row.names of a data.frame are ordered by default. When they are stored in an SQL column they are also ordered. The SELECT query can use ORDER BY row_names to fetch the ordered set.

Value of row.names in dbReadTable() argument can be changed to NA in case the SQL table does not contain the row_names column.[2]

[1] REF: DBI::dbWriteTable

```<code> The interpretation of rownames depends on the ‘row.names’
 argument, see ‘sqlRownamesToColumn()’ for details:

    • If ‘FALSE’ or ‘NULL’, row names are ignored.

    • If ‘TRUE’, row names are converted to a column named
      "row_names", even if the input data frame only has natural
      row names from 1 to ‘nrow(...)’.

    • If ‘NA’, a column named "row_names" is created if the data
      has custom row names, no extra column is created in the case
      of natural row names.

    • If a string, this specifies the name of the column in the
      remote table that contains the row names, even if the input
      data frame only has natural row names.
</code>```

[2] REF: DBI::dbReadTable

```<code> The presence of rownames depends on the ‘row.names’ argument, see
 ‘sqlColumnToRownames()’ for details:

    • If ‘FALSE’ or ‘NULL’, the returned data frame doesn't have
      row names.

    • If ‘TRUE’, a column named "row_names" is converted to row
      names.

    • If ‘NA’, a column named "row_names" is converted to row names
      if it exists, otherwise no translation occurs.

    • If a string, this specifies the name of the column in the
      remote table that contains the row names.
</code>```

ContentLicense: CC BY-SA 3.0

Body:
I am just starting to use MySQL to handle data that is currently in R dataframe objects.  I was hoping for a simple round-trip to and from SQL that would recreate an R dataframe exactly:

library("compare",pos=2)
library("RMySQL",pos=2)
conR <- dbConnect(MySQL(),
             user = '...',
             password = '...',
             host = '...',
             dbname='r2014')
a3 <- data.frame(x=5:1,y=letters[1:5],z=ordered(c("NEVER","ALWAYS","NEVER","SOMETIMES","NEVER"),levels=c("NEVER","SOMETIMES","ALWAYS")))
a3
dbWriteTable(conn = conR, name = 'a3', value = a3)
a4 <- dbReadTable(conn = conR, name = 'a3')
compare(a3,a4)$detailedResult
a3$z
a4$z

the result shows that factors end up as strings (columns y and z), and that the ordering information for ordered factors is lost (column z):

> a3
  x y         z
1 5 a     NEVER
2 4 b    ALWAYS
3 3 c     NEVER
4 2 d SOMETIMES
5 1 e     NEVER
> compare(a3,a4)$detailedResult
    x     y     z
  TRUE FALSE FALSE
> a3$z
[1] NEVER     ALWAYS    NEVER     SOMETIMES NEVER
Levels: NEVER < SOMETIMES < ALWAYS
> a4$z
[1] "NEVER"     "ALWAYS"    "NEVER"     "SOMETIMES" "NEVER"
> a3$y
[1] a b c d e
Levels: a b c d e
> a4$y
[1] "a" "b" "c" "d" "e"

Is there some way to specify the information in the ordered factors in the creation of the table a3 in the database?

Body:
I would change the code to:

dbWriteTable(conn = conR, name = 'a3', value = a3, row.names=TRUE)
a4 <- dbReadTable(conn = conR, name = 'a3', row.names=TRUE)

row.names of a data.frame are ordered by default. When they are stored in an SQL column they are also ordered. The SELECT query can use ORDER BY row_names to fetch the ordered set.

Value of row.names in dbReadTable() argument can be changed to NA in case the SQL table does not contain the row_names column.[2]

[1] REF: DBI::dbWriteTable

 The interpretation of rownames depends on the ‘row.names’
 argument, see ‘sqlRownamesToColumn()’ for details:

    • If ‘FALSE’ or ‘NULL’, row names are ignored.

    • If ‘TRUE’, row names are converted to a column named
      "row_names", even if the input data frame only has natural
      row names from 1 to ‘nrow(...)’.

    • If ‘NA’, a column named "row_names" is created if the data
      has custom row names, no extra column is created in the case
      of natural row names.

    • If a string, this specifies the name of the column in the
      remote table that contains the row names, even if the input
      data frame only has natural row names.

[2] REF: DBI::dbReadTable

 The presence of rownames depends on the ‘row.names’ argument, see
 ‘sqlColumnToRownames()’ for details:

    • If ‘FALSE’ or ‘NULL’, the returned data frame doesn't have
      row names.

    • If ‘TRUE’, a column named "row_names" is converted to row
      names.

    • If ‘NA’, a column named "row_names" is converted to row names
      if it exists, otherwise no translation occurs.

    • If a string, this specifies the name of the column in the
      remote table that contains the row names.


问:
Unable to seed users and roles after a new database is created in .Net 5
Once the database is created I am unable to get the users and roles to seed.
I am using a multi-tenancy app that needs to create a database first and then seed the user and roles.

I do not get any error messages. The code completes however nothing is put into the database.

Main api

```<code>public class systemController : ControllerBase
{
    private readonly SignInManager<User> _signInManager;
    private readonly UserManager<User> _userManager;
    private readonly RoleManager<Role> _roleManager;
    private IServiceProvider _serviceProvider;
    private readonly IMapper _mapper;
    private readonly IUserRepository _repo;

    public systemController(UserManager<User> userManager, SignInManager<User> signInManager,
        RoleManager<Role> roleManager, IServiceProvider serviceProvider, IMapper mapper, IUserRepository repo)
    {
        _userManager = userManager;
        _signInManager = signInManager;
        _roleManager = roleManager;
        _serviceProvider = serviceProvider;
        _mapper = mapper;
        _repo = repo;

    }
    //create the new database
    [HttpGet("CreateNewDb/{dbName}")]
    public ActionResult CreateNewDb(string dbName)
    {
        var __roleManager = _serviceProvider.GetRequiredService<RoleManager<Role>>();
        var __userManager = _serviceProvider.GetRequiredService<UserManager<User>>();
        string serverDataString = "server=localhost; user=user; password=password; database=";

        // this method works and created the database
        createDb(dbName, serverDataString);

        // this method does not work
        createUserRole(dbName, serverDataString, __roleManager, __userManager);
    }
</code>```

createUserRole method

```<code>    private static void createUserRole(string dbName, string serverDataString, RoleManager<Role> __roleManager, UserManager<User> __userManager)
    {
        var connectionString = serverDataString.Replace("database=", "database=" + dbName);

        // build a DbContext for this tenant
        var optionsBuilder = new DbContextOptionsBuilder<DataContext>()
            .UseMySql(connectionString,
            new MySqlServerVersion(new Version(8, 0, 21)));

        DataContext context = new DataContext(optionsBuilder.Options);

        Seed.SeedRoles(context, __roleManager);
        Seed.SeedUsers(context, __userManager);
    }
</code>```

role in seed class

```<code>public class Seed
{
    public static void SeedRoles(DataContext context, RoleManager<Role> roleManager)
    {
        if (!context.Roles.Any())
        {
            //create roles
            var roles = new List<Role>
            {
                new Role{Name = "Admin"},
                new Role{Name = "Standard"},
                new Role{Name = "Basic"}
            };
            foreach (var role in roles)
            {
                roleManager.CreateAsync(role).Wait();
            }
        }

    }
</code>```

答:
I think you might  don't add your roles to context and/or you don't commit changes in a right way. You can just do like this ,</p>
```<code>public class Seed
{
  public static async Task SeedRoles(DataContext context)
  {
      if (!context.Roles.Any())
      {
          //create roles
          var roles = new List<Role>
          {
             new Role{Name = "Admin"},
             new Role{Name = "Standard"},
             new Role{Name = "Basic"}
          };

         context.AddRange(roles);
         await  context.SaveChangesAsync();
    }

}
</code>```

Body:
Once the database is created I am unable to get the users and roles to seed.
I am using a multi-tenancy app that needs to create a database first and then seed the user and roles.

I do not get any error messages. The code completes however nothing is put into the database.

Main api

public class systemController : ControllerBase
{
    private readonly SignInManager<User> _signInManager;
    private readonly UserManager<User> _userManager;
    private readonly RoleManager<Role> _roleManager;
    private IServiceProvider _serviceProvider;
    private readonly IMapper _mapper;
    private readonly IUserRepository _repo;

    public systemController(UserManager<User> userManager, SignInManager<User> signInManager,
        RoleManager<Role> roleManager, IServiceProvider serviceProvider, IMapper mapper, IUserRepository repo)
    {
        _userManager = userManager;
        _signInManager = signInManager;
        _roleManager = roleManager;
        _serviceProvider = serviceProvider;
        _mapper = mapper;
        _repo = repo;

    }
    //create the new database
    [HttpGet("CreateNewDb/{dbName}")]
    public ActionResult CreateNewDb(string dbName)
    {
        var __roleManager = _serviceProvider.GetRequiredService<RoleManager<Role>>();
        var __userManager = _serviceProvider.GetRequiredService<UserManager<User>>();
        string serverDataString = "server=localhost; user=user; password=password; database=";

        // this method works and created the database
        createDb(dbName, serverDataString);

        // this method does not work
        createUserRole(dbName, serverDataString, __roleManager, __userManager);
    }

createUserRole method

    private static void createUserRole(string dbName, string serverDataString, RoleManager<Role> __roleManager, UserManager<User> __userManager)
    {
        var connectionString = serverDataString.Replace("database=", "database=" + dbName);

        // build a DbContext for this tenant
        var optionsBuilder = new DbContextOptionsBuilder<DataContext>()
            .UseMySql(connectionString,
            new MySqlServerVersion(new Version(8, 0, 21)));

        DataContext context = new DataContext(optionsBuilder.Options);

        Seed.SeedRoles(context, __roleManager);
        Seed.SeedUsers(context, __userManager);
    }

role in seed class

public class Seed
{
    public static void SeedRoles(DataContext context, RoleManager<Role> roleManager)
    {
        if (!context.Roles.Any())
        {
            //create roles
            var roles = new List<Role>
            {
                new Role{Name = "Admin"},
                new Role{Name = "Standard"},
                new Role{Name = "Basic"}
            };
            foreach (var role in roles)
            {
                roleManager.CreateAsync(role).Wait();
            }
        }

    }

Body:
I think you might  don't add your roles to context and/or you don't commit changes in a right way. You can just do like this ,

public class Seed
{
  public static async Task SeedRoles(DataContext context)
  {
      if (!context.Roles.Any())
      {
          //create roles
          var roles = new List<Role>
          {
             new Role{Name = "Admin"},
             new Role{Name = "Standard"},
             new Role{Name = "Basic"}
          };

         context.AddRange(roles);
         await  context.SaveChangesAsync();
    }

}


问:
array in array filter - javascript
hello i need help with array , as you can see my data

```<code>{
"age" : "18",
"altKategoriler" : [ "Dramalar" ],
"category" : [ "Aksiyon", "Heyecanlı", "Gerilim" ],
"id" : 5240718100,
"img" : "https://i.ibb.co/k8wx5C8/AAAABW9-ZJQOg-MRljz-Zwe30-JZw-Hf4vq-ERHq6-HMva5-ODHln-Ci-OEV6ir-Rcjt88tcnm-QGQCKpr-K9h-Oll-Ln-Sbb-EI.jpg",
"izlenilmeSayisi" : 0,
"logo" : "https://i.ibb.co/Rb2SrcB/AAAABfcrhh-Rni-Ok-Ct2l-Rys-ZYk-Oi-T0-XTeagkrw-Mkm-U0h-Lr-WIQZHEHg-VXihf-OWCwz-Vv-Qd7u-Ffn-DFZEX2-Ob.webp",
"oyuncuKadrosu" : [ "Diego Luna", "Michael Pena", "Scoot McNairy", "Tenoch Huerta", "Joaquin Cosio" ],
"senarist" : [ "Doug Miro" ],
"time" : "3 Sezon",
"title" : "Narcos: Mexico",
"type" : "Dizi",
"videoDescription" : "Guadalajara Karteli'nin yükselişinin gerçek öyküsünü anlatan bu yeni ve cesur Narcos hikâyesinde, Meksika'daki uyuşturucu savaşının 1980'lerdeki doğuşuna tanıklık edin.",
"videoQuality" : "HD",
"videosrc" : "https://tr.vid.web.acsta.net/uk/medias/nmedia/90/18/10/18/19/19550785_hd_013.mp4",
"year" : "2021",
"yonetmen" : [ "Carlo Bernard", "Chris Brancato" ]
}
</code>```

I can access elements such as id , title or logo because they are not arrays.

How can I loop through the data inside the array since there is an array in the category in yield?

```<code>var data = this.database.filter((item) => item.type == searchType)

var data = this.database.filter((item) => item.category == searchCategory)
</code>```

It's okay because my type value doesn't have an array.

But when I enter my category value, it only gets the first index[0]. It does not look at other indexes.

in summary,
item.category[0] , item.category[1] , item.category[2]...........

How can I get index browsing like

答:
if your data looks like this :

```<code>let data ={
"age" : "18",
"altKategoriler" : [ "Dramalar" ],
"category" : [ "Aksiyon", "Heyecanlı", "Gerilim" ],
"id" : 5240718100,
"img" : "https://i.ibb.co/k8wx5C8/AAAABW9-ZJQOg-MRljz-Zwe30-JZw-Hf4vq-ERHq6-HMva5-ODHln-Ci-OEV6ir-Rcjt88tcnm-QGQCKpr-K9h-Oll-Ln-Sbb-EI.jpg",
"izlenilmeSayisi" : 0,
"logo" : "https://i.ibb.co/Rb2SrcB/AAAABfcrhh-Rni-Ok-Ct2l-Rys-ZYk-Oi-T0-XTeagkrw-Mkm-U0h-Lr-WIQZHEHg-VXihf-OWCwz-Vv-Qd7u-Ffn-DFZEX2-Ob.webp",
"oyuncuKadrosu" : [ "Diego Luna", "Michael Pena", "Scoot McNairy", "Tenoch Huerta", "Joaquin Cosio" ],
"senarist" : [ "Doug Miro" ],
"time" : "3 Sezon",
"title" : "Narcos: Mexico",
"type" : "Dizi",
"videoDescription" : "Guadalajara Karteli'nin yükselişinin gerçek öyküsünü anlatan bu yeni ve cesur Narcos hikâyesinde, Meksika'daki uyuşturucu savaşının 1980'lerdeki doğuşuna tanıklık edin.",
"videoQuality" : "HD",
"videosrc" : "https://tr.vid.web.acsta.net/uk/medias/nmedia/90/18/10/18/19/19550785_hd_013.mp4",
"year" : "2021",
"yonetmen" : [ "Carlo Bernard", "Chris Brancato" ]
}
</code>```

and if we have array of data you can do something like this :

```<code>myArray.filter(item=>item.category.indexOf(searchCategory)>=0)
</code>```

but if you want to explore in object rather than array you can do this :

```<code>data.category.indexOf(searchCategory)>=0
</code>```

Title: array in array filter - javascript

Body:
hello i need help with array , as you can see my data

{
"age" : "18",
"altKategoriler" : [ "Dramalar" ],
"category" : [ "Aksiyon", "Heyecanlı", "Gerilim" ],
"id" : 5240718100,
"img" : "https://i.ibb.co/k8wx5C8/AAAABW9-ZJQOg-MRljz-Zwe30-JZw-Hf4vq-ERHq6-HMva5-ODHln-Ci-OEV6ir-Rcjt88tcnm-QGQCKpr-K9h-Oll-Ln-Sbb-EI.jpg",
"izlenilmeSayisi" : 0,
"logo" : "https://i.ibb.co/Rb2SrcB/AAAABfcrhh-Rni-Ok-Ct2l-Rys-ZYk-Oi-T0-XTeagkrw-Mkm-U0h-Lr-WIQZHEHg-VXihf-OWCwz-Vv-Qd7u-Ffn-DFZEX2-Ob.webp",
"oyuncuKadrosu" : [ "Diego Luna", "Michael Pena", "Scoot McNairy", "Tenoch Huerta", "Joaquin Cosio" ],
"senarist" : [ "Doug Miro" ],
"time" : "3 Sezon",
"title" : "Narcos: Mexico",
"type" : "Dizi",
"videoDescription" : "Guadalajara Karteli'nin yükselişinin gerçek öyküsünü anlatan bu yeni ve cesur Narcos hikâyesinde, Meksika'daki uyuşturucu savaşının 1980'lerdeki doğuşuna tanıklık edin.",
"videoQuality" : "HD",
"videosrc" : "https://tr.vid.web.acsta.net/uk/medias/nmedia/90/18/10/18/19/19550785_hd_013.mp4",
"year" : "2021",
"yonetmen" : [ "Carlo Bernard", "Chris Brancato" ]
}

I can access elements such as id , title or logo because they are not arrays.

How can I loop through the data inside the array since there is an array in the category in yield?

var data = this.database.filter((item) => item.type == searchType)

var data = this.database.filter((item) => item.category == searchCategory)

It's okay because my type value doesn't have an array.

But when I enter my category value, it only gets the first index[0]. It does not look at other indexes.

in summary,
item.category[0] , item.category[1] , item.category[2]...........

How can I get index browsing like

Body:
if your data looks like this :

let data ={
"age" : "18",
"altKategoriler" : [ "Dramalar" ],
"category" : [ "Aksiyon", "Heyecanlı", "Gerilim" ],
"id" : 5240718100,
"img" : "https://i.ibb.co/k8wx5C8/AAAABW9-ZJQOg-MRljz-Zwe30-JZw-Hf4vq-ERHq6-HMva5-ODHln-Ci-OEV6ir-Rcjt88tcnm-QGQCKpr-K9h-Oll-Ln-Sbb-EI.jpg",
"izlenilmeSayisi" : 0,
"logo" : "https://i.ibb.co/Rb2SrcB/AAAABfcrhh-Rni-Ok-Ct2l-Rys-ZYk-Oi-T0-XTeagkrw-Mkm-U0h-Lr-WIQZHEHg-VXihf-OWCwz-Vv-Qd7u-Ffn-DFZEX2-Ob.webp",
"oyuncuKadrosu" : [ "Diego Luna", "Michael Pena", "Scoot McNairy", "Tenoch Huerta", "Joaquin Cosio" ],
"senarist" : [ "Doug Miro" ],
"time" : "3 Sezon",
"title" : "Narcos: Mexico",
"type" : "Dizi",
"videoDescription" : "Guadalajara Karteli'nin yükselişinin gerçek öyküsünü anlatan bu yeni ve cesur Narcos hikâyesinde, Meksika'daki uyuşturucu savaşının 1980'lerdeki doğuşuna tanıklık edin.",
"videoQuality" : "HD",
"videosrc" : "https://tr.vid.web.acsta.net/uk/medias/nmedia/90/18/10/18/19/19550785_hd_013.mp4",
"year" : "2021",
"yonetmen" : [ "Carlo Bernard", "Chris Brancato" ]
}

and if we have array of data you can do something like this :

myArray.filter(item=>item.category.indexOf(searchCategory)>=0)

but if you want to explore in object rather than array you can do this :

data.category.indexOf(searchCategory)>=0


问:
git fatal: Authentication failed
I tried to contribute to Bioconda. However, I got the following error:

```<code>$ git clone https://github.com/mictadlo/bioconda-recipes.git
Cloning into 'bioconda-recipes'...
remote: Enumerating objects: 312339, done.
remote: Total 312339 (delta 0), reused 0 (delta 0), pack-reused 312339
Receiving objects: 100% (312339/312339), 332.37 MiB | 9.93 MiB/s, done.
Resolving deltas: 100% (172897/172897), done.
Updating files: 100% (18842/18842), done.

$ cd bioconda-recipes/
$ git remote add upstream https://ghp_eWgtJStONjyA40srOLuSCnEQ0rrdza@github.com/bioconda/bioconda-recipes.git
$ git checkout master
Already on 'master'

$ git push origin master
Username for 'https://github.com': mictadlo
Password for 'https://mictadlo@github.com':
remote: Support for password authentication was removed on August 13, 2021. Please use a personal access token instead.
remote: Please see https://github.blog/2020-12-15-token-authentication-requirements-for-git-operations/ for more information.
fatal: Authentication failed for 'https://github.com/mictadlo/bioconda-recipes.git/'

</code>```

What did I miss?

Thank you in advance.

答:
Double-check the "password" you have entered when trying to push back to your own repository.

Since Aug. 2021, you would need to use a PAT (Personal Access Token).

You can also, for testing, try with an SSH URL, provided you have registered a public SSH key to your profile.

Title: git fatal: Authentication failed

Body:
I tried to contribute to Bioconda. However, I got the following error:

$ git clone https://github.com/mictadlo/bioconda-recipes.git
Cloning into 'bioconda-recipes'...
remote: Enumerating objects: 312339, done.
remote: Total 312339 (delta 0), reused 0 (delta 0), pack-reused 312339
Receiving objects: 100% (312339/312339), 332.37 MiB | 9.93 MiB/s, done.
Resolving deltas: 100% (172897/172897), done.
Updating files: 100% (18842/18842), done.

$ cd bioconda-recipes/
$ git remote add upstream https://ghp_eWgtJStONjyA40srOLuSCnEQ0rrdza@github.com/bioconda/bioconda-recipes.git
$ git checkout master
Already on 'master'

$ git push origin master
Username for 'https://github.com': mictadlo
Password for 'https://mictadlo@github.com':
remote: Support for password authentication was removed on August 13, 2021. Please use a personal access token instead.
remote: Please see https://github.blog/2020-12-15-token-authentication-requirements-for-git-operations/ for more information.
fatal: Authentication failed for 'https://github.com/mictadlo/bioconda-recipes.git/'

What did I miss?

Thank you in advance.


问:
BottomSheetDialog with transparent background
I would like to display a bottom sheet dialog less wide than the screen width.

For instance, the Share option from Google Play Music on a Nexus 9.

Do you know how to achieve this ?

For now I just succed to reduce the width of the sheet content but the background is still at the screen width and display a white background.

Some code:

build.gradle

```<code>compile 'com.android.support:design:23.3.0'
</code>```

MainActivity

```<code>@Override
protected void onCreate(Bundle savedInstanceState) {
    ...

    mBottomSheetDialog = new BottomSheetDialog(this);
    mBottomSheetDialog.setContentView(R.layout.sheet_test);
    mBottomSheetDialog.setOnDismissListener(new DialogInterface.OnDismissListener() {
        @Override
        public void onDismiss(DialogInterface dialog) {
            mBottomSheetDialog = null;
        }
    });
    mBottomSheetDialog.show();
}
</code>```

sheet_test

```<code><?xml version="1.0" encoding="utf-8"?>
<android.support.v4.widget.NestedScrollView xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="100dp"
    android:layout_height="match_parent"
    android:orientation="vertical">

    <LinearLayout
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:orientation="vertical">

        <TextView
            style="@style/TextAppearance.AppCompat.Body1"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:layout_margin="16dp"
            android:text="Some Text"
            android:textColor="@color/colorPrimary" />

        <View
            android:layout_width="match_parent"
            android:layout_height="1dp"
            android:background="#ddd" />

        <TextView
            style="@style/TextAppearance.AppCompat.Body1"
            android:layout_width="match_parent"
            android:layout_height="match_parent"
            android:layout_margin="16dp"
            android:text="Some Text" />

        <View
            android:layout_width="match_parent"
            android:layout_height="1dp"
            android:background="#ddd" />

    </LinearLayout>

</android.support.v4.widget.NestedScrollView>
</code>```

答:
For me below worked:</p>
```<code>    override fun onViewCreated(view: View, savedInstanceState: Bundle?) {
    super.onViewCreated(view, savedInstanceState)
    (view.parent as? View)?.setBackgroundColor(Color.TRANSPARENT)
}
</code>```

Body:
I would like to display a bottom sheet dialog less wide than the screen width.

For instance, the Share option from Google Play Music on a Nexus 9.

Do you know how to achieve this ?

For now I just succed to reduce the width of the sheet content but the background is still at the screen width and display a white background.

Some code:

build.gradle

compile 'com.android.support:design:23.3.0'

MainActivity

@Override
protected void onCreate(Bundle savedInstanceState) {
    ...

    mBottomSheetDialog = new BottomSheetDialog(this);
    mBottomSheetDialog.setContentView(R.layout.sheet_test);
    mBottomSheetDialog.setOnDismissListener(new DialogInterface.OnDismissListener() {
        @Override
        public void onDismiss(DialogInterface dialog) {
            mBottomSheetDialog = null;
        }
    });
    mBottomSheetDialog.show();
}

sheet_test

<?xml version="1.0" encoding="utf-8"?>
<android.support.v4.widget.NestedScrollView xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="100dp"
    android:layout_height="match_parent"
    android:orientation="vertical">

    <LinearLayout
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:orientation="vertical">

        <TextView
            style="@style/TextAppearance.AppCompat.Body1"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:layout_margin="16dp"
            android:text="Some Text"
            android:textColor="@color/colorPrimary" />

        <View
            android:layout_width="match_parent"
            android:layout_height="1dp"
            android:background="#ddd" />

        <TextView
            style="@style/TextAppearance.AppCompat.Body1"
            android:layout_width="match_parent"
            android:layout_height="match_parent"
            android:layout_margin="16dp"
            android:text="Some Text" />

        <View
            android:layout_width="match_parent"
            android:layout_height="1dp"
            android:background="#ddd" />

    </LinearLayout>

</android.support.v4.widget.NestedScrollView>

Body:
For me below worked:

    override fun onViewCreated(view: View, savedInstanceState: Bundle?) {
    super.onViewCreated(view, savedInstanceState)
    (view.parent as? View)?.setBackgroundColor(Color.TRANSPARENT)
}


问:
How to Export Jupyter Notebook by VSCode in PDF format? (Windows 10)
When I try to export my Jupyter Notebook in pdf format in VSCode like this:

then I got this error:

and jupyter output panel says:

so i tried to install MikTeX and update the required packages, but still I can't export Jupyter Notebooks in PDF format by VSCode!

how can I fix this problem?

Note That I know i can do it by convert it to HTML and then with ctrl+p try to save it as pdf! but I want to convert it to pdf in straight way!

答:
Since I'm using conda venvs, I did these steps:

Activate conda venv using: conda activate <NAME_OF_VENV> in Anaconda prompt.

Install nbconvert using conda install -c anaconda nbconvert

Now it's all okay, and I can export Jupyter notebooks in HTML and PDF format both.

Update

Also, you should consider that nbconvert is compatible with Python 3.7-3.9 based on the official doc.


问:
What is the return type of Specification after creating a join with criteria API
I don't understand how to return the query after making the join and adding the predicate in the specification in this code. Can Anyone help me?

```<code>public class QueriesEntity {
    @Column(name = "query_id")
    @Id
    private int queryId;

    private String name;

    private String description;

    private String query;

    private Integer owner;

    @Column(name = "conn_id")
    private Integer dataSourceId;

    @Column(name = "created_dt")
    @Temporal(TemporalType.TIMESTAMP)
    private Date createdDt;

    @Column(name = "updated_dt")
    @Temporal(TemporalType.TIMESTAMP)
    private Date updatedDt;

    @Column(name = "container_id")
    private Integer containerId;

    @OneToOne
    @JoinColumn(name = "conn_id", updatable = false, insertable = false)
    private DataSourceEntity dataSourceEntity;
</code>```

public static Specification filters(Integer containerId, Integer queryID, Integer datasourceId) {

```<code>    return new Specification<QueriesEntity>() {
        private static final long serialVersionUID = 1L;

        @Override
        public Predicate toPredicate(Root<QueriesEntity> root, CriteriaQuery<?> query,
                CriteriaBuilder criteriaBuilder) {

             query = criteriaBuilder.createQuery(QueriesEntity.class);
             root = query.from(QueriesEntity.class);
            Join<QueriesEntity, DataSourceEntity> join = root.join("dataSourceEntity", JoinType.INNER);

            List<Predicate> outerPreds = new ArrayList<>();
            if (containerId != null) {
                Predicate containerIdPred = criteriaBuilder.equal(root.get("containerId"), containerId);
                outerPreds.add(containerIdPred);
            }
            if(queryID > 0) {
                Predicate queryIdPred = criteriaBuilder.equal(root.get("queryId"), queryID);
                outerPreds.add(queryIdPred);
            }
            if(datasourceId != null) {
                Predicate datasourceIdPred = criteriaBuilder.equal(root.get("dataSourceId"), datasourceId);
                outerPreds.add(datasourceIdPred);
            }
            return criteriaBuilder.and(outerPreds.toArray(new Predicate[outerPreds.size()]));
        }
    };
}
</code>```

答:
When using the Join option you shouldn't use the root to get fields, use the Join instead:

```<code>if (containerId != null) {
    Predicate containerIdPred = criteriaBuilder.equal(join.get("containerId"), containerId);
    outerPreds.add(containerIdPred);
}
</code>```

Example code:

```<code>public static Specification<QueriesEntity> filters(Integer containerId, Integer queryID, Integer datasourceId) {
    return (root, query, cb) -> {
        Join<QueriesEntity, DataSourceEntity> join = root.join("dataSourceEntity", JoinType.INNER);
        List<Predicate> outerPreds = new ArrayList<>();
        if (containerId != null) {
            Predicate containerIdPred = cb.equal(join.get("containerId"), containerId);
            outerPreds.add(containerIdPred);
        }
        if(queryID > 0) {
            Predicate queryIdPred = cb.equal(join.get("queryId"), queryID);
            outerPreds.add(queryIdPred);
        }
        if(datasourceId != null) {
            Predicate datasourceIdPred = cb.equal(root.get("dataSourceId"), datasourceId);
            outerPreds.add(datasourceIdPred);
        }
        return cb.and(outerPreds.stream().toArray(Predicate[]::new));
    };
}
</code>```

Body:
I don't understand how to return the query after making the join and adding the predicate in the specification in this code. Can Anyone help me?

public class QueriesEntity {
    @Column(name = "query_id")
    @Id
    private int queryId;

    private String name;

    private String description;

    private String query;

    private Integer owner;

    @Column(name = "conn_id")
    private Integer dataSourceId;

    @Column(name = "created_dt")
    @Temporal(TemporalType.TIMESTAMP)
    private Date createdDt;

    @Column(name = "updated_dt")
    @Temporal(TemporalType.TIMESTAMP)
    private Date updatedDt;

    @Column(name = "container_id")
    private Integer containerId;

    @OneToOne
    @JoinColumn(name = "conn_id", updatable = false, insertable = false)
    private DataSourceEntity dataSourceEntity;

public static Specification filters(Integer containerId, Integer queryID, Integer datasourceId) {

    return new Specification<QueriesEntity>() {
        private static final long serialVersionUID = 1L;

        @Override
        public Predicate toPredicate(Root<QueriesEntity> root, CriteriaQuery<?> query,
                CriteriaBuilder criteriaBuilder) {

             query = criteriaBuilder.createQuery(QueriesEntity.class);
             root = query.from(QueriesEntity.class);
            Join<QueriesEntity, DataSourceEntity> join = root.join("dataSourceEntity", JoinType.INNER);

            List<Predicate> outerPreds = new ArrayList<>();
            if (containerId != null) {
                Predicate containerIdPred = criteriaBuilder.equal(root.get("containerId"), containerId);
                outerPreds.add(containerIdPred);
            }
            if(queryID > 0) {
                Predicate queryIdPred = criteriaBuilder.equal(root.get("queryId"), queryID);
                outerPreds.add(queryIdPred);
            }
            if(datasourceId != null) {
                Predicate datasourceIdPred = criteriaBuilder.equal(root.get("dataSourceId"), datasourceId);
                outerPreds.add(datasourceIdPred);
            }
            return criteriaBuilder.and(outerPreds.toArray(new Predicate[outerPreds.size()]));
        }
    };
}

Body:
When using the Join option you shouldn't use the root to get fields, use the Join instead:

if (containerId != null) {
    Predicate containerIdPred = criteriaBuilder.equal(join.get("containerId"), containerId);
    outerPreds.add(containerIdPred);
}

Example code:

public static Specification<QueriesEntity> filters(Integer containerId, Integer queryID, Integer datasourceId) {
    return (root, query, cb) -> {
        Join<QueriesEntity, DataSourceEntity> join = root.join("dataSourceEntity", JoinType.INNER);
        List<Predicate> outerPreds = new ArrayList<>();
        if (containerId != null) {
            Predicate containerIdPred = cb.equal(join.get("containerId"), containerId);
            outerPreds.add(containerIdPred);
        }
        if(queryID > 0) {
            Predicate queryIdPred = cb.equal(join.get("queryId"), queryID);
            outerPreds.add(queryIdPred);
        }
        if(datasourceId != null) {
            Predicate datasourceIdPred = cb.equal(root.get("dataSourceId"), datasourceId);
            outerPreds.add(datasourceIdPred);
        }
        return cb.and(outerPreds.stream().toArray(Predicate[]::new));
    };
}


问:
augment stack dimension on shiny server
I'm working on a shiny server and I'm doing some quite complex operations.
The problem I've found is that, after some minutes working on large data frames, the app crashed with this error:
Error: C stack usage  7969194 is too close to the limit. Execution halted.
I googled it and I've found that from the cmd in ubuntu with this command ulimit -s 16384 you can augment the stack size but I didn't find how to do it in the shiny server.

Please, does someone have some ideas?

Thank you

答:
TLDR:

Add the following lines to the end of /etc/security/limits.conf:

```<code>* soft memlock unlimited
* hard memlock unlimited

* soft stack unlimited
* hard stack unlimited
</code>```

Credited to this answer. Bascially this change will adjust the size of stack and memlock memories to unlimited.

More Background:

As OP points out, one can use ulimit to adjust available stack size before entering an R session.

The R package rlimit is supposed to work just like the command line utility ulimit, however, I cannot produce the same behavior after entering into an R session:

```<code>Cstack_info()["size"] # prints 7969177
rlimit_stack(16384, 16384) # expect 16777216
Cstack_info()["size"] # still prints 7969177
</code>```

Eventually I found the above TLDR solution and Error: C stack usage 7969194 is too close to the limit. Execution halted. goes away.

Title: augment stack dimension on shiny server

Body:
TLDR:

Add the following lines to the end of /etc/security/limits.conf:

* soft memlock unlimited
* hard memlock unlimited

* soft stack unlimited
* hard stack unlimited

Credited to this answer. Bascially this change will adjust the size of stack and memlock memories to unlimited.

More Background:

As OP points out, one can use ulimit to adjust available stack size before entering an R session.

The R package rlimit is supposed to work just like the command line utility ulimit, however, I cannot produce the same behavior after entering into an R session:

Cstack_info()["size"] # prints 7969177
rlimit_stack(16384, 16384) # expect 16777216
Cstack_info()["size"] # still prints 7969177

Eventually I found the above TLDR solution and Error: C stack usage 7969194 is too close to the limit. Execution halted. goes away.


问:
How to parallelize a training loop ever samples of a batch when CPU is only available in pytorch?
I want to parallelize over single examples or batch of example (in my situation is that I only have cpus, I have up to 112). I tried it but I get a bug that the losses cannot have the gradient out of separate processes (which entirely ruins my attempt). I still want to do it and it essential that after the multiproessing happens that I can do an optimizer step. How do I get around it? I made a totally self contained example:

```<code>
import torch
import torch.nn as nn
from torch.optim.lr_scheduler import StepLR
from torch.utils.data import Dataset, DataLoader

from torch.multiprocessing import Pool

class SimpleDataSet(Dataset):

    def __init__(self, Din, num_examples=23):
        self.x_dataset = [torch.randn(Din) for _ in range(num_examples)]
        # target function is x*x
        self.y_dataset = [x**2 for x in self.x_dataset]

    def __len__(self):
        return len(self.x_dataset)

    def __getitem__(self, idx):
        return self.x_dataset[idx], self.y_dataset[idx]

def get_loss(args):
    x, y, model = args
    y_pred = model(x)
    criterion = nn.MSELoss()
    loss = criterion(y_pred, y)
    return loss

def get_dataloader(D, num_workers, batch_size):
    ds = SimpleDataSet(D)
    dl = DataLoader(ds, batch_size=batch_size, num_workers=num_workers)
    return dl

def train_fake_data():
    num_workers = 2
    Din, Dout = 3, 1
    model = nn.Linear(Din, Dout).share_memory()

    optimizer = torch.optim.Adam(model.parameters(), lr=0.1)

    batch_size = 2
    num_epochs = 10
    # num_batches = 5
    num_procs = 5
    dataloader = get_dataloader(Din, num_workers, batch_size)
    scheduler = StepLR(optimizer, step_size=1, gamma=0.7)
    for epoch in range(num_epochs):
        for _, batch in enumerate(dataloader):
            batch = [(torch.randn(Din), torch.randn(Dout), model) for _ in batch]
            with Pool(num_procs) as pool:
                optimizer.zero_grad()

                losses = pool.map(get_loss, batch)
                loss = torch.mean(losses)
                loss.backward()

                optimizer.step()
            # scheduler
            scheduler.step()

if __name__ == '__main__':
    # start = time.time()
    # train()
    train_fake_data()
    # print(f'execution time: {time.time() - start}')
</code>```

Error:

```<code>Traceback (most recent call last):
  File "/Users/brando/anaconda3/envs/coq_gym/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3427, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-ea57e03ba088>", line 1, in <module>
    runfile('/Users/brando/ML4Coq/playground/multiprocessing_playground/multiprocessing_cpu_pytorch.py', wdir='/Users/brando/ML4Coq/playground/multiprocessing_playground')
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/brando/ML4Coq/playground/multiprocessing_playground/multiprocessing_cpu_pytorch.py", line 95, in <module>
    train_fake_data()
  File "/Users/brando/ML4Coq/playground/multiprocessing_playground/multiprocessing_cpu_pytorch.py", line 83, in train_fake_data
    losses = pool.map(get_loss, batch)
  File "/Users/brando/anaconda3/envs/coq_gym/lib/python3.7/multiprocessing/pool.py", line 290, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/Users/brando/anaconda3/envs/coq_gym/lib/python3.7/multiprocessing/pool.py", line 683, in get
    raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '[tensor(0.5237, grad_fn=<MseLossBackward>)]'. Reason: 'RuntimeError('Cowardly refusing to serialize non-leaf tensor which requires_grad, since autograd does not support crossing process boundaries.  If you just want to transfer the data, call detach() on the tensor before serializing (e.g., putting it on the queue).')'
</code>```

I am sure I want to do this. How should I be doing this?

New attempt using DDP

```<code>"""
Based on: https://pytorch.org/tutorials/intermediate/ddp_tutorial.html

Note: as opposed to the multiprocessing (torch.multiprocessing) package, processes can use
different communication backends and are not restricted to being executed on the same machine.
"""
import torch
from torch import nn, optim
import torch.distributed as dist
import torch.multiprocessing as mp
from torch.nn.parallel import DistributedDataParallel as DDP

import os

num_epochs = 5
batch_size = 8
Din, Dout = 10, 5
data_x = torch.randn(batch_size, Din)
data_y = torch.randn(batch_size, Dout)
data = [(i*data_x, i*data_y) for i in range(num_epochs)]

class OneDeviceModel(nn.Module):
    """
    Toy example for a model ran in parallel but not distributed accross gpus
    (only processes with their own gpu or hardware)
    """
    def __init__(self):
        super().__init__()
        self.net1 = nn.Linear(Din, Din)
        self.relu = nn.ReLU()
        self.net2 = nn.Linear(Din, Dout)

    def forward(self, x):
        return self.net2(self.relu(self.net1(x)))

def setup_process(rank, world_size, backend='gloo'):
    """
    Initialize the distributed environment (for each process).

    gloo: is a collective communications library (https://github.com/facebookincubator/gloo). My understanding is that
    it's a library/API for process to communicate/coordinate with each other/master. It's a backend library.
    """
    # set up the master's ip address so this child process can coordinate
    # os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'

    # - use NCCL if you are using gpus: https://pytorch.org/tutorials/intermediate/dist_tuto.html#communication-backends
    if torch.cuda.is_available():
        backend = 'nccl'
    # Initializes the default distributed process group, and this will also initialize the distributed package.
    dist.init_process_group(backend, rank=rank, world_size=world_size)

def cleanup():
    """ Destroy a given process group, and deinitialize the distributed package """
    dist.destroy_process_group()

def run_parallel_training_loop(rank, world_size):
    """
    Distributed function to be implemented later.

    This is the function that is actually ran in each distributed process.

    Note: as DDP broadcasts model states from rank 0 process to all other processes in the DDP constructor,
    you don’t need to worry about different DDP processes start from different model parameter initial values.
    """
    print()
    print(f"Start running DDP with model parallel example on rank: {rank}.")
    print(f'current process: {mp.current_process()}')
    print(f'pid: {os.getpid()}')
    setup_process(rank, world_size)

    # create model and move it to GPU with id rank
    model = OneDeviceModel().to(rank) if torch.cuda.is_available() else OneDeviceModel().share_memory()
    # ddp_model = DDP(model, device_ids=[rank])
    ddp_model = DDP(model)

    for batch_idx, batch in enumerate(data):
        x, y = batch
        loss_fn = nn.MSELoss()
        optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

        optimizer.zero_grad()
        outputs = ddp_model(x)
        labels = y.to(rank) if torch.cuda.is_available() else y
        # Gradient synchronization communications take place during the backward pass and overlap with the backward computation.
        loss_fn(outputs, labels).backward()  # When the backward() returns, param.grad already contains the synchronized gradient tensor.
        optimizer.step()  # TODO how does the optimizer know to do the gradient step only once?

    print()
    print(f"Start running DDP with model parallel example on rank: {rank}.")
    print(f'current process: {mp.current_process()}')
    print(f'pid: {os.getpid()}')
    # Destroy a given process group, and deinitialize the distributed package
    cleanup()

def main():
    print()
    print('running main()')
    print(f'current process: {mp.current_process()}')
    print(f'pid: {os.getpid()}')
    # args
    world_size = mp.cpu_count()
    mp.spawn(run_parallel_training_loop, args=(world_size,), nprocs=world_size)

if __name__ == "__main__":
    print('starting __main__')
    main()
    print('Done!\a\n')
</code>```

it seems it works but my question is in line 74 do I need to do this

```<code>    model = OneDeviceModel().to(rank) if torch.cuda.is_available() else OneDeviceModel().share_memory()
</code>```

or

```<code>    model = OneDeviceModel().to(rank) if torch.cuda.is_available() else OneDeviceModel()
</code>```

for it to work properly in multiple CPUs?

Serial is faster than parallel even if I have 112 cpu cores?

My current issue is that the cpu parallel job is slower than the serially running one when only cpus are available.

I want to know how to set up python and parallel cpus. e.g. if I have X cpus how many processes should I be running...X? or what? How do I choose this number, even if its heursitics rough.

related links from research:

https://discuss.pytorch.org/t/multiprocessing-for-loop-on-cpu/59836

How to use multiprocessing in PyTorch?

https://discuss.pytorch.org/t/how-to-parallelize-a-loop-over-the-samples-of-a-batch/32698/7

https://www.reddit.com/r/pytorch/comments/sm073v/how_to_parallelize_a_training_loop_ever_samples/

答:
Torch will use multiple CPU to parallelize operations, so your serial is maybe using multi-core vectorization.

Take this simple example

import torch
c = 0;
for i in range(10000):
    A = torch.randn(1000, 1000, device='cpu');
    B = torch.randn(1000, 1000, device='cpu');
    c += torch.sum(A @ B)
```
<p>No code is used to parallelize, however 80% of 12 CPUs with the default configuration.</p>
<p><img src="https://i.stack.imgur.com/5FUNX.png" alt="enter image description here" /></p>
<p>You can use torch.set_num_threads to set intraop parallelism on CPU. In particular if you are running multiple process and you want each process to use a single CPU you may want to set in each process the intraop parallelism to <code>1</code>.</p>
<p>However, parallelizing the operations has a cost, I am unable go into the implementation details but we can run a quick experiment that shows the overhead of using multiple threads.</p>
<pre class="lang-py prettyprint-override"><code>import matplotlib.pyplot as plt
import numpy as np
import torch;
import time;
A = torch.randn(1000, 1000, device='cpu');
B = torch.randn(1000, 1000, device='cpu');
funcs = {
    'sin': lambda a,b: torch.sin(A),
    'tanh': lambda a,b: torch.tanh(A),
    'log': lambda a,b: torch.log(A),
    'matmul': lambda a,b: A @ B.T
}
t = np.zeros(20)
for k,f in funcs.items():
    for i in range(1, len(t) + 1):
        torch.set_num_threads(i)
        c = 0;
        t0 = time.time();
        for _ in range(100):
            f(A,B)
        tf = time.time()
        t[i-1] = (tf - t0)*i;
    plt.plot(np.arange(1, len(t)+1), t, '-o', label=k)
plt.xlabel('Number of threads')
plt.legend()
plt.ylabel('Core x time')
</code>```

The operations tends to run faster with parallelism

But if we take the total CPU time, by multiplying by the number of threads, we see that the single thread version is more efficient.

If you are able to parallelize your experiment at a higher level, by running independent processes, you should try that with a single core for each process, otherwise each process will try to use all the CPUs and all of them will run very slowly because your system is overloaded.

Tweaking DDP example

I modified hyperparameters of your example scripts intentionally in a way that weights in favor of multi process.

comparably less initialization overhead

comparably less communication between processes

```<code>"""
Based on: https://pytorch.org/tutorials/intermediate/ddp_tutorial.html

Note: as opposed to the multiprocessing (torch.multiprocessing) package, processes can use
different communication backends and are not restricted to being executed on the same machine.
"""
import torch
from torch import nn, optim
import torch.distributed as dist
import torch.multiprocessing as mp
from torch.nn.parallel import DistributedDataParallel as DDP
import argparse
import os

# More than one epoch so that the initialization is less significant
# than compared to the model processing time
num_epochs = 10
# for the experiment select a number that has a lot of divisors
# as I want to test with equal number of batches
num_batches = 16*9*5
# Uses a larger batch so that more work is done in each process
# between two gradient synchronizations
# apparently the intraop optimization is not helping
# (at least not too much) in the batch dimension
batch_size = 10000
# Use smaller dimensions, so that the intraop parallelization becomes less
# helpful
Din, Dout = 3, 5
data_x = torch.randn(batch_size, Din)
data_y = torch.randn(batch_size, Dout)
data = [(i*data_x, i*data_y) for i in range(num_batches)]

class OneDeviceModel(nn.Module):
    """
    Toy example for a model ran in parallel but not distributed accross gpus
    (only processes with their own gpu or hardware)
    """
    def __init__(self):
        super().__init__()
        # -- Use more layers
        self.net = [nn.Linear(Din, Din) for _ in range(10)]
        # -- Bob: use more complex activation
        self.tanh = nn.Tanh()
        self.sigmoid = nn.Sigmoid()
        self.relu = nn.ReLU()
        self.net2 = nn.Linear(Din, Dout)

    def forward(self, x):
      # apply the 10 layers sequentially
      for i in range(10):
        x = self.net[i](x)
        x = self.sigmoid(x)
        x = self.tanh(x)
        x = self.relu(x)
      return self.net2(x)

def setup_process(rank, world_size, backend='gloo'):
    """
    Initialize the distributed environment (for each process).

    gloo: is a collective communications library (https://github.com/facebookincubator/gloo). My understanding is that
    it's a library/API for process to communicate/coordinate with each other/master. It's a backend library.
    """
    # set up the master's ip address so this child process can coordinate
    # os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'

    # - use NCCL if you are using gpus: https://pytorch.org/tutorials/intermediate/dist_tuto.html#communication-backends
    if torch.cuda.is_available():
        backend = 'nccl'
    # Initializes the default distributed process group, and this will also initialize the distributed package.
    dist.init_process_group(backend, rank=rank, world_size=world_size)

def cleanup():
    """ Destroy a given process group, and deinitialize the distributed package """
    dist.destroy_process_group()

def run_parallel_training_loop(rank, world_size):
    """
    Distributed function to be implemented later.

    This is the function that is actually ran in each distributed process.

    Note: as DDP broadcasts model states from rank 0 process to all other processes in the DDP constructor,
    you don’t need to worry about different DDP processes start from different model parameter initial values.
    """
    print()
    print(f"Start running DDP with model parallel example on rank: {rank}.")
    print(f'current process: {mp.current_process()}')
    print(f'pid: {os.getpid()}')
    setup_process(rank, world_size)
    torch.set_num_threads(mp.cpu_count() // world_size)
    # create model and move it to GPU with id rank
    model = OneDeviceModel().to(rank) if torch.cuda.is_available() else OneDeviceModel().share_memory()
    # ddp_model = DDP(model, device_ids=[rank])
    ddp_model = DDP(model)
    for _ in range(num_epochs):
      for batch_idx, batch in enumerate(data[rank::world_size]):
          x, y = batch
          loss_fn = nn.MSELoss()
          optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

          optimizer.zero_grad()
          outputs = ddp_model(x)
          labels = y.to(rank) if torch.cuda.is_available() else y
          # Gradient synchronization communications take place during the backward pass and overlap with the backward computation.
          loss_fn(outputs, labels).backward()  # When the backward() returns, param.grad already contains the synchronized gradient tensor.
          optimizer.step()  # TODO how does the optimizer know to do the gradient step only once?

    print()
    print(f"Start running DDP with model parallel example on rank: {rank}.")
    print(f'current process: {mp.current_process()}')
    print(f'pid: {os.getpid()}')
    # Destroy a given process group, and deinitialize the distributed package
    cleanup()

def main():
    print()
    print('running main()')
    print(f'current process: {mp.current_process()}')
    print(f'pid: {os.getpid()}')
    parser = argparse.ArgumentParser()
    parser.add_argument('--world-size', default=1, type=int)
    args = parser.parse_args()
    assert num_batches % args.world_size == 0
    mp.spawn(run_parallel_training_loop, args=(args.world_size,), nprocs=args.world_size)

if __name__ == "__main__":
    print('starting __main__')
    main()
    print('Done!\a\n')
</code>```
```<code>$ time python3 ddp.py --world-size 1 > /dev/null

real    0m59.092s
user    8m46.589s
sys     0m7.320s

$ time python3 ddp.py --world-size 1 > /dev/null

real    1m11.124s
user    10m54.209s
sys     0m9.595s

$ time python3 ddp.py --world-size 6 > /dev/null

real    0m18.348s
user    2m28.799s
sys     0m18.068s
$ time python3 ddp.py --world-size 12 > /dev/null

real    0m26.352s
user    4m3.074s
sys     0m39.179s
$ time python3 ddp.py --world-size 3 > /dev/null

real    0m23.047s
user    3m51.172s
sys     0m11.483s
$ time python3 ddp.py --world-size 4 > /dev/null

real    0m18.195s
user    2m55.241s
sys     0m12.841s
$ time python3 ddp.py --world-size 2 > /dev/null

real    0m26.955s
user    4m15.837s
sys     0m7.127s
</code>```

If I remove the line

torch.set_num_threads(mp.cpu_count() // world_size)
```
```$ time python3 ddp.py --world-size 4 > /dev/null

real    0m40.574s
user    6m39.176s
sys     0m19.025s

$ time python3 ddp.py --world-size 2 > /dev/null

real    0m28.066s
user    3m17.775s
sys     0m8.410s

$ time python3 ddp.py --world-size 1 > /dev/null

real    0m37.114s
user    2m19.743s
sys     0m4.866s
```
<p>Using</p>
<pre class="lang-py prettyprint-override"><code>torch.set_num_threads(mp.cpu_count() // world_size // 2)
</code>```

$ time python3 ddp.py --world-size 6 > /dev/null

real    0m16.399s
user    1m38.915s
sys     0m20.780s

$ time python3 ddp.py --world-size 4 > /dev/null

real    0m15.649s
user    1m1.821s
sys     0m13.589s

$ time python3 ddp.py --world-size 3 > /dev/null

real    0m16.947s
user    1m29.696s
sys     0m10.069s

$ time python3 ddp.py --world-size 2 > /dev/null

real    0m21.851s
user    2m4.564s
sys     0m7.486s
```

My Opinion

DDP in a single node seems not particularly advantageous. Unless you have a model that does a lot of work that is particularly not well handled by pytorch intraop parallelism, have large batches, and preferrably models with less parameters and more operations, meaning less gradients to synchronize, e.g. a convolutional model on a very large input.

Other scenarios where DDP might be helpful is if you are using too much python in your model, instead of vectorized operations.

Body:
I want to parallelize over single examples or batch of example (in my situation is that I only have cpus, I have up to 112). I tried it but I get a bug that the losses cannot have the gradient out of separate processes (which entirely ruins my attempt). I still want to do it and it essential that after the multiproessing happens that I can do an optimizer step. How do I get around it? I made a totally self contained example:

import torch
import torch.nn as nn
from torch.optim.lr_scheduler import StepLR
from torch.utils.data import Dataset, DataLoader

from torch.multiprocessing import Pool

class SimpleDataSet(Dataset):

    def __init__(self, Din, num_examples=23):
        self.x_dataset = [torch.randn(Din) for _ in range(num_examples)]
        # target function is x*x
        self.y_dataset = [x**2 for x in self.x_dataset]

    def __len__(self):
        return len(self.x_dataset)

    def __getitem__(self, idx):
        return self.x_dataset[idx], self.y_dataset[idx]

def get_loss(args):
    x, y, model = args
    y_pred = model(x)
    criterion = nn.MSELoss()
    loss = criterion(y_pred, y)
    return loss

def get_dataloader(D, num_workers, batch_size):
    ds = SimpleDataSet(D)
    dl = DataLoader(ds, batch_size=batch_size, num_workers=num_workers)
    return dl

def train_fake_data():
    num_workers = 2
    Din, Dout = 3, 1
    model = nn.Linear(Din, Dout).share_memory()

    optimizer = torch.optim.Adam(model.parameters(), lr=0.1)

    batch_size = 2
    num_epochs = 10
    # num_batches = 5
    num_procs = 5
    dataloader = get_dataloader(Din, num_workers, batch_size)
    scheduler = StepLR(optimizer, step_size=1, gamma=0.7)
    for epoch in range(num_epochs):
        for _, batch in enumerate(dataloader):
            batch = [(torch.randn(Din), torch.randn(Dout), model) for _ in batch]
            with Pool(num_procs) as pool:
                optimizer.zero_grad()

                losses = pool.map(get_loss, batch)
                loss = torch.mean(losses)
                loss.backward()

                optimizer.step()
            # scheduler
            scheduler.step()

if __name__ == '__main__':
    # start = time.time()
    # train()
    train_fake_data()
    # print(f'execution time: {time.time() - start}')

Error:

Traceback (most recent call last):
  File "/Users/brando/anaconda3/envs/coq_gym/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3427, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-ea57e03ba088>", line 1, in <module>
    runfile('/Users/brando/ML4Coq/playground/multiprocessing_playground/multiprocessing_cpu_pytorch.py', wdir='/Users/brando/ML4Coq/playground/multiprocessing_playground')
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/brando/ML4Coq/playground/multiprocessing_playground/multiprocessing_cpu_pytorch.py", line 95, in <module>
    train_fake_data()
  File "/Users/brando/ML4Coq/playground/multiprocessing_playground/multiprocessing_cpu_pytorch.py", line 83, in train_fake_data
    losses = pool.map(get_loss, batch)
  File "/Users/brando/anaconda3/envs/coq_gym/lib/python3.7/multiprocessing/pool.py", line 290, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/Users/brando/anaconda3/envs/coq_gym/lib/python3.7/multiprocessing/pool.py", line 683, in get
    raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '[tensor(0.5237, grad_fn=<MseLossBackward>)]'. Reason: 'RuntimeError('Cowardly refusing to serialize non-leaf tensor which requires_grad, since autograd does not support crossing process boundaries.  If you just want to transfer the data, call detach() on the tensor before serializing (e.g., putting it on the queue).')'

I am sure I want to do this. How should I be doing this?

New attempt using DDP

"""
Based on: https://pytorch.org/tutorials/intermediate/ddp_tutorial.html

Note: as opposed to the multiprocessing (torch.multiprocessing) package, processes can use
different communication backends and are not restricted to being executed on the same machine.
"""
import torch
from torch import nn, optim
import torch.distributed as dist
import torch.multiprocessing as mp
from torch.nn.parallel import DistributedDataParallel as DDP

import os

num_epochs = 5
batch_size = 8
Din, Dout = 10, 5
data_x = torch.randn(batch_size, Din)
data_y = torch.randn(batch_size, Dout)
data = [(i*data_x, i*data_y) for i in range(num_epochs)]

class OneDeviceModel(nn.Module):
    """
    Toy example for a model ran in parallel but not distributed accross gpus
    (only processes with their own gpu or hardware)
    """
    def __init__(self):
        super().__init__()
        self.net1 = nn.Linear(Din, Din)
        self.relu = nn.ReLU()
        self.net2 = nn.Linear(Din, Dout)

    def forward(self, x):
        return self.net2(self.relu(self.net1(x)))

def setup_process(rank, world_size, backend='gloo'):
    """
    Initialize the distributed environment (for each process).

    gloo: is a collective communications library (https://github.com/facebookincubator/gloo). My understanding is that
    it's a library/API for process to communicate/coordinate with each other/master. It's a backend library.
    """
    # set up the master's ip address so this child process can coordinate
    # os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'

    # - use NCCL if you are using gpus: https://pytorch.org/tutorials/intermediate/dist_tuto.html#communication-backends
    if torch.cuda.is_available():
        backend = 'nccl'
    # Initializes the default distributed process group, and this will also initialize the distributed package.
    dist.init_process_group(backend, rank=rank, world_size=world_size)

def cleanup():
    """ Destroy a given process group, and deinitialize the distributed package """
    dist.destroy_process_group()

def run_parallel_training_loop(rank, world_size):
    """
    Distributed function to be implemented later.

    This is the function that is actually ran in each distributed process.

    Note: as DDP broadcasts model states from rank 0 process to all other processes in the DDP constructor,
    you don’t need to worry about different DDP processes start from different model parameter initial values.
    """
    print()
    print(f"Start running DDP with model parallel example on rank: {rank}.")
    print(f'current process: {mp.current_process()}')
    print(f'pid: {os.getpid()}')
    setup_process(rank, world_size)

    # create model and move it to GPU with id rank
    model = OneDeviceModel().to(rank) if torch.cuda.is_available() else OneDeviceModel().share_memory()
    # ddp_model = DDP(model, device_ids=[rank])
    ddp_model = DDP(model)

    for batch_idx, batch in enumerate(data):
        x, y = batch
        loss_fn = nn.MSELoss()
        optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

        optimizer.zero_grad()
        outputs = ddp_model(x)
        labels = y.to(rank) if torch.cuda.is_available() else y
        # Gradient synchronization communications take place during the backward pass and overlap with the backward computation.
        loss_fn(outputs, labels).backward()  # When the backward() returns, param.grad already contains the synchronized gradient tensor.
        optimizer.step()  # TODO how does the optimizer know to do the gradient step only once?

    print()
    print(f"Start running DDP with model parallel example on rank: {rank}.")
    print(f'current process: {mp.current_process()}')
    print(f'pid: {os.getpid()}')
    # Destroy a given process group, and deinitialize the distributed package
    cleanup()

def main():
    print()
    print('running main()')
    print(f'current process: {mp.current_process()}')
    print(f'pid: {os.getpid()}')
    # args
    world_size = mp.cpu_count()
    mp.spawn(run_parallel_training_loop, args=(world_size,), nprocs=world_size)

if __name__ == "__main__":
    print('starting __main__')
    main()
    print('Done!\a\n')

it seems it works but my question is in line 74 do I need to do this

    model = OneDeviceModel().to(rank) if torch.cuda.is_available() else OneDeviceModel().share_memory()

or

    model = OneDeviceModel().to(rank) if torch.cuda.is_available() else OneDeviceModel()

for it to work properly in multiple CPUs?

Serial is faster than parallel even if I have 112 cpu cores?

My current issue is that the cpu parallel job is slower than the serially running one when only cpus are available.

I want to know how to set up python and parallel cpus. e.g. if I have X cpus how many processes should I be running...X? or what? How do I choose this number, even if its heursitics rough.

related links from research:

https://discuss.pytorch.org/t/multiprocessing-for-loop-on-cpu/59836

How to use multiprocessing in PyTorch?

https://discuss.pytorch.org/t/how-to-parallelize-a-loop-over-the-samples-of-a-batch/32698/7

https://www.reddit.com/r/pytorch/comments/sm073v/how_to_parallelize_a_training_loop_ever_samples/

Body:
Torch will use multiple CPU to parallelize operations, so your serial is maybe using multi-core vectorization.

Take this simple example

import torch
c = 0;
for i in range(10000):
    A = torch.randn(1000, 1000, device='cpu');
    B = torch.randn(1000, 1000, device='cpu');
    c += torch.sum(A @ B)

No code is used to parallelize, however 80% of 12 CPUs with the default configuration.

You can use torch.set_num_threads to set intraop parallelism on CPU. In particular if you are running multiple process and you want each process to use a single CPU you may want to set in each process the intraop parallelism to 1.

However, parallelizing the operations has a cost, I am unable go into the implementation details but we can run a quick experiment that shows the overhead of using multiple threads.

import matplotlib.pyplot as plt
import numpy as np
import torch;
import time;
A = torch.randn(1000, 1000, device='cpu');
B = torch.randn(1000, 1000, device='cpu');
funcs = {
    'sin': lambda a,b: torch.sin(A),
    'tanh': lambda a,b: torch.tanh(A),
    'log': lambda a,b: torch.log(A),
    'matmul': lambda a,b: A @ B.T
}
t = np.zeros(20)
for k,f in funcs.items():
    for i in range(1, len(t) + 1):
        torch.set_num_threads(i)
        c = 0;
        t0 = time.time();
        for _ in range(100):
            f(A,B)
        tf = time.time()
        t[i-1] = (tf - t0)*i;
    plt.plot(np.arange(1, len(t)+1), t, '-o', label=k)
plt.xlabel('Number of threads')
plt.legend()
plt.ylabel('Core x time')

The operations tends to run faster with parallelism

But if we take the total CPU time, by multiplying by the number of threads, we see that the single thread version is more efficient.

If you are able to parallelize your experiment at a higher level, by running independent processes, you should try that with a single core for each process, otherwise each process will try to use all the CPUs and all of them will run very slowly because your system is overloaded.

Tweaking DDP example

I modified hyperparameters of your example scripts intentionally in a way that weights in favor of multi process.

comparably less initialization overhead

comparably less communication between processes

"""
Based on: https://pytorch.org/tutorials/intermediate/ddp_tutorial.html

Note: as opposed to the multiprocessing (torch.multiprocessing) package, processes can use
different communication backends and are not restricted to being executed on the same machine.
"""
import torch
from torch import nn, optim
import torch.distributed as dist
import torch.multiprocessing as mp
from torch.nn.parallel import DistributedDataParallel as DDP
import argparse
import os

# More than one epoch so that the initialization is less significant
# than compared to the model processing time
num_epochs = 10
# for the experiment select a number that has a lot of divisors
# as I want to test with equal number of batches
num_batches = 16*9*5
# Uses a larger batch so that more work is done in each process
# between two gradient synchronizations
# apparently the intraop optimization is not helping
# (at least not too much) in the batch dimension
batch_size = 10000
# Use smaller dimensions, so that the intraop parallelization becomes less
# helpful
Din, Dout = 3, 5
data_x = torch.randn(batch_size, Din)
data_y = torch.randn(batch_size, Dout)
data = [(i*data_x, i*data_y) for i in range(num_batches)]

class OneDeviceModel(nn.Module):
    """
    Toy example for a model ran in parallel but not distributed accross gpus
    (only processes with their own gpu or hardware)
    """
    def __init__(self):
        super().__init__()
        # -- Use more layers
        self.net = [nn.Linear(Din, Din) for _ in range(10)]
        # -- Bob: use more complex activation
        self.tanh = nn.Tanh()
        self.sigmoid = nn.Sigmoid()
        self.relu = nn.ReLU()
        self.net2 = nn.Linear(Din, Dout)

    def forward(self, x):
      # apply the 10 layers sequentially
      for i in range(10):
        x = self.net[i](x)
        x = self.sigmoid(x)
        x = self.tanh(x)
        x = self.relu(x)
      return self.net2(x)

def setup_process(rank, world_size, backend='gloo'):
    """
    Initialize the distributed environment (for each process).

    gloo: is a collective communications library (https://github.com/facebookincubator/gloo). My understanding is that
    it's a library/API for process to communicate/coordinate with each other/master. It's a backend library.
    """
    # set up the master's ip address so this child process can coordinate
    # os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'

    # - use NCCL if you are using gpus: https://pytorch.org/tutorials/intermediate/dist_tuto.html#communication-backends
    if torch.cuda.is_available():
        backend = 'nccl'
    # Initializes the default distributed process group, and this will also initialize the distributed package.
    dist.init_process_group(backend, rank=rank, world_size=world_size)

def cleanup():
    """ Destroy a given process group, and deinitialize the distributed package """
    dist.destroy_process_group()

def run_parallel_training_loop(rank, world_size):
    """
    Distributed function to be implemented later.

    This is the function that is actually ran in each distributed process.

    Note: as DDP broadcasts model states from rank 0 process to all other processes in the DDP constructor,
    you don’t need to worry about different DDP processes start from different model parameter initial values.
    """
    print()
    print(f"Start running DDP with model parallel example on rank: {rank}.")
    print(f'current process: {mp.current_process()}')
    print(f'pid: {os.getpid()}')
    setup_process(rank, world_size)
    torch.set_num_threads(mp.cpu_count() // world_size)
    # create model and move it to GPU with id rank
    model = OneDeviceModel().to(rank) if torch.cuda.is_available() else OneDeviceModel().share_memory()
    # ddp_model = DDP(model, device_ids=[rank])
    ddp_model = DDP(model)
    for _ in range(num_epochs):
      for batch_idx, batch in enumerate(data[rank::world_size]):
          x, y = batch
          loss_fn = nn.MSELoss()
          optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

          optimizer.zero_grad()
          outputs = ddp_model(x)
          labels = y.to(rank) if torch.cuda.is_available() else y
          # Gradient synchronization communications take place during the backward pass and overlap with the backward computation.
          loss_fn(outputs, labels).backward()  # When the backward() returns, param.grad already contains the synchronized gradient tensor.
          optimizer.step()  # TODO how does the optimizer know to do the gradient step only once?

    print()
    print(f"Start running DDP with model parallel example on rank: {rank}.")
    print(f'current process: {mp.current_process()}')
    print(f'pid: {os.getpid()}')
    # Destroy a given process group, and deinitialize the distributed package
    cleanup()

def main():
    print()
    print('running main()')
    print(f'current process: {mp.current_process()}')
    print(f'pid: {os.getpid()}')
    parser = argparse.ArgumentParser()
    parser.add_argument('--world-size', default=1, type=int)
    args = parser.parse_args()
    assert num_batches % args.world_size == 0
    mp.spawn(run_parallel_training_loop, args=(args.world_size,), nprocs=args.world_size)

if __name__ == "__main__":
    print('starting __main__')
    main()
    print('Done!\a\n')

$ time python3 ddp.py --world-size 1 > /dev/null

real    0m59.092s
user    8m46.589s
sys     0m7.320s

$ time python3 ddp.py --world-size 1 > /dev/null

real    1m11.124s
user    10m54.209s
sys     0m9.595s

$ time python3 ddp.py --world-size 6 > /dev/null

real    0m18.348s
user    2m28.799s
sys     0m18.068s
$ time python3 ddp.py --world-size 12 > /dev/null

real    0m26.352s
user    4m3.074s
sys     0m39.179s
$ time python3 ddp.py --world-size 3 > /dev/null

real    0m23.047s
user    3m51.172s
sys     0m11.483s
$ time python3 ddp.py --world-size 4 > /dev/null

real    0m18.195s
user    2m55.241s
sys     0m12.841s
$ time python3 ddp.py --world-size 2 > /dev/null

real    0m26.955s
user    4m15.837s
sys     0m7.127s

If I remove the line

torch.set_num_threads(mp.cpu_count() // world_size)

$ time python3 ddp.py --world-size 4 > /dev/null

real    0m40.574s
user    6m39.176s
sys     0m19.025s

$ time python3 ddp.py --world-size 2 > /dev/null

real    0m28.066s
user    3m17.775s
sys     0m8.410s

$ time python3 ddp.py --world-size 1 > /dev/null

real    0m37.114s
user    2m19.743s
sys     0m4.866s

Using

torch.set_num_threads(mp.cpu_count() // world_size // 2)

$ time python3 ddp.py --world-size 6 > /dev/null

real    0m16.399s
user    1m38.915s
sys     0m20.780s

$ time python3 ddp.py --world-size 4 > /dev/null

real    0m15.649s
user    1m1.821s
sys     0m13.589s

$ time python3 ddp.py --world-size 3 > /dev/null

real    0m16.947s
user    1m29.696s
sys     0m10.069s

$ time python3 ddp.py --world-size 2 > /dev/null

real    0m21.851s
user    2m4.564s
sys     0m7.486s

My Opinion

DDP in a single node seems not particularly advantageous. Unless you have a model that does a lot of work that is particularly not well handled by pytorch intraop parallelism, have large batches, and preferrably models with less parameters and more operations, meaning less gradients to synchronize, e.g. a convolutional model on a very large input.

Other scenarios where DDP might be helpful is if you are using too much python in your model, instead of vectorized operations.


问:
Run maven building profile without building, just run exec-maven-plugin
I created 4 run profiles that contain only the maven-exec plugin with goal java to run my war with different startup parameters. Problem is that profiles are always triggering rebuilding my war file, and it causes failure because the war file is in use when at least one of these profiles is already running. I don't want to rebuild war when I run these profiles. Is it possible to run the build profile without building war and just run the plugins in the profile?

```<code><profiles>
    <profile>
        <id>devAdmin</id>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.codehaus.mojo</groupId>
                    <artifactId>exec-maven-plugin</artifactId>
                    <executions>
                        <execution>
                            <phase>package</phase>
                            <goals>
                                <goal>java</goal>
                            </goals>
                        </execution>
                    </executions>
                    <configuration>
                        <mainClass>main.Main</mainClass>
                        <arguments>
                            <argument>-admin</argument>
                        </arguments>
                        ...
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>
    <profile>
        <id>devCluster</id>
        <activation>
            <activeByDefault>false</activeByDefault>
        </activation>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.codehaus.mojo</groupId>
                    <artifactId>exec-maven-plugin</artifactId>
                    <executions>
                        <execution>
                            <phase>package</phase>
                            <goals>
                                <goal>java</goal>
                            </goals>
                        </execution>
                    </executions>
                    <configuration>
                        <mainClass>main.Main</mainClass>
                        <arguments>
                            <argument>-cluster</argument>
                        </arguments>
                        ...
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>
    <profile>
        <id>devClient</id>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.codehaus.mojo</groupId>
                    <artifactId>exec-maven-plugin</artifactId>
                    <executions>
                        <execution>
                            <phase>package</phase>
                            <goals>
                                <goal>java</goal>
                            </goals>
                        </execution>
                    </executions>
                    <configuration>
                        <mainClass>main.Main</mainClass>
                        <arguments>
                            <argument>-client</argument>
                        </arguments>
                       ...
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>
    <profile>
        <id>devWeb</id>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.codehaus.mojo</groupId>
                    <artifactId>exec-maven-plugin</artifactId>
                    <executions>
                        <execution>
                            <phase>package</phase>
                            <goals>
                                <goal>java</goal>
                            </goals>
                        </execution>
                    </executions>
                    <configuration>
                        <mainClass>main.Main</mainClass>
                        <arguments>
                            <argument>-web</argument>
                        </arguments>
                        ...
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>
</profiles>
</code>```

Thank you so much for help :)

答: Execute just the single goal: mvn exec:java -PdevClient

Body:
I created 4 run profiles that contain only the maven-exec plugin with goal java to run my war with different startup parameters. Problem is that profiles are always triggering rebuilding my war file, and it causes failure because the war file is in use when at least one of these profiles is already running. I don't want to rebuild war when I run these profiles. Is it possible to run the build profile without building war and just run the plugins in the profile?

<profiles>
    <profile>
        <id>devAdmin</id>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.codehaus.mojo</groupId>
                    <artifactId>exec-maven-plugin</artifactId>
                    <executions>
                        <execution>
                            <phase>package</phase>
                            <goals>
                                <goal>java</goal>
                            </goals>
                        </execution>
                    </executions>
                    <configuration>
                        <mainClass>main.Main</mainClass>
                        <arguments>
                            <argument>-admin</argument>
                        </arguments>
                        ...
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>
    <profile>
        <id>devCluster</id>
        <activation>
            <activeByDefault>false</activeByDefault>
        </activation>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.codehaus.mojo</groupId>
                    <artifactId>exec-maven-plugin</artifactId>
                    <executions>
                        <execution>
                            <phase>package</phase>
                            <goals>
                                <goal>java</goal>
                            </goals>
                        </execution>
                    </executions>
                    <configuration>
                        <mainClass>main.Main</mainClass>
                        <arguments>
                            <argument>-cluster</argument>
                        </arguments>
                        ...
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>
    <profile>
        <id>devClient</id>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.codehaus.mojo</groupId>
                    <artifactId>exec-maven-plugin</artifactId>
                    <executions>
                        <execution>
                            <phase>package</phase>
                            <goals>
                                <goal>java</goal>
                            </goals>
                        </execution>
                    </executions>
                    <configuration>
                        <mainClass>main.Main</mainClass>
                        <arguments>
                            <argument>-client</argument>
                        </arguments>
                       ...
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>
    <profile>
        <id>devWeb</id>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.codehaus.mojo</groupId>
                    <artifactId>exec-maven-plugin</artifactId>
                    <executions>
                        <execution>
                            <phase>package</phase>
                            <goals>
                                <goal>java</goal>
                            </goals>
                        </execution>
                    </executions>
                    <configuration>
                        <mainClass>main.Main</mainClass>
                        <arguments>
                            <argument>-web</argument>
                        </arguments>
                        ...
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>
</profiles>

Thank you so much for help :)


问:
Why ismy nameValue, amountValue and dateValue undefined?
I try to append a tr element to a tbody element, it works but when the tr is shown, the td is undefined, NaN and undefined. I try to get the value from the input elements and submit the tr of data to tbody.

Do you guys have any idea of where i went wrong?

```<code><!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Expense tracker remake</title>
</head>
<body>
    <h1>Expense tracker</h1>
  <div class="bold" id="name">Name of Item: </div>  <input type="text">
  <div class="bold" id="amount">Amount: </div> <input type="text">
  <div class="bold" id="date">Date: </div> <input type="date">
  <button id="submit">Submit</button> <button id="clear">Clear all</button>
  <table>
      <thead>
          <th>Name</th>
          <th>Amount</th>
          <th>Date</th>
      </thead>
      <tbody id="tbody">
      </tbody>
  </table>

      <script>

    var tbody = document.getElementById("tbody");
    var name = document.getElementById("name");
    var amount = document.getElementById("amount");
    var date = document.getElementById("date");
    var clear = document.getElementById("clear");
    var submit = document.getElementById("submit");

    submit.addEventListener("click",function(){
      var nameValue = name.value;
      var amountValue = amount.value;
      var dateValue = date.value;

         var tr = document.createElement("tr");
            tr.innerHTML += "<td>"+nameValue+ " </td><td>"+ + amountValue+ "</td><td>" + dateValue+ "</td>";
            tbody.appendChild(tr);

            console.log(name.value);

    })

      </script>
</code>```

答:
You made two mistakes:

Your name, amount and date elements are a divs, not an inputs, so when you trying to read from name.value you are trying to read from empty div.

Some words in javascript are reserved and sometimes when you using those your code could works in strange way. One of those words is name. Source.

The working demo of your code you can find here: jsFiddle.net.

As you can see, I've just pointed to input instead of div (by calling nextElementSibling because your input is a next element of your div) and changing variables names for avoiding name conflicts. Also, I have removed unnecessary doubled plus signs from tr.innerHtml += ....

Body:
I try to append a tr element to a tbody element, it works but when the tr is shown, the td is undefined, NaN and undefined. I try to get the value from the input elements and submit the tr of data to tbody.

Do you guys have any idea of where i went wrong?

<!DOCTYPE html>

    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Expense tracker remake</title>

Expense tracker

Name of Item:
  <input type="text">

Amount:
 <input type="text">

Date:
 <input type="date">
  <button id="submit">Submit</button> <button id="clear">Clear all</button>

          Name
          Amount
          Date