答:
Hierarchical databases used to be very popular, but they went out of fashion in the 80s because they are not very good at supporting ad hoc querying, and setting them up could be difficult, I believe. The best known standard for hierarchical databases was the CODASYL data model (see http://en.m.wikipedia.org/wiki/CODASYL for details of this), which was integrated into the COBOL language. Commercial implementations are still available, but I don't anyone does new work in it any more.

A more modern equivalent is the document-store database, of which mongodb is the most popular. Mongodb stores and handles queries about documents in BSON, which is a data format roughly equivalent to JSON in capability. This means that while not all xml can be easily mapped to it, a lot can, including the examples you link. See http://bsonspec.org/ formore detail about  BSON.

Title: What is the hierarchical DB schema for this xml?

Body:
As per this link, this is the xml that maintains "BookStore" data.

I see that number of occurences of author are varying for any book.

xml's hierarchical representation of BookStore looks more meaningful/intuitive for me to visualise the data unlike relational table tuples. It is rather more difficult(kind of tweek) to map such hierarchical information in tables.

It is strange to see that real world(hierarchical) data(for example, SNMP MIBs of network devices or BookStore etc...) are mapped to tables of records in many commercial softwares. For me, it is not-intuitive(rather unnecessary skill) to convert such hierarchical data to tables, despite relational DB(like MSSQL) support massive/safe/multi-user/convenient/efficient/reliable unlike  xml file format.

So, we are trying to fit hierarchical ordered data into table tuples which is an overhead. Do we have any commercial database that companies use to maintain schema in hierarchical fashion?

Note: am currently part of database course.


问:
How do I deal with imported modules in GPL?
I wish to release a non-commercial python application under GPL3 in both source and (PyInstaller) executable form. I'm struggling to find a consistent answer to the problem of imported modules.

Can I just put the usual GPL headers on my source files, include the GPL in COPYING and leave it at that?

Or do I have to collect together all the licences and source code (both python and, e.g. C/C++ dependencies) for everything that I import and include these in my distribution? I'm assuming that I don't, but reading the GPL3 licence suggests* that I do. Surely I don't have to ship the source for Qt, PyQt4, Numpy, Scipy, Matplotlib, Gdal etc. etc. in my distribution?

I also want to distribute a windows binary because the end users are unlikely to use the software if they have to run from source. This will package and re-distribute python and all the shared library dependancies. What are my legal obligations here with GPL? Are the imports 'Major Components' and therefore do not need to have the source distributed?

Section 1, Source code:

... The “Corresponding Source” for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code...

答:
You asked:

Can I just put the usual GPL headers on my source files, include the GPL in COPYING and leave it at that?

Yes, that's fine.

Or do I have to collect together all the licences and source code (both python and, e.g. C/C++ dependencies) for everything that I import and include these in my distribution?

No, because they won't be released as their previous license

To be a little less cryptic about that.

So long as the incorporated projects have GPL compatible licenses, when you include the modules within your program, you are effectively re-licensing those modules as GPL'd modules.  That's why you don't have to1 worry about including their previous license state.

As far as distributing the code - The GPL doesn't require you to distribute the code along with the binary.  It only requires you to make the code available for distribution.  That's an important nuance to keep in mind.

Given that your project is going to subsume other projects, I would encourage you to make available for distribution the source code of the projects that you incorporated into your application.

How all of the re-licensing works is a little complicated.  Have a look at an earlier answer of mine to understand how the GPL and copyleft apply to compatibly licensed projects that are incorporated into another.

The shortest summary from my earlier answer is the FSF quote explaining how it works.

It means that the other license and the GNU GPL are compatible; you can combine code released under the other license with code released under the GNU GPL in one larger program.

1  You really, really do need to make sure that their current license is a GPL compatible license though. You are not legally allowed to incorporate a non-compatiblely licensed project within your GPL application.  You would need to look at possibly the LGPL or AGPL instead.

ContentLicense: CC BY-SA 3.0


答:
The GPL primarily puts obligations on persons redistributing your code. You can't sue yourself for breaching the GPL anyway, so what you do with your own code is your own decision.

Now, whether you need to redistribute source code for those libraries depends on their licenses, not yours. You may be able to rely on subsection 6c in case they use (L)GPL 3: "Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b.".

Where it gets tricky is when you mix MIT/BSD code with existing GPL'ed code. This may force you to provide all sources per the GPL license, as all that code is needed to build your GPL-licensed executable. You can't refuse to distribute MIT/BSD code when GPL code depends on it.

Title: How do I deal with imported modules in GPL?

Body:
I wish to release a non-commercial python application under GPL3 in both source and (PyInstaller) executable form. I'm struggling to find a consistent answer to the problem of imported modules.

Can I just put the usual GPL headers on my source files, include the GPL in COPYING and leave it at that?

Or do I have to collect together all the licences and source code (both python and, e.g. C/C++ dependencies) for everything that I import and include these in my distribution? I'm assuming that I don't, but reading the GPL3 licence suggests* that I do. Surely I don't have to ship the source for Qt, PyQt4, Numpy, Scipy, Matplotlib, Gdal etc. etc. in my distribution?

I also want to distribute a windows binary because the end users are unlikely to use the software if they have to run from source. This will package and re-distribute python and all the shared library dependancies. What are my legal obligations here with GPL? Are the imports 'Major Components' and therefore do not need to have the source distributed?

Section 1, Source code:

... The “Corresponding Source” for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code...


问:
Does distributing non-GPLd assets with a GPL application violate the license?
This is somewhat related to my other question, but is actually different.

I would like to license a Windows Phone application under the GPL. All other Windows Phone Marketplace issues aside (I'll ask those on the forums), I'd like to include icons that ship with the SDK in my application. While this is common practice (documentation points to the icons' location), I'm not sure if I'd be forcing GPL on the icons (a move expressly forbidden by the Application Provider Agreement).

How is this usually handled in GPL or am I simply out of luck?

EDIT

The icons in question are copied into the project's source tree and distributed with the application.

The license that ships with the SDK states that  you may not:

modify or distribute the source code of any Distributable Code [incl. the icons] so that any part of it becomes subject to an Excluded License. An Excluded License is one that requires, as a condition of use, modification or distribution, that

the code be disclosed or distributed in source code form; or

others have the right to modify it.

My question now becomes: can I distribute the icons with my GPL'd application in any way (for example, by including provisions in my license text) that would not violate either the GPL or the SDK agreement?

答:
Yes, this violates the license. The SDK doesn't spell out the GPL by name, but there's no doubt that the GPL is one of those Excluded Licenses. Hence, the SDK specifically bans you from distributing Distributable Code [incl. the icons] as part of your GPL application. As you have no other right to do so, regular copyright law kicks in and forbids you to distribute those icons.

As the original author of your own code, you could effectively eliminate all the clauses that cause the GPL to be an Excluded License, but you cannot claim that the resulting license is still GPL. Also, you would end with a license that's no longer OSF recognized.


问:
Ideal data structure/techniques for storing generic scheduler data in C#
I am trying to implement a generic scheduler object in C# 4 which will output a table in HTML. Basic aim is to show some object along with various attributes, and whether it was doing something in a given time period.

The scheduler will output a table displaying the headers:

Detail Field 1 ....N| Date1.........N

I want to initialise the table with a start date and an end date to create the date range (ideally could also do other time periods e.g. hours but that isn't vital). I then want to provide a generic object which will have associated events.

Where an object has events within the period I want a table cell to be marked

E.g.

```<code>Name Height Weight 1/1/2011 2/1/2011 3/1/20011...... 31/1/2011
Ben  5.11    75       X        X                       X
Bill 5.7     83                X        X
</code>```

So I created scheduler with Start Date=1/1/2011 and end date 31/1/2011

I'd like to give it my person object (already sorted) and tell it which fields I want displayed (Name, Height, Weight)

Each person has events which have a start date and end date. Some events will start and end outwith but they should still be shown on the relevant date etc.

Ideally I'd like to have been able to provide it with say a class booking object as well. So I'm trying to keep it generic.

I have seen Javasript implementations etc of similar.

What would a good data structure be for this? Any thoughts on techniques I could use to make it generic. I am not great with generics so any tips appreciated.

答:
In general, I wouldn't store the data for your solution based on the way that you want to display it.  That leads to very specific solutions that will make things difficult when your needs change.  I would break things down in terms of entities in your solution and then create a set of LINQ Queries that would generate your display data when it is time to generate the report.

All the code below is structural and omitting initialization, etc.

```<code>public class Person
{
   public string Name {get;set;}
   public double Height {get;set;}
   public double Weight {get;set;}
}

public class ScheduledEvent
{
   public Person Attendee {get;set;}
   public DateTime EventDate {get;set;}
}

public class Schedule
{
   public DateTime StartDate {get;set; }
   public DateTIme EndDate {get;set;}
   List<Person> Persons {get;set;}
   List<ScheduledEvent> SheduledEvents {get;set;}
}
</code>```

Now, in practice, I would store these in a database of some kind and do all the queries at runtime with something like NHibernate, or Entity Framework.  But for in-memory demo purposes, queries to produce your table lines would be like the following:

```<code>class TableRow
{
   string Name { get;set; }
   string Height {get;set; }
   string Weight {get;set; }
   List<string> DatesWithEvents {get; }
}

var columns = List<string>{ "Name", "Height", "Weight", }.Concat(
                schedule.ScheduledEvents
                  .Select(e=> e.EventDate.ToShortDate())
                  .Distinct())
              .ToList();
var rows = new List<TableRow>();

foreach(var p in schedule.Persons)
{
   var row = new TableRow();
   row.Name = p.Name;
   row.Height = p.Height;
   row.Weight = p.Weight;
   row.DatesWithEvents = schedule.ScheduledEvents
     .Where(e => e.Person == p)
     .Select(e => e.EventDate.ToShortDate()).Distinct().ToList();
   rows.Add(row);
}
</code>```

Then in your renderer, just match up the columns to know where to stick the 'X' to mark that the date is filled.  With more effort, you can refactor this into a much more elegant solution, but this is the basics.

Body:
In general, I wouldn't store the data for your solution based on the way that you want to display it.  That leads to very specific solutions that will make things difficult when your needs change.  I would break things down in terms of entities in your solution and then create a set of LINQ Queries that would generate your display data when it is time to generate the report.

All the code below is structural and omitting initialization, etc.

public class Person
{
   public string Name {get;set;}
   public double Height {get;set;}
   public double Weight {get;set;}
}

public class ScheduledEvent
{
   public Person Attendee {get;set;}
   public DateTime EventDate {get;set;}
}

public class Schedule
{
   public DateTime StartDate {get;set; }
   public DateTIme EndDate {get;set;}
   List<Person> Persons {get;set;}
   List<ScheduledEvent> SheduledEvents {get;set;}
}

Now, in practice, I would store these in a database of some kind and do all the queries at runtime with something like NHibernate, or Entity Framework.  But for in-memory demo purposes, queries to produce your table lines would be like the following:

class TableRow
{
   string Name { get;set; }
   string Height {get;set; }
   string Weight {get;set; }
   List<string> DatesWithEvents {get; }
}

var columns = List<string>{ "Name", "Height", "Weight", }.Concat(
                schedule.ScheduledEvents
                  .Select(e=> e.EventDate.ToShortDate())
                  .Distinct())
              .ToList();
var rows = new List<TableRow>();

foreach(var p in schedule.Persons)
{
   var row = new TableRow();
   row.Name = p.Name;
   row.Height = p.Height;
   row.Weight = p.Weight;
   row.DatesWithEvents = schedule.ScheduledEvents
     .Where(e => e.Person == p)
     .Select(e => e.EventDate.ToShortDate()).Distinct().ToList();
   rows.Add(row);
}

Then in your renderer, just match up the columns to know where to stick the 'X' to mark that the date is filled.  With more effort, you can refactor this into a much more elegant solution, but this is the basics.


问:
Developer Branch vs. Feature Branch
I saw around that development teams usually use source control in one of 2 schemes:

Either "Feature Branch", where each feature is developed in a new branch, or "Developer Branch", where each developer in the team has a dedicated branch for his work.

I understand the technical differences, but I don't really understand when it's better to use each scheme.

I mainly use HG, but I don't think it matters

答:
From my experience, feature branches are strictly more general-purpose and flexible. The advantage is that feature branches allow for a developer to work on multiple branches concurrently - and even for multiple developers to be working on the same branch, if needed, without confusion ("Why is Joe writing commits on the Bob branch...?").

Using developer-specific branches, although it is simpler initially, can cause friction when many features are being worked on simultaneously (things like git stash can be used to somewhat avoid this, but at the cost of lowered transparency).

And perhaps one of the more important issues is that you lose a bit of semantic information when using developer-specific branches. The line:

```<code>git merge topnav-style-change
</code>```

is simple and can help to avoid mistakes (it could not, for example, be easily confused with the branch massive-refactoring). On the other hand,

```<code>git merge joe
</code>```

does not convey much by itself ("Was it Joe or Bob who was working on the style changes...?").

For these kinds of reasons, in general, I see more projects nowadays using feature branches. However, the two choices are not necessarily exclusive - they could be used together as-is, or even in some kind of hybrid mix. It really depends on the complexity and needs of your project.

Title: Developer Branch vs. Feature Branch

Body:
From my experience, feature branches are strictly more general-purpose and flexible. The advantage is that feature branches allow for a developer to work on multiple branches concurrently - and even for multiple developers to be working on the same branch, if needed, without confusion ("Why is Joe writing commits on the Bob branch...?").

Using developer-specific branches, although it is simpler initially, can cause friction when many features are being worked on simultaneously (things like git stash can be used to somewhat avoid this, but at the cost of lowered transparency).

And perhaps one of the more important issues is that you lose a bit of semantic information when using developer-specific branches. The line:

git merge topnav-style-change

is simple and can help to avoid mistakes (it could not, for example, be easily confused with the branch massive-refactoring). On the other hand,

git merge joe

does not convey much by itself ("Was it Joe or Bob who was working on the style changes...?").

For these kinds of reasons, in general, I see more projects nowadays using feature branches. However, the two choices are not necessarily exclusive - they could be used together as-is, or even in some kind of hybrid mix. It really depends on the complexity and needs of your project.


问:
Toggling "maintenance mode" in Java EE app
I've been toying with the idea of feature toggles for various configuration/access purposes, but have been a little unsure of myself when it comes to toggling something like a downtime or maintenance mode, where users should be unable to access anything during that time.

It seems like such a simple thing to do implementation-wise, but I wanted to get some input from those more experienced than I on the subject.

My first thought was just checking a flag in a database (this is a clustered environment, so a file-based flag seems nonideal in our setup), but it feels wrong with the database hit on every request. So, I moved to do it similar to a timed in-memory cache, but that also doesn't feel right to me, as I wouldn't want the cache to be so short-lived that it has to update too frequently from the db, but I also don't want to turn on maintenance mode and potentially have users able to come in during the time before the update occurs. Should I maybe have a cache with a separate forced update trigger as mentioned here?

How have you handled toggling an app-wide maintenance or downtime mode?

答:
Two-step switch

One possibility is to use two-step switch.

Imagine you need to perform a maintenance task during which users should not be able to access the database.

Step 1

You start by changing a flag in a database. This flag is not a maintenance mode flag, but indicates that clients should not use the database any longer since it will soon be put into maintenance mode.

The value of this flag can be cached for, say, five minutes to avoid repeatedly querying the database.

Step 2

Five minutes later, you start the actual maintenance task.

You're now sure that everyone got their cache invalidated, so either the client already queried for the flag and know that the database is about to go into maintenance mode, or will necessarily query for this value before doing any other queries.

Pros

You don't need to query the database too often for the value of the pre-maintenance flag.

There is no way for the client to start querying the database while maintenance is performed.

Cons

You should need to wait for several minutes (five in my example) before performing the maintenance task. This is OK if this is a scheduled task or something which can be planned, but would be a problem for maintenance tasks which should be performed as soon as possible, such as changes to configuration related to a newly discovered security issue or measures used to prevent an ongoing attack.

Two-ways connection

If it is necessarily to be able to perform maintenance tasks quickly without additional planning (see Cons above), you may need to implement a two-ways connection which makes it possible for you—the service provider—to push a message to your clients, notifying them that the service will be switched to maintenance mode.

Although this sort of communication becomes more and more popular for web applications with the support of Web sockets by all major browsers, if I understand well, you're dealing with desktop applications. If Java ecosystem doesn't have a similar technology for desktop apps (I'm pretty sure it does) or if for some reason you can't use it, you may be interested in legacy techniques used in web apps, such as long polling. You may read about those techniques on Comet page, especially how they affect bandwidth and what side effects they have.

Pros

You can perform maintenance task nearly immediately while knowing that the clients are informed that you have switched to maintenance mode.

Cons

Those techniques may be difficult to implement correctly (for example, how would you handle the case where you sent a message to a client, but the client neither haven't responded, nor dropped the connection?)


答:
There is a specific tool within Java EE for doing this.  Java Management Extensions aka JMX (Oracle best practices)

With, you would set up a 'managed bean' (aka MBean) which you can then either poll externally (useful for setting up notifications on performance, resources, problems) or used to poke data back into the application (go into maintain mode now).

Many servlet containers support JMX administration.  For example, Tomcat.  Its right there - no external dependencies of a database or caching or other approaches.  This is what it is meant for - the remote administration of an application (be it stand alone Java or a web application within some container).

The reason that I'm so high level with this is that it is a fairly large technology area of its own.  It is the right way to handle this type of operation, but this is also akin to saying "for a CRUD app use REST" - the problem domain is very large and a more specific answer can't really drill down too deeply.

Title: Toggling "maintenance mode" in Java EE app

Body:
I've been toying with the idea of feature toggles for various configuration/access purposes, but have been a little unsure of myself when it comes to toggling something like a downtime or maintenance mode, where users should be unable to access anything during that time.

It seems like such a simple thing to do implementation-wise, but I wanted to get some input from those more experienced than I on the subject.

My first thought was just checking a flag in a database (this is a clustered environment, so a file-based flag seems nonideal in our setup), but it feels wrong with the database hit on every request. So, I moved to do it similar to a timed in-memory cache, but that also doesn't feel right to me, as I wouldn't want the cache to be so short-lived that it has to update too frequently from the db, but I also don't want to turn on maintenance mode and potentially have users able to come in during the time before the update occurs. Should I maybe have a cache with a separate forced update trigger as mentioned here?

How have you handled toggling an app-wide maintenance or downtime mode?


问:
Use of underscores and earmuffs
I have seen underscores and earmuffs (like _good_ or *good*) frequently used in conversations among programmers. All I know is that underscores are used to denote private variables or methods in languages like C, Python etc. and  earmuffs are the preferred way of indicating that something is a global variable.

Do these have any other punctuational meaning when they are used outside code, in mailing lists etc? Does the highlighted word have any hidden meaning or expression added to it? Or, are they just like air quotes?

答:
They tend to be used for emphasis:

```No, that's *my* monkey.```

or

```No, that's _my_ monkey.```

Asterisks can also be used to denote an action the person is supposedly doing at the time of the comment:

```No, that's my monkey. *snatches monkey*```

Or a description to apply to the comment

```*firmly* No, that's my monkey.```

In the examples you give though think of them as bold or italic for text that doesn't support rich text.

ContentLicense: CC BY-SA 2.5

Title: Use of underscores and earmuffs

Body:
They tend to be used for emphasis:

No, that's *my* monkey.

or

No, that's _my_ monkey.

Asterisks can also be used to denote an action the person is supposedly doing at the time of the comment:

No, that's my monkey. *snatches monkey*

Or a description to apply to the comment

*firmly* No, that's my monkey.

In the examples you give though think of them as bold or italic for text that doesn't support rich text.


问:
Is reinventing the wheel really all that bad?
Its common knowledge in programming that reinventing the wheel is bad or evil.

But why is that?

I am not suggesting that it's good. I believe it to be wrong. However, I once read an article that said, if someone is doing something wrong (programming wise) explain to them why its wrong, if you can't, then maybe you should be asking yourself if it is really wrong.

That leads me to this question:

If I see someone is clearly reinventing the wheel by building their own method of something that is already built into the language/framework. First, for arguments sake, lets assume that their method is just as efficient as the built in method. Also the developer, aware of the built in method, prefers his own method.

Why should he use the built in one over his own?

答:
If people didn't reinvent wheels, the world would be filled with these..

Here is a dialogue from my workplace:

```<code>- I would like to add some colors to the output of this program.
- Oh, there is this library called Colorama ..
</code>```

There are two options: either reinvent the wheel OR use Colorama. Here is what each option would result into:

Using Colorama

Maybe a bit faster to get running

Adding a third party dependency for something trivial

You continue being as stupid as before using Colorama

Reinventing the wheel

You understand how some programs are able to show color

You learn that special characters can be used to color on any terminal

You are able to color in any programming language you might use in the future

Your project is less likely to break

As you see, it's all up to context. Reinventing the wheel is something I do very often because I want to be able and think for myself and not rely on other people thinking for me. If however you are running on a deadline or what you try to implement is huge and already exists, then you're better off using what is there.

Update (after a few years of experience)

I update my answer since after some years of working on multiple projects, my opinion has slightly changed. Also to answer some of the comments.

First of all with re-invent the wheel we mean essentially creating a library for a specific functionality, when there's already at least a 3rd party library that already does that.

When to re-invent the wheel

When you want to learn.

When you know exactly what you need, but an existing library becomes overly complex to give you that.

When the existing library overhead over-saturates the functionality you need. E.g. using NLP when you just need to count a few words in a document.

When the functionality you're looking for, is not overly complex to do from scratch (e.g. rebuilding a whole new OS kernel).

When cross-platform is not essential (if given by the 3rd library).

And ofcourse it all depends on context. Sometimes a 3rd party library might not be maintained properly, or maybe it makes more sense to just fork it and build on top.

Body:
If people didn't reinvent wheels, the world would be filled with these..

Here is a dialogue from my workplace:

- I would like to add some colors to the output of this program.
- Oh, there is this library called Colorama ..

There are two options: either reinvent the wheel OR use Colorama. Here is what each option would result into:

Using Colorama

Maybe a bit faster to get running

Adding a third party dependency for something trivial

You continue being as stupid as before using Colorama

Reinventing the wheel

You understand how some programs are able to show color

You learn that special characters can be used to color on any terminal

You are able to color in any programming language you might use in the future

Your project is less likely to break

As you see, it's all up to context. Reinventing the wheel is something I do very often because I want to be able and think for myself and not rely on other people thinking for me. If however you are running on a deadline or what you try to implement is huge and already exists, then you're better off using what is there.

Update (after a few years of experience)

I update my answer since after some years of working on multiple projects, my opinion has slightly changed. Also to answer some of the comments.

First of all with re-invent the wheel we mean essentially creating a library for a specific functionality, when there's already at least a 3rd party library that already does that.

When to re-invent the wheel

When you want to learn.

When you know exactly what you need, but an existing library becomes overly complex to give you that.

When the existing library overhead over-saturates the functionality you need. E.g. using NLP when you just need to count a few words in a document.

When the functionality you're looking for, is not overly complex to do from scratch (e.g. rebuilding a whole new OS kernel).

When cross-platform is not essential (if given by the 3rd library).

And ofcourse it all depends on context. Sometimes a 3rd party library might not be maintained properly, or maybe it makes more sense to just fork it and build on top.

ContentLicense: CC BY-SA 4.0


问:
Architecture for monitoring lots of mini-games
In my project, I'm going to create lots of mini games. Those mini games can last 30 seconds to 1 week (the duration of the game is decided when it is created and is fixed).

The number of concurrent games can be large (more than 1 game per user and we hope to have many users of course :-)).

Nothing really happens during the game. They are composed of just a winning condition and end time.

Once a game reaches its end, I need the server to check the winning condition (using a remote service) and finalize the game (decide who won and give points). During the time the game takes place, the server does not need to manage it.

You can think of the games as bets. Someone places a bet that something will happen in 30 seconds and after 30 seconds I need to check if that something really happened and update the system.

Latency of the game finalization should be very fast, especially for short games. Not more than a second or two. I.e, for a game of 30 seconds, I should be able to update if it won or lost a second or two after it ended.

I thought about persisting each game to a row in a sql table and schedule a task to run when it ends and finalize it.

For example, a user starts a game of 1 minute. The server will create the game in the db and schedule a task to run in 1 minute (and pass it the game's id). When the task triggers, it will finalize the game and update its status in the db.

Are there other ways to solve this problem? Maybe something that does not require having a task per game? I thought also about having 1 task that will check for all finished games but for short games (e.g. 30 secs) the polling time must be very short and I'm not sure how well it can scale.

答:
Since you mentioned scale-ability and speed is important to you, I would like to propose the Message Queue as the architecture for your application. You will need to research examples in your technology of choice. Here's a Wiki for the general description (http://en.wikipedia.org/wiki/Message_queue).

The idea is simple. Clients (your games) send messages(game results) to the Queue Manager. The Manager dispatches these messages to subroutines that process them. As your application grows you can add more managers to process messages. Therefore, scale horizontally. As you add more games (that most likely will have different result formats and subroutines to process those results...i.e chess, poker or Go) you can add more subroutines that can process those results. Here you scale vertically.


问:
Why convert to assembly language instead of machine code?
When I compile a C program, it is compiled first to assembly code, then assembled into machine code. I'm curious why it doesn't just convert straight to machine code in the first place.</p>

答: It's possible for compilers to generate code which is suitable for use as input to a processor (for direct execution), a linker, an assembler, or some other kind of building tool.  Which format is useful often depends upon what other code it will need to be combined with.  If a compiler for language X which generates output suitable for one brand of linker, and a compiler for language Y generates output suitable for a different one, it may be hard to use a mixture of code X and Y in the same project.  If a compiler for language X, however, generates assembly code instead of a linkable object file, it may be easier to use its output with the tool set associated with language Y.  Because assembly-language formats aren't completely standardized some massaging may be necessary, but massaging an assembly-language file may be easier than converting a binary object file.</p>


问:
Test Driven Development when the specifications change
These days, it's fairly common for me to be tasked to make a change that actually breaks the prior spec. I know that one of the major ideas behind TDD is to have a suite that verifies all your changes didn't actually break anything, but what do you do when the change you need to do, by it's very nature, actually must break the unit test?

How to do Test Driven Development is a similar, related question but it is subtlety different from my question.  In my case, I already have the test and the specification has changed.  In that question, the OP does not have the test yet.

答:
The advertised cycle of TDD is write tests until they fail then hack at the code until they pass again and then refactor while keeping all test succeeding.

When the spec changes you will need to remove the old tests that would verify a violation of the new spec and write new tests that will verify the new spec.


问:
How detailed should QA aim to be?
When QA discovers an issue during testing should they...

Log the bug with the test case and scenario information and move on testing?

Try investigating to determine why the bug is happening?

I would think it is normally both, but some people are saying it is not QA's job to even try to find root causes. QA is just meant to point out bugs. Interestingly this sentiment is coming from QA not developers.

Some background

Imagine this is happening in a very small team (3-4 people) working in an agile-ish environment. The developers are continually adding new features and fixes. Unit test coverage is very low. This is mostly due to the fact the developers have been pushed to deliver features first. QA is relied heavily on verifying correctness and functionality. Generally the developers are open to working closely with QA on things.

The QA team is testing in parallel. As soon as new builds show up they start their regression and new functional tests. However there are no automated tests. All tests are manual. Luckily most tests are quick to execute since they are simple comparisons against expected results.

Added

In the overarching organization, there are mostly two breeds of QA. There are the automation QA who do what their title suggests. And there are testers. Some float between the two from project to project, but for the purposes of this question, the QA are meant to be testing.

Perhaps to rephrase, what is a appropriate level of detail QA should be recording about bugs when they come across it (only the Devs will read these details)?

Is "Calculation X is wrong when using attached data file" sufficient?

Or, "Calculation X was off by a some constant. When we change value Y1 in the data file, the difference is equal to the value of Y1."

答:
Saying "QA" as a job role is akin to saying "health care professional."  You've got nurses, nurses assistants, physicians, physicians assistants, nurse practitioner,  surgeons, midwives, etc...

Trying to answer "how far QA should go" is much like trying to answer "what does a health care professional do?" - it depends on the job description the person was hired for and their capabilities.

I've seen QA be people that are just supposed to click and verify.  I've seen QA where its programmers who are writing automated tests.  I've seen QA where they have to put their stamp of approval on every release (or it doesn't go out).  I've seen QA that has tried to do as little as possible and not claim any responsibility for the end product.  I've seen QA where they take the A part of the title very seriously and are involved in the process of assurance of quality (making sure code reviews are done, getting deep into ISO 9000).

So what should your QA department do?  What they need to. This is completely dependent on what you need them to do and what they are hired to do and what their responsibilities are.  This is probably a discussion you should have with your management chain if it is necessary.  Many programers will find they have had similar discussions about if they should be gathering requirements, writing documentation, preforming tests themselves - the same needs to be done and said of QA.

For issues of how detailed a bug report QA should give, it is an issue of quality vs quantity.  If they are finding one bug a cycle and the test suite takes less than one cycle to do, then they have the opportunity (given that they have the appropriate skill set - because you're hiring good employees and paying competitively) they can then write a sufficiently detailed bug reports such that programmers can work from them more easily.

On the other hand, if they are expected to do 7 days of testing in 5, there is no way that they can do a complete root cause analysis (RCA) on the bugs they find - being under time pressure to finish the regression suite.

If QA is finding that each cycle there are sufficiently many bugs that while they do finish the tests, they can't do an RCA on each one, you will likely find the quality of the bug reports varying greatly.  This becomes especially prevalent when there are no automated tests that the coders run - each time a new build is being done QA is finding itself having to make sure that each bit of functionality is still the same.  Depending on the size of the application, this can mean lots of testing.

Related reading:

Qa Is Not Qc and Quality Assurance Is Not Quality Control

Software quality assurance

Minor rant - we work in an industry where people go for "code ninja" or "java wizard" as job title and such and then try to tell the "QA" people they should or shouldn't do something because of their job title? Read the job description.  Do what is needed. If the job description needs to change, change it. If something needs to be done, do it.

Title: How detailed should QA aim to be?


问:
Are all magic numbers created the same?
On a recent project, I needed to convert from bytes to <s>kilobytes</s> kibibyte. The code was straightforward enough:

var kBval = byteVal / 1024;
```

After writing that, I got the rest of the function working & moved on.

But later on, I started to wonder if I had just embedded a magic number within my code. Part of me says it was fine because the number is a fixed constant and should be readily understood. But another part of me thinks it would have been super clear if wrapped in a defined constant like BYTES_PER_KBYTE.

So are numbers that are well known constants really all that magical or not?

Related questions:

When is a number a magic number? and Is every number in the code considered a "magic number"? - are similar, but are much broader questions than what I'm asking.  My question is focused on well-known constant numbers which is not addressed in those questions.

Eliminating Magic Numbers: When is it time to say "No"? is also related, but is focused on refactoring as opposed to whether or not a constant number is a magic number.

答:
Other posters have mentioned that the conversion happening is 'obvious', but I disagree.  The original question, at this point in time, includes:

<s>kilobytes</s> kibibytes

So already I know the author is or was confused.  The Wikipedia page adds to the confusion:

```<code>1000 = KB kilobyte (metric)
1024 = kB kilobyte (JEDEC)
1024 = KiB kibibyte (IEC)
</code>```

So "Kilobyte" can be used to mean both a factor of 1000 and 1024, with the only difference in shorthand being the capitalization of the 'k'.  On top of that, 1024 can mean kilobyte(JEDEC) or kibibyte(IEC).  Why not shatter all of that confusion outright with a constant with a meaningful name?  BTW, this thread has used "BYTES_PER_KBYTE" frequently, and that's no less ambiguous.  KBYTE: is it KIBIBYTE or KILOBYTE?  I'd prefer to ignore JEDEC and have BYTES_PER_KILOBYTE = 1000 and BYTES_PER_KIBIBYTE = 1024.  No more confusion.

The reason why people like me, and many others out there, have 'militant' (to quote a commenter in here) opinions on naming magic numbers is all about documenting what you intend to do, and removing ambiguity. And you actually picked a unit that has lead to a lot of confusion.

If I see:

```<code>int BYTES_PER_KIBIBYTE = 1024;
...
var kibibytes = bytes / BYTES_PER_KIBIBYTE;
</code>```

Then it's immediately obvious what the author intended to do, and there's no ambiguity.  I can check the constant in a matter of seconds(even if it's in another file), so even though it's not 'instant', it's close enough to instant.

In the end, it might be obvious when you're writing it, but it'll be less obvious when you come back to it later, and it may be even less obvious when someone else edits it.  It takes 10 seconds to make a constant; it could take half an hour or more to debug an issue with units(the code isn't going to jump out at you and tell you the units are wrong, you're going to have to do the math yourself to figure that out, and you'll likely hunt down 10 different avenues before you check units).

Title: Are all magic numbers created the same?

Body:
Other posters have mentioned that the conversion happening is 'obvious', but I disagree.  The original question, at this point in time, includes:

<s>kilobytes</s> kibibytes

So already I know the author is or was confused.  The Wikipedia page adds to the confusion:

1000 = KB kilobyte (metric)
1024 = kB kilobyte (JEDEC)
1024 = KiB kibibyte (IEC)

So "Kilobyte" can be used to mean both a factor of 1000 and 1024, with the only difference in shorthand being the capitalization of the 'k'.  On top of that, 1024 can mean kilobyte(JEDEC) or kibibyte(IEC).  Why not shatter all of that confusion outright with a constant with a meaningful name?  BTW, this thread has used "BYTES_PER_KBYTE" frequently, and that's no less ambiguous.  KBYTE: is it KIBIBYTE or KILOBYTE?  I'd prefer to ignore JEDEC and have BYTES_PER_KILOBYTE = 1000 and BYTES_PER_KIBIBYTE = 1024.  No more confusion.

The reason why people like me, and many others out there, have 'militant' (to quote a commenter in here) opinions on naming magic numbers is all about documenting what you intend to do, and removing ambiguity. And you actually picked a unit that has lead to a lot of confusion.

If I see:

int BYTES_PER_KIBIBYTE = 1024;
...
var kibibytes = bytes / BYTES_PER_KIBIBYTE;

Then it's immediately obvious what the author intended to do, and there's no ambiguity.  I can check the constant in a matter of seconds(even if it's in another file), so even though it's not 'instant', it's close enough to instant.

In the end, it might be obvious when you're writing it, but it'll be less obvious when you come back to it later, and it may be even less obvious when someone else edits it.  It takes 10 seconds to make a constant; it could take half an hour or more to debug an issue with units(the code isn't going to jump out at you and tell you the units are wrong, you're going to have to do the math yourself to figure that out, and you'll likely hunt down 10 different avenues before you check units).


问:
Why aren't there other programming languages that compile to Python bytecode?
In Java, there are multiple languages that compile to Java bytecode and can run on the JVM -- Clojure, Groovy, and Scala being the main ones I can remember off the top of my head.

However, Python also turns into bytecode (.pyc files) before being run by the Python interpreter.  I might just be ignorant, but why aren't there any other programming languages that compile to python bytecode?

Is it just because nobody bothered to, or is there some kind of inherent restriction or barrier in place that makes doing so difficult?

答:
The other answers make a lot of sense, but there are actually languages now that compile to Python. Where there's a will...

I don't know anything about these languages, but they seem to work by transpiling their source code to Python ASTs and letting Python compile the trees to bytecode, avoiding the problems mentioned in other answers.

Based on the comments, we currently know of three alternative languages that use the Python VM (feel free to add any others here):

Mochi Describes itself as a dynamically typed programming language for functional programming and actor-style programming.

Hy: Describes itself as a dialect of Lisp that's embedded in Python.

dg: Describes itself as a (technically) simple language that compiles to CPython bytecode.


答:
In terms of "how deep should QA go", it really depends on the people involved and what kind of teams you have, as well as the project code base. At my current job, we don't really like the term QA any more because we feel that we are all responsible for ensuring quality. Our developers practice TDD and unit test as much of the code base as we can (as well as integration and pretty much any other automated tests we can build). We then have a small team of testers, these guys have been using the products pretty much since day one and know it inside and out (from a user perspective).

When they find a bug that slipped through the automatic testing they report it to us with as much detail as they can muster from a user view. They effectively attempt to pin down the issue as fast as possible into recreation steps and these steps then become the acceptance criteria. Ie: "if x y and z no longer cause j and k to happen, then the bug is resolved."

In terms of who's job is it to find the root cause, we say whoever finds the bug should attempt to give as much detail as possible to whoever works on a fix. This resolves the issue as fast as possible and knowing the extent of the issue up front will only help to speed this process up.

If your testers who act from a user view won't attempt to recreate the bug in a number of different ways to gain more information to resolve things quicker, then I'd say you've got an unhealthy separation of teams going on. Development and QA go hand in hand, friendly rivalry is great and beneficial but when one team starts getting awkward about things then you have a problem. Phrases like "well the acceptance criteria is technically met" or "I found the bug, it's not my job to suggest a fix", I think you need to be looking at how your teams are working together to achieve high quality rather than questioning who should be doing what specifically.

“I expect QA will find nothing [...] QA ought to wonder why they exist” - Uncle Bob.

Title: How detailed should QA aim to be?

Body:
When QA discovers an issue during testing should they...

Log the bug with the test case and scenario information and move on testing?

Try investigating to determine why the bug is happening?

I would think it is normally both, but some people are saying it is not QA's job to even try to find root causes. QA is just meant to point out bugs. Interestingly this sentiment is coming from QA not developers.

Some background

Imagine this is happening in a very small team (3-4 people) working in an agile-ish environment. The developers are continually adding new features and fixes. Unit test coverage is very low. This is mostly due to the fact the developers have been pushed to deliver features first. QA is relied heavily on verifying correctness and functionality. Generally the developers are open to working closely with QA on things.

The QA team is testing in parallel. As soon as new builds show up they start their regression and new functional tests. However there are no automated tests. All tests are manual. Luckily most tests are quick to execute since they are simple comparisons against expected results.

Added

In the overarching organization, there are mostly two breeds of QA. There are the automation QA who do what their title suggests. And there are testers. Some float between the two from project to project, but for the purposes of this question, the QA are meant to be testing.

Perhaps to rephrase, what is a appropriate level of detail QA should be recording about bugs when they come across it (only the Devs will read these details)?

Is "Calculation X is wrong when using attached data file" sufficient?

Or, "Calculation X was off by a some constant. When we change value Y1 in the data file, the difference is equal to the value of Y1."


问:
Strongly typed views, ViewModel and MVC
I'm learning MVC and the book I'm using favors strongly-typed views. Some of these concepts are new to me and I'm trying to better understand the design philosophy. Strongly typed views typically result in having to create a ViewModel since the domain entities are unlikely to have all the information a particular view may need. The problem is now I need to update my ViewModel whenever the associated domain entities or view changes. I also don't see the ViewModel being much use when porting an application to another platform, say web to desktop, or web to mobile.

Wouldn't it be better if we could declare our objects, collections, etc and populate them in the controller and then be able to access these variables directly in the view?

The controller would look something like:

```<code>    public ActionResult Index() {
        User user = db.Users.Where(u => u.UserId = userId);
        List<Car> carList = db.Cars.Where(c => c.OwnerId = userId).ToList();
        return View("UserCars");
    }
</code>```

The view would look like:

```<code><!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width" />
    <title>Title</title>
</head>
<body>
    <div id="username">
        @{user.UserName;}
    </div>
    <div id="carlist">
        @foreach (var car in carList) {
            car.Name
            <br />
        }
    </div>
</body>
</html>
</code>```

What is wrong with this approach and what makes strongly typed views, and by extension, ViewModels better?

Edit: Strongly typed view explanation: http://www.c-sharpcorner.com/UploadFile/abhikumarvatsa/strongly-typed-views-in-mvc/

Edit: I know that code won't work, it's there to illustrate my question.

My concern is keeping a model, viewmodel and view in sync. If the view changes, I have to change the viewmodel. If I change a bool to a nullable bool in the model, I have to change that in any viewmodels that reference that model. If I have a separate viewmodel for every view and my web site is reasonably large, that's a lot of places to update. I don't just change things in one place, I change them in 50 places, or 100 places. I thought good programming practices try to avoid this sort of issue.

答:
I also prefer strongly-typed views with dedicated ViewModels.

To get around the problem you are seeing I sometimes construct my ViewModels like:

```<code>public class UserCarsViewModel
{
    public User User { get; set; }
    public List<Car> Cars { get; set; }

    public bool SomeCalculationIDontWantToHaveToDoInRazor
    {
        get { return _some mildly complex code here_ }
    }

    public List<Car> SomeFilteringIDontWantToHaveToDoInRazor
    {
        get { return _some mildly complex code here_ }
    }

    public bool SomeOtherConditionIDontWantToHaveToDoInRazor
    {
        get { return _some mildly complex code here_ }
    }
}
</code>```

All of the more-than-dead-simple processing that would otherwise have to be in the view code (Razor) would be placed in the ViewModel, where is can be more easily tested. The Razor code is, in turn, much easier to read.

But, my Models are also in there, ready to be used as-is. If the Models change, but the changes don't affect my ViewModel calculations, my ViewModels don't need to change.

My controller ends up looking like:

```<code>public ActionResult Index()
{
    var viewModel = new UserCarsViewModel();
    viewModel.User = db.Users.Where(u => u.UserId = userId);
    viewModel.Cars = db.Cars.Where(c => c.OwnerId = userId).ToList();
    return View("UserCars", viewModel);
}
</code>```

Title: Strongly typed views, ViewModel and MVC

Body:
I also prefer strongly-typed views with dedicated ViewModels.

To get around the problem you are seeing I sometimes construct my ViewModels like:

public class UserCarsViewModel
{
    public User User { get; set; }
    public List<Car> Cars { get; set; }

    public bool SomeCalculationIDontWantToHaveToDoInRazor
    {
        get { return _some mildly complex code here_ }
    }

    public List<Car> SomeFilteringIDontWantToHaveToDoInRazor
    {
        get { return _some mildly complex code here_ }
    }

    public bool SomeOtherConditionIDontWantToHaveToDoInRazor
    {
        get { return _some mildly complex code here_ }
    }
}

All of the more-than-dead-simple processing that would otherwise have to be in the view code (Razor) would be placed in the ViewModel, where is can be more easily tested. The Razor code is, in turn, much easier to read.

But, my Models are also in there, ready to be used as-is. If the Models change, but the changes don't affect my ViewModel calculations, my ViewModels don't need to change.

My controller ends up looking like:

public ActionResult Index()
{
    var viewModel = new UserCarsViewModel();
    viewModel.User = db.Users.Where(u => u.UserId = userId);
    viewModel.Cars = db.Cars.Where(c => c.OwnerId = userId).ToList();
    return View("UserCars", viewModel);
}


问:
What is the hierarchical DB schema for this xml?
As per this link, this is the xml that maintains "BookStore" data.

I see that number of occurences of author are varying for any book.

xml's hierarchical representation of BookStore looks more meaningful/intuitive for me to visualise the data unlike relational table tuples. It is rather more difficult(kind of tweek) to map such hierarchical information in tables.

It is strange to see that real world(hierarchical) data(for example, SNMP MIBs of network devices or BookStore etc...) are mapped to tables of records in many commercial softwares. For me, it is not-intuitive(rather unnecessary skill) to convert such hierarchical data to tables, despite relational DB(like MSSQL) support massive/safe/multi-user/convenient/efficient/reliable unlike  xml file format.

So, we are trying to fit hierarchical ordered data into table tuples which is an overhead. Do we have any commercial database that companies use to maintain schema in hierarchical fashion?

Note: am currently part of database course.

答:
Some (much?) data, such as the book example, isn't inherently hierarchical. That the bookstore example uses hierarchical storage is a consequence of XML's tree structure, not the inherent structure of the data. Consider that a book can have many authors, and an author can write many books, which means neither can strictly belong to the other. XML gets around this by using identity attributes by which one node can reference another; the same technique is used in other serializations of circular data structures.

A full relational model can deal with truly hierarchical, homological data using closure properties; specifically, the transitive closure allows tree paths to be retrieved using a parent-child relationship. The real problem is that SQL and most production RDBMSs don't support closure properties in general. Transitive closures are available in SQL with Common Table Expressions/the WITH RECURSIVE clause but are relatively new in implementation and don't seem to get used as much (and aren't supported by all RDBMSs). More typically, you see the full path stored in the table (Farey Fractions can be considered paths using their decimal expansions and special markers for repeated trailing digits, similar to quote notation).

Another data model you used to see often was the network model, where nodes are datums and edges are relationships. In the book model, there'd be an edge from each author node to each book node for a book that the author wrote. The database reports & specifications created by CODASYL used a network model. There are various issues with using a network model that the relational model specifically addresses; Codd's seminal paper has more.

If the relational model seems non-intuitive, perhaps it's because you have yet to grok the relational model. Don't think of it as tables and rows (which are more what you find in a spreadsheet) but as declarative relationships:

```
Hector Garcia-Molina wrote "A First Course in Database Systems"
Jeffrey Ullman wrote "A First Course in Database Systems"
"Database Systems: The Complete Book" is a book with ISBN '0-13-815504-6' and price $85
...
```

From there, you write the statements using predicates:

```
Wrote(Hector Garcia-Molina, "A First Course in Database Systems")
Wrote(Jeffrey Ullman, "A First Course in Database Systems")
Book("Database Systems: The Complete Book", ISBN:0-13-815504-6, price:$85)
...
```

Note that the Book example isn't a simple predicate because two of the datums are tagged with names; this is part of what distinguishes a relationship from predicates, relations and other similar mathematical objects. Predicates define sets, so you can use set operations to define new relationships. This brief overview is very informal and imprecise, but should give you a starting point.

Body:
Some (much?) data, such as the book example, isn't inherently hierarchical. That the bookstore example uses hierarchical storage is a consequence of XML's tree structure, not the inherent structure of the data. Consider that a book can have many authors, and an author can write many books, which means neither can strictly belong to the other. XML gets around this by using identity attributes by which one node can reference another; the same technique is used in other serializations of circular data structures.

A full relational model can deal with truly hierarchical, homological data using closure properties; specifically, the transitive closure allows tree paths to be retrieved using a parent-child relationship. The real problem is that SQL and most production RDBMSs don't support closure properties in general. Transitive closures are available in SQL with Common Table Expressions/the WITH RECURSIVE clause but are relatively new in implementation and don't seem to get used as much (and aren't supported by all RDBMSs). More typically, you see the full path stored in the table (Farey Fractions can be considered paths using their decimal expansions and special markers for repeated trailing digits, similar to quote notation).

Another data model you used to see often was the network model, where nodes are datums and edges are relationships. In the book model, there'd be an edge from each author node to each book node for a book that the author wrote. The database reports & specifications created by CODASYL used a network model. There are various issues with using a network model that the relational model specifically addresses; Codd's seminal paper has more.

If the relational model seems non-intuitive, perhaps it's because you have yet to grok the relational model. Don't think of it as tables and rows (which are more what you find in a spreadsheet) but as declarative relationships:

Hector Garcia-Molina wrote "A First Course in Database Systems"
Jeffrey Ullman wrote "A First Course in Database Systems"
"Database Systems: The Complete Book" is a book with ISBN '0-13-815504-6' and price $85
...

From there, you write the statements using predicates:

Wrote(Hector Garcia-Molina, "A First Course in Database Systems")
Wrote(Jeffrey Ullman, "A First Course in Database Systems")
Book("Database Systems: The Complete Book", ISBN:0-13-815504-6, price:$85)
...

Note that the Book example isn't a simple predicate because two of the datums are tagged with names; this is part of what distinguishes a relationship from predicates, relations and other similar mathematical objects. Predicates define sets, so you can use set operations to define new relationships. This brief overview is very informal and imprecise, but should give you a starting point.


问:
Algorithm to test file equivalence
I'm writing a program that needs to test if a file is equivalent to one or more other files. To accomplish this, every time we see a new file we stat the file and get the size.

We use the size as a key and if we've never seen the size before than we know that's a new file.

If we have seen that size before we were planning to md5 the first 4k and last 4k of the target file and check to see if that hash has been seen in list associated with the size key.

I don't want to hash the whole file since these files can be rather large (largest I've seen so far is 90G).

The goal is to avoid spending a lot of time on redundant file comparisons. Will this algorithm work, and can it be improved?

More details on my particular problem: I'm attempting to deduplicate a decently sized set of data ( 2Pb ) that contains a large amount of time stamped files (about 40%) created from syslog on FreeBSD machines. Before I start chewing through that many files line by line I wanted to make sure that the file I was looking at hadn't been seen before.

答:
Elaborating a bit on Snowman's answer, I think I would go for a hierarchy of hash values over (exponentially) increasing subsets of the file, computed on-demand whenever collisions occur, and memorized in a suitable data structure (hash table, but even a simple prefix tree would do) for quick future access. This should ensure quick failure in case of 'almost identity', and retain the worst-case complexity (up to a log factor) and achieve good average-case complexity.

It would go as follows in pseudo-Python, taking as input a file f, a set of files D and a dictionary H (again, one could do better here, but it should not matter too much) acting as a cache for previously computed hash values:

```<code>collisions = [fc for fc in D if size(fc)==size(f)]
size_hash = 4*1024
while (len(collisions) > 0) and (size_hash<=size(f)):
  H[(f,size_hash)] = md5(f,size_hash)
  for fc in collisions:
    if (fc,size_hash) not in H:
      H[(fc,size_hash)] = md5(fc,size_hash)
  collisions = [fc for fc in collisions if H[(fc,size_hash)]==H[(f,size_hash)]]
  size_hash *= 2
for fc in collisions:
  # Painstakingly read and compare content to that of f...
</code>```

Worst-case complexities: In the worst case, all of the files have equal length n, and have different yet MD5 identical (unlucky!) contents, so one ends up computing MD5 hashes for chunks of size 4k, 8k, 16k... n in each of the files, only to read them in full afterward.

In term of time, the first 4k of each file are read to compute the first hash, then the first 8k for the second, 16k for the third ... then the full size n. Computing an MD5 can be done in linear time, so the total time-consumption is (up to a constant) 4k+8k+16k+...+n < 2n operations, i.e. it remains in the order of magnitude of the final (unavoidable) comparison of the files.

In term of memory, log(n) MD5 hash values (one for 4k, one for 8k...), each of constant size will be stored, so the overhead should be reasonable.

Average-case complexities: I won't detail the analysis (maths are not permitted by the markdown system here anyway :) ), but even assuming a large number of files having equal size, the expected number of computed hash values should be constant on average, so this algorithm will neither read significant chunks of the files, nor clotter the memory.

Title: Algorithm to test file equivalence

Body:
Elaborating a bit on Snowman's answer, I think I would go for a hierarchy of hash values over (exponentially) increasing subsets of the file, computed on-demand whenever collisions occur, and memorized in a suitable data structure (hash table, but even a simple prefix tree would do) for quick future access. This should ensure quick failure in case of 'almost identity', and retain the worst-case complexity (up to a log factor) and achieve good average-case complexity.

It would go as follows in pseudo-Python, taking as input a file f, a set of files D and a dictionary H (again, one could do better here, but it should not matter too much) acting as a cache for previously computed hash values:

collisions = [fc for fc in D if size(fc)==size(f)]
size_hash = 4*1024
while (len(collisions) > 0) and (size_hash<=size(f)):
  H[(f,size_hash)] = md5(f,size_hash)
  for fc in collisions:
    if (fc,size_hash) not in H:
      H[(fc,size_hash)] = md5(fc,size_hash)
  collisions = [fc for fc in collisions if H[(fc,size_hash)]==H[(f,size_hash)]]
  size_hash *= 2
for fc in collisions:
  # Painstakingly read and compare content to that of f...

Worst-case complexities: In the worst case, all of the files have equal length n, and have different yet MD5 identical (unlucky!) contents, so one ends up computing MD5 hashes for chunks of size 4k, 8k, 16k... n in each of the files, only to read them in full afterward.

In term of time, the first 4k of each file are read to compute the first hash, then the first 8k for the second, 16k for the third ... then the full size n. Computing an MD5 can be done in linear time, so the total time-consumption is (up to a constant) 4k+8k+16k+...+n < 2n operations, i.e. it remains in the order of magnitude of the final (unavoidable) comparison of the files.

In term of memory, log(n) MD5 hash values (one for 4k, one for 8k...), each of constant size will be stored, so the overhead should be reasonable.

Average-case complexities: I won't detail the analysis (maths are not permitted by the markdown system here anyway :) ), but even assuming a large number of files having equal size, the expected number of computed hash values should be constant on average, so this algorithm will neither read significant chunks of the files, nor clotter the memory.


答:
Another reason is that the JVM is highly optimized, well-evolved, and extremely complete ecosystem. On it's own, it competes extremely well with any of the other compiled languages. (I won't say that it's the best general purpose VM out there, but I've certainly banked my career on that.) So getting access to the JVM, short of writing bytecode, is desirable in itself.

However, the Python VM is good, but (nothing against Python) has some serious shortcomings. The Python runtime environment suits the dynamic nature of the language well, but can really surprise you when you get familiar with its memory usage, global locking, or threading model.

In head-to-head comparisons, the JVM is typically twice as fast as the Python VM. The JVM (surprizingly) even competes well with natively compiled code, based on the "hot" optimizations it performs. And that's not even counting the more sophisticated thread handling, etc.

I love Python, I really do, and hate to say it, but sometimes the performance just kicks me in the teeth -- otherwise, why would critical Python libraries like numpy or scipy have to fall back into C code?

In other words, people who gravitate to Python do so because they like the language. But if you want to write a brand new language to suit your preferences, you're much better off compiling to the JVM, because your new idiosyncratic language will start off in one of the best (subjectively, maybe the best) operating environments available.

Title: Why aren't there other programming languages that compile to Python bytecode?

Body:
In Java, there are multiple languages that compile to Java bytecode and can run on the JVM -- Clojure, Groovy, and Scala being the main ones I can remember off the top of my head.

However, Python also turns into bytecode (.pyc files) before being run by the Python interpreter.  I might just be ignorant, but why aren't there any other programming languages that compile to python bytecode?

Is it just because nobody bothered to, or is there some kind of inherent restriction or barrier in place that makes doing so difficult?


问:
How early does a hashable type need to become immutable?
Suppose I have a class that looks vaguely like this:

```<code>class Foo(object):
    _value = None  # XXX: see below

    def __init__(self, value):
        self._value = value

    @property
    def value(self):
        """The value of this Foo."""
        return self._value

    def __hash__(self):
        return hash(self.value)

    def __eq__(self, other):
        return type(self) is type(other) and self.value == other.value
</code>```

(I know the _value = None line is wrong; in my actual code, _value is a data descriptor with a default value.  It never raises AttributeError even if it hasn't been set yet, and I'm just trying to emulate that behavior in as few lines of code as possible.  I could rip the behavior out, but it would significantly complicate things in other areas of my codebase.)

Have I violated the rules?  Should I be setting _value in Foo.__new__() instead?  Clearly it needs to be set somewhere; there will be a point in time at which the hash() of our instance changes.  When should that happen, exactly?

If there is not anything wrong with the above code, then why is the following code wrong?  Logically, this combination is illegal, but I'm not sure which class has broken the rules:

```<code>bar = set()

class Baz(object):
    def __init__(self, *args, **kwargs):
        bar.add(self)
        super(Baz, self).__init__(*args, **kwargs)  # play nice with MI

class Qux(Baz, Foo):
    pass

assert Qux('quuux') in bar  # assertion fires because the hash() changed
</code>```

答:
The instance should be immutable at the moment it is used as a key in a dict. Before that, it can mutate. But preferably it should be incapable of mutation after the moment of construction, just to avoid programming errors.

The builder pattern fits this well: the final .build() step returns an immutable object. You can additionally prevent inadvertent attribute modification by setting instance's __setattr__ to something raising an exception.

Unfortunately, you did not provide a minimal runnable example of the problem in the second code fragment. Next time please do. I only can suspect that somewhere in the real problematic code you conflate self of the original object with something else. (The idea of updating a global hash table from a constructor does not strike me as a particularly elegant one, though memoizing decorators usually do this reliably.)


问:
Log to file or to database table?
I'm developing a web application which uses MS SQL for various data: including users, user accounts, user licenses, license prices, invoices.

I need to log users' real-time usage of the system, and use that for monthly billing: e.g. log whenever a user gets a specific page/URL and bill users at the end of the month based on the number of pages they fetched.

Should I write these log events into a table in my MS SQL database ?

Should I write these log events into a non-SQL append-only log file?

Should I write these log events into a different log file for each user?

This is not an especially high-volume web site: for example, a maximum of 10,000 users each doing an average of 5 loggable events/day => 50,000 events / day = 30 events / minute = 18,000,000 events / year.

I'm asking because either option seems viable and I don't see whether one has a clear advantage.

The data associated with a billable event is simple, e.g.:

User ID (foreign key relationship to the Users table in SQL)

Date and time

URL of the billable page

My own answer to this question is as follows:

Some benefits of writing the log to a database table:

Relational integrity: e.g. logged events are associated with valid user IDs (by defining the user ID as a foreign key between the tables)

Easy to read for billing: e.g. SELECT COUNT GROUP BY to get a count of the number of log events per user

Some benefits of writing to log file:

Easier performance: SQL is used less often e.g. only for user log-in events, and mostly only used for reading

Easier management: easier to archive old data e.g. at year end, by moving old log files instead of by deleting/archiving from the database

Please let me know if my answer is wrong; or exaggerates the importance of something; or has forgotten some important consideration.

And/or please let me know what your answer is, if it's different from mine.

答:
Since you are using this information for billing purposes, I do not see why you would not want it in the database where it can be easily queried, aggregated, reported on, and joined to other data.

I also think it's much easier to maintain a single database table containing the log information than a bunch of separate log files.  Same with your concern about the load on the server - there are much better ways of dealing with that than resorting to keeping data in flat files.

Your third option, by the way, is to do both.  Use the database for most needs, but have the log file for auditing purposes.

Title: Log to file or to database table?


问:
Maintenance cost of SIMD programming code base

Question:

The software industry's consensus is that clean and simple code is fundamental to the long-term viability of the code base and the organization that owns it. These properties lead to lower maintenance costs and increased likelihood of the code base being continued.

However, SIMD code is different than general application code, and I would like to know if there is a similar consensus regarding clean and simple code applying specifically to SIMD code.

Background to my question.

I write plenty of SIMD (single-instruction, multiple data) code for various image processing and analysis tasks. Recently I also had to port a small number of these functions from one architecture (SSE2) to another (ARM NEON).

The code is written for shrink-wrapped software, therefore it cannot depend on proprietary languages without unrestricted redistribution rights such as MATLAB.

An example of typical code structure:

Using OpenCV's matrix type (Mat) for all memory, buffer and lifetime management.

After checking the size (dimensions) of input arguments, the pointers to the start address of each row of pixels is taken.

The pixel count, and the start addresses of each row of pixels from each input matrix is passed into some low-level C++ functions.

These low-level C++ functions use SIMD intrinsics (for Intel Architecture, and ARM NEON), loading from and saving to raw pointer addresses.

Characteristics of these low-level C++ functions:

Exclusively one-dimensional (consecutive in memory)

Does not deal with memory allocations.
(Every allocation, including temporaries, are handled by the outer code using OpenCV facilities.)

The range of name lengths of symbols (intrinsics, variable names, etc) are roughly 10 - 20 characters, which is quite excessive.
(Reads like techno-babble.)

Reuse of SIMD variables is discouraged because compilers are quite buggy in correctly parsing code that is not written in the "single-assignment" coding style.
(I've filed several compiler bug reports.)

What aspects of SIMD programming would cause the discussion to differ from the general case?  Or, why is SIMD different?

In terms of initial development cost

It is well-known that the initial development cost of C++ SIMD code with good performance is about 10x - 100x (with a wide margin) compared to casually-written C++ code.

As noted in the answers to Choosing between performance vs readable/cleaner code? , most code (including casually-written code and SIMD code) is initially neither clean nor fast.

Evolutionary improvements in code performance (in both scalar and SIMD code) is discouraged (because it is seen as a kind of software rework), and the cost and benefit is not tracked.

In terms of propensity

(e.g. the Pareto principle, a.k.a. the 80-20 rule)

Even if image processing only comprises 20% of a software system (in both code size and functionality), image processing is comparatively slow (when viewed as a percentage of CPU time spent), taking more than 80% of time.

This is due to the data size effect: A typical image size is measured in megabytes, whereas the typical size of non-image data is measured in kilobytes.

Within the image processing code, a SIMD programmer is trained to automatically recognize the 20% code comprising the hotspots by identifying the loop structure in the C++ code. Thus, from a SIMD programmer's perspective, 100% of the "code that matters" is performance bottleneck.

Often in an image processing system, multiple hotspots exist and take up comparable proportions of time. For example, there may be 5 hotspots each taking up (20%, 18%, 16%, 14%, 12%) of total time. To achieve a high performance gain, all of the hotspots need to be rewritten in SIMD.

This is summarized as the balloon-popping rule: a balloon cannot be popped twice.

Suppose there are some balloons, say 5 of them. The only way to decimate them is to pop them one by one.

Once the first balloon is popped, the remaining 4 balloons now comprises a higher percentage of total execution time.

To make further gains, one must then pop another balloon.
(This is in defiance to the 80-20 rule of optimization: a good economical outcome can be achieved after the 20% of lowest-hanging fruits has been picked.)

In terms of readability and maintenance

SIMD code is patently hard to read.

This is true even if one follows every software engineering best-practice e.g. naming, encapsulation, const-correctness (and making side-effects obvious), function decomposition, etc.

This is true even for experienced SIMD programmers.

Optimal SIMD code is very contorted, (see remark) compared to its equivalent C++ prototype code.

There are many ways to contort SIMD code, but only 1 out of 10 such attempts will attain acceptably fast results.

(That is, in the tunes of 4x-10x performance gains in order to justify to high development cost. Even higher gains have been observed in practice.)

(Remark)

This is the main thesis of the MIT Halide project - quoting the paper's title verbatim:

"decoupling algorithms from schedules for easy optimization of image processing pipelines"

In terms of forward applicability

SIMD code is strictly tied to a single architecture. Each new architecture (or each widening of SIMD registers) requires a rewrite.

Unlike the majority of software development, each piece of SIMD code is typically written for a single purpose that never changes.
(With the exception of porting to other architectures.)

Some architectures maintain perfect backward compatibility (Intel); some fall short by a trivial amount (ARM AArch64, replacing vtbl with vtblq) but which is sufficient to cause some code to fail to compile.

In terms of skills and training

It is not clear what knowledge prerequisites are required to properly train a new programmer to write and maintain SIMD code.

College graduates who have learned SIMD programming in school seems to despise and dismiss it as an impractical career track.

Disassembly-reading and low-level performance profiling are cited as two fundamental skills for writing high-performance SIMD code. However, it is unclear how to systematically train programmers in these two skills.

Modern CPU architecture (which diverges significantly from what is taught in textbooks) makes training even more difficult.

In terms of correctness and defect-related costs

A single SIMD processing function is actually cohesive enough that one can establish correctness by:

Applying formal methods (with pen-and-paper), and

Verifying output integer ranges (with prototype code and performed outside run-time).

The verification process is, however, very costly (spends 100% time on code review and 100% time on prototype model checking), which triples the already-expensive development cost of SIMD code.

If a bug somehow manages to slip through this verification process, it is nearly impossible to "repair" (fix) except to replace (rewrite) the suspected defective function.

SIMD code suffers from the blunt of defects in the C++ compiler (optimizing code generator).

SIMD code generated using C++ expression templates also suffers greatly from defects of the compiler.

In terms of disruptive innovations

Many solutions have been proposed from academia, but few are seeing widespread commercial usage.

MIT Halide

Stanford Darkroom

NT2 (Numerical Template Toolbox) and the related Boost.SIMD

Libraries with widespread commercial usage do not seem to be heavily SIMD-enabled.

Open-source libraries seem lukewarm to SIMD.

Recently I have this first-hand observation of this after profiling a large number of OpenCV API functions, as of version 2.4.9.

Many other image processing libraries I have profiled also do not make heavy use of SIMD, or they miss the true hotspots.

Commercial libraries seem to avoid SIMD altogether.

In a few cases, I have even seen image processing libraries reverting SIMD-optimized code in an earlier version to non-SIMD code in a later version, resulting in severe performance regressions.
(The vendor's response is that it was necessary to avoid compiler bugs.)

This Programmer's question:  Does low latency code sometimes have to be "ugly"?  is related, and I previously wrote an answer to that question to explain my view points a few years ago.

However, that answer is pretty much "appeasement" to the "premature optimization" viewpoint, i.e. to the viewpoint that:

All optimizations are premature by definition (or, short-term by nature), and

The only optimization that has long-term benefit is toward simplicity.

But such viewpoints are contested in this ACM article.

All of that leads me to ask:

SIMD code is different than general application code, and I would like to know if there is a similar industry consensus regarding the value of clean and simple code for SIMD code.

答:
I've done assembly programming in the past, not SIMD programming recently.

Have you consider using a SIMD-aware compiler like Intel's? Is A Guide to
Vectorization with
Intel® C++ Compilers interesting?

Several of your comments like "balloon-popping" suggest using a compiler (to get benefits throughout if you don't have a single hot-spot).


问:
How to get out of the support rut and start repaying the technical debt!
I have a "friend". Yes good start I know but honestly this isn't me!

Basically he's been working on a successful project for about 4 years now, the difficulty is the technical debt has caught up and he's finding it almost impossible to stop supporting the product (tweaking this and that) and actually move on with real development.

I've made various suggestions, log all of your time, create tickets, don't answer emails etc etc. The problem with this is that it seems to only serve as a reminder that he's not getting anything "useful" done.

The technical debt has largely occurred because in the first instance it was a big benefit to the product, to take requests and phone calls from users and just quickly implement them.

What I would like to know is does anybody have any suggestions for how he might get out of this rut, a large part of which would be changing the perceptions of the users so that they don't think they can just ring and expect something to be done then and there.

It's all very well saying plan better, though I understand it's very difficult to plan actual development given the requirement for support and the relative pressures from users (see above).

答:
Your friend's organization desperately needs someone to do change management. This person or group would take change requests and bug fixes and prioritize them according to business impact and amount of effort required. That way, tasks that are more important for the organization as a whole would be done first, as opposed to tasks that are more important for whoever is bothering your friend at the moment.

EDIT: As an example of how this would work, most organizations have a severity scale. The highest severity level is a business-critical application or function that doesn't work. If there's something the business can do to work around the problem, that lowers the severity to the next level. If the application isn't business-critical, that makes the severity even lower. Requests for new enhancements are normally prioritized separately.


答:
TDD "by the book" has a specific cycle, and specficially for changing specs, this cycle should ideally look like this:

write one test according to the new spec (-> "red")

change the SUT ("subject under test") to match the new test; depending on the change, this
might break some old tests (-> the new test goes "green", but the old tests become "red")

Change the old tests if they can be modified to test the SUT according to the new spec. Or, if that is not possible, delete them (-> "green")

repeat from step 1 until you have enough tests for validating the whole spec, and the SUT implements it

Of course, in a real world scenario you will try to avoid situations where adding one test in step 1 will lead to the deletion of 100 existing tests in step 3. If that is the case, the tests show up that your original design was not evolvable enough, or your tests were not DRY enough, or the spec change was so big that you actually had to drop most of the original SUT and write a new one.

Title: Test Driven Development when the specifications change

Body:
These days, it's fairly common for me to be tasked to make a change that actually breaks the prior spec. I know that one of the major ideas behind TDD is to have a suite that verifies all your changes didn't actually break anything, but what do you do when the change you need to do, by it's very nature, actually must break the unit test?

How to do Test Driven Development is a similar, related question but it is subtlety different from my question.  In my case, I already have the test and the specification has changed.  In that question, the OP does not have the test yet.


问:
UML: Queue processor in a sequence diagram
Which is the appropriate way of representing a queue processor in a UML sequence diagram?

I want to represent in the same diagram two systems that are loosely coupled through a queue. The first system, performs some actions and insert an action in a queue, the other system is always listening and when the event appears in the queue, it dequeues the event and processes it, generating new actions. They are both in the same diagram because it is to represent an integration use case that shows how both work together.

Should I create a life line representing the queue?

Should I represent the "listening" process as a loop box containing: a self-message with the label "listen", self-message for processing message and messages representing the sending of the outcome.

答:
Implementing Bart's answer in the wonderful PlantUML could look like this, (ab)using an entity to distinguish the queue from the participating threads:

The corresponding source code:

```<code>participant Producer
entity Queue

Consumer->Queue : dequeue
activate Consumer
...
Producer->Queue : enqueue
activate Producer
Queue-->Consumer
deactivate Consumer
Queue-->Producer
deactivate Producer
</code>```

Body:
Implementing Bart's answer in the wonderful PlantUML could look like this, (ab)using an entity to distinguish the queue from the participating threads:

The corresponding source code:

participant Producer
entity Queue

Consumer->Queue : dequeue
activate Consumer
...
Producer->Queue : enqueue
activate Producer
Queue-->Consumer
deactivate Consumer
Queue-->Producer
deactivate Producer


问:
Would be good to include your freelancer account in your Resume / CV when applying for a job?
I've been working as a freelancer for about two years in vWorker. Any person can visit a coder's profile, and see in how many projects the coder has worked on, (if the coders allows) see how much the coder obtained in each project, ratings, feedbacks, etc.

Would be good to include a freelancer account in your Resume / CV when applying for a job?

Is it something you would do if you have finished several projects there?

答:
Rather dont bother including it into your resume and instead upload the apps/programs that you've created to a github or bitbucket account. That when whoever views your resume can instead view the code that you've written.

Also, you don't want whoever is viewing your resume to be able to see how much you where getting paid for those projects as it will most likely be much less than what you will be paid in the programming job that you are applying for.


问:
Term to describe pre-canned objects for use in testing
I have forgotten the term for a class that provides pre-canned objects for use as test data.

e.g. If I'm testing a payment module and I need credit card objects of different types, I might have a class in my test scope called CreditCard__something__ with static members or factory methods for creating cards of different flavours.

What is this term?

答:
Fixtures.

The setUp and tearDown methods are for creating or cleaning up your fixtures. If they live in memory, cleanup is easy, but if your fixtures live in the database, then both setting them up and tearing them down can be a little more involved.


问:
Is using multiple static classes with maximum one public method is a good idea
I'm writing a structural detailing (CAD) software for concrete buildings in C#. I have defined like hundreds of static classes each with one public method and if needed some private methods. Each one of these methods in these static classes does part of the job. They are called from one God static class named Building.cs.

For example one class looks like this:

```<code>public static partial class GetMainRebars
{
    public static void GetMainRebars()
    {
    }

    // other possible variables and private methods
}
</code>```

The program is running very very fast compared to other similar programs. But deep in my heart, looking at the above code I feel that something still may be wrong with this design pattern. Maybe I will encounter problems in terms of maintenance, or ...? I've taught myself programming, so I may have missed many core principles during the fast self teaching process. Can you elaborate the cons and pros of the above pattern?

And one very rookie question relevant to the pattern. Let's say I define a static variable inside such a class, will it remain in memory during the life time of the program? No disposing and whatsoever? Is it OK?

Update:
When the program starts I'm loading the required variables in relevant fields and properties in the Building.cs for example listOfBeams, listOfColumns, listOfRebars, listOfConnectivities and etc. Now that these variables are in place, I used the methods in static classes to do various operations on these. Let's say I get the listOfRebars and listOfColumns in one static class and generate some variables to output. Then I get listOfRebars and listOfBeams and do some other stuff.

Update2:
From what I have learned based on the answers and comments, I have come up with the below pattern. I was wondering if I'm on the right path?

I have one static class: Building.cs When I create/or load a project this class holds all the properties of the building like list of columns, beams, points, etc. All of these are stored as private fields. I can access these using the class's public methods like GetColumns or GetPoints ...

Now I also have non-static classes. They contain 2-3 public methods. and do some stuff on various parts of the building.

```<code>public static class Building
{
    private static List<Column> columns;
    private static List<Beams> beams;
    private static List<Points> points;

    public static List<Column> GetColumns()
    {
        return Columns;
    }
}

public class ColumnsService()
{
    private List<Columns> columns;
    public GroupColumns(List<Columns> columns)
    {
        this.columns = columns;
    }

    public void Group()
    {
        // group columns
    }
}

var columns = Building.GetColumns();
var columnsService = new ColumnsService(columns);
columnsService.Group();
</code>```

答:
Using static classes like this is simple and efficient. Like most things that are simple and efficient it comes with drawbacks:

your code will be difficult to unit test because any object that uses these static methods cannot easily be separated from them, so tests will have to include them, whereas usually we'd aim to use either stub or mock implementations of the collaborators of the object being tested.

it increases coupling, which has the effect of making your code hard to change.

it leads to needing large amounts of global state, which in itself also leads to testing being hard (global state can make tests dependent on each other, which is undesirable) and making changes harder (global state can lead to hard-to-identify interactions between apparently unrelated areas of the program)

The best approach would be to change the objects to non-static and use dependency injection to manage the dependencies between objects.


答:
I am not familiar with C#, but it is my conviction that one should try to limit global values and classes as programming in general. Of course depending on the sircumstances.
Always try and keep your classes to a minmum (but logical) responsibility. You should do some reading on the subject "low coupling high cohesion".

If you have a lot of classes interacting or depending on eachother, maybe you could implement a mediator design pattern or a observer pattern.

But it could sound to me that you haven't spent time doing some design of your project.
We can give a better answer to your question, if you could provide us with an explanation of the domain or how the objects relates/should relate to eachoter.

Title: Is using multiple static classes with maximum one public method is a good idea

Body:
I'm writing a structural detailing (CAD) software for concrete buildings in C#. I have defined like hundreds of static classes each with one public method and if needed some private methods. Each one of these methods in these static classes does part of the job. They are called from one God static class named Building.cs.

For example one class looks like this:

```<code>public static partial class GetMainRebars
{
    public static void GetMainRebars()
    {
    }

    // other possible variables and private methods
}
</code>```

The program is running very very fast compared to other similar programs. But deep in my heart, looking at the above code I feel that something still may be wrong with this design pattern. Maybe I will encounter problems in terms of maintenance, or ...? I've taught myself programming, so I may have missed many core principles during the fast self teaching process. Can you elaborate the cons and pros of the above pattern?

And one very rookie question relevant to the pattern. Let's say I define a static variable inside such a class, will it remain in memory during the life time of the program? No disposing and whatsoever? Is it OK?

Update:
When the program starts I'm loading the required variables in relevant fields and properties in the Building.cs for example listOfBeams, listOfColumns, listOfRebars, listOfConnectivities and etc. Now that these variables are in place, I used the methods in static classes to do various operations on these. Let's say I get the listOfRebars and listOfColumns in one static class and generate some variables to output. Then I get listOfRebars and listOfBeams and do some other stuff.

Update2:
From what I have learned based on the answers and comments, I have come up with the below pattern. I was wondering if I'm on the right path?

I have one static class: Building.cs When I create/or load a project this class holds all the properties of the building like list of columns, beams, points, etc. All of these are stored as private fields. I can access these using the class's public methods like GetColumns or GetPoints ...

Now I also have non-static classes. They contain 2-3 public methods. and do some stuff on various parts of the building.

```<code>public static class Building
{
    private static List<Column> columns;
    private static List<Beams> beams;
    private static List<Points> points;

    public static List<Column> GetColumns()
    {
        return Columns;
    }
}

public class ColumnsService()
{
    private List<Columns> columns;
    public GroupColumns(List<Columns> columns)
    {
        this.columns = columns;
    }

    public void Group()
    {
        // group columns
    }
}

var columns = Building.GetColumns();
var columnsService = new ColumnsService(columns);
columnsService.Group();
</code>```