答:
You compile each source file into object files using the appropriate complier for its language, then link the object files into an executable, the same way you would for any other program.

For this to work, all the compilers have to agree on how the modules communicate with each other.  This includes things such as: what symbol names the objects export and import, what format the data is in, how arguments get passed to functions, who allocates and frees memory, when and how.  Sometimes, the languages normally do things differently, and you need a special keyword.  There used to be a lot of different variants, but modern ABIs tend to be designed to make this easier than it used to be.

A specific example: C++ and C are very similar languages, but in order for a C and a C++ program to call each other’s functions, the C++ program needs to declare them as extern "C".  This is because C can only export one symbol named foo, but C++ lets you declare both int foo(int) and int foo(double) as different functions, so it needs to give them different names internally.  To communicate with modules in other languages, it has to turn that off. extern "C" tells the C++ compiler to use the same name a C compiler would for that function.  In Fortran, all function arguments can be modified, which means that to call f(x) in Fortran, a C program would need to call f(&x).  In C, a string is terminated by a null byte that you have to search for, but in many other languages, the first few bytes of each string are its length.  C lets functions, most famously printf(), take any number and type of arguments, and that doesn’t let it pass arguments in some of the ways that other languages have done it.

There are a lot of variants, but if your compilers can write code that talks to each other, there will be some convention both mutually know how to use.

Title: How are complex programs made?

Body:
I know a few languages, and can program in them. How do multiple languages blend together in a single program? For ex. https://github.com/facebook/watchman. This uses C, PHP, Python, Javascript, etc. How exactly do these languages combine to give a single result? How do they interact with each other? How do I learn to do this?

And I have seen this question but I don't think this is the same: How to use multiple programming languages together in the same program? As my question is about the exact way these things interact and how to learn it whereas the above question is just about how they interact, which also has no detailed answers.


问:
Manipulating data for a single file
I'm looking for a best practice solution for creating an object whose responsibility it is to handle file storage for a single file. I want to be able to read data from the file, write to the file, delete the file etc.

I have seen the repository pattern in use but that appears to be a solution for handling multiple files. I imagine creating an object that is named after the file it will be representing, in this case RenderOutput (or similar), that has functions such as .delete() which don't reveal anything about HOW they delete or WHERE the file is stored. This means that it doesn't matter from the outside how the file is persisted.

My question is whether or not there is a pattern or such that I can follow for naming, responsibilities, functionality etc.

答:
So if I understand correctly, you want a sort of Data class that allows you to read and write to it without knowing the underlying details of how it works.

You can do this by using an AbstractFactory pattern which provides these Data classes for writing/reading.  Each factory will have its own implementation with a method to initialize and a method to shutdown.  While your file system factory won't likely need to do much, your database factory will need to establish a connection and then close it, for instance.  You must be able to do all this without passing parameters to the factory.  I'll be honest, I'm not too familiar with actionscript, but in Java you could say, load a configuration file and the configuration file provides connection info or whatever else required for the factory to initialize.

Each factory type would return a specific implementation of Data class that does the actual work of writing/reading from that source.  My advice is that the location is provided upon calling the factory and that location remains immutable.  In other words, to write to a different location, you'd need to call the factory again with a different address.

In this way you're hiding implementation details from its usage in your program, which is fine.  However, this only works so long as the usage remains the same between all Data class implementations.  For instance, if you're having to determine the type of a Data class implementation so you can cast it to a DatabaseData instance and then call "BulkInsert", you should not be using this pattern nor should you be hiding implementation details.  The whole point is that you should only ever need an instance of Data class.

I hope that answers your question!

ContentLicense: CC BY-SA 3.0

Title: Manipulating data for a single file


问:
Class with only one execution
I'm implementing a long algorithm (about 500 lines of C++).
The algorithm consists currently of about 10 different methods. Each method I have to hand over multiple data-structures as parameters, which are calling each other. Only one method is can (should) be used by the end-consumer.

I was wondering if a class would be better. This way I can hold all data-structures in one place (as instance variables) and can access them from each method without passing all these as parameters.

A typical usage would be:

```<code>MyAlg myalg_instance(input_param1, input_param2, pointer to computation_results);
myalg_instance.do();
</code>```

And afterwards the instance is useless and can be deleted. Is this patter good? Are there better patterns?

答:
So we're looking at a big fat static method broken down into bite-sized pieces.  I would definitely recommend putting it into a class, even if it is for a single purpose.  Your process has a state and that state can be maintained through class members, so yes absolutely.

Just be careful to distinguish between operation parameters and process parameters.  Process parameters are what you pass to your algorithm and what is required in almost every method throughout.  Operation parameters are results of previous calls to methods that are necessary for the next operation, but only for the next operation.

A good example of a process parameter would be a list of items to sort. If you were to create a recursive sort algorithm, then the entire list would be your process parameter so you don't have continually pass the same array recursively.  The start and end indices would be a good example of operation parameters that don't need to be class members.  Using class members to hold information is an excellent way of reducing method parameters, but do it only if it is useful throughout.

If you don't like launching the algorithm in this way, as a formality you could create a public static method that does:

```<code>ProcessClass process = new ProcessClass(<parameters>);
process.start();
</code>```

So that the caller would only have to call Process.start(<parameters>) so you still have a neat way to call it externally and internally your code is cleaner than if you had one gigantic method.

Hope that helps!

Title: Class with only one execution

Body:
So we're looking at a big fat static method broken down into bite-sized pieces.  I would definitely recommend putting it into a class, even if it is for a single purpose.  Your process has a state and that state can be maintained through class members, so yes absolutely.

Just be careful to distinguish between operation parameters and process parameters.  Process parameters are what you pass to your algorithm and what is required in almost every method throughout.  Operation parameters are results of previous calls to methods that are necessary for the next operation, but only for the next operation.

A good example of a process parameter would be a list of items to sort. If you were to create a recursive sort algorithm, then the entire list would be your process parameter so you don't have continually pass the same array recursively.  The start and end indices would be a good example of operation parameters that don't need to be class members.  Using class members to hold information is an excellent way of reducing method parameters, but do it only if it is useful throughout.

If you don't like launching the algorithm in this way, as a formality you could create a public static method that does:

ProcessClass process = new ProcessClass(<parameters>);
process.start();

So that the caller would only have to call Process.start(<parameters>) so you still have a neat way to call it externally and internally your code is cleaner than if you had one gigantic method.

Hope that helps!


答:
Consider the concept of a closure (see e.g. this question and its answers). A closure saves you from passing around extra arguments that are known in a given context. For example, instead of writing:

```<code>def addTax(taxPercentage, amounts):
    def computeTax(perc, amount):
        return perc * amount / 100.0

    return [a + computeTax(taxPercentage, a) for a in amounts]
</code>```

you can write

```<code>def addTax(taxPercentage, amounts):
    def computeTax(amount):
        return taxPercentage * amount / 100.0

    return [a + computeTax(a) for a in amounts]
</code>```

The closure computeTax() implicitly gets the variable taxPercentage from its context (the context in which it is defined) so that it is not necessary to pass around the same variable again and again.

Now you have several functions that need to manipulate some common data. In fact, an object can be considered as a collection of closures that close over some common variables (the object's member variables). In this way, these functions need not pass their common data to each other explicitly.

So, yes, IMO it is definitely appropriate to make an object containing the data to be used during the computation and to implement all your functions as methods (closures) that access that common data.

Title: Class with only one execution

Body:
I'm implementing a long algorithm (about 500 lines of C++).
The algorithm consists currently of about 10 different methods. Each method I have to hand over multiple data-structures as parameters, which are calling each other. Only one method is can (should) be used by the end-consumer.

I was wondering if a class would be better. This way I can hold all data-structures in one place (as instance variables) and can access them from each method without passing all these as parameters.

A typical usage would be:

```<code>MyAlg myalg_instance(input_param1, input_param2, pointer to computation_results);
myalg_instance.do();
</code>```

And afterwards the instance is useless and can be deleted. Is this patter good? Are there better patterns?

Body:
Consider the concept of a closure (see e.g. this question and its answers). A closure saves you from passing around extra arguments that are known in a given context. For example, instead of writing:

def addTax(taxPercentage, amounts):
    def computeTax(perc, amount):
        return perc * amount / 100.0

    return [a + computeTax(taxPercentage, a) for a in amounts]

you can write

def addTax(taxPercentage, amounts):
    def computeTax(amount):
        return taxPercentage * amount / 100.0

    return [a + computeTax(a) for a in amounts]

The closure computeTax() implicitly gets the variable taxPercentage from its context (the context in which it is defined) so that it is not necessary to pass around the same variable again and again.

Now you have several functions that need to manipulate some common data. In fact, an object can be considered as a collection of closures that close over some common variables (the object's member variables). In this way, these functions need not pass their common data to each other explicitly.

So, yes, IMO it is definitely appropriate to make an object containing the data to be used during the computation and to implement all your functions as methods (closures) that access that common data.


问:
Refactor presentation logic from model
I've seen this pattern used in a considerable amount of MVC applications. Let's assume we have a User model with the methods:

hasSessionExpired

displaySessionExpired

The first method (hasSessionExpired) looks up in the database if the user's session has expired and returns true or false and the second calls the first method and returns an HTML string that basically says "Expired" with a red font or "Active" with a green font depending on the output of hasSessionExpired.

In the controller there is a call to the User object to fetch all users which are returned as an array of User objects, this array is sent to the view.

Then in the view, the array is iterated and the method displaySessionExpired is called for every object in the array.

There is something that smells fundamentally wrong with this approach, since the model has presentation logic in the displaySessionExpired method and the outcome of that logic is based on the outcome of a database query made at that moment. This of course makes the model classes have thousands of lines plagued with mixes of presentation and business logic that in the long run makes the application more difficult to maintain.

How should this be refactored so there is a clear separation of concerns?

答: The 'displaySessionExpired' on your model simply does not belong there. It is the responsibility of your model to handle logic, but HOW it is displayed (what text and what color) is the responsibility of the view. It is up to the view to call hasSessionExpired and take the result to be displayed.</p>

Title: Refactor presentation logic from model


答:
I think there's a missing concept here, which is UserSession. An user can have multiple UserSession (but only one active session at a time)

So if I do the refactoring, it would be something like this (in .NET):

```<code>public class UserSession
{
    private DateTime _expireTime;
    //Other properties

    public bool IsActive()
    {
        return _expireTime > DateTime.Now;
    }
}

public class User
{
    private List<UserSession> _sessions;

    public bool IsLoggedIn()
    {
        return _sessions.Any(s => s.IsActive());
    }
}
</code>```

Those two are in domain layer

MVC controllers will get list of users from database, with UserSession loaded in User. If you are worrying about performance impact of loading all UserSession, you can just load active UserSession.

In MVC view, to display who's logging in, who's not:

```<code>@foreach (var user in users)
{
    if(user.IsLoggedIn())
    {
        <span style='background-color: green' >@user.Name</span>
    }
    else
    {
        <span style='background-color: red' >@user.Name</span>
    }
}
</code>```

Or even better if you have a view model that contains all data needed to be displayed in the view:

```<code>public class UserStatusViewModel
{
    public string Name { get; set; }
    public bool IsLoggedIn { get; set; }
}
</code>```

The mapping from User to UserStatusViewModel is done in controller/application service. The view now depends on view model instead of model

Title: Refactor presentation logic from model

Body:
I've seen this pattern used in a considerable amount of MVC applications. Let's assume we have a User model with the methods:

hasSessionExpired

displaySessionExpired

The first method (hasSessionExpired) looks up in the database if the user's session has expired and returns true or false and the second calls the first method and returns an HTML string that basically says "Expired" with a red font or "Active" with a green font depending on the output of hasSessionExpired.

In the controller there is a call to the User object to fetch all users which are returned as an array of User objects, this array is sent to the view.

Then in the view, the array is iterated and the method displaySessionExpired is called for every object in the array.

There is something that smells fundamentally wrong with this approach, since the model has presentation logic in the displaySessionExpired method and the outcome of that logic is based on the outcome of a database query made at that moment. This of course makes the model classes have thousands of lines plagued with mixes of presentation and business logic that in the long run makes the application more difficult to maintain.

How should this be refactored so there is a clear separation of concerns?

Body:
I think there's a missing concept here, which is UserSession. An user can have multiple UserSession (but only one active session at a time)

So if I do the refactoring, it would be something like this (in .NET):

public class UserSession
{
    private DateTime _expireTime;
    //Other properties

    public bool IsActive()
    {
        return _expireTime > DateTime.Now;
    }
}

public class User
{
    private List<UserSession> _sessions;

    public bool IsLoggedIn()
    {
        return _sessions.Any(s => s.IsActive());
    }
}

Those two are in domain layer

MVC controllers will get list of users from database, with UserSession loaded in User. If you are worrying about performance impact of loading all UserSession, you can just load active UserSession.

In MVC view, to display who's logging in, who's not:

@foreach (var user in users)
{
    if(user.IsLoggedIn())
    {
        <span style='background-color: green' >@user.Name</span>
    }
    else
    {
        <span style='background-color: red' >@user.Name</span>
    }
}

Or even better if you have a view model that contains all data needed to be displayed in the view:

public class UserStatusViewModel
{
    public string Name { get; set; }
    public bool IsLoggedIn { get; set; }
}

The mapping from User to UserStatusViewModel is done in controller/application service. The view now depends on view model instead of model


问:
Is reinventing the wheel really all that bad?
Its common knowledge in programming that reinventing the wheel is bad or evil.

But why is that?

I am not suggesting that it's good. I believe it to be wrong. However, I once read an article that said, if someone is doing something wrong (programming wise) explain to them why its wrong, if you can't, then maybe you should be asking yourself if it is really wrong.

That leads me to this question:

If I see someone is clearly reinventing the wheel by building their own method of something that is already built into the language/framework. First, for arguments sake, lets assume that their method is just as efficient as the built in method. Also the developer, aware of the built in method, prefers his own method.

Why should he use the built in one over his own?

答:
One useful reason to reinvent the wheel is for learning purposes -- but I'd recommend doing it on your own time.  As more pre-canned solutions become available, and more levels of abstraction is provided, we become a lot more productive.  We can focus on the business problem rather than the generic stuff that's been tweaked time and again.  BUT, for that reason, you can sharpen your skills and learn a lot by trying to re-implement a solution on your own.  Just not necessarily for production use.

One other thing -- if a concern is dependency on a third party library from a company that may disappear, make sure there's an option to get the source code, or at least a couple of other choices out there to fall back on.

By the way, if you do choose to implement your own, avoid doing this for cryptography or other security-related functionality.  Established (and fully tested) tools are available for that, and in this day and age, it is wayyyyyy too risky to roll your own.  That is never worth it, and it's scary that I still hear about teams doing this.

ContentLicense: CC BY-SA 2.5


问:
AngularJS and ASP MVC
We have started to implement AngularJS in our web-applications using ASP MVC.
I am wondering what approach to use for building application in the future.

Keep using our current stategy: Build the website and provide the pages with ASP MVC. Authentication and authorization is provided out of the box by MVC.

Possible new strategy: Provide a the web pages through a normal website and put all the request behind a WebAPI.

Any toughts or other suggestions?

答:
Using ASP.Net MVC for its authentication and authorization capabilities is perfectly fine.  If you're using Angular, you will almost certainly also be using Web API to get the data into the templates (see note below), and you will probably want to reuse the same authentication and authorization code and capability on that side.  Remember that this level of checking must happen on the server (and optionally on the client side).

A quick note: it's not 100% clear from your question whether you are suggesting combining MVC's templating capabilities with Angular's.  Hopefully not, as this doesn't seem to make a lot of sense (unless you really need to be dynamically generating Angular templates), and you will likely end up with the worst of both worlds.  So in a situation where both are used, the MVC part should only be serving up the Angular templates, and Angular doing what it's good at by combining it with data it pulls via Web API.

Title: AngularJS and ASP MVC


问:
What is an accepted method of timestamping in a distributed system?
In a normal application, a simple call to time.Now() in most languages will suffice for getting a timestamp, but what about in a web application spread over multiple servers? As I understand it time.Now() will use the current system time, but there's no reliable way of telling after the fact if that time was synchronised with the rest of the machines. By this point you're throwing data into persistent storage with inconsistent timestamps which is not such a problem for created/modified times when inserting into a database, but a bigger problem when tracking event timing).

The best way I see to mitigate this is to have all machines request a timestamp from one specific 'master' machine, but then I thought what about latency? What if the network is busy and the response is delayed? There's several flaws with this method but I can't think of a more reliable one other than using CPU and network resources synchronising the time every few seconds.

What is the accepted best practice/industry standard here?

答:
apart from having a time server that every timestamp is supposed to come from (with the massive performance overhead in network traffic that may generate)?

The usual practice is to either a) not bother or b) have procedures in place that ensure the clocks on all machines are kept in synch from a single central NTP server.


问:
3D visualization of moment using accelerometer on Android device
Is it possible to reconstruct the movement of an Android device in 3D space using data from accelerometer?

For example, I may draw a circle with my hand in the air, holding an Android device.

 Is it theoretically possible to reconstruct this circle using accelerometer data in 3D space, relative to the ground accurately?

 Here is an example of how this circle could be represented:

 Are results like in this video possible: https://www.youtube.com/watch?v=6ijArKE8vKU

Are there some sort of algorithms that can perform such tasks available online?

答:
As amon said, this is theoretically possible with the addition of orientation/rotation sensors.

In practice, it really depends on the accurracy required. The calculations are all integrals, so they tend to accumulate errors very fast. This means, the calculated endpoint of the circle in your example will be away from your origin, even if you carefully move your phone. How much away it will be depends on

how fast you move

how fast you can measure (sample frequency)

how accurrate you can measure

how accurrate your math implementation is (rounding errors etc..)

As a master thesis, I have created an assisted inertial navigation system for use in trains, with a quite good commercial sensor. The results were quite disappointing after a short while, though.


问:
Does music increase your productivity?
I wanted to know whether you think and audio background is just a distraction or could be of some help while programming.

For me, it works like this:

when I'm writing pseudocode, or implementing procedures that are very clear in my mind in a language I can write without looking up docs, having music playing in the background (very straightforward electronics, mostly) helps me a lot: it kind of gives some rhythm to my coding and, if the procedures are trivial, avoids the distractions that boredom can cause.

When I incur in a problem, or I have to try to fix a bug, instead, it just distracts me.

Also, if the song is really regular and I know it well enough not to cause me to focus on it, it eliminates unexpected environmental noise that sometimes make me waste precious seconds before I can concentrate on the code again.

Guidelines checklist: 1 ok, 2 I hope; 3 I suppose; 4 Definitely; 5 I would like that to happen; 6 Doch (English should implement a way to disambiguate double negatives in answers)

答:
This is a case of sometimes fierce debate at work.

I work better with certain kinds of music - either light stuff, classical, or metal. (Death/black/extreme metal also acts as a very effective sound barrier when played through headphones.)

But others here in the office cannot help concentrating on the music rather than programming.

Title: Does music increase your productivity?


问:
Class design, responsibility granulation, efficiency and performance
I am in a design dilemma. I have a set of data that can be interpreted in numerous ways, but I cannot really decide how finely grained should it be. To illustrate it with some simple code:

class Base {
    Datamembers data;
    virtual void doStuff() = 0;
};

class T1 : public Base {
    void doStuff() {
        // do T1 stuff
    }
};

class T2 : public Base {
    void doStuff() {
        // do T2 stuff
    }
};

class ...

// ---------------------------

class Master {
    enum Role {
        T1, T2 ...
    };

    Role r;
    Datamembers data;

    void doStuff() {
        switch (r) {
        case T1:
            // do T1 stuff
            return;
        case T2:
            // do T2 stuff
            return;
        case ...
        }
    }
};
```

In reality the number of responsibilities is much higher than 2, currently still in the two digit range, but may grow further. Also, it is not one, but several doStuff()s at play.

In the first case, the data is encapsulated in a base class, and doStuff() is a pure virtual, which each derived class implements. Each responsibility is encapsulated in its own, dedicated type.

In the second case doStuff() is not a virtual function, and the responsibility is stored as an extra member in a Role, and the method switches the role and executes the appropriate code.

The two designs achieve the same goal in a different way, but what are the consequences?

Code maintenance:

The first approach seems cleaner, but even though usually clean implies maintainable, I feel it is kind of fragmented, which may end up reducing the ease of maintenance.

In the second approach all the code for the different responsibilities is crammed in the same place, which may sound dirty, but in many cases having it all in the same place may actually be easier, since all you have to do is scroll rather than jump around different files. But then again... it is still a big wall of code, not obvious in the example code above, but in practice it will be 4-5 digit lines of code. Although there is an easy remedy for that, just implement a function for each case that takes this as a parameter to access the data and force inlining to eliminate the overhead of extra function calls (even if not virtual).

Performance:

In my case, I deal with massive trees of objects, where doStuff() will be executed in tight loops, traversing that tree.

In approach one, the method is virtual, this means I get penalty for the indirection. Also, the code is spread across multiple types, which may end up fragmenting it in memory, reducing cache efficiency.

In the second approach the method is not virtual, and all code is essentially in a single function, which means it will be tighter in memory, and potentially more cache friendly. There is the switching penalty vs the indirection and fragmentation of the first approach.

Memory usage:

Although it is missing from the example code above, the hierarchy does have a constructor with plenty of arguments. Meaning that in the first approach each derived class will have its own constructor with the same parameters, redirecting those to the base class constructor. This, aside from tedious to implement for each type, also adds some code bloat. And there is the overhead of the v-tables. Both the extra constructors and v-tables are avoided in the second approach.

Obviously, I've made some considerations, and had this been a smaller undertaking, I'd do both and compare. But it is actually a lot of work and it is not really possible, so I'd appreciate some experienced opinions - are my considerations valid, am I missing something, is there another, better way to do it?

As it seems the scale of the problem isn't entirely clear, I will just summarize it. I am talking about an object tree, potentially tens of millions of nodes big, a responsibility hierarchy of currently 30 roles, expected to grow to over 100. Right now I have approximately about 200 lines of code per role, although the number varies. Parallelization is not an option, since node result depends on parent or children nodes results, so I am limited to a single thread traversing the tree, currently processing 200-300k nodes a second.

答:
You can use Visitor pattern. Visitor pattern with C++ templete is highly efficient.

```<code>class T1 {
    public: void do(Node& node) {...}
};

class T2 {
    public: void do(Node& node) {...}
};

class Master {
public:
    template <typename T>
    void traverse() {
        T t;
        for each node in the tree {
            t.do(node);
        }
    }
};

Master m;
m.traverse<T1>();
</code>```

If you have many traversal algorithms, Strategy pattern can be used together.

Body:
You can use Visitor pattern. Visitor pattern with C++ templete is highly efficient.

class T1 {
    public: void do(Node& node) {...}
};

class T2 {
    public: void do(Node& node) {...}
};

class Master {
public:
    template <typename T>
    void traverse() {
        T t;
        for each node in the tree {
            t.do(node);
        }
    }
};

Master m;
m.traverse<T1>();

If you have many traversal algorithms, Strategy pattern can be used together.


答:
The usual practice is to set the servers to all keep their time updated using NTP.  There are limitations to the accuracy when using NTP time syncing which means that you should only rely on the time stamp to give a general idea of when events occurred, which is likely to be good enough for identifying the set of events you are interested in.

Timestamps are no good for determining the sequence of events, even on a single server at a precision level, as even with microsecond timing, each CPU core will process 1000's of assembler instructions during the same timestamp.  Your server may have 40 such cores.

The best approach I know of, to keep track of what is happening in a distributed system is to use transaction ID's and log files. A single log file written sequentially allows you to determine event order on a single server.  Anything that passes work between servers will need to appear in the logs on both servers recording the transfer of the work with a shared transaction ID.

Title: What is an accepted method of timestamping in a distributed system?

Body:
In a normal application, a simple call to time.Now() in most languages will suffice for getting a timestamp, but what about in a web application spread over multiple servers? As I understand it time.Now() will use the current system time, but there's no reliable way of telling after the fact if that time was synchronised with the rest of the machines. By this point you're throwing data into persistent storage with inconsistent timestamps which is not such a problem for created/modified times when inserting into a database, but a bigger problem when tracking event timing).

The best way I see to mitigate this is to have all machines request a timestamp from one specific 'master' machine, but then I thought what about latency? What if the network is busy and the response is delayed? There's several flaws with this method but I can't think of a more reliable one other than using CPU and network resources synchronising the time every few seconds.

What is the accepted best practice/industry standard here?


答: As mentioned in @jwenting's comment, NTP is basically the "accepted standard".  But if latency is highly critical for you, maybe TICSync is a useful option. It achieves millisecond accuracy very quickly and is used in robotics systems where a high synchronisation precision, beyond what NTP can offer, is required.</p>


问:
What to do about large svn history when moving to git?
Edit: unlike some similar questions such as Moving a multi-GB SVN repo to Git
or
https://stackoverflow.com/questions/540535/managing-large-binary-files-with-git
My scenario doesn't involve several subprojects that can be easily converted into git submoduels, nor a few very large binary files that are well suited for git-annex. It is a single repository where the binaries are the test suite which tightly coupled to the main source code of the same revision, much like if they were compile time assets such as graphics.

I'm investigating switching an old medium/large sized (50 users, 60k revisions, 80Gb history, 2Gb working copy) code repository from svn. As the number of users have grown, there is a lot of churn in trunk, and features are often spread out on multiple commits making code review hard to do. Also without branching there is no way to "gate" bad code out, reviews can only be done after it is committed to trunk. I'm investigating alternatives. I was hoping we could move to git, but I'm having some problems.

The problem with the current repo as far as git goes is size. There is a lot of old cruft in there, and cleaning it with --filter-branch when converting to git can cut it down in size by an order of magnitude, to around 5-10GB. This is still too big. The biggest reason for the large repository size is that there are a lot of binary documents being inputs to tests. These files vary between .5mb and 30mb, and there are hundreds. They also have quite a lot of changes. I have looked at submodules, git-annex etc, but having the tests in a submodule feels wrong, as does having annex for many files for which you want full history.

So the distributed nature of git is really what's blocking me from adopting it. I don't really care about distributed, I just want the cheap branching and powerful merging features. Like I assume 99.9% of git users do, we will use a blessed, bare central repository.

I'm not sure I understand why each user has to have a full local history when using git? If the workflow isn't decentralized, what is that data doing on the users' disks? I know that in recent versions of git you can use a shallow clone with only recent history. My question is: is it viable to do this as the standard mode of operation for an entire team? Can git be configured to always be shallow so you can have a full history only centrally, but users by default only have 1000 revs of history? The option to that of course would be to just convert 1000 revs to git, and keep the svn repo for archeology. In that scenario however, we'd encounter the same problem again after the next several thousand revisions to the test documents.

What is a good best practice for using git with large repos containing many binary files that you do want history for? Most best practices and tutorials seem to avoid this case. They solve the problem of few huge binaries, or propose dropping the binaries entirely.

Is shallow cloning usable as a normal mode of operation or is it a "hack"?

Could submodules be used for code where you have a tight dependency between the main source revision and the submodule revision (such as in compile time binary dependencies, or a unit test suite)?

How big is "too big" for a git repository (on premises)? Should we avoid switching if we can get it down to 4GB? 2GB?

答:
Wow, that's a long question (and a complex problem). I'll try to have a
go at it.

I'm not sure I understand why each user has to have a full local
  history when using git?

This is a central design decision with git. For the exact reasons you'd
need to ask the author (Linus Torvalds), but as far as I know, the main
reason is speed: Having everything local (on a fast disk or even cached
in RAM) makes operations on history much faster by avoiding network
access.

The biggest reason for the large repository size is that there are a
  lot of binary documents being inputs to tests. These files vary
  between .5mb and 30mb, and there are hundreds. They also have quite a
  lot of changes.

That is the point I would think about first. Having so many constantly
changing binary files in source control seems problematic to me (even
with SVN). Can't you use a different approach? Ideas:

Unlike source code, a 3 MB binary file is probably not written by
hand. If some tool/process generates it, consider integrating that
into your build, instead of storing the data.

If that is not practical, binary files are typically better off in an
artifact repository (such as Artifactory for Maven & co.). Maybe that
is an option for you.

I have looked at submodules, git-annex etc, but having
  the tests in a submodule feels wrong, as does having annex for many
  files for which you want full history.

Actually, this looks like git-annex would fit perfectly. git-annex
basically allows you to store file contents outside a git repository
(the repository contains a placeholder instead). You can store the file
contents in a variety of ways (central git repo, shared drive, cloud storage...), and you can control which contents you want to have locally.

Did you maybe misunderstand how git-annex works? git-annex does store
full history for all the files it manages - it just lets you choose
which file contents you want to have locally.

Finally, about your questions:

What is a good best practice for using git with large repos containing
  many binary files that you do want history for?

In my experience, the options usually are:

avoid the need for binaries in the repo (generate them on demand,
store them elsewhere)

use git-annex (or a similar solution, such as Git LFS)

live with a big repo (not all git operations are affected by big
files, and if you have a fast computer and drive, it can be quite
workable)

Is shallow cloning usable as a normal mode of operation or is it a
  "hack"?

That might be doable; however, I don't think this will solve your
problem:

you'd lose of git's benefits that come from having full history, such
as quick searching of the history

merges can become tricky, because AKAIK you must have at least the
history back to the branch point to merge

users would need to re-clone periodically to keep the size of their
clone small

it's just an uncommon way of using git, so you'd likely run into
problems with many tools

How big is "too big" for a git repository (on premises)? Should we
  avoid switching if we can get it down to 4GB? 2GB?

That depends on the structure of the repo (few/many files etc.), on what
you want to do, on how beefy your computers are, and on your patience
:-).

To give you a quick idea: On my (newish, but low-spec) laptop,
committing a 500 MB file takes 30-60s. Just listing history (git log
etc.) is not affected by big files; things like "git log -S" which must
scan file content are very slow - however, the speed is mainly dominated
by I/O, so it's not really git's fault.

On a 3 GB repo with a handful of revisions, "git log -S" takes about a
minute.

So I'd say a couple of GB is ok, though not ideal. More than 10-20 GB is
probably pushing it, but it might be doable - you'd have to try it.


答: Look into GCC mailing list. Migrating the GCC compiler's source tree from SVN to GIT is discussed right now (august & september 2015), while keeping GCC's history. See e.g. repository for the conversion machinery & Acceptance criteria for the git conversion mail threads; you'll find references to tools and procedures related to the conversion (which is not as straightforward as it seems; the conversion of such a large code base history needs 36 hours and about 64Gbytes of RAM, IIRC)</p>

Title: What to do about large svn history when moving to git?

Body:
Edit: unlike some similar questions such as Moving a multi-GB SVN repo to Git
or
https://stackoverflow.com/questions/540535/managing-large-binary-files-with-git
My scenario doesn't involve several subprojects that can be easily converted into git submoduels, nor a few very large binary files that are well suited for git-annex. It is a single repository where the binaries are the test suite which tightly coupled to the main source code of the same revision, much like if they were compile time assets such as graphics.

I'm investigating switching an old medium/large sized (50 users, 60k revisions, 80Gb history, 2Gb working copy) code repository from svn. As the number of users have grown, there is a lot of churn in trunk, and features are often spread out on multiple commits making code review hard to do. Also without branching there is no way to "gate" bad code out, reviews can only be done after it is committed to trunk. I'm investigating alternatives. I was hoping we could move to git, but I'm having some problems.

The problem with the current repo as far as git goes is size. There is a lot of old cruft in there, and cleaning it with --filter-branch when converting to git can cut it down in size by an order of magnitude, to around 5-10GB. This is still too big. The biggest reason for the large repository size is that there are a lot of binary documents being inputs to tests. These files vary between .5mb and 30mb, and there are hundreds. They also have quite a lot of changes. I have looked at submodules, git-annex etc, but having the tests in a submodule feels wrong, as does having annex for many files for which you want full history.

So the distributed nature of git is really what's blocking me from adopting it. I don't really care about distributed, I just want the cheap branching and powerful merging features. Like I assume 99.9% of git users do, we will use a blessed, bare central repository.

I'm not sure I understand why each user has to have a full local history when using git? If the workflow isn't decentralized, what is that data doing on the users' disks? I know that in recent versions of git you can use a shallow clone with only recent history. My question is: is it viable to do this as the standard mode of operation for an entire team? Can git be configured to always be shallow so you can have a full history only centrally, but users by default only have 1000 revs of history? The option to that of course would be to just convert 1000 revs to git, and keep the svn repo for archeology. In that scenario however, we'd encounter the same problem again after the next several thousand revisions to the test documents.

What is a good best practice for using git with large repos containing many binary files that you do want history for? Most best practices and tutorials seem to avoid this case. They solve the problem of few huge binaries, or propose dropping the binaries entirely.

Is shallow cloning usable as a normal mode of operation or is it a "hack"?

Could submodules be used for code where you have a tight dependency between the main source revision and the submodule revision (such as in compile time binary dependencies, or a unit test suite)?

How big is "too big" for a git repository (on premises)? Should we avoid switching if we can get it down to 4GB? 2GB?


问:
Is it OK to use (WTF) forms to validate against stuff from DB?
Form classes are intended (IMO) for submitted data against rules. For example: are passwords equal, is end date later than start date.

submitted data--->|Form|

Is it okay for Form classes to validate submitted data against data retrieved from a database as well? This seems like a responsibility issue, the Form class should responsible for validating and cleaning the submission alone, and checking against the DB belongs outside the class.

submitted data--->|Form|<---data from DB

Here is my example in WTForms in Python: it checks if the user id is an integer, then queries the DB to verify that it exists.

```<code>class ClientForm(Form):
    name = StringField(validators=[DataRequired(message="Invalid value")])
    user = IntegerField(validators=[DataRequired(message="Invalid value")])
    zone = IntegerField(validators=[AnyOf([i[0] for i in ZONES], message="Invalid value")])

    #validate user as real
    def validate_user(form, field):
        user = db.session.query(User).filter(User.id==form.user.data).first()
        if not user:
            raise ValidationError('Invalid user')
</code>```

The upside of adding this check to the form is convenience, WTForms handles all aspects of errors, even rendering them:

```<code>form = ClientForm(post_data)
if not form.validate():
    # render the form again with errors highlighted
    return render("client_registration_form.html", form=form)
else:
    return render("success.html")
</code>```

答:
I think it's fine to use WTForms to validate data not directly submitted by a user. It seems very strange to validate data from your database: the database and DAL should have validated it already.

It's not unreasonable to use WTForms to validate a combination of user-submitted data and data from the database. Still, for an interactive form, I'd prefer the user see a form which by its very structure prevents the entry of invalid data, based on the contents of the database. Probably, sometimes it is not possible or practical.

I can easily see using WTForms to validate data coming from e.g. external systems which are not form submissions, like parsed input files.


问:
Are data type declarators like "int" and "char" stored in RAM when a C program executes?
When a C program is running, the data is stored on the heap or the stack. The values are stored in RAM addresses. But what about the type indicators (e.g., int or char)? Are they also stored?

Consider the following code:

```<code>char a = 'A';
int x = 4;
</code>```

I read that A and 4 are stored in RAM addresses here. But what about a and x? Most confusingly, how does the execution know that a is a char and x is an int? I mean, is the int and char mentioned somewhere in RAM?

Let's say a value is stored somewhere in RAM as 10011001; if I am the program which executes the code, how will I know whether this 10011001 is a char or an int?

What I don't understand is how the computer knows, when it reads a variable's value from an address such as 10001, whether it is an int or char. Imagine I click on a program called anyprog.exe. Immediately the code starts executing. Does this executable file include information on whether the variables stored are of the type int or char?

答:
but why some says here in C# case, it is not the story? i read some others comments and they say in C# and C++ the story(info on data types) is different and even CPU does not do the computing. Any ideas about that?

In type-checked languages like C#, the type-checking is done by the compiler. The code benji wrote:

```<code>int main()
{
    int a = 65;
    char b = 'A';
    if(a == b)
    {
        printf("Well, what do you know. A char can equal an int.\n");
    }
    return 0;
}
</code>```

Would simply refuse to compile. Similarly if you tried to multiply a string and an integer (I was going to say add, but the operator '+' is overloaded with string concatenation and it might just work).

```<code>int a = 42;
string b = "Compilers are awesome.";
double[] c = a * b;
</code>```

The compiler would simply refuse to generate machine code from this C#, no matter how much your string kissed up to it.

Body:
but why some says here in C# case, it is not the story? i read some others comments and they say in C# and C++ the story(info on data types) is different and even CPU does not do the computing. Any ideas about that?

In type-checked languages like C#, the type-checking is done by the compiler. The code benji wrote:

int main()
{
    int a = 65;
    char b = 'A';
    if(a == b)
    {
        printf("Well, what do you know. A char can equal an int.\n");
    }
    return 0;
}

Would simply refuse to compile. Similarly if you tried to multiply a string and an integer (I was going to say add, but the operator '+' is overloaded with string concatenation and it might just work).

int a = 42;
string b = "Compilers are awesome.";
double[] c = a * b;

The compiler would simply refuse to generate machine code from this C#, no matter how much your string kissed up to it.


答: I find its easier to code with music I know and like, but harder to code with music I either dislike or don't know. It comes down to maintaining focus, its easier when the music fits a regular pattern and consumes the background noise from the office and the brain.</p>

Title: Does music increase your productivity?

Body:
I wanted to know whether you think and audio background is just a distraction or could be of some help while programming.

For me, it works like this:

when I'm writing pseudocode, or implementing procedures that are very clear in my mind in a language I can write without looking up docs, having music playing in the background (very straightforward electronics, mostly) helps me a lot: it kind of gives some rhythm to my coding and, if the procedures are trivial, avoids the distractions that boredom can cause.

When I incur in a problem, or I have to try to fix a bug, instead, it just distracts me.

Also, if the song is really regular and I know it well enough not to cause me to focus on it, it eliminates unexpected environmental noise that sometimes make me waste precious seconds before I can concentrate on the code again.

Guidelines checklist: 1 ok, 2 I hope; 3 I suppose; 4 Definitely; 5 I would like that to happen; 6 Doch (English should implement a way to disambiguate double negatives in answers)


问:
Manual Dependency Injection Examples Still Tightly Coupled
I've been using AutoFac for some time now to take care of the dependency injection we're using in a web project. All is well, but I now want to demonstrate DI outside of an AutoFac context for reasons irrelevant to this question.

In doing so, I'm having to read up on 'manual' methods of achieving this pattern. All of the examples in tutorials work (obviously), but the issue of tightly coupled objects still exist.

Take this for example: http://www.devx.com/dotnet/Article/34066/0/page/2

The constructor of the dependent class takes an interface:

```<code>public class BusinessFacade {
    private IBusinessLogic businessLogic;
    public BusinessFacade(IBusinessLogic businessLogic) {
        this.businessLogic = businessLogic;
    }
}
</code>```

But then the instantiation of BusinessFacade still requires a concrete implementation to be passed, which means an implementation needs to be instantiated:

```<code>IBusinessLogic productBL = new ProductBL();
BusinessFacade businessFacade = new BusinessFacade(productBL);
</code>```

Question

Now I know, somewhere at some point, something (whether it is a container or manually written code) will have to decide on the implementation to choose, but how can I come to terms with the benefit of moving such decision outside of the dependent class? Of course the BusinessFacade class is no longer tightly coupled, but we're still very explicit about which implementation we choose - so what's been gained here? Is it purely for the fact we can mock out the dependencies?

答:
The fact that a BusinessFacade object can take any such implementation of IBusinessLogic is such that you are decoupled so long as you follow SOLID principles of design in your objects.

The typical idea that you brought up that ProductBL property can now be mocked in a unit test is one particular benefit that you mentioned.  The other is that if you have requirements in the future for additional IBusinessLogic implementations then you can simply use the same BusinessFacade class and create many container managed objects of type BusinessFacade that could be wrapping many various implementations.  You probably will organize them and name them properly in your DI configuration.

It may seem pointless, but probably only because of what it is I think you are doing.

All is well, but I now want to demonstrate DI outside of an AutoFac context for reasons irrelevant to this question.

If you are newing up objects outside of your container like in your code example then how is that Dependency Injection?  If AutoFac doesn't monopolize DI in your project then how can any of your code have any reasonable assurance as to unstated and unconfigured dependencies that could exist in an object?  This is really beyond the scope of your question but I thought it was worth mentioning that you might not be seeing the benefits of DI simply because it sounds to me from an outsiders perspective that you are purposely trying to circumvent it and work against it.

Now I know, somewhere at some point, something (whether it is a container or manually written code) will have to decide on the implementation to choose,

The container should always choose or what is the point of using DI at all.


问:
How do distributed applications verify that the data was processed correctly?
There are a lot of projects out there that rely on the public sphere to process data for them, but I do wonder how they ensure that every computer participating has processed the data properly.

The first one that comes to mind is Bitcoin mining. I know that the Bitcoin network has a mathematical way of verifying that the data has been processed correctly (I'm not going to go into that here).

But, when I think about other projects, such as the SETI@home project, I often wonder how they know that every participant is behaving correctly; how is it that they know the data was actually processed, or that the results being returned are accurate. The only validation I can think of would require them to also process the same data, meaning that the distributed model is moot.

The game of connect four has already been solved, but let's say that someone wanted to develop a project that solved the game of connect four using a distributed public network (the internet). The server would send down a large chunk of data to one machine to process, for example it would send down a starting point and the machine would have to process all possible boards with a depth of 5 and use a predetermined formula for scoring, then return the best move to the server.

How does the server know that this was the best move using the algorithm?

How do we guarantee that there isn't a rogue actor that is consuming the service and replacing the predetermined scoring algorithm with a junk one that yields the incorrect best move?

答:
The use of a global snapshot, in conjunction with an analysis algorithm can (and is) used for this type of verification.

The Snapshot Algorithm works like this:

```<code>The observer process (the process taking a snapshot):
    Saves its own local state
    Sends a snapshot request message bearing a snapshot token to all other processes
A process receiving the snapshot token for the first time on any message:
    Sends the observer process its own saved state
    Attaches the snapshot token to all subsequent messages (to help propagate the
    snapshot token)
Should a process that has already received the snapshot token receive
a message that does not bear the snapshot token, this process
will forward that message to the observer process. This message was obviously sent before
the snapshot “cut off” (as it does not bear a snapshot token and thus must have come from
before the snapshot token was sent out) and needs to be included in the snapshot.
</code>```

From this, the observer builds up a complete snapshot: a saved state
  for each process and all messages “in the ether” are saved.

Once the observers have all generated a snapshot of their local state, a central analyzer can pull these snapshots, reconstruct them, and run verification on them.

Body:
The use of a global snapshot, in conjunction with an analysis algorithm can (and is) used for this type of verification.

The Snapshot Algorithm works like this:

The observer process (the process taking a snapshot):
    Saves its own local state
    Sends a snapshot request message bearing a snapshot token to all other processes
A process receiving the snapshot token for the first time on any message:
    Sends the observer process its own saved state
    Attaches the snapshot token to all subsequent messages (to help propagate the
    snapshot token)
Should a process that has already received the snapshot token receive
a message that does not bear the snapshot token, this process
will forward that message to the observer process. This message was obviously sent before
the snapshot “cut off” (as it does not bear a snapshot token and thus must have come from
before the snapshot token was sent out) and needs to be included in the snapshot.

From this, the observer builds up a complete snapshot: a saved state
  for each process and all messages “in the ether” are saved.

Once the observers have all generated a snapshot of their local state, a central analyzer can pull these snapshots, reconstruct them, and run verification on them.


问:
What are the valid uses of static classes?
I noticed that nearly every time I see programmers using static classes in object oriented languages such as C#, they are doing it wrong. The major problems are obviously the global state and the difficulty to swap implementations at runtime or with mocks/stubs during tests.

By curiosity, I've looked at a few of my projects selecting ones which are well tested and when I made an effort to actually think about architecture and design. The two usages of static classes I've found are:

The utility class—something I would rather avoid if I was writing this code today,

The application cache: using static class for that is plainly wrong. What if I want to replace it by MemoryCache or Redis?

Looking at .NET Framework, I don't see any example of a valid usage of static classes either. For example:

File static class makes it really painful to switch to alternatives. For example, what if one needs to switch to Isolated Storage or store files directly in memory or needs a provider which can support NTFS transactions or at least be able to deal with paths longer than 259 characters? Having a common interface and multiple implementations appears as easy as using File directly, with the benefit of not having to rewrite most of the code base when requirements change.

Console static class makes testing overly complicated. How should I ensure within a unit test that a method outputs a correct data to console? Should I modify the method to send the output to a writeable Stream which is injected at run time? Seems that a non-static console class would be as simple as a static one, just without all the drawbacks of a static class, once again.

Math static class is not a good candidate either; what if I need to switch to arbitrary precision arithmetic? Or to some newer, faster implementation? Or to a stub which will tell, during a unit test, that a given method of a Math class was indeed called during a test?

On Programmer.SE, I've read:

Don't Use “Static” in C#? Some answers are quite against static classes. Others assert that “Static methods are fine to use and have a rightful place in programming.”, but don't bake it with arguments.

When to use a Singleton and when to use a static class. Utility classes are mentioned, given that I mentioned above that utility classes are problematic.

Why and when should I make a class 'static'? What is the purpose of 'static' keyword on classes? The only valid usage which is mentioned is a container for extension methods. Great.

Aside extension methods, what are the valid uses of static classes, i.e. cases where Dependency Injection or singletons are either impossible or will result in a lower quality design and harder extensibility and maintenance?

答:
The case for static methods: This:

```<code>var c = Math.Max(a, Int32.Parse(b));
</code>```

is much simpler and clearer and more readable than this:

```<code>IMathLibrary math = new DefaultMathLibrary();
IIntegerParser integerParser = new DefaultIntegerParser();
var c = math.Max(a, integerParser.Parse(b));
</code>```

Now add dependency injection to hide the explicit instantiations, and see the one-liner grow into tens or hundreds of lines - all to support the unlikely scenario that you invent your own Max-implementation which is significantly faster than the one in the framework and you need to be able to transparently switch between the alternative Max-implementations.

API design is a trade-off between simplicity and flexibility. You can always introduce additional layers of indirection (ad infinitum), but you have to weigh the convenience of the common case against the benefit of additional layers.

For example, you can always write your own instance wrapper around the file system API if you need to be able to transparently switch between storage providers. But without the simpler static methods, everybody would be forced to create an instance every time they had to do a simple thing like saving a file. It would be really a bad design trade-off to optimize for an extremely rare edge case, and making everything more convoluted for the common case. Same with Console - you can easily create your own wrapper if you need to abstract or mock the console, but quite often you use the console for quick solutions or debugging purposes, so you just want the simplest possible way to output some text.

Furthermore, while "a common interface with multiple implementations" is cool, there is not necessarily a "right" abstraction which unifies all the storage providers you want. You might want to switch between files and isolated storage, but someone else might want to switch between files, database and cookies for storage. The common interface would look different, and it is impossible to predict all possible abstractions users could want. It is much more flexible to provide static methods as the "primitives", and then let users build their own abstractions and wrappers.

Your Math example is even more dubious. If you want to change to arbitrary precision math you would presumably need new numeric types. This would be a fundamental change of you whole program, and having to change Math.Max() to ArbitraryPrecisionMath.Max() is certainly the least of you worries.

That said, I agree that stateful static classes are in most cases evil.

Body:
The case for static methods: This:

var c = Math.Max(a, Int32.Parse(b));

is much simpler and clearer and more readable than this:

IMathLibrary math = new DefaultMathLibrary();
IIntegerParser integerParser = new DefaultIntegerParser();
var c = math.Max(a, integerParser.Parse(b));

Now add dependency injection to hide the explicit instantiations, and see the one-liner grow into tens or hundreds of lines - all to support the unlikely scenario that you invent your own Max-implementation which is significantly faster than the one in the framework and you need to be able to transparently switch between the alternative Max-implementations.

API design is a trade-off between simplicity and flexibility. You can always introduce additional layers of indirection (ad infinitum), but you have to weigh the convenience of the common case against the benefit of additional layers.

For example, you can always write your own instance wrapper around the file system API if you need to be able to transparently switch between storage providers. But without the simpler static methods, everybody would be forced to create an instance every time they had to do a simple thing like saving a file. It would be really a bad design trade-off to optimize for an extremely rare edge case, and making everything more convoluted for the common case. Same with Console - you can easily create your own wrapper if you need to abstract or mock the console, but quite often you use the console for quick solutions or debugging purposes, so you just want the simplest possible way to output some text.

Furthermore, while "a common interface with multiple implementations" is cool, there is not necessarily a "right" abstraction which unifies all the storage providers you want. You might want to switch between files and isolated storage, but someone else might want to switch between files, database and cookies for storage. The common interface would look different, and it is impossible to predict all possible abstractions users could want. It is much more flexible to provide static methods as the "primitives", and then let users build their own abstractions and wrappers.

Your Math example is even more dubious. If you want to change to arbitrary precision math you would presumably need new numeric types. This would be a fundamental change of you whole program, and having to change Math.Max() to ArbitraryPrecisionMath.Max() is certainly the least of you worries.

That said, I agree that stateful static classes are in most cases evil.


问:
Adding new features to software while keeping the same database
My little team is just a few days from launching our web page. Its another user content type page so a lot of our data will be about user preferences and behavior. We decided to get it out of the door as soon as minimal functionality is ready and tested but have grand plans of more stuff to be implemented while its live.

These things will most likely include new columns in data tables and maybe even complete reinterpretations of existing ones. Those kinds of changes will likely take a long time to implement and we'd like to keep the downtime to a minimum. We think we have it figured out by

creating a parallel database with the new structure

swapping the software to a transitory one that would duplicate new user submissions to both the old and the new database while still using the old database

simultaneously converting data from the old db to the new one

swap the software to the final release that only uses the new db

discard the old db

This would result in 2 very small downtimes (less than a second really) separated by whatever amount of time it takes to complete the transfer of old data (likely being throttled and done at low activity times to not disturb the response performance).

We have all our db access methods set up to make whipping up the transitory version as easy as possible, sounds like a good plan? How do the big guys(like google or facebook) do it?

We're using asp.net + MariaDB if that matters.

答:
For telecoms customers where downtime directly affects revenue and multicomponent migrations are often done in a one night process, we have used versioned table names and background scripts that replicate and convert the data prior to a cutover. That way testing can be done against the new version of the table which has a complete set of data against whict to test. At the same time, queues can drained and the software can bounced to immediately start using the new version of the table at the time of cutover.

It also allows a relatively easy backout if something was missed and the software needs to be reverted because the data is all there. Once the migration has been confirmed successful, the old table can be dropped.


问:
How to design a time tracking or activity monitoring application?
I am trying to build something like Manic Time - which is an application that tracks what the user is currently working on. It worked flawlessly on Windows, but doesn't support Linux.

It has mad features, but the core is basically just tracking what the current 'active' window is, it's process, window title etc.

I've been thinking about this problem for some time and here's the Pythonic pseudo-code that I've come up with, but I'm not sure if this is the way to go.

```<code># The script will probably run as a daemon
while True:

    # Get process, window title, etc.
    wnd_details = get_active_window_details()

    # Save the current timestamp and the details to a database (SQLite)
    insert_in_db(current_timestamp, wnd_details)

    # Wait for a second
    sleep(1000)
</code>```

Will executing a write query per second affect performance?

An optimization might be to remember what the previous window details were and write to database only when the window changes (the user has switched to another application) but that will add unnecessary complexity to the code.

Yet another thing to look into might be some sort of hooks or callbacks, so my Python code gets called whenever a Window change occurs (like a new window is created or active window is changed) I guess Windows had something similar to this, but have no idea about Linux.

答: I have found an open source application that seems to be doing EXACTLY what I wanted. https://github.com/gurgeh/selfspy/


问:
AngularJS: structuring a web application with multiple ng-apps
The blogosphere has a number of articles on the topic of AngularJS app structuring guidelines such as these (and others):

http://www.johnpapa.net/angular-app-structuring-guidelines/

http://codingsmackdown.tv/blog/2013/04/19/angularjs-modules-for-great-justice/

http://danorlando.com/angularjs-architecture-understanding-modules/

http://henriquat.re/modularizing-angularjs/modularizing-angular-applications/modularizing-angular-applications.html

However, one scenario I have yet to come across for guidelines and best practices is the case where you have a large web application containing multiple "mini-spa" apps, and the mini-spa apps all share a certain amount of code.

I am not referring to the case of trying to have multiple ng-app declarations on the same page; rather, I mean different sections of a large site that have their own, unique ng-app declaration.

As Scott Allen writes in his OdeToCode blog:

One scenario I haven't found addressed very well is the scenario where
  multiple apps exist in the same greater web application and require
  some shared code on the client.

Are there any recommended approaches to take, pitfalls to avoid, or good sample structures of this scenario that you can point to?

Update - 9/10/2015

One project with an interesting organization strategy is MEAN.JS and its modules folder.

https://github.com/meanjs/mean

https://github.com/meanjs/mean/tree/master/modules

Another example is from the ASP.NET Music Store SPA example.
https://github.com/aspnet/MusicStore
https://github.com/aspnet/MusicStore/tree/master/src/MusicStore.Spa/ng-apps

答:
Here is the design that I work with. I found it useful on two larger projects I built and haven't hit any road blocks so far.

Folder Structure

```<code>your-project/
  apps/
    global.html
    app1/
      index.html
      app1.module.js
      app1.js
      parts/
        foo.js
        foo.html
        ...
    app2/
  libs
    lib1/
      lib1.module.js
      parts/
        directive1.js
        directive1.html
    lib2/
  third-party/
</code>```

Configure your server web framework to find apps/app1/index.html when a request for /app1 comes in. Use friendly URLs  (e.g. the-first-application/ instead of app1/ and map them using your server technology if required.

Your server technology must include global.html in index.html because it contains the vendor includes (see below).

Include the assets required by the respective app in index.html (see below).

Put the ng-app and the root

 in the index.html.

Every app and lib is a separate angular module.

Every app gets a <app-name>.module.js file that contains the angular module definition and dependency list.

Every app gets a <app-name>.js file that contains the modules' config and run blocks, and the routing config as part of it.

Every app gets a parts folder that contains the applications' controllers, views, services and directives in a  structure that makes sense for the specific app. I don't consider sub-folders like controllers/, views/ etc helpful, because they don't scale, but YMMV.

Libs follow the same structure as apps, but leave things out that they don't need (obviously).

Start out with services and directives within the app where they are used. As soon as you need something in another app as well, refactor to a library. Try to create functionally consistent libraries, not just all-directive or all-services libraries.

Assets

I minify both JS and CSS files for release builds but work with unminified files during development. Here's a setup that support this:

Manage vendor includes globally in global.html. This way, the heavy stuff can be loaded from cache when navigating between SPAs. Make sure caching works appropriately.

Per-SPA assets are defined in index.html. This should only include the app-specific files as well as the used libraries.

The folder structure above makes it easy to find the right files for the minification build steps.

Body:
Here is the design that I work with. I found it useful on two larger projects I built and haven't hit any road blocks so far.

Folder Structure

your-project/
  apps/
    global.html
    app1/
      index.html
      app1.module.js
      app1.js
      parts/
        foo.js
        foo.html
        ...
    app2/
  libs
    lib1/
      lib1.module.js
      parts/
        directive1.js
        directive1.html
    lib2/
  third-party/

Configure your server web framework to find apps/app1/index.html when a request for /app1 comes in. Use friendly URLs  (e.g. the-first-application/ instead of app1/ and map them using your server technology if required.

Your server technology must include global.html in index.html because it contains the vendor includes (see below).

Include the assets required by the respective app in index.html (see below).

Put the ng-app and the root <div ui-view></div> in the index.html.

Every app and lib is a separate angular module.

Every app gets a <app-name>.module.js file that contains the angular module definition and dependency list.

Every app gets a <app-name>.js file that contains the modules' config and run blocks, and the routing config as part of it.

Every app gets a parts folder that contains the applications' controllers, views, services and directives in a  structure that makes sense for the specific app. I don't consider sub-folders like controllers/, views/ etc helpful, because they don't scale, but YMMV.

Libs follow the same structure as apps, but leave things out that they don't need (obviously).

Start out with services and directives within the app where they are used. As soon as you need something in another app as well, refactor to a library. Try to create functionally consistent libraries, not just all-directive or all-services libraries.

Assets

I minify both JS and CSS files for release builds but work with unminified files during development. Here's a setup that support this:

Manage vendor includes globally in global.html. This way, the heavy stuff can be loaded from cache when navigating between SPAs. Make sure caching works appropriately.

Per-SPA assets are defined in index.html. This should only include the app-specific files as well as the used libraries.

The folder structure above makes it easy to find the right files for the minification build steps.


问:
Server-Side Development for Android Developers
I'm a Java/Android Developer, I recently graduated college and currently working as a Jr. Android Developer. I feel very comfortable around Android and Java in general, also I'm always reading and exploring new things. Lately a colleague advised me to look into server-side development as well.

After doing some research online, the following options came-up:

Java EE (JAX-RS for example), it probably has a much smaller learning curve than the others as I'm already comfortable with Java.

PHP, it seems that PHP is solid and everywhere, the code base is huge, even most Android tutorials that I've seen use PHP for the server-side.

JavaScript (Node.js with Express).

NONE!, just sticking to Android development and improving my skills and knowledge.

Does a mobile developer HAVE to know about server-side development ?? which of the choices above is the way to go ?? (if none of them, please let me know your suggestion(s)).

答: Rather than providing this as career advice, I will say that knowing both the client and server side in development to at least some degree makes you a better programmer because you can learn best practices on what to do and not to do in terms of performance, maintainability, standard approaches etc. Without learning the server side, a programmer is missing out on understanding how to solve several classes of problems.</p>


答:
A couple of points about optimization, first:

the overhead for a virtual method dispatch vs entering a case statement is roughly the same; unless the rest of your code is extremely efficient, it is unlikely to make a noticeable difference.

regarding cache usage, unless the per-role routines are small enough to fit more than one in a single cache line, there will be no difference between the two. Only locality within a cache line matters; for objects larger than a cache line they can be randomly spread over the entire system memory and the cache will be just as effective.

if your class already has virtual members, the first option you present will use less memory. This may result in a lower memory bandwidth and therefore a slight improvement in performance.

Regarding maintainability of the code, however, you seem to have neglected a third possibility, which is to have Role being an abstract class with a doStuff method to which your main (non-virtual) doStuff method can then delegate. You would then create a subclass for each role you have, and a single static instance of each subclass would be shared between each node on your tree with that particular role.

This has all the advantages of your first approach without the cited disadvantage of duplicated code for constructors. Furthermore, as the main class's doStuff will get inlined, it should be exactly equivalent performance-wise to your first approach.

Title: Class design, responsibility granulation, efficiency and performance

Body:
I am in a design dilemma. I have a set of data that can be interpreted in numerous ways, but I cannot really decide how finely grained should it be. To illustrate it with some simple code:

class Base {
    Datamembers data;
    virtual void doStuff() = 0;
};

class T1 : public Base {
    void doStuff() {
        // do T1 stuff
    }
};

class T2 : public Base {
    void doStuff() {
        // do T2 stuff
    }
};

class ...

// ---------------------------

class Master {
    enum Role {
        T1, T2 ...
    };

    Role r;
    Datamembers data;

    void doStuff() {
        switch (r) {
        case T1:
            // do T1 stuff
            return;
        case T2:
            // do T2 stuff
            return;
        case ...
        }
    }
};
```

In reality the number of responsibilities is much higher than 2, currently still in the two digit range, but may grow further. Also, it is not one, but several doStuff()s at play.

In the first case, the data is encapsulated in a base class, and doStuff() is a pure virtual, which each derived class implements. Each responsibility is encapsulated in its own, dedicated type.

In the second case doStuff() is not a virtual function, and the responsibility is stored as an extra member in a Role, and the method switches the role and executes the appropriate code.

The two designs achieve the same goal in a different way, but what are the consequences?

Code maintenance:

The first approach seems cleaner, but even though usually clean implies maintainable, I feel it is kind of fragmented, which may end up reducing the ease of maintenance.

In the second approach all the code for the different responsibilities is crammed in the same place, which may sound dirty, but in many cases having it all in the same place may actually be easier, since all you have to do is scroll rather than jump around different files. But then again... it is still a big wall of code, not obvious in the example code above, but in practice it will be 4-5 digit lines of code. Although there is an easy remedy for that, just implement a function for each case that takes this as a parameter to access the data and force inlining to eliminate the overhead of extra function calls (even if not virtual).

Performance:

In my case, I deal with massive trees of objects, where doStuff() will be executed in tight loops, traversing that tree.

In approach one, the method is virtual, this means I get penalty for the indirection. Also, the code is spread across multiple types, which may end up fragmenting it in memory, reducing cache efficiency.

In the second approach the method is not virtual, and all code is essentially in a single function, which means it will be tighter in memory, and potentially more cache friendly. There is the switching penalty vs the indirection and fragmentation of the first approach.

Memory usage:

Although it is missing from the example code above, the hierarchy does have a constructor with plenty of arguments. Meaning that in the first approach each derived class will have its own constructor with the same parameters, redirecting those to the base class constructor. This, aside from tedious to implement for each type, also adds some code bloat. And there is the overhead of the v-tables. Both the extra constructors and v-tables are avoided in the second approach.

Obviously, I've made some considerations, and had this been a smaller undertaking, I'd do both and compare. But it is actually a lot of work and it is not really possible, so I'd appreciate some experienced opinions - are my considerations valid, am I missing something, is there another, better way to do it?

As it seems the scale of the problem isn't entirely clear, I will just summarize it. I am talking about an object tree, potentially tens of millions of nodes big, a responsibility hierarchy of currently 30 roles, expected to grow to over 100. Right now I have approximately about 200 lines of code per role, although the number varies. Parallelization is not an option, since node result depends on parent or children nodes results, so I am limited to a single thread traversing the tree, currently processing 200-300k nodes a second.


问:
How to unit test method that returns a collection while avoiding logic in the test
I am test-driving a method that is to generate a collection of data objects. I want to verify that the properties of the objects are being set correctly. Some of the properties will be set to the same thing; others will be set to a value which is dependent on their position in the collection. The natural way to do this seems to be with a loop. However, Roy Osherove strongly recommends against using logic in unit tests (Art of Unit Testing, 178). He says:

A test that contains logic is usually testing more than one thing at a time, which isn't recommended, because the test is less readable and more fragile. But test logic also adds complexity that may contain a hidden bug.

Tests should, as a general rule, be a series of method calls with no control flows, not even try-catch, and with assert calls.

However, I can't see anything wrong with my design (how else do you generate a list of data objects, some of whose values are depended on where in the sequence they are?—can't exactly generate and test them separately). Is there something non-test-friendly with my design? Or am being too rigidly devoted to Osherove's teaching? Or is there some secret unit test magic I don't know about that circumvents this problem? (I'm writing in C#/VS2010/NUnit, but looking for language-agnostic answers if possible.)

答:
I'm adding a new answer because my perspective is different from when I wrote the original question and answer; it doesn't make sense to mesh them together into one.

I said in the original question

However, I can't see anything wrong with my design (how else do you generate a list of data objects, some of whose values are depended on where in the sequence they are?—can't exactly generate and test them separately)

This is where I went wrong. After doing functional programming for the last year, I now realize that I just needed a collection operation with an accumulator. Then I could write my function as a pure function that operated on one thing and use some standard library function to apply it to the collection.

So my new answer is: use functional programming techniques and you will avoid this problem entirely most of the time. You can write your functions to operate on single things and only apply them to collections of things at the last moment. But if they are pure you can test them without reference to collections.

For more complex logic, lean on property-based tests. When they do have logic, it should be less than and inverse to the logic of the code under test, and each test verifies so much more than a case-based unit test that the small amount of logic is worth it.

Above all always lean on your types. Get the strongest types you can and use them to your advantage. This will reduce the number of tests you have to write in the first place.

ContentLicense: CC BY-SA 4.0


问:
Is it still an antipattern if we log an exception message and throw a different exception?
Our webapplication is using an ExceptionMapper to map some exceptions to Response. We log the exception messages before throwing a new exception as follows:

```<code>catch (SomeException ex) {
  LOG.error(ex.getMessage());
  throw new MyException(ex.getMessage());
}
</code>```

We are not re-throwing the same exception, so my question is if this would be considered a Log and Throw antipattern. And thus, would it be better to remove the logging in similar places and move them to the several ExceptionMapper classes as follows:

```<code>@Provider
public class MyExceptionMapper implements ExceptionMapper<MyException> {

  // bla bla

  @Override
  public Response toResponse(final MyException ex) {
    LOG.error(ex.getMessage());
    return Response.status(400).entity("something").build();
  }
}
</code>```

答:
Your code actually sports not one, but three antipatterns:

log and rethrow;

rethrow without wrapping the original cause;

log only the message and not the stacktrace (this is the worst one).

If you followed the best practice to:

not catch at all (let the exception propagate on its own);

if forced to catch a checked exception, wrap into unchecked and rethrow;

never log anything except at the top level;

log the entire exception stacktrace with log.error("Error occurred", e);

then you would not face any dilemmas, including your current one, because the logged stacktrace would include all the wrapped exceptions as well.

OwnerDisplayName: Diyarbakir

OwnerDisplayName: Marko Topolnik


答: I think any programmer who does mobile or web app development should do server side work. As Jason already said, it should make better at what you do. But to me, a big reason to do it is so you can debug and fix your own issues. If all you ever do is client side work, then you are at the mercy of others to do the server-side work. You won't have a choice in some situations, and you'll have to deal with a black box. But where all the development is in-house, try to at least get access to the server side so you can be more efficient.</p>

Title: Server-Side Development for Android Developers

Body:
I'm a Java/Android Developer, I recently graduated college and currently working as a Jr. Android Developer. I feel very comfortable around Android and Java in general, also I'm always reading and exploring new things. Lately a colleague advised me to look into server-side development as well.

After doing some research online, the following options came-up:

Java EE (JAX-RS for example), it probably has a much smaller learning curve than the others as I'm already comfortable with Java.

PHP, it seems that PHP is solid and everywhere, the code base is huge, even most Android tutorials that I've seen use PHP for the server-side.

JavaScript (Node.js with Express).

NONE!, just sticking to Android development and improving my skills and knowledge.

Does a mobile developer HAVE to know about server-side development ?? which of the choices above is the way to go ?? (if none of them, please let me know your suggestion(s)).


问:
Multiple Canvas in the Browser -- How to Proceed?
Here's something I wrote ages ago as Java applet.  I want to convert it into JavaScript and run in html5 canvas in the browser. From what I can see, canvas makes the most sense here.  (SVG / DIV techniques aren't so friendly for what I want to do.)  The canvas does allow for mouse drag and visual update, something you need to be able to understand the concepts displayed.

Actually that is four separate applets.  This doesn't translate perfectly well to canvas.  I'm not sure the best way to make this update to JavaScript/Canvas.  I think I have three choices:

One large canvas, divided into fourths.  (The problem here is when you grab and drag the control elements there is nothing stopping that drag from hitting a neighboring technique. I could program internal boundaries, but where is the fun in that? This is my least favorite option.)

Four individual canvases (spelling?), each with its own #id.

four iframes with each canvas as a stand alone in its own html file.  These would all be hosted at the same server.

Anybody got an opinion on best technique to proceed for displaying four functional graphic boxes (and why?)  As I understand it, because I'm asking for an opinion, this question doesn't really fit over at stackoverflow...

答:
I think you'd be best with 4 canvas elements.  That would give each its own context, and allow you to respond to mouse events on them more simply. Plus from the drawing perspective, you won't have to worry about clipping.  iframes would be completely unnecessary here.

By the way, you may want to look into some Canvas based libraries like Fabric.js, as they may make interactions with the graphics easier to do.


问:
DVCS and different versions of the same Product
Let's say that we have an application, with two (or more) different versions. E.g. Professional and Lite editions.

What would be the best way to keep the source under version control (DVCS in particular). The assumption here is that some or most of the code is common, but the Professional edition has some extra code as well.

What would be the optimum way of versioning the code with a system like git or Mercurial?

答:
Whether it is with a DVCS or a CVCS (centralized), the idea is to define two "modules" (set of files) with:

one being the common code

the other being the features for the "pro" version

Then you can organize them as sub-repos (Mercurial) or submodules (Git)

(or "external" with SVN", or...)


问:
Alternatives to cron jobs or other ways of improving scheduled task performance
I'm working on a social networking website where users gain ratings after specific actions taken and proper conditions are met on the tables. The ratings are calculated depending on 'total hours worked' and 'total points gained'. The cron php has an sql that INNER JOINs 7 tables. To keep this up-to-date we run a cron to check the table entries once a day to update values.

This is just one cron of 15 in total. Some of them only sends emails with Mandrill, one sends a newsletter and there will be more newsletter in near future.

Currently, when there is an active event on the site (it's kind of an event networking site but with administration and onsite management features), around 2000+ users and their data are involved in these processes. But there will be more and more soon.

We had two server crashes that really ruined the server because of two crons before, one of them is the one which updates user ratings. The second cron that was having problems was updating the total hours a user worked in two tables. We will need to have even more scheduled tasks for other data like these to be updated automatically.

After some research I've found The Fat Controller which is -

A parallel execution handler, used to repeatedly run other programs, usually scripts, a bit like CRON. It was designed to handle the execution of scripts which perform background processing for websites which generally need to repeat, react to how much work there is to do at any one time or run as a daemon.

(As stated here - http://fat-controller.sourceforge.net/index.html )

Any ideas on how the performance of running 5 php scripts previously used as cron jobs would be compared to running them as cron jobs? Also, some of the cron jobs we currently have need to run in specified hours/days (i.e send mail to a user - not weekend - at 9.30 etc.) is this possible with a tool like the fat controller? Or is it possible to set the task to run once a year?

Any recommendations on the best way to handle this kind of processes would be really appreciated.

答:
I've always liked cron.  I think it's easy to understand, and (like most *nix apps) it only does one thing - launch scripts when the current date/time matches a pattern.

What those scripts do, however, is a completely different story.  If they are well written and tested, they can be awesome - critical components of your day-to-day operations.

If poorly written, they can (like any app) can bog the system or worse.

Most of the cron replacements that I've seen were written to be easier to use, not to be "more stable" or "more performant".  cron itself is very stable and since it doesn't do any heavy lifting, performance really isn't an issue.

If I were you I'd take a good look at the scripts you're asking cron to run and I'd also look at your database.  You might have some expensive database calls, be missing an index, etc.


问:
Is it okay to return the "wrong" HTTP status code in order to show a more user-friendly error page?
Located here, RFC 7231 is all about the HTTP protocol (status codes)

I'm developing an in-house web application for my university's department and I'm at a (probably unimportant) impasse.

I'm reducing users' access to resources based on role (student, faculty, staff) and have it working just fine. However, when a user attempts to access a resource and is denied access, the HTTP specification says that I should use a 403: Forbidden, marking that the user does not have access to the resource, or "hide" the resource by returning a 404: Not Found.

I could do that, but I want to make the site a bit more user-friendly and instead redirect them to their previous page with a flash (NOTE that I did not say 'Adobe Flash'; see comments) message instead of having a 403/4 page with links back to different parts of the site. I think it's a pretty UX-friendly feature.

However, that would involve redirecting via a 303: See Other, which more or less violates the standard.

Obviously it really doesn't matter as this is an in-house project for a lab at my university and nobody here is going to be complaining, but imagine this site were under the Google umbrella.

So, is it bad practice to deviate from the [HTTP] standard like this?

Edit: There seems to be a lot of confusion, so let me give an example:

User with id = 1 attempts to access mydomain.com/projects/2 through their profile (by clicking a link). User 1 is the owner of project 2, so the server accepts the requests and renders the page with the project.

User 1 then types mydomain.com/projects/3 into the URL. User 1 is not the owner of project 3. Therefore, when the request gets to the server, the server denies it.

Now, there are two options:

1.) The server redirects the user to /projects/2 (their previously visited page, which would of course always be a valid one) with a helpful notification at the top such as "Sorry, but you do not have permission to access XXXX" or

2.) The server renders a 403: Forbidden page, or a 404: Not Found page, which would probably have that same text (if it was a 403), and maybe even a JavaScript-enabled "back" link

Personally, the redirect seems way more user friendly to me, and eliminates seeing an unfriendly page like 403: Forbidden and having to click a "back" button. However, doing the redirect goes against the HTTP protocol standard.

答:
Consider that returning a 403 may itself be leaking privileged information, i.e. how many total projects exist and how frequently they are being added.

I agree with the suggestions to redirect the user to a helpful list of what they are allowed to see. I would not reveal that they have located something that they are not allowed to access.

This may seem paranoid, since the leaked information may indeed be trivial. However, this actually makes the problem worse. People tend to value what they can measure. If they have poor metrics, they will make poorly informed decisions.

Consider scenarios such as:

A department's productivity being measured by the number of projects that exist or the rate they are being added.

A faculty member attempting to evaluate their position in the pecking order based on the percentage of total projects that they are allowed to access.

A faculty member attempting to determine if they are being sandbagged by the rest of the department based on a proliferation of new projects that are being initiated without their involvement.

Too paranoid for an academic setting?

Moreover, I think it is good to make a habit of never allowing a system to reveal information that people do not need to know. If there is a way to abuse it (and there usually is), some clever person will figure it out.

In the same vein, it would be wise to identify the projects by something other than a predictably incrementing number.