问:
Distributed long-running jobs

Scenario

Various types of files containing customer's data are being uploaded via FTP, and a POST is being made to a REST API to update a database table, which tells the system (a web app) which of those files are new.

Users of the web app are notified of those new files, and can choose to import them into the app's db at any time. They can only import them from the oldest to the newest, but can freely choose to import only a subset of them (don't ask me why, this is a requirement and it also happens to be a key aspect of the issue I'm facing).
When the user chooses which files to import, said table gets updated with a timestamp which signals to the system that the file has to be imported.

A Windows service queries the table every X seconds and, if it finds files flagged with the timestamp, retrieves all the enabled files for the customer with the oldest one, sorts them by creation date (which is embedded in the name of the file) for good measure, and imports them. Then the process repeats for the next customer with enabled files, and so on and so forth.

Problem

Now, importing those files can take a rather long time, with the net result that import requests queue up but, being processed serially, can take hours to be fulfilled. Since it's accounting and payroll data we're talking about, customers aren't willing to wait that much.

We definitely need to speed things up.

What can't be done

The low-hanging fruit would appear to be attempting to parallelize the import of individual files belonging to the same customer, but that can't be done, because a file contains data about a specific month and, in many cases, e.g. July's file has to be imported before August's. In other words, most the file types we're dealing with have a temporal dependency on one another (and if they don't now, they might in the future).

What could be done

Another idea is to keep a queue of the import requests, have some workers available, and parallelize the import by customer.

But here comes into play the requirement I wrote about before: the customer, while constrained to make individual import requests in chronological order, can choose to only import a subset of them.

Say Alice sees there are 6 new files to import, and decides to import 3 of them. Ok, the background job gets queued, worker X is available and grabs the task and executes it. But let's say that Alice decides to import the other 3 files while worker X is still busy. The request gets queued and worker Y, who happens to be available, starts to work on those 3 files.

What would happen is that data could be imported in the wrong order, leading to all sorts of problems.

What I need, I guess, is a way to somehow say

If job A is currently being executed for customer X, a new one B for the same customer X needs to be kept in the queue until A is done. If, however, customer Y creates a new job C, it can be picked up as soon as a worker frees up".

But why don't you make the import faster?

It's a huge and complex piece of software and, while I've been able to make it faster, it has to execute a lot of IronPython scripts, which implies a great overhead. It could for sure be made faster, but from the profiling I did, script execution is the chief bottleneck, and we simply can't afford to move all that logic to C# (even because it's heavily customized for each customer).

Then why don't you make the import itself parallel?

While it's true that the import process is typically composed of many multiple steps, some of those steps again have a temporal dependency, meaning that some of them need to be executed first (e.g. I can't import data about payrolls if new hires haven't been imported first).
I went through the trouble of making it transactional, so interrupting any of those steps won't actually do much harm, but that was very little fun. Making it parallel would require extensive changes to the code, and I would prefer to leave it untouched, as it's pretty hairy and without tests whatsoever.

Why don't you just use Hangfire?

Because Hangfire can't handle dependencies between tasks. In the future I will have to enqueue tasks that don't have absolutely anything to do with one another, and in that case I will be able to use Hangfire.

Ok then...

It's clear that things could, in theory, be parallelized at one or more of these three levels:

customer, which owns

files, each of which takes many simpler

steps to be imported

All of those three levels have the same problem: temporal coupling. If they didn't, things would be that much easier.
The third would imply changes to the import code, which I would rather not do. It takes a list of files to import, so I would much rather parallelize at the first two levels, the easiest being the customer.

My questions

Assume I'm going to use ZeroMQ/RabbitMQ or something of that sort. I could store the IDs of the files to import and enqueue them.

But:

how do you build one of those workers, anyway? Is it going to be a thread spawned by my already-existing Windows service which sits there idle polling the queue until there's a task to execute?

could such a design scale across machines, should the need arise to throw more hardware at it?

if that's the case, I guess I should deploy a new instance of my service on the other machines, right?

how do you establish dependencies between jobs?

how can I see if a job for customer X is already executing and, in that case, leave any other job for the same customer in the queue? Sounds like writing a flag to a database table would lead to race conditions.

I guess I should take measures to somehow take and persist a snapshot of the queue every X minutes, in case the server goes down or needs to be rebooted.

Bear with me, I never had the chance to do any distributed programming, and reading here and there I only found vague explanations.

Sorry about the length of this question, but I wanted to narrow down the specific scenario.

答:
how do you build one of those workers, anyway? Is it going to be a thread spawned by my already-existing Windows service which sits there idle polling the queue until there's a task to execute?

how can I see if a job for customer X is already executing and, in that case, leave any other job for the same customer in the queue? Sounds like writing a flag to a database table would lead to race conditions.

With a per-customer worker, the worker itself is pretty straightforward since it just contains a running job and a FIFO queue of items to process. On the .NET platform, the worker could be represented by:

ActionBlock from the TPL DataFlow library

MailboxProcessor from F#

Task + Queue<T> + locking (on enqueue, dequeue, and task complete)

Note that there does exist ConcurrentQueue<T>, but in this type of code, the benefits are negligible in my experience. It's concurrent for the queue, but you still have to lock to update your own statistics variables and then the usage pattern gets uncomfortable.

A number of other more basic components like threads, wait handles, etc. (not recommended).

The first 2 components are pre-built to take care of only executing one job at a time while queuing up the others.

You can then use a Dictionary<CustomerId, Worker> to represent what's happening overall.

I actually built this in F# using MailboxProcessors. It takes care of keeping an agent per whatever criteria you can pull from the message like CustomerId. It cleans up idle agents automatically.

could such a design scale across machines, should the need arise to throw more hardware at it?

Scaling across machines can be done in various ways. Likely what you will want is partitioning. The easiest way is by hashing on some request value to decide which server to send to. For instance, if your customer ID is an integer (or can be consistently reduced to one... e.g. with GetHashCode) and you have 3 machines processing tasks, you can use a simple modulus to decide which customer should go to which machine. machineNumber = customerId % 3. CustomerId 1,4,7, etc will always go to machine 1. However, if Machine 1 goes down, 1/3 of the customer requests will not get processed until it comes back up. Since these are long running imports anyway, that's likely not a big deal. The load will also not be distributed evenly, since there are usually some customers who are heavier users. Again, probably not a huge deal. Measure to make sure.

Another way that is resilient to failure is to use a distributed directory. It  keeps track of which node currently owns which customer. Project Orleans uses a mechanism like this. It allows for nodes to fail and customers to be transitioned to another node. Before allocating a new customer on a node, you can also query the node to see which is the least loaded. However, I'm not aware of a pre-built component for this purpose, and building it yourself is perilous to your time.

if that's the case, I guess I should deploy a new instance of my service on the other machines, right?

Correct. Having partitioning be separate from the worker, the same worker service code could be running on all servers. The worker service is ignorant of how it is partitioned.

how do you establish dependencies between jobs?

The dependency you've described so far is only temporal, and your application prevents the user from submitting files out of order. So I am not understanding your need here. The worker components described above process jobs one at a time in order. So long as you submit them in temporal order, you're good.

When I have done dependencies in the past (example: if FileTypeB and FileTypeA are both in queue, then FileTypeA must be processed first), I was doing a custom worker where I kept multiple data structures to represent the queue of work. For example, a Dictionary<CustomerId, Tuple<Worker, List<FileRequest>>> (not that I would actually use tuple here). Basically, each customer had a worker and a list of outstanding requests. When the worker came available, I would scan the outstanding requests, determine dependencies, and pick one with the highest priority (the most depended on) -- all under lock. Then the chosen file was processed.

For the F# MailboxProcessor, there also exist the Scan method which can be used to scan the submitted messages until a desirable one is chosen. However, this probably wouldn't work as well for an exhaustive search like a finding the lowest date.

As far as how to represent dependencies, a tree structure is common.

```<code>FileTypeC -> FileTypeB -> FileTypeA
</code>```

I guess I should take measures to somehow take and persist a snapshot of the queue every X minutes, in case the server goes down or needs to be rebooted.

Instead of periodically persisting the queue, just save each request as it comes in and update it as changes phases... example: queued/complete. On restart, you can query the database for incomplete jobs and try to resubmit them to workers. I might add some guards into the worker (especially if it is on a separate server) so that when a job is submitted, it will check its internal state for the same job (e.g. by job id) and respond with a "No, this job is already queued/running/complete." as necessary.

Title: Distributed long-running jobs

Body:
how do you build one of those workers, anyway? Is it going to be a thread spawned by my already-existing Windows service which sits there idle polling the queue until there's a task to execute?

how can I see if a job for customer X is already executing and, in that case, leave any other job for the same customer in the queue? Sounds like writing a flag to a database table would lead to race conditions.

With a per-customer worker, the worker itself is pretty straightforward since it just contains a running job and a FIFO queue of items to process. On the .NET platform, the worker could be represented by:

ActionBlock from the TPL DataFlow library

MailboxProcessor from F#

Task + Queue<T> + locking (on enqueue, dequeue, and task complete)

Note that there does exist ConcurrentQueue<T>, but in this type of code, the benefits are negligible in my experience. It's concurrent for the queue, but you still have to lock to update your own statistics variables and then the usage pattern gets uncomfortable.

A number of other more basic components like threads, wait handles, etc. (not recommended).

The first 2 components are pre-built to take care of only executing one job at a time while queuing up the others.

You can then use a Dictionary<CustomerId, Worker> to represent what's happening overall.

I actually built this in F# using MailboxProcessors. It takes care of keeping an agent per whatever criteria you can pull from the message like CustomerId. It cleans up idle agents automatically.

could such a design scale across machines, should the need arise to throw more hardware at it?

Scaling across machines can be done in various ways. Likely what you will want is partitioning. The easiest way is by hashing on some request value to decide which server to send to. For instance, if your customer ID is an integer (or can be consistently reduced to one... e.g. with GetHashCode) and you have 3 machines processing tasks, you can use a simple modulus to decide which customer should go to which machine. machineNumber = customerId % 3. CustomerId 1,4,7, etc will always go to machine 1. However, if Machine 1 goes down, 1/3 of the customer requests will not get processed until it comes back up. Since these are long running imports anyway, that's likely not a big deal. The load will also not be distributed evenly, since there are usually some customers who are heavier users. Again, probably not a huge deal. Measure to make sure.

Another way that is resilient to failure is to use a distributed directory. It  keeps track of which node currently owns which customer. Project Orleans uses a mechanism like this. It allows for nodes to fail and customers to be transitioned to another node. Before allocating a new customer on a node, you can also query the node to see which is the least loaded. However, I'm not aware of a pre-built component for this purpose, and building it yourself is perilous to your time.

if that's the case, I guess I should deploy a new instance of my service on the other machines, right?

Correct. Having partitioning be separate from the worker, the same worker service code could be running on all servers. The worker service is ignorant of how it is partitioned.

how do you establish dependencies between jobs?

The dependency you've described so far is only temporal, and your application prevents the user from submitting files out of order. So I am not understanding your need here. The worker components described above process jobs one at a time in order. So long as you submit them in temporal order, you're good.

When I have done dependencies in the past (example: if FileTypeB and FileTypeA are both in queue, then FileTypeA must be processed first), I was doing a custom worker where I kept multiple data structures to represent the queue of work. For example, a Dictionary<CustomerId, Tuple<Worker, List<FileRequest>>> (not that I would actually use tuple here). Basically, each customer had a worker and a list of outstanding requests. When the worker came available, I would scan the outstanding requests, determine dependencies, and pick one with the highest priority (the most depended on) -- all under lock. Then the chosen file was processed.

For the F# MailboxProcessor, there also exist the Scan method which can be used to scan the submitted messages until a desirable one is chosen. However, this probably wouldn't work as well for an exhaustive search like a finding the lowest date.

As far as how to represent dependencies, a tree structure is common.

FileTypeC -> FileTypeB -> FileTypeA

I guess I should take measures to somehow take and persist a snapshot of the queue every X minutes, in case the server goes down or needs to be rebooted.

Instead of periodically persisting the queue, just save each request as it comes in and update it as changes phases... example: queued/complete. On restart, you can query the database for incomplete jobs and try to resubmit them to workers. I might add some guards into the worker (especially if it is on a separate server) so that when a job is submitted, it will check its internal state for the same job (e.g. by job id) and respond with a "No, this job is already queued/running/complete." as necessary.


问:
Make: Greedy search for sources vs. a more selective approach... or a hybrid?
I rewrote my makefile a while back to get the benefit of greedily searching for and compiling all sources under some root folder. I'm very happy with it except for the fact that the source libraries I include as submodules tend to have a lot of junk files, e.g. examples, tests, or other sources I don't need... and the make process chokes on these. This is where I am at currently.

So I've come back around to seeing why it can be better to explicitly list sources. The problem with that is that one must always remember to amend the Makefile whenever new sources are added to the tree. That's tedious, and the reason I went initially with the above.

I'm now beginning to wonder if there isn't some method to allow a mix of the two. For example, what if certain folders had a manifest in them that specified certain files to compile, while other folders without a manifest caused the makefile to default to "compile all herein"?

I'm newish to make, so does this sound wise or unwise? Advice on how to set it up? Pitfalls?

P.S. I've considered sparse checkout of submodules to solve the matter without changing my current makefile, but SourceTree does not support this yet, and I rely heavily on it for my use of git.

答:
You are asking for a technological solution to a non-technological problem.

You have these constraints:

All files with certain extensions below a directory are source code files that are part of the project.

Except people randomly make additional files that violates the first constraint.

Except some random additional files do not violate the first constraint.

Your options include:

Teach people to use a naming convention for temporary files.

Ignore the file system and compile the source code control system.

Compile only what compiled last time and ask about new files.

Make lots of different combinations of compilations and keep only the ones that compile.

Figure out how to compile quickly enough that it doesn't matter.   (The Go language has this as a design constraint.)

There is really no way to completely have your cake and fully eat is as well.

ContentLicense: CC BY-SA 3.0


问:
Implementing incremental builds
I'm working on a toy language.

Until now, multiple files have been compiled by merging the ASTs of each file and then running the semantic analysis and code generation phase on the merged AST.

Now I'd like to implement incremental builds. My language does not have header files. Therefore, I need to persist the symbol table between the compilation different files. One way would be to create a SQLite database that holds track of all symbols. When a file changes I could simply update the database.

My question is:

Is this a good idea? What is "state of the art" for compiling incrementally in a language with no header files?

答: Do as you like.  You will probably find that a strict schema will get in your way as you will be changing it constantly.  You are basically serializing the internal state of the compilation somehow and reading it in later.  </p>

Title: Implementing incremental builds


答:
A hybrid solution was indeed the final outcome.

My makefile now contains two separate variables:

a list of directories which may be searched both exhaustively AND recursively for sources to compile - this is used for the main application code over which I have control and which also changes very often, thus I need not explicitly add/remove sources to/from the makefile whenever I add/remove to/from the application's source tree;

a list of explicit, cherry-picked sources - this used for libraries over which I have no control and where I thus must pick individual sources in order to avoid compiling unwanted sources like tests and examples.

...By combining these two into a single list, I now have my full source list to pass to $CC, with greater control and only marginally less convenience.

Title: Make: Greedy search for sources vs. a more selective approach... or a hybrid?

Body:
I rewrote my makefile a while back to get the benefit of greedily searching for and compiling all sources under some root folder. I'm very happy with it except for the fact that the source libraries I include as submodules tend to have a lot of junk files, e.g. examples, tests, or other sources I don't need... and the make process chokes on these. This is where I am at currently.

So I've come back around to seeing why it can be better to explicitly list sources. The problem with that is that one must always remember to amend the Makefile whenever new sources are added to the tree. That's tedious, and the reason I went initially with the above.

I'm now beginning to wonder if there isn't some method to allow a mix of the two. For example, what if certain folders had a manifest in them that specified certain files to compile, while other folders without a manifest caused the makefile to default to "compile all herein"?

I'm newish to make, so does this sound wise or unwise? Advice on how to set it up? Pitfalls?

P.S. I've considered sparse checkout of submodules to solve the matter without changing my current makefile, but SourceTree does not support this yet, and I rely heavily on it for my use of git.


答:
Some defer the symbol checking until runtime. This avoids the problem because each individual source file can be compiled without reference to any other file. Of course, you also lose compile time checking.

Some convert each source file into one or more compiled files. For example, the java compiler will produce one or more .class files for each .java file you ask it to compile. If the compiler needs to reference other files it can just read the .class files from disk.

Others use a single file to store all the needed data. Data pertaining to modules that didn't need to be recompiled can simply be read in from the old copy of the file. I understand that Microsoft's C# compiler did this before support for incremental mode was removed.

Title: Implementing incremental builds

Body:
I'm working on a toy language.

Until now, multiple files have been compiled by merging the ASTs of each file and then running the semantic analysis and code generation phase on the merged AST.

Now I'd like to implement incremental builds. My language does not have header files. Therefore, I need to persist the symbol table between the compilation different files. One way would be to create a SQLite database that holds track of all symbols. When a file changes I could simply update the database.

My question is:

Is this a good idea? What is "state of the art" for compiling incrementally in a language with no header files?


问:
What is the bug workflow on your agile/Scrum team?
What is the bug workflow on your agile/Scrum team?

Here is ours:
- If the bug is related to a story in the current sprint, we fix it.
- If the bug is not related to a story in the current sprint and it is not critical, it is sent to the product owner for prioritization.
- If the bug is not related to a story in the sprint and it is critical, we fix it.

答:
Anything related to work in the current sprint is fixed, we don't even consider them bugs and do not write them up as such. We only consider something a bug if it is part of something we already considered Done.

When a new bug arises, we add it to the backlog and it get prioritized by our stakeholders. If we have time remaining in a sprint, we tend to tackle easier bugs that may have lower priority but are something we can complete in the time remaining.

ContentLicense: CC BY-SA 2.5


问:
Does returning pointer to composed objects violate encapsulation
When I want to create an object which aggregates other objects, I find myself wanting to give access to the internal objects instead of revealing the interface to the internal objects with passthrough functions.

For example, say we have two objects:

```<code>class Engine;
using EnginePtr = unique_ptr<Engine>;
class Engine
{
public:
    Engine( int size ) : mySize( 1 ) { setSize( size ); }
    int getSize() const { return mySize; }
    void setSize( const int size ) { mySize = size; }
    void doStuff() const { /* do stuff */ }
private:
    int mySize;
};

class ModelName;
using ModelNamePtr = unique_ptr<ModelName>;
class ModelName
{
public:
    ModelName( const string& name ) : myName( name ) { setName( name ); }
    string getName() const { return myName; }
    void setName( const string& name ) { myName = name; }
    void doSomething() const { /* do something */ }
private:
    string myName;
};
</code>```

And lets say we want to have a Car object which is composed of both an Engine and a ModelName (this is contrived obviously).  One possible way to do so would be to give access to each of these

```<code>/* give access */
class Car1
{
public:
    Car1() : myModelName{ new ModelName{ "default" } }, myEngine{ new Engine{ 2 } } {}
    const ModelNamePtr& getModelName() const { return myModelName; }
    const EnginePtr& getEngine() const { return myEngine; }
private:
    ModelNamePtr myModelName;
    EnginePtr myEngine;
};
</code>```

Using this object would look like this:

```<code>Car1 car1;
car1.getModelName()->setName( "Accord" );
car1.getEngine()->setSize( 2 );
car1.getEngine()->doStuff();
</code>```

Another possibility would be to create a public function on the Car object for each of the (desired) functions on the internal objects, like this:

```<code>/* passthrough functions */
class Car2
{
public:
    Car2() : myModelName{ new ModelName{ "default" } }, myEngine{ new Engine{ 2 } } {}
    string getModelName() const { return myModelName->getName(); }
    void setModelName( const string& name ) { myModelName->setName( name ); }
    void doModelnameSomething() const { myModelName->doSomething(); }
    int getEngineSize() const { return myEngine->getSize(); }
    void setEngineSize( const int size ) { myEngine->setSize( size ); }
    void doEngineStuff() const { myEngine->doStuff(); }
private:
    ModelNamePtr myModelName;
    EnginePtr myEngine;
};
</code>```

The second example would be used like this:

```<code>Car2 car2;
car2.setModelName( "Accord" );
car2.setEngineSize( 2 );
car2.doEngineStuff();
</code>```

My concern with the first example is that it violates OO encapsulation by giving direct access to the private members.

My concern with the second example is that, as we get to higher levels in the class hierarchy, we could wind up with "god-like" classes that have very large public interfaces (violates the "I" in SOLID).

Which of the two examples represents better OO design?  Or do both examples demonstrate a lack of OO comprehension?

答:
I don’t think it necessarily violates encapsulation to return references to the wrapped object, particularly if they’re const.  Both std::string and std::vector do this.  If you can start altering the object’s internals from under it without going through its interface, that’s more questionable, but if you could already effectively do that with setters, encapsulation was an illusion anyway.

Containers are especially hard to fit into this paradigm; it’s difficult to imagine a useful list that can’t be decomposed into a head and tail.  To some extent, you can write interfaces such as std::find() that are orthogonal to the internal layout of the data structure.  Haskell goes further with classes such as Foldable and Traversible.  But at some point, you’ve ended up saying that everything you wanted to break encapsulation to do is now inside the encapsulation barrier.


问:
Does Continuous Integration imply one monolithic VCS repo?
I am part of a small team which develops several internal apps for our company. We are in a process of becoming more Agile, this especially includes lots of automated testing. Now we are in a situation where, for the one or two apps which we have worked on most recently, we generally make small, reversible changes, run lots of tests, and deploy quite fast with not much human intervention.

I would consider us to be still a long way off from doing 'real' CI. For the one or two apps mentioned above we could probably get very close to it pretty soon. However, I am having a hard time imagining what our setup is going to look like, sometime in the future where we have started using CI for most of our legacy code.

Suppose we have several standalone apps, which all can be deployed independently to different servers. We also have some shared code which is used by many of them, and which we want to use consistently between them. This includes utility functions, code which enforces things like a consistent interface over some parts of all apps, and the ORM definitions for our database (shared across all apps).

I see two alternatives, neither of which looks very easy or elegant:

All code gets merged into one huge repo. Deployment means running all the tests for all the code, integration tests for everything and acceptance tests for all the apps, before rolling out everything in one go. This seems to make deployment a much Bigger Deal than it was before, contrary to the philosophy of CI, which suggests that it should be fast and easy. It also means we don't have any separation between different parts of our codebase, with things which never work together in the same repo forever, just because they both depend on some third thing.

We keep each app and each component of the shared code separate. Deployment means testing the latest version of one component extensively, before 'dropping it in' to a working system consisting of all other components. This seems like a cleaner design. However, it seems to imply that we have to manage dependencies and versioning for all these things. Every integration/acceptance test has to have some foreknowledge of what versions of the other components it is going to be used with and can tolerate. In other words, although every component becomes a lot more reliable, we have to worry loads more about how the pieces fit together, and integration bugs. When the pieces which everything else depends on change, there is potential for breakage eveywhere.

The way out of this dilemma is probably to do a limp version of CI, where the individual components get tested extensively and deployed quickly, but we stick to having big 'flag day' releases of the utilities and database schema. But it really seems like there are lots of benefits of proper CI/CD which we would be missing out on. It isn't an aesthetically pleasing solution, and we aren't looking to satisfy our bosses that we have ticked a box, but we want to improve our working practices.

How should we organize the code for proper CI, and what are the key lessons to learn about planning and designing the architecture, both for legacy and future newly written code?

答:
You could also work with a "lineup" of all your individual component repositories (your 2nd alternative) in a monolithic manner (see my answer to this Q for a possible approach), treating them as a single mega-project, which in a sense would allow you to ignore the issue of managing the dependencies between components (much more complex IMHO).

Every change to the shared code components would be considered to be impacting all the components dependent on them. This would allow you to focus on testing the actual apps instead of doing extensive testing on the shared components (which IMHO can't always make guarantees for the apps using those shared comps).


答:
I find myself wanting to give access to the internal objects instead of revealing the interface to the internal objects with passthrough functions.

So, why then is it internal?

The objective is not to "reveal the interface to the internal object" but to create a coherent, consistent, expressive interface. If an internal object's functionality needs to be exposed and a simple pass-through will do, then pass-through. Good design is the goal, not "avoid trivial coding."

Giving access to an internal object means:

The client has to know about those internals to use them.

The above means desired abstraction is blown out of the water.

You expose the internal object's other public methods and properties, allowing the client to manipulate your state in unintended ways.

Significantly increased coupling. Now you are at risk of breaking client code should you modify the internal object, change it's method signature, or even replace the whole object (change the type).

All of this is why we have the Law of Demeter. Demeter does not say "well, if it's just passing through, then it's ok to ignore this principle."

Title: Does returning pointer to composed objects violate encapsulation

Body:
When I want to create an object which aggregates other objects, I find myself wanting to give access to the internal objects instead of revealing the interface to the internal objects with passthrough functions.

For example, say we have two objects:

```<code>class Engine;
using EnginePtr = unique_ptr<Engine>;
class Engine
{
public:
    Engine( int size ) : mySize( 1 ) { setSize( size ); }
    int getSize() const { return mySize; }
    void setSize( const int size ) { mySize = size; }
    void doStuff() const { /* do stuff */ }
private:
    int mySize;
};

class ModelName;
using ModelNamePtr = unique_ptr<ModelName>;
class ModelName
{
public:
    ModelName( const string& name ) : myName( name ) { setName( name ); }
    string getName() const { return myName; }
    void setName( const string& name ) { myName = name; }
    void doSomething() const { /* do something */ }
private:
    string myName;
};
</code>```

And lets say we want to have a Car object which is composed of both an Engine and a ModelName (this is contrived obviously).  One possible way to do so would be to give access to each of these

```<code>/* give access */
class Car1
{
public:
    Car1() : myModelName{ new ModelName{ "default" } }, myEngine{ new Engine{ 2 } } {}
    const ModelNamePtr& getModelName() const { return myModelName; }
    const EnginePtr& getEngine() const { return myEngine; }
private:
    ModelNamePtr myModelName;
    EnginePtr myEngine;
};
</code>```

Using this object would look like this:

```<code>Car1 car1;
car1.getModelName()->setName( "Accord" );
car1.getEngine()->setSize( 2 );
car1.getEngine()->doStuff();
</code>```

Another possibility would be to create a public function on the Car object for each of the (desired) functions on the internal objects, like this:

```<code>/* passthrough functions */
class Car2
{
public:
    Car2() : myModelName{ new ModelName{ "default" } }, myEngine{ new Engine{ 2 } } {}
    string getModelName() const { return myModelName->getName(); }
    void setModelName( const string& name ) { myModelName->setName( name ); }
    void doModelnameSomething() const { myModelName->doSomething(); }
    int getEngineSize() const { return myEngine->getSize(); }
    void setEngineSize( const int size ) { myEngine->setSize( size ); }
    void doEngineStuff() const { myEngine->doStuff(); }
private:
    ModelNamePtr myModelName;
    EnginePtr myEngine;
};
</code>```

The second example would be used like this:

```<code>Car2 car2;
car2.setModelName( "Accord" );
car2.setEngineSize( 2 );
car2.doEngineStuff();
</code>```

My concern with the first example is that it violates OO encapsulation by giving direct access to the private members.

My concern with the second example is that, as we get to higher levels in the class hierarchy, we could wind up with "god-like" classes that have very large public interfaces (violates the "I" in SOLID).

Which of the two examples represents better OO design?  Or do both examples demonstrate a lack of OO comprehension?


问:
Does Mono have a place in the enterprise world?
For enterprise windows-based solutions, .NET is the best choice sometimes. How is Mono looked at by the enterprises who have to use Linux (or rather prefer to use Linux) ? Assuming that the developers aren't a problem and they are familiar with .NET/Mono and other possible competitors such as Java.   </p>

<p>Would a medium/large company run Mono on their servers as opposite to technologies such as Java? Do you know of any such company ? </p>

答:
I think the main concern many corporates would have is the license issues between Mono and Microsoft. My understanding is that while Microsoft have formally agreed that Mono can use the core .NET technologies, but outside that, including about of very commonly used stuff, is more of a grey area legally with Microsoft stating no firm position on it one way or the other.

That obviously leaves open the possibility that down the line they'll demand license fees or just sue for patent violation. It's unlikely that either would happen from the way they're behaving right now but most corporates don't like that sort of uncertainty, particularly not when it comes with potential liabilities and costs attached.


问:
Performance concern in object oriented languages
I recently moved into web development using ASP.NET MVC. The language I use is C#. Having considerable experience in C makes me look for optimized coding standards (memory, efficient data structures and speed of the loops to name a few).

But my peers who have years of experience in this domain do not seem to take into account all these. So my question is, does the code need to be optimized when working with such languages as the servers we will be deploying these applications on has huge memory and processing speeds? Or is ignoring them a bad practice?

答:
In most business application development the focus is on delivering functionality and writing code as cleanly and maintainable as possible in order to avoid bugs and make it easier to react to business changes and to deliver new functionality in the future. Performance optimizations usually have a cost in development time and in more complex code, which is then harder to modify later. Therefore optimizations can be counterproductive.

The usual rule of thumb is to only optimize if there is an actual observable performance problem, rather than optimize "by default" as is more common in systems development.


答:
The culture of programming has changed considerably since the times that C was the language of choice and the hardware was so wimpy that C was actually a necessity.

Luckily, we do not have to worry about optimization so much nowadays.  The general rule is to never optimize unless you have a good reason to do so.  And in order to have a good reason, you have to:

Have a pre-established performance requirement for your product, something like "server response time must be less than 200 milliseconds".  (Vague requirements like "as fast as possible" are generally frowned upon.)

Measure the actual response time of your server and actually witness that it is failing to meet the requirement.

Exhaust all options of meeting the requirement by reconfiguring your system to make it work more optimally.  (You would be surprised.  Some people don't know the difference between running a web server in debug mode and in production mode.)

Exhaust all options of meeting the requirement by buying better hardware and/or more hardware.  Hardware nowadays generally costs far less than developers' salaries.

Throw the profiler on your system and determine that the bottleneck is in fact in code that you are responsible for and have the power to change.

Exhaust all options for algorithmic optimizations. (Introduction of a caching layer somewhere; restructuring code so that something happens asynchronously rather than synchronously; etc.)

Then, and only then, is it advisable to try your luck by tweaking code to make it work more optimally.  And as you might imagine, we hardly ever reach this stage.

Title: Performance concern in object oriented languages

Body:
I recently moved into web development using ASP.NET MVC. The language I use is C#. Having considerable experience in C makes me look for optimized coding standards (memory, efficient data structures and speed of the loops to name a few).

But my peers who have years of experience in this domain do not seem to take into account all these. So my question is, does the code need to be optimized when working with such languages as the servers we will be deploying these applications on has huge memory and processing speeds? Or is ignoring them a bad practice?


问:
Is it always wrong to optimize before profiling?
I see the mantra of "profiling before optimization" repeated again and again here, on SO, and elsewhere. Although I certainly use profiling tools, I'm only occasionally surprised by the results. It seems like, as often as not, the profiler is just giving the same information that you could reasonably deduce by knowing the likely execution path of your program, understanding how your architecture works, and having a good idea of what optimization techniques the compiler can employ for you.

Because of this, I generally find that when I'm developing, I see areas of the code that I can sense are going to be bottlenecks (often times when writing the code I will think to myself "hey, this is going to be a critical part of the code, and needs to be fast, but I will use a slower implementation first to prove the concept, then optimize it later") and I just go ahead and optimize these areas before I bother doing much profiling.

Is this really such a bad practice, or is it just the natural result of gaining experience doing optimization?

答:
The critical distinction is between:

The optimized code is as simple or simpler than the un-optimized.

The optimized code is more complex (and therefore more error prone and harder to modify in the future) than the un-optimized code.

In the first case, sure go ahead. In the second case you have to weigh the investment in development time (including opportunity cost on not using the same time to fix bugs or deliver features) and the future higher cost of maintenance for a more complex solution. You have to weigh this cost against the observable improvements to performance. How will you perform this judgement if you have no idea what the performance cost is? A function might be obviously inefficient, but if it only takes a few milliseconds anyway, a 1000x performance optimization will not provide any value. Not to mention the opportunity cost of not working on an optimization where it actually matters.

Second, your intuition about performance might very well be wrong - and you will never know if you "optimize" before measuring. For example many developers tend to think that say a O(log n) algorithm is faster than a O(n). But you don't know that. The O(n) algorithm might be faster as long as n is below some threshold. What is that threshold? You probably don't know. And what is n actually in your particular program? Is it usually above or below this threshold? How will you find out?

What if you intuition is wrong and you optimization actually makes the program run slower? If you profiled before and after, you realize your mistake and roll the changes back. At worst you wasted some time. If you don't profile you are none the wiser and have caused long term damage to the program.

The really difficult decision is when you can go down different roads in the architecture. Should the network communication be JSON over HTTP(simple) or protocol buffers over TCP (performant)? The problem here is you have to decide up front, before you can even measure performance, if you don't want to waste work by having to change protocol later. In this case you cannot just start out with the simple version and then optimize later when it turns out to be a problem. But you shouldn't just choose the most performant version by default either. You will have to do some educated guesses and projections.

I note you state that profiling "as often as not" gives the same result as you intuition based on your understanding of the program. I take that to mean that you have a 50% success rate in predicting the best way to allocate resources for optimizing. Given the short and long term cost of misapplied optimization, that is not really a very good rate to rely on.


问:
Using own ArrayList in general purpose C libraries, but allowing a different implementation
I've written a couple of libraries since beginning to use C again last year. One thing that still needs addressing in both of these is the correct use of array-based lists - hereafter, ArrayLists.

At present I tend to do one-off array allocations (usually static allocation) based on library constants (defines) that may be tweaked by the user before compile. Usually I use arrays of pointers or small values, so memory lost by allocating excessive sizes is of little concern.

This approach is fine for many cases but not for all. For example, right now I need to implement the Observer pattern as part of one these libraries, and it seems wise to have a dynamically-sized array for the subscriber lists stored on the publisher. So it seems the right time to set up ArrayLists for all cases, replacing my old fixed-size arrays for the most part.

My question is about the details of this as it pertains to general-use libraries. There are many ArrayList implementations out there and it would be naive of me to assume that mine is any better than those others. I do not want to force my implementation on anyone; or, put another way, just because you choose to use my library does not mean I want you to use my naive ArrayList. Not only that, but in many cases, if my library is used in a given project, it would seem natural to be using your own ArrayList implementation across all code in that project, rather than mine for certain stuff, and yours for other stuff - it should be uniform.

Here's how I think I should tackle this in C, using publisher/subscriber as the example:

In the header, declare interface functions to wrap different implementations, so e.g. publisher_add_subscriber() would wrap an add() call for some ArrayList, could be yours, mine, or someone else's: point being that different implementations could be swapped in/out;

ensure that any functions thus implemented using my default ArrayList implementation are not part of the library's main source file, but rather are kept in a separate source file so that it is trivial to reimplement them.

Are there any obvious problems with this assumed course? Suggestions or alternatives?

P.S. FWIW, my C is very object-oriented in nature; perhaps this will better inform your responses.

P.P.S. One obvious answer might appear to be, "Your ArrayList is an implementation detail and so should anyway be hidden behind a suitably opaque interface, thus making it irrelevant." There is some truth to this, except for two things: (1) C exposes all data, and for this reason I like having access to similar data structures across the board when choosing to access internals; (2) one could choose to swap in another ArrayList implementation for performance reasons.

答: I decided to hardcode a fast, robust, third-party arraylist solution in for now.  I may in future go with the approach outlined in my question. Ultimately though, if all ArrayList accesses are hidden behind an opaque interface, it should not matter too much which implementation is chosen.</p>


问:
What is the difference between WCF service and a simple Web service in developing using .NET Framework?
My questions are:

What is the difference between WCF service and a simple Web service in .NET Framework?

What a WCF Service can do which a .NET Web service cant? In other words, what are the limitation of .NET Web services which were overcome in WCF services?

I understand that WCF are REST based and .NET web services are SOAP based. But I need to know more than that. How a developer will make a design decision whether to developer a Web service or a WCF service?

答:
What a WCF Service can do which a .Net Web Service cant?

One of the things we learned (the hard way) was that WCF lets you modify messages before they are transmitted or after they are received. We had an issue where nothing we were able to do to deserialize fault messages was working, so we had to create a message handler to inspect incoming messages for faults and then throw the relevant exception. Sample.


问:
Does this structure satisfy the terms of the Inversion of Control principle?
I'm trying, as a test, to create Dependency Injection from scratch in JavaScript. Consider the following code structure, which outputs an Alert:

Does it satisfy the terms of the Dependency Inversion Principle?

Is it technically Dependency Injection?

and if not, what structural changes are necessary?

-

```<code>function App(name, services, clients) {
    var that = this;
    this.name = name;
    this.services = services || {};
    this.clients = clients || {};
    this.Service = function (func) {
        func();
    }
    this.Client = function () {
        this.dependencies = {};
        for (var i = 0; i < arguments.length; i++) {
            if (that.services[arguments[i]]) {
                this.dependencies[arguments[i]] = that.services[arguments[i]];
            }
        }
    }
}

var fooBar = new App('fooBar');
fooBar.services.alert = new fooBar.Service(function() {
    alert('Yep!');
});
fooBar.clients.alerter = new fooBar.Client('alert');
fooBar.clients.alerter.alert();
</code>```

One of the structural aspects of this structure that I'm not sure about is the fact that in the above code, the client's constructor knows about the code to inject dependencies.

Wikipedia says:

Passing the service to the client, rather than allowing a client to
  build or find the service, is the fundamental requirement of the
  pattern. The client is not allowed to call the injector code.

However, an answer on this site explaining the naming of dependency injection mentions the constructor can inject the dependency:

Let's say you have some sort of "repository" class, and that
  repository is responsible for handing data to you from a data source.

The repository could establish a connection to the data source by
  itself. But what if it allowed you to pass in a connection to the data
  source through the repository's constructor?

Note, the above is shortened (working) code, I omitted code involved in defining services in order to make the code sample simpler (also omitted strict error handling in case of a missing dependency) - the overall structure should be apparently correct or incorrect - that's the focus of the question, not the specific implementation.

答: I'm not quite sure what your code is doing, but I'm pretty sure it's not something I'd recognise as dependency injection. Specifically, it seems as though your client object is responsible for finding its dependencies from a central repository during construction (albeit using a dependency name that's phased as a constructor parameter), which strikes me as closer to the Service Locator pattern than Dependency Injection.</p>


问:
How do things other than services fit into the Dependency Injection pattern?
Wikipedia explains:

A dependency is an object that can be used (a service). An injection
  is the passing of a dependency to a dependent object (a client) that
  would use it.

But studying AngularJS's DI implementation, I noticed it includes many different structures to be injected such as:

service

value

factory

provider

constant

How do these things other than "services" fit into the Dependency Injection pattern?

答:
A Dependency is any piece of logic your code depends on. Its inversion of control because the alternative method of implementing the logic would be

```<code>dependency.Logic(myObject)
</code>```

vs

```<code>myObject.Logic(dependency)
</code>```

the first requires that dependancy knows about and has access to the properties of myObject, where as the second allows you to construct and use myObject in an OOP style way and still keep the dependency logic in a seperate class

Body:
A Dependency is any piece of logic your code depends on. Its inversion of control because the alternative method of implementing the logic would be

dependency.Logic(myObject)

vs

myObject.Logic(dependency)

the first requires that dependancy knows about and has access to the properties of myObject, where as the second allows you to construct and use myObject in an OOP style way and still keep the dependency logic in a seperate class


问:
Is a large static initializer a code smell?
I am extending SimpleExpandableListAdapter in Android. I don't think Android's adapter is implemented very well, in that its constructors have a large number of rather complicated arguments and it has no setters or builder. In my class, most of those arguments are not dependent on the calling class so I want to build them internally. However, the arguments are nested Lists and arrays of ints and strings that need to be built programmatically.

Since nothing can be called before the super constructor, and instance methods can not be called before the super call returns, I currently have several static methods that I call from the super call:

```<code>super(getContext(), initGroupData(), groupLayout, initGroupFrom(), initGroupTo(),
        initChildData(), childLayout, initChildFrom(), initChildTo());
</code>```

I see three ways of handling this: calling static methods like I am now, having a large static initializer that probably calls these same methods to initialize static variables that are then used in the super call, or encapsulating all of these methods into a builder.

I think right now I'm leaning towards the builder, but I'm wondering if there's any better way to handle this.

答:
Having static helper functions to create the constructor argument is a perfectly sane solution, but these functions are limited in what operations they can perform since they must produce exactly one argument each, and cannot communicate with each other.

In the most general case where you want to adapt an interface Constructor(A, B, C) to a more usable interface Constructor(X, Y), you can define a private helper constructor that takes a private ArgumentObject and chains to the existing constructor. The more usable constructor then chains to the helper constructor via a static helper function to create the argument object:

class Constructor {
  // constructor you want to wrap
  public Constructor(A a, B b, C c) { ... }
  // better constructor you are defining
  public Constructor(X x, Y y) { this(createArgumentObject(x, y)); }
  // helper constructor using an ArgumentObject
  private Constructor(ArgumentObject ao) { this(ao.a, ao.b, ao.c); }
  // helper to create the argument object
  private static ArgumentObject createArgumentObject(X x, Y y) { ... }
  private static class ArgumentObject { ... }
}
```

In languages that don't have constructor chaining within the same class (such as C++03), you would have to define an intermediate subclass for the helper constructor.

However, this technique is merely a generalization of your usage of static functions in the constructor arguments. The other solutions you have proposed have various drawbacks which is why I would avoid them unless there is a very good reason to prefer them:

implementing good builders takes a lot of effort for very little value. If the new constructor is simple enough, you can do without a builder. Assuming the class you are wrapping performs solid validation, you could make the argument object public so that arguments can be passed using the Constructor(new Constructor.Arguments {{ foo = 42; bar = baz; }}) idiom.

using static variables that are computed in a static initializer block makes sense for truly static data, but care must be taken to avoid global state. Unless the initialization is very simple, you should be using static functions to initialize these variables in order for the initialization to be testable. Now, the only advantage over directly using static methods is that the values are only computed once and reused for all initializations.

Since your question indicates these initializations could be more complex, using static initializer blocks is a big no-no if testability is important for you. (If it isn't, you have more urgent problems.)

Body:
Having static helper functions to create the constructor argument is a perfectly sane solution, but these functions are limited in what operations they can perform since they must produce exactly one argument each, and cannot communicate with each other.

In the most general case where you want to adapt an interface Constructor(A, B, C) to a more usable interface Constructor(X, Y), you can define a private helper constructor that takes a private ArgumentObject and chains to the existing constructor. The more usable constructor then chains to the helper constructor via a static helper function to create the argument object:

class Constructor {
  // constructor you want to wrap
  public Constructor(A a, B b, C c) { ... }
  // better constructor you are defining
  public Constructor(X x, Y y) { this(createArgumentObject(x, y)); }
  // helper constructor using an ArgumentObject
  private Constructor(ArgumentObject ao) { this(ao.a, ao.b, ao.c); }
  // helper to create the argument object
  private static ArgumentObject createArgumentObject(X x, Y y) { ... }
  private static class ArgumentObject { ... }
}

In languages that don't have constructor chaining within the same class (such as C++03), you would have to define an intermediate subclass for the helper constructor.

However, this technique is merely a generalization of your usage of static functions in the constructor arguments. The other solutions you have proposed have various drawbacks which is why I would avoid them unless there is a very good reason to prefer them:

implementing good builders takes a lot of effort for very little value. If the new constructor is simple enough, you can do without a builder. Assuming the class you are wrapping performs solid validation, you could make the argument object public so that arguments can be passed using the Constructor(new Constructor.Arguments {{ foo = 42; bar = baz; }}) idiom.

using static variables that are computed in a static initializer block makes sense for truly static data, but care must be taken to avoid global state. Unless the initialization is very simple, you should be using static functions to initialize these variables in order for the initialization to be testable. Now, the only advantage over directly using static methods is that the values are only computed once and reused for all initializations.

Since your question indicates these initializations could be more complex, using static initializer blocks is a big no-no if testability is important for you. (If it isn't, you have more urgent problems.)


答:
Everything fits into the Dependency injection pattern.

When ol' Wirth was proclaiming that every Pascal program should begin with

```<code>PROGRAM ProgramName(INPUT,OUTPUT)
</code>```

it was dependency injection that he had in mind, albeit in a very rudimentary form, and we did not even have a word for it yet, so generations upon generations of programmers never understood it, and instead copied it from program to program as some apocryphal words that just had to be written this way for the magic to work.

Explicitly passing the input stream and the output stream to your program instead of having your program rely on some pre-determined, fixed, globally available entities is, in a sense, dependency injection.

Allegedly for the sake of convenience, modern languages offer many globally available features that are at your disposal simply with an #include or a using or an import statement.  These import statements do not import just interfaces and primitive constants;  they also import entire subsystems declared as public static, ready to be used by anyone.

Does your sorting routine sometimes become bored while sorting, and feels the need to check out its facebook account?  Just have your sorting routine import the io.net package from the SDK, and it can now access any gossip all over the entire planet.

Do your web services feel that their job is kind of lackluster and they'd prefer to do something more glamorous while waiting for the remote host to respond?  They can just import java.awt from the SDK, and display some slick 2D graphics for the amusement of whoever happens to walk by the server console at that moment.

All this insanity exists because the concept of dependency injection has not yet gained the traction it deserves.

Everything could, and in my opinion should, be done using dependency injection; instead of your console application's entry point being

```<code>int main( String args[] )
</code>```

it could be

```<code>void main( ConsoleRuntimeEnvironment environment )
</code>```

in which case you would be invoking the environment to get your arguments, and invoking it again to exit returning a result code.

If your application is a graphics application, it could be

```<code>void main( Graphical2DRuntimeEnvironment environment )
</code>```

If it is a web service it could be

```<code>void main( WebServiceRuntimeEnviroment environment )
</code>```

...and so on and so forth.

Whatever services, features, libraries, call them what you like you need, you should be able to obtain by asking your environment for them, and the public static keyword combination should be prohibited for anything but primitives.

But while nobody enforces this style of doing things, nothing prevents you from doing it voluntarily.  You can inject anything as a dependency.  Want some examples that you can use right now in applications that are currently sitting in your hard drive?

Come up with a Logger interface and pass it to all of your classes instead of magically using some omnipresent, omnipotent, and ultimately evil, statically available logging facility.

Come up with a Clock interface and pass it to any code that needs the current time, so that you do not obtain the current time using new Instance(), thus making your time-dependent code testable.

Come up with a FileSystem interface and pass it to any code of yours that needs to deal with files, so that it does not use the new operator to directly instantiate classes from java.io.  This way, your file-manipulation code can be tested using a filesystem implemented in memory, so your tests can be much faster.

And the list goes on.

Title: How do things other than services fit into the Dependency Injection pattern?

Body:
Wikipedia explains:

A dependency is an object that can be used (a service). An injection
  is the passing of a dependency to a dependent object (a client) that
  would use it.

But studying AngularJS's DI implementation, I noticed it includes many different structures to be injected such as:

service

value

factory

provider

constant

How do these things other than "services" fit into the Dependency Injection pattern?

Body:
Everything fits into the Dependency injection pattern.

When ol' Wirth was proclaiming that every Pascal program should begin with

PROGRAM ProgramName(INPUT,OUTPUT)

it was dependency injection that he had in mind, albeit in a very rudimentary form, and we did not even have a word for it yet, so generations upon generations of programmers never understood it, and instead copied it from program to program as some apocryphal words that just had to be written this way for the magic to work.

Explicitly passing the input stream and the output stream to your program instead of having your program rely on some pre-determined, fixed, globally available entities is, in a sense, dependency injection.

Allegedly for the sake of convenience, modern languages offer many globally available features that are at your disposal simply with an #include or a using or an import statement.  These import statements do not import just interfaces and primitive constants;  they also import entire subsystems declared as public static, ready to be used by anyone.

Does your sorting routine sometimes become bored while sorting, and feels the need to check out its facebook account?  Just have your sorting routine import the io.net package from the SDK, and it can now access any gossip all over the entire planet.

Do your web services feel that their job is kind of lackluster and they'd prefer to do something more glamorous while waiting for the remote host to respond?  They can just import java.awt from the SDK, and display some slick 2D graphics for the amusement of whoever happens to walk by the server console at that moment.

All this insanity exists because the concept of dependency injection has not yet gained the traction it deserves.

Everything could, and in my opinion should, be done using dependency injection; instead of your console application's entry point being

int main( String args[] )

it could be

void main( ConsoleRuntimeEnvironment environment )

in which case you would be invoking the environment to get your arguments, and invoking it again to exit returning a result code.

If your application is a graphics application, it could be

void main( Graphical2DRuntimeEnvironment environment )

If it is a web service it could be

void main( WebServiceRuntimeEnviroment environment )

...and so on and so forth.

Whatever services, features, libraries, call them what you like you need, you should be able to obtain by asking your environment for them, and the public static keyword combination should be prohibited for anything but primitives.

But while nobody enforces this style of doing things, nothing prevents you from doing it voluntarily.  You can inject anything as a dependency.  Want some examples that you can use right now in applications that are currently sitting in your hard drive?

Come up with a Logger interface and pass it to all of your classes instead of magically using some omnipresent, omnipotent, and ultimately evil, statically available logging facility.

Come up with a Clock interface and pass it to any code that needs the current time, so that you do not obtain the current time using new Instance(), thus making your time-dependent code testable.

Come up with a FileSystem interface and pass it to any code of yours that needs to deal with files, so that it does not use the new operator to directly instantiate classes from java.io.  This way, your file-manipulation code can be tested using a filesystem implemented in memory, so your tests can be much faster.

And the list goes on.


问:
Exception handling in Python - Am I doing this wrong (and why?)
I've read many questions and articles on exception handling in Python (and in general), but I still think that it's the most confusing thing ever. I ended up doing something like this:

```<code># error class for exceptions specific to this script
class MyError(Exception): pass

# error class for exceptions in MyClass,
# which can be used from outside of this script
class MyClassError(Exception): pass

class MyClass:
    ...
    # class has method that takes a function as a parameter
    def method(self, param_function):
        ...
        param_function(a, b, c, d)
        try:
            some_other_funtion(arg)
        except ValueError, e:
            raise MyClassError("Some message")
        ...

# outside of MyClass
if __name__ == "__main__":

    # read input from console etc.
    # ...

    def function(a, b, c, d):
       # ....
        try:
            # something with a
        except ValueError, e:
            raise MyError("My message:" + a)
        try:
            # something with b
        except ValueError, e:
            raise MyError("My message:" + b)
       try:
            # something with c
        except ValueError, e:
            raise MyError("My message:" + c)
        try:
            # something with d
        except ValueError, e:
            raise My Error("My slightly different message:" + d)

    x = MyClass()
    try:
        x.method(function)
    except (IOError, MyError, MyClassError) as e:
        print e
</code>```

Explanation:

I want the user of the script to just get a user-friendly message if the error is a result of him passing the wrong input.

The messages with which I raise my custom exceptions are user-friendly, and they (in my opinion) contain all the information I want the user to see.

That's also the reason why I'm handling IOError in a similar manner. I expect an IOError only if the user provides a wrong filename, and I want him to see something like "File not found: name of the file" and nothing more.

Why this seems wrong:

In my main function, in function, I'm basically surrounding every line of code with a try-catch, and handling every exception in a similar manner - sometimes the message the user should see is a little different, but mostly the only difference is the parameter in the message.

Questions:

Am I handling my exceptions in the right places?

Am I handling them in the right way?

Is the code duplication that I'm doing in function neccessary?

答:
You can probably simplify your code by centralising the try/except validation
of function args, and the conversion of exceptions to your exception class,
into one or two decorators, which you apply to each of your methods and
functions.  google for python decorators for exceptions, and python decorators to validate args and you'll find stackoverflow examples like
  this
and this.

You often don't need to explicitly validate method arguments as your code is
going to cause exceptions naturally, and this might be good enough, as
you cannot test for all eventualities.

Remember when writing a script, that often that your code will be even more useful
if someone can include it as a library module, so make the main be specific to what a user would like from the command line, but in the library part don't try too hard to obscure where an exception stems from and so on.


答:
I haven't used mono commercially, but I use it privately, because I work in a Windows company, but privately am a Linux user (so I can reuse what I do at work).

Overall, I agree with Miguel de Icaza who says:

25 % of .NET applications work out of the box with mono

another 25% can be made to work within a day or less

further 25% can be made to work within a week

The last 25% require a complete rewrite of the application
  (WinForms/COM)

Mono works fairly well, but there are some problems:

VB.NET support only for .NET <= 2.0

Windows authentication not implemented

WPF not implemented

WCF support incomplete

Entity Framework not implemented and no plans to implement

"ASP.NET Web Parts" not implemented

No COM-interop support

Sybase connection for version 15.5 (latest) doesn't work

Bugs and incompleteness in the C# class library (e.g. XML was buggy in mono < 2.6)

The Linux web-browser control requires GTK#

Then the minor problems:

Windows Forms work, but aren't always rendered properly

MonoDevelop can't design windows forms

MonoDevelop 'step through' debugging doesn't really work

Mono-Service crashes after 5 hours...

Form what I can say:

WebServices function excellently

If you run a WebApplication, it works fairly well (if it doesn't use
  WebParts).

If you run WindowsForms, it won't always look very nice (to say the least).

There is no working equivalent for Microsoft Reporting Service
  (FYIreporting is the closest thing to
  it, but it's slow, buggy and very
  incomplete, plus no activity since
  more than a year)

You will experience problems if you need to create Word or Excel
  documents.

If you want to develop .NET on Linux

You could develop ASP.NET there (debugging & step through works very
  badly)

You can't really develop WinForms on Linux

You need to use GTK# instead of WinForms

In other words:

Mono has it's place in running web applications and WebServices and
  MailServers.

But it's not viable to run WindowsForms applications, you need to
  write applications with GTK#

It lacks a reporting solution and MS file format support (or working
  libraries therefore)

Edit (2015 update):

I wanted to add that by now, 'step through' debugging works excellently, and you can use MonoDevelop to develop web applications on Linux, even with nuGet dependencies. The problem with Excel and Word libraries is also gone, and entity-framework is now open-source. The rest is pretty much "as is" (don't know if mono-service is fixed, but I would hope so).

What has improved as well is that you can now have current packages for your distro, meaning you don't need to wait until the next release of, say Debian/Ubuntu, until you get the latest mono version (without having to compile them yourselfs).  This is a major time-safer.

Also, with the release of Roslyn, VB.NET support should get a lot better in the near future.

Title: Does Mono have a place in the enterprise world?

Body:
For enterprise windows-based solutions, .NET is the best choice sometimes. How is Mono looked at by the enterprises who have to use Linux (or rather prefer to use Linux) ? Assuming that the developers aren't a problem and they are familiar with .NET/Mono and other possible competitors such as Java.   </p>

<p>Would a medium/large company run Mono on their servers as opposite to technologies such as Java? Do you know of any such company ? </p>


问:
The present and future of programming in local languages
Can programming become language-independent? Will we always program in English? Will we ever be able to program in our local languages? Is there any research that addresses the matter? Are there any such developments?</p>

答:
I think it is very hard to program without the english keywords / functionnames and also, without the concepts they bring. Because many of the concepts are learned from English resources. Finding a translation will not make things easier, because you have to learn the concepts anyway. When working with a local flavor of Excel, I couldn't for example get used to something different than 'if'. Because 'if' is across all programming languages and is hardwired into my brain by now. ('If' being a simplistic version. But it goes to memory allocation of cpu cycles - those are domain specific terms so they are new for anyone who doesn't know the domain, and those who know it will know the english terms anyway.)

But I do not think that you are only programming abstract principles. When writing applications or datamodels, you relate to 'the real world', and then it makes sense to pick names that describe what they relate to out there, and sometimes, those concepts are only useful in your own language because they refer (for example) to legal terms.

So some point or other, languages will cross.


答:
Any object can be registered with DI. The various types of registration methods in angular are simply conventions. .service doesn't really have to register a service. Nor does .controller have to register a controller. Angular is targeted at user interface, so its registration methods target predefined types of components which are commonly used in angular.

All of these methods just register a function / object, and it doesn't matter if a service is really registered with .service. The only differences between them (generally) are:

Whether the object is new every time (called the object's lifetime here)

How the object is created

Lifetime

So an object registered with .service is created new only once during the whole application. That same object is kept in memory (with all its current values) and injected every time another component asks for it. Changes made by ComponentA can be seen by ComponentB and vice versa.

But a .controller object is created new every time before it is injected. ComponentA and ComponentB get their own copies of the object. If ComponentA sets a value on the controller, those changes only belong to ComponentA. ComponentB will never see them.

That's the difference in lifetimes. (In other frameworks there are additional lifetimes. Not sure if angular has more than these two.)

Object Creation

.service and .factory are both "new only once", but they differ in how the object is created. For a service, the angular framework uses new to create the object.

```<code>var createdObject = new MyService();
</code>```

But a factory is called as a method.

```<code>var createdObject = FactoryFunction();
</code>```

Other Frameworks

Many other DI containers are used more generically. Instead of having different registration methods to specify a combination of lifetime and creation, they allow you to provide them discretely when you register the object. Consider the following.

```<code>public class WarriorModule : NinjectModule
{
    public override void Load()
    {
        // IWeapon will be created with: new Sword()
        // the same Sword will be used for every IWeapon requested
        // similar to angular's .service registration
        this.Bind<IWeapon>().To<Sword>().InSingletonScope();

        // will be created by calling a function
        // similar to .factory registration
        this.Bind<IWeapon>().ToMethod(context => new Sword()).InSingletonScope();

        // new sword created every time it is injected for IWeapon
        // similar to .controller registration
        this.Bind<IWeapon>().To<Sword>();
    }
}
</code>```

Body:
Any object can be registered with DI. The various types of registration methods in angular are simply conventions. .service doesn't really have to register a service. Nor does .controller have to register a controller. Angular is targeted at user interface, so its registration methods target predefined types of components which are commonly used in angular.

All of these methods just register a function / object, and it doesn't matter if a service is really registered with .service. The only differences between them (generally) are:

Whether the object is new every time (called the object's lifetime here)

How the object is created

Lifetime

So an object registered with .service is created new only once during the whole application. That same object is kept in memory (with all its current values) and injected every time another component asks for it. Changes made by ComponentA can be seen by ComponentB and vice versa.

But a .controller object is created new every time before it is injected. ComponentA and ComponentB get their own copies of the object. If ComponentA sets a value on the controller, those changes only belong to ComponentA. ComponentB will never see them.

That's the difference in lifetimes. (In other frameworks there are additional lifetimes. Not sure if angular has more than these two.)

Object Creation

.service and .factory are both "new only once", but they differ in how the object is created. For a service, the angular framework uses new to create the object.

var createdObject = new MyService();

But a factory is called as a method.

var createdObject = FactoryFunction();

Other Frameworks

Many other DI containers are used more generically. Instead of having different registration methods to specify a combination of lifetime and creation, they allow you to provide them discretely when you register the object. Consider the following.

public class WarriorModule : NinjectModule
{
    public override void Load()
    {
        // IWeapon will be created with: new Sword()
        // the same Sword will be used for every IWeapon requested
        // similar to angular's .service registration
        this.Bind<IWeapon>().To<Sword>().InSingletonScope();

        // will be created by calling a function
        // similar to .factory registration
        this.Bind<IWeapon>().ToMethod(context => new Sword()).InSingletonScope();

        // new sword created every time it is injected for IWeapon
        // similar to .controller registration
        this.Bind<IWeapon>().To<Sword>();
    }
}


问:
Is mutually referencing classes a consequence of bad design?
I am currently working on a small game I did in about ten days. Since I am programming for myself, I am being lazy and going to the easiest solution with (too) little concern about bad code quality.

However I am reaching a point where it begins to matter. I am starting to move booleans that references states into classes etc. and now I wanted to know experienced programmers opinion about mutual referencing objects.

What I mean by mutual referencing objects is this: we have a server which can create multiple games, such as chess games when two clients connect to play. Everytime a player makes a move, his connection handler must update a specific board on the server. After the update, once the board is updated, a broadcast of the new state needs to be done to each client.

I found convenient to have two Client objects in the Board (so Board can Client.send() to both) and a Board in each Client (so Client can call Board.update (move)). For some reason, I'm inclined to think this is generally bad and could be avoided, but what is your opinion ?

Is this fine as it is, or am I missing something here ?

答:
Yes, this can be a problem.

When A depends on B, that means changes in B often require changes in A as well. When A and B both depend on each other, that means changes in either one often require changes in the other. For maintenance purposes, once two classes mutually reference each other, they might as well be a single big class.

Like many software architecture issues, just one mutual reference like the example you describe is unlikely to be a major problem by itself, but if you never remove this mutual reference when it does become a nuisance, it will make many future changes needlessly fiddly. And if you let yourself add more mutual references, you risk turning all of your nice small classes into a de facto god class you can no longer detangle.

As a very, very simplistic example of what I'm talking about, imagine you want to add a spectator mode or an online leaderboard or some other feature that requires sending the board state to "clients" that aren't the other player. Client has to change no matter what since it needs to know how to call SpectatorBoard.update() and Leaderboard.addScore() as well as Board.update(), but because of the mutual reference, Board also has to change so that it can call Spectator.send() and Leaderboard.sendScore(). Now your complex networking logic is scattered across two classes, one of which clearly has no business knowing about it in this much detail.

Without knowing more about your requirements, current code and programming language it's impossible to say what the correct alternative is, but I'm going to guess that the need to send updates "originated" in the Board somehow (perhaps it's the one handling user input), in which case one obvious alternative is to teach the Board to fire an event (assuming your language supports events, a.k.a. the Observer pattern), and then simply have the Client subscribe to that event. Then the Client can remain the only class that needs to know your networking logic.


答:
User input sucks. You can't trust those users to get anything right, and so you've got to handle all kinds of special cases that make your life difficult. Having said that, we can minimize the difficulty with general principles.

Validate early, not often

Check input for validity as soon as it read into your program. If you read in a string that should be a number, convert it into a number right away and complain to the user if it isn't a number. Any rogue data you don't verify at input will make its way into the rest of the program and produce bugs.

Now, you can't always do this. There will be cases where you can't verify the correct properties right away, and you'll have to verify them during later processing. But you want as much verification to happen as early as possible so that you have your special cases around input logic centralized to one location as much as possible.

Use Schemas

Let's consider a function that parses some json.

```<code>def parse_student(text):
    try:
        data = json.parse(text)
    except ValueError as error:
        raise ParseError(error)

    if not isinstance(data, dict):
        raise ParseError("Expected an object!")

    try:
        name = data['name']
    except KeyError:
        raise ParseError('Expected a name')

    if not isinstance(name, dict):
       raise ParseError("Expected an object for name")

    try:
        first = name['first']
    except KeyError:
        raise ParseError("Expected a first name")

    if not isinstance(first, basestring):
        raise ParseError("Expected first name to be a string")

    if first == '':
        raise ParseError("Expected non-empty first name")
</code>```

That was a lot of work just to extract the first name, let alone any other attributes. We can make this a lot better if we can use a json-schema. See: http://json-schema.org/.

I can describe what my student object looks like:

```<code>{
    "type": "object",
    "properties": {
        "name": {
            "type": "object",
            "properties": {
                  "first" : {
                       "type" : "string"
                  }
            },
            "required": "first"
        },
    }
    "required": ["name"]
}
</code>```

When I parse I then do something like:

```<code>def parse_student(text):
    try:
        data = json.parse(text)
    except ValueError as error:
        raise ParseError(error)

    try:
        validate(data, STUDENT_SCHEMA)
    except ValidationError as error:
        raise ParseError(error)

    first = data['name']['first']
</code>```

Checking against the schema verifies most of the structure that I need. If the user input does not match the schema, the schema validator will produce a nice error message explaining exactly what was wrong. It will do so far more consistently and correctly then if I wrote the checking code by hand. Once the validation has been passed, I can just grab data out of the json object, because I know that it will have the correct structure.

Now, you probably aren't parsing JSON. But you may find that you can do something similar for your format that lets you reuse the basic validation logic across the different pieces of information that you fetch.

Title: Exception handling in Python - Am I doing this wrong (and why?)

Body:
I've read many questions and articles on exception handling in Python (and in general), but I still think that it's the most confusing thing ever. I ended up doing something like this:

```<code># error class for exceptions specific to this script
class MyError(Exception): pass

# error class for exceptions in MyClass,
# which can be used from outside of this script
class MyClassError(Exception): pass

class MyClass:
    ...
    # class has method that takes a function as a parameter
    def method(self, param_function):
        ...
        param_function(a, b, c, d)
        try:
            some_other_funtion(arg)
        except ValueError, e:
            raise MyClassError("Some message")
        ...

# outside of MyClass
if __name__ == "__main__":

    # read input from console etc.
    # ...

    def function(a, b, c, d):
       # ....
        try:
            # something with a
        except ValueError, e:
            raise MyError("My message:" + a)
        try:
            # something with b
        except ValueError, e:
            raise MyError("My message:" + b)
       try:
            # something with c
        except ValueError, e:
            raise MyError("My message:" + c)
        try:
            # something with d
        except ValueError, e:
            raise My Error("My slightly different message:" + d)

    x = MyClass()
    try:
        x.method(function)
    except (IOError, MyError, MyClassError) as e:
        print e
</code>```

Explanation:

I want the user of the script to just get a user-friendly message if the error is a result of him passing the wrong input.

The messages with which I raise my custom exceptions are user-friendly, and they (in my opinion) contain all the information I want the user to see.

That's also the reason why I'm handling IOError in a similar manner. I expect an IOError only if the user provides a wrong filename, and I want him to see something like "File not found: name of the file" and nothing more.

Why this seems wrong:

In my main function, in function, I'm basically surrounding every line of code with a try-catch, and handling every exception in a similar manner - sometimes the message the user should see is a little different, but mostly the only difference is the parameter in the message.

Questions:

Am I handling my exceptions in the right places?

Am I handling them in the right way?

Is the code duplication that I'm doing in function neccessary?

Body:
User input sucks. You can't trust those users to get anything right, and so you've got to handle all kinds of special cases that make your life difficult. Having said that, we can minimize the difficulty with general principles.

Validate early, not often

Check input for validity as soon as it read into your program. If you read in a string that should be a number, convert it into a number right away and complain to the user if it isn't a number. Any rogue data you don't verify at input will make its way into the rest of the program and produce bugs.

Now, you can't always do this. There will be cases where you can't verify the correct properties right away, and you'll have to verify them during later processing. But you want as much verification to happen as early as possible so that you have your special cases around input logic centralized to one location as much as possible.

Use Schemas

Let's consider a function that parses some json.

def parse_student(text):
    try:
        data = json.parse(text)
    except ValueError as error:
        raise ParseError(error)

    if not isinstance(data, dict):
        raise ParseError("Expected an object!")

    try:
        name = data['name']
    except KeyError:
        raise ParseError('Expected a name')

    if not isinstance(name, dict):
       raise ParseError("Expected an object for name")

    try:
        first = name['first']
    except KeyError:
        raise ParseError("Expected a first name")

    if not isinstance(first, basestring):
        raise ParseError("Expected first name to be a string")

    if first == '':
        raise ParseError("Expected non-empty first name")

That was a lot of work just to extract the first name, let alone any other attributes. We can make this a lot better if we can use a json-schema. See: http://json-schema.org/.

I can describe what my student object looks like:

{
    "type": "object",
    "properties": {
        "name": {
            "type": "object",
            "properties": {
                  "first" : {
                       "type" : "string"
                  }
            },
            "required": "first"
        },
    }
    "required": ["name"]
}

When I parse I then do something like:

def parse_student(text):
    try:
        data = json.parse(text)
    except ValueError as error:
        raise ParseError(error)

    try:
        validate(data, STUDENT_SCHEMA)
    except ValidationError as error:
        raise ParseError(error)

    first = data['name']['first']

Checking against the schema verifies most of the structure that I need. If the user input does not match the schema, the schema validator will produce a nice error message explaining exactly what was wrong. It will do so far more consistently and correctly then if I wrote the checking code by hand. Once the validation has been passed, I can just grab data out of the json object, because I know that it will have the correct structure.

Now, you probably aren't parsing JSON. But you may find that you can do something similar for your format that lets you reuse the basic validation logic across the different pieces of information that you fetch.

ContentLicense: CC BY-SA 4.0


问:
Do people modify open source code to hack?
I am trying to create an authentication client and server for my project, and I am trying to think of ways on how a potential hacker could modify my code and use it to hack the servers. Do people actually do this often and if so, how do programmers try to get around it in open source projects?</p>

答: People do this sometimes, yes.  For example: http://m.slashdot.org/story/40281. One mitigation is to require cryptographic signing of commits from maintainers.  But you still have to trust your maintainers.</p>


答:
You cannot trust input from the outside world, whether it's messages in your authentication scheme or data files.

Attackers can and will feed you invalid messages to trigger bugs or information leaks in your public-facing endpoints. They will also happily spoof credentials if your (assumedly homebrew) authentication scheme blindly trusts the foreign party.

There is no universal security or trust, all established mechanisms rely on establishing trust relationships up-front (pre-sharing keys) or by deferring to existing trust vendors (certificate authorities for TLS).

The only thing that differs between attackers leveraging your source and attackers reverse-engineering protocols and binaries is that it's slightly easier for them to get a grip on what your program does from decently written source code.

Title: Do people modify open source code to hack?

Body: I am trying to create an authentication client and server for my project, and I am trying to think of ways on how a potential hacker could modify my code and use it to hack the servers. Do people actually do this often and if so, how do programmers try to get around it in open source projects?</p>


答:
I don't think there's much space for Mono there:

Java is already established platform on Linux, with huge community and abundance of enterprise quality libraries and tools, both commercial and open-source. I don't see any reason to pick Mono over Java when starting new project targeting Linux (or Mac), unless there's some specific circumstance (see Walter's answer).


问:
What is the best design principle to add behavior in future to your API?
I'm designing an API. And, this API will be used by client.

Currently I've two behaviors in a class definition. To illustrate the scenario, sample implementation looks like the following-

```<code>public class Session {

    // instance field declaration goes here

    void connect() {
        // method definition goes here
    }

    void submitJob() {
        // method definition goes here
    }
}
</code>```

I want to add more behaviors to this class definition in future to offer more functionality. And, I don't want the client code to change (i.e. My client will keep instantiating this single class and invoke behaviors as per their need.) But, I don't want to modify this class definition. Is there any design pattern and/or principle that can be followed to add behaviors to achieve the goal?

Note: I do not intend to add behaviors at runtime.

Appreciate any explanation and/or guides.

答:
Without knowing much about your actual structure and the behavioral changes you anticipate, I'd say, maybe the Command pattern might be of help.

Basically, instead of declaring lots of different methods in your Session class and therefor changing its API all the time, you'd provide yourself a Command interface and all your actual commands, i.e. what actually needs to be done (and would otherwise be in different methods), go into concrete implementations of Command. Actually, it looks a bit like, your submitJob method is already on its way to receiving commands like those; just add a Command parameter... :-)

```<code>class Session {

    public void submitJob(JobCommand job) {
        job.execute();
    }
}

interface JobCommand {

    public void execute();
}
</code>```

Body:
Without knowing much about your actual structure and the behavioral changes you anticipate, I'd say, maybe the Command pattern might be of help.

Basically, instead of declaring lots of different methods in your Session class and therefor changing its API all the time, you'd provide yourself a Command interface and all your actual commands, i.e. what actually needs to be done (and would otherwise be in different methods), go into concrete implementations of Command. Actually, it looks a bit like, your submitJob method is already on its way to receiving commands like those; just add a Command parameter... :-)

class Session {

    public void submitJob(JobCommand job) {
        job.execute();
    }
}

interface JobCommand {

    public void execute();
}