问:
python - differences between reusable code vs. code for solving specific tasks
Reusable code (ex. libraries and frameworks) and code written to solve a specific task and not meant to be reused as a general tool (for example, code being used only by my 6 person team in a private repository) differ greatly in purpose, so I imagine there could be vastly different strategies, best practices, advice, etc. for each.

In Python, what are the biggest differences between these two types of coding?

Possible Examples: My team ends up writing long functions and we don’t have many classes, which I think makes sense because the code is not meant to be reused by external programmers but rather only to complete a known task. But on the other hand, when I read Python libraries on Github they use plenty of classes, and in talks about writing reusable code in Python they encourage very short methods and functions.

答:
There's reasons to change and there's difficulty of change.

The worst-case scenario in a codebase is when these two have anything close to a proportional relationship, when the hardest things to change have the most reasons to change.

That covers a lot of SE principles (albeit coarsely and oversimplified) in a nutshell.

Difficulty of Change: Efferent Coupling

Code that's reused by a lot of places (ex: the public interface of a widely-used class, the signature and return of a widely-used function), is always going to be difficult to change, since changing their design breaks everything using their design (rippling/cascading need for changes).

For example, at a basic level, this package (design):

... is going to be difficult to change. This package (design) generally wouldn't be anywhere near as difficult to change:

It sounds to me like a lot of your code fits more in the second kind of diagram, and that kind of code tends to be easier to change without breaking external dependencies.

Reasons to Change: Responsibilities

While the worst-case scenario we want to avoid is for a design that's very difficult to change to also have many reasons to change, just about anything benefits from having fewer reasons to change if possible.

One of the things that increases the number of reasons for something to change is when that something is doing more things.

As a blatant example, if you have a single function (or even class) that is responsible for playing music, calculating physics, and drawing stuff to a screen, then that has a lot of reasons to change. Someone might want to improve the way it's drawing stuff to a screen. Another might want to correct a minor glitch in the physics calculation. Another might want the music side of the functionality to handle a new audio format. Some of these might want to change not just the implementation of the function, but the design of the function (and thus break all the code using it).

As a result, it's worth splitting up these concerns into more functions. Each function then tackling a single responsibility will have fewer reasons to change (and also generally be easier to change since only the parts of your code that need audio will use the function that plays music, e.g., and for other reasons mentioned below).

As some point, perhaps your application still ultimately wants to play music, calculate physics, and draw stuff to the screen. But it doesn't have to do this all in one function, let alone one class, let alone one module. You might be able to push these disparate needs all the way towards the main entry point of your application, which just constructs a GUI which calls various functions to play music, calculate physics, and draw stuff to the screen. That's a lot easier to manage than having some function (let alone class) at the core of your system trying to do all this at once.

Another way to look at this is that some part of your codebase will always find reasons to change, provided that your software is evolving and faces new demands and needs and finds new versions to release. The best possible scenario is when the code that needs to change the most is the easiest to change. Often the code that is easiest to change will be simple code, not widely-used, that's just using a bunch of stuff that don't need to change.

Difficulty of Change: Complexity

Typically the basic instability metric covering reasons to change/difficulty of change only tackles coupling between designs and abstracting away details, but I want to get into another broad reason that makes changes difficult even at the implementation level: complexity.

If you have a five hundred line function, it's generally going to be difficult to change in addition to often having more reasons to change (because it's doing so much: something that does more has more reasons to change as covered above).

It's more difficult to change when it has a reason to change because it's complex. It's hard to digest 500 lines of code all at once trying to figure out exactly how the function works in order to make meaningful and valid changes to it. That also makes it difficult to debug as well as even test thoroughly.

So you can also end up with that nightmarish scenario of something that is very difficult to change still having reasons to change if it's complex. The complexity will tend to invite changes, while also making them more difficult to apply.

Reasons to Change: Unreliability

Let's say you had to implement something inevitably complex just at the conceptual level. It involved digesting a 40-page cutting-edge paper on a groundbreaking algorithm riddled with complex mathematics and algorithms and data structures.

After weeks, you finally get what looks like a reasonable implementation. Phew! You give it some basic manual testing from within the application, and it seems to do what it's supposed to do. Yay!

Except your software goes into alpha and, after being exposed to a number of users, they find various edge cases your code didn't handle. Argh!!! Now you have to go back and revisit your original code as well as that complex paper that gave you nightmares in the first place, trying to figure out what's wrong and how to properly solve the issue.

Here even if you organize your functions and classes neatly into designs that have few reasons to change (tackle a singular, clear responsibility), the unreliability of your code (the edge cases you didn't spot in advance) can provide a reason to change (a bug reason). And the difficulty of change can still be tremendous, even if you only need to add a dozen lines of code to correct the issue.

The solution to this is to start testing your code more exhaustively in advance, including automated unit and integration testing which runs the tests you've established each time you commit a change (continuous integration). Problems spotted sooner tend to be less costly to fix, as well as improving your sanity.

Reasons to Change: Details

Designs that mention more details tend to have more reasons to change. If the design of a robot offers a function that tells it to walk to new places, you might write a lot of code to tell it to walk to various rooms.

But what happens when you encounter a new robot design that can allow it to instantly teleport to a room with a molecular teleporter? Now we're screwed two ways. First, we're screwed because all that code telling the robot to walk to a room now has to change to fit the new robot's design which can teleport to a room. Likewise, your new robot can't use the previous design because it can't walk to a room.

The problem here is that the design is mentioning too many details. Walking implies that the robot has legs. All we really needed was an abstract design that can tell a robot to go to a room by whatever means possible. That abstract design would have been compatible with any means of transportation for any model of robot.

So again, a design that mentions lots of concrete details like this has more reasons to change (something very undesirable when we can avoid it). This then takes us into concepts like dependency injection and the dependency inversion principle which might seem a little advanced at this stage (at least the latter). However, a key idea to take out of this is that you want your designs to mention fewer concrete details when they can avoid it. Go to a room, not walk to a room, not fly to a room. By doing that, you shape your code using such a design to be compatible with more underlying changes, and therefore find fewer reasons for such coding using the design to change because there will be fewer reasons for the design itself to change.

Conclusion

So anyway, I hope this little simplified introduction helps a bit. If your designs are not being reused by many places, then that mitigates at least the
difficulty of change part.

But if you take that as an excuse to design monolithic functions or classes, then you are still making maintenance more difficult than it needs to be by increasing both the difficulty of change at another level as well as the reasons to change.

10 smaller, simpler functions that each do one thing can be a lot easier to change, as well as having fewer reasons to change, than one combined huge function. In that case, if we had a reason to change, it might be for one small, simple function, and thus also make the change easier to apply to that one little function.

So software engineering principles don't really change much between reusable code that's used a lot, and between code that's not being used a lot. It's always worth keeping in mind the dynamics that increases difficulty of change and reasons to change. We want to mitigate both.

The practical difference, perhaps, is that the code which is the absolute hardest to change (ex: code being reused all over the place, but is very complex at a basic conceptual level) should seek even harder to find fewer and fewer reasons to change (improving reliability through testing, having fewer responsibilities, abstracting away details, etc. -- the whole shebang).

Yet that doesn't mean you should drop your guard and start ignoring all these engineering principles even for the code that isn't being used by many places.


问:
Refactoring a long method which is based on large number of switch cases
We are using Java as a backend development language.

One year back, we wrote a method which uses switch cases based on Enums values. Since we are continuously adding enum members and according adding cases in the method, the method has grown to very large extent. Currently, we have around 100 enum fields and corresponding number of switch cases.e.g.

```<code> class AClass{

   enum option{ o1, o2, o3...on}

   Value method(Option o){

      switch(o){
        case o1:
          value = deriveValue(p1,p2,p3);
        case o2:
          value = deriveValue(p2,p3,p4);
        .
        .
        .
        case on:
          value = deriveValue(p1,p2,p3);
      }
   }
 }
</code>```

Thus each time a business requirement comes, we add enum and corresponding switch case. Now the method has become too long and looks unmanageable, if we keep on adding the same logic in future.

To clean, we thought of replacing switch case with polymorphism by creating classes for the same, but again, we will end up creating n number of classes.

We are looking for small, simple and manageable solution.

-----------------  Update ---------------------

As suggested, to elaborate more, Enum values are fields names for which a client needs a value. Thus if a client need the value of a new field, we add the field in enum, and the corresponding definition of how to fetch value for the newly added field, by adding the corresponding switch case.

In the switch case, we have a common reusable method e.g. deriveValue() (Please refer to the example given) to which we pass parameters required for deriving the value for the newly added field.

答:
You have more and more options, leading to more and more enum cases, and more and more code implementing something for each option. You can try to do something about the growing number of options - well, maybe you can't.

If you can't prevent the number of options from growing, then a switch statement is just as good as any other method to choose between the different options. And it looks like very little additional code is needed beyond the actual code to evaluate each option. A decent compiler will tell you about missing cases.

If you don't like this advice: One day you reach the point where you worry about simple and maintainable code, and not about violations of some abstract principle that you just heard about at your latest course. Rules are there to be broken unless you are a novice. Now if instead of "this is a violation of the open-closed principle" you could actually give a practical example...

ContentLicense: CC BY-SA 3.0


问:
Throwing an exception inside finally
Static code analyzers like Fortify "complain" when an exception might be thrown inside a finally block, saying that Using a throw statement inside a finally block breaks the logical progression through the try-catch-finally. Normally I agree with this. But recently I've come across this code:

```<code>SomeFileWriter writer = null;
try {
     //init the writer
     //write into the file
} catch (...) {
     //exception handling
} finally {
     if (writer!= null) writer.close();
}
</code>```

Now if the writer cannot be closed properly the writer.close() method will throw an exception. An exception should be thrown because (most probably) the file wasn't saved after writing.

I could declare an extra variable, set it if there was an error closing the writer and throw an exception after the finally block. But this code works fine and I'm not sure whether to change it or not.

What are the drawbacks of throwing an exception inside the finally block?

答:
This is going to be more of a conceptual answer to why these warnings might exist even with try-with-resources approaches. It's also unfortunately not the easy kind of solution you might be hoping to achieve.

Error Recovery Cannot Fail

finally models a post-transaction control flow that is executed regardless of whether a transaction succeeds or fails.

In the fail case, finally captures logic that is executed in the middle of recovering from an error, before it has been fully recovered (before we reach our catch destination).

Imagine the conceptual problem it presents to encounter an error in the middle of recovering from an error.

Imagine a database server where we're trying to commit a transaction, and it fails halfway through (say the server ran out of memory in the middle). Now the server wants to roll back the transaction to a point as though nothing happened. Yet imagine it encounters yet another error in the process of rolling back. Now we end up having a half-committed transaction to the database -- the atomicity and the indivisible nature of the transaction is now broken, and the integrity of the database will now be compromised.

This conceptual problem exists in any language that deals with errors whether it's C with manual error code propagation, or C++ with exceptions and destructors, or Java with exceptions and finally.

finally cannot fail in languages that provide it in the same way destructors cannot fail in C++ in the process of encountering exceptions.

The only way to avoid this conceptual and difficult problem is to make sure that the process of rolling back transactions and releasing resources in the middle cannot possibly encounter a recursive exception/error.

So the only safe design here is a design where writer.close() cannot possibly fail. There are usually ways in the design to avoid scenarios where such things can fail in the middle of recovery, making it impossible.

It's unfortunately the only way -- error recovery cannot fail. The easiest way to ensure this is to make those kinds of "resource release" and "reverse side effects" functions incapable of failing. It's not easy -- proper error recovery is hard and also unfortunately hard to test. But the way to achieve it is to make sure that any functions that "destroy", "close", "shutdown", "roll back", etc. cannot possibly encounter an external error in the process, since such functions will often need to be called in the middle of recovering from an existing error.

Example: Logging

Let's say you want to log stuff inside a finally block. This is often going to be a huge problem unless logging cannot fail. Logging almost certainly can fail, since it might want to append more data to file, and that can easily find many reasons to fail.

So the solution here is to make it so any logging function used in finally blocks cannot throw to the caller (it might be able to fail, but it won't throw). How might we do that? If your language allows throwing within the context of finally provided that there's a nested try/catch block, that would be one way to avoid throwing to the caller by swallowing exceptions and turning them into error codes, e.g. Maybe logging can be done in a separate process or thread which can fail separately and outside of an existing error-recovery stack unwind. As long as you can communicate with that process without the possibility of running into an error, that would also be exception-safe, since the safety issue only exists in this scenario if we're recursively throwing from within the same thread.

In this case, we can get away with logging failure provided that it does not throw since failing to log and doing nothing isn't the end of the world (it's not leaking any resources or failing to roll back side effects, e.g.).

Anyway, I'm sure you can already begin to imagine how incredibly difficult it is to truly make a software exception-safe. It might not be necessary to seek this to the fullest degree in all but the most mission-critical software. But it's worth taking note of how to truly achieve exception-safety, since even very general-purpose library authors often fumble here and wreck the whole exception-safety of your application using the library.

SomeFileWriter

If SomeFileWriter can throw inside close, then I'd say it's generally incompatible with exception-handling, unless you never try to close it in a context that involves recovering from an existing exception. If the code for it is outside of your control, we might be SOL but it would be worth notifying the authors of this blatant exception-safety issue. If it's within your control, my main recommendation is to make sure that closing it cannot possibly fail by whatever means necessary.

Imagine if an operating system could actually fail to close a file. Now any program that tries to close a file on shut down will fail to shut down. What are we supposed to do now, just keep the application open and in limbo (probably no), just leak the file resource and ignore the problem (maybe okay if it's not so critical)? The safest design: make it so it's impossible to fail to close a file.

Title: Throwing an exception inside finally


问:
Should I log errors on exception throwing constructors?
I was building an application for a few months and I come to realize a pattern that emerged:

```<code>logger.error(ERROR_MSG);
throw new Exception(ERROR_MSG);
</code>```

Or, when catching:

```<code>try {
    // ...block that can throw something
} catch (Exception e) {
    logger.error(ERROR_MSG, e);
    throw new MyException(ERROR_MSG, e);
}
</code>```

So, whenever I was throwing or catching an exception, I would log it. In fact, that was almost all logging I did on the application (besides something for application initialization).

So, as a programmer, I shun repetition; So I decided to move the logger calls to the exception construction, so, whenever I was building an exception, things would be logged. I could also, certainly, create a ExceptionHelper that throwed the exception for me, but that would make my code harder to interpret and, even worse, the compiler wouldn't deal well with it, failing to realize that a call to that member would throw immediately.

So, is this an anti-pattern? If so, why?

答:
I think the problem is on a more basic level: you log the error and throw it as an exception at the same place. That's the anti-pattern. This means the same error is logged many times if it is caught, perhaps wrapped into another exception and and re-thrown.

Instead of this I suggest to log errors not when the exception is created, but when it is caught. (For this, of course, you must make sure it is always caught somewhere.) When an exception is caught, I'd only log its stacktrace if it is not re-thrown or wrapped as cause into another exception. The stacktraces and messages of wrapped exceptions are logged in stack traces as "Caused by ...", anyway. And the catcher could also decide e.g. to retry without logging the error on the first failure, or just treat it as a warning, or whatever.


答:
You could use abstract methods in the enum, thereby having better code locality (for better legibility and maintenance, not talking performance here).</p>

```<code>class AClass{
    enum Option{
        o1 {
            @Override
            Value deriveValue() {
                return deriveValue(p1, p2, p3);
            }
        }, ... on {
            @Override
            Value deriveValue() {
                return deriveValue(p1, p2, p3);
            }
        };
        abstract Value deriveValue();
    }

    Value method(Option o){
        return o.deriveValue();
    }
}
</code>```

Title: Refactoring a long method which is based on large number of switch cases

Body:
We are using Java as a backend development language.

One year back, we wrote a method which uses switch cases based on Enums values. Since we are continuously adding enum members and according adding cases in the method, the method has grown to very large extent. Currently, we have around 100 enum fields and corresponding number of switch cases.e.g.

```<code> class AClass{

   enum option{ o1, o2, o3...on}

   Value method(Option o){

      switch(o){
        case o1:
          value = deriveValue(p1,p2,p3);
        case o2:
          value = deriveValue(p2,p3,p4);
        .
        .
        .
        case on:
          value = deriveValue(p1,p2,p3);
      }
   }
 }
</code>```

Thus each time a business requirement comes, we add enum and corresponding switch case. Now the method has become too long and looks unmanageable, if we keep on adding the same logic in future.

To clean, we thought of replacing switch case with polymorphism by creating classes for the same, but again, we will end up creating n number of classes.

We are looking for small, simple and manageable solution.

-----------------  Update ---------------------

As suggested, to elaborate more, Enum values are fields names for which a client needs a value. Thus if a client need the value of a new field, we add the field in enum, and the corresponding definition of how to fetch value for the newly added field, by adding the corresponding switch case.

In the switch case, we have a common reusable method e.g. deriveValue() (Please refer to the example given) to which we pass parameters required for deriving the value for the newly added field.

Body:
You could use abstract methods in the enum, thereby having better code locality (for better legibility and maintenance, not talking performance here).

class AClass{
    enum Option{
        o1 {
            @Override
            Value deriveValue() {
                return deriveValue(p1, p2, p3);
            }
        }, ... on {
            @Override
            Value deriveValue() {
                return deriveValue(p1, p2, p3);
            }
        };
        abstract Value deriveValue();
    }

    Value method(Option o){
        return o.deriveValue();
    }
}


答:
Adding more and more classes is not really a problem, if they are parts of a well defined structure that allows you to ignore them when you want to: just put them in a module, and only access the module when you have to modify it.

IMHO, each option represents a way to compute a value, therefore in OO, each opinion should be represented by an object having a specific implementation of deriveValue(). In Java, that means that each object must have a specific class defining the implementation. So your code becomes something like:

In your option package:

```<code>public interface Option {
     Value deriveValue(? p1, ? p2, ? p3);
}

public FirstOption implements Option {
    Value deriveValue(? p1, ? p2, ? p3){ ... }
}
</code>```

In an other package;

```<code>class AClass{
   private Option o;

   Value method(){ o.deriveValue(p1,p2,p3); }
}
</code>```

If you have different behaviors to specify, each of them can be represented by a single call instead of a big switch. Caveat: each time you want to add a new behavior, you have to open each option class to add a new method.

Body:
Adding more and more classes is not really a problem, if they are parts of a well defined structure that allows you to ignore them when you want to: just put them in a module, and only access the module when you have to modify it.

IMHO, each option represents a way to compute a value, therefore in OO, each opinion should be represented by an object having a specific implementation of deriveValue(). In Java, that means that each object must have a specific class defining the implementation. So your code becomes something like:

In your option package:

public interface Option {
     Value deriveValue(? p1, ? p2, ? p3);
}

public FirstOption implements Option {
    Value deriveValue(? p1, ? p2, ? p3){ ... }
}

In an other package;

class AClass{
   private Option o;

   Value method(){ o.deriveValue(p1,p2,p3); }
}

If you have different behaviors to specify, each of them can be represented by a single call instead of a big switch. Caveat: each time you want to add a new behavior, you have to open each option class to add a new method.


答:
It's worth noting that, if you have this kind of situation where there's only one single function in the system always which will perform such checks, and it does not have any need whatsoever to be open for external extension, then throwing a polymorphic solution at it may actually leave you finding that your daily maintenance costs have actually increased rather than decreased.

Here, excluding a need for open extension, the practical difference between polymorphic and conditionals might boil down to whether you prefer more centralized, unstable code or more decentralized, stable code.

More decentralized, stable code might actually be favorable in some cases if it leads to less version control toe-stepping, e.g. More centralized, unstable code here would tend to be shorter, if only as a result of avoiding the boilerplate associated with defining new classes for each little simple case.

Currently, we have around 100 enum fields and corresponding number of
  switch cases.e.g. Thus each time a business requirement comes, we add
  enum and corresponding switch case.

This feels like the problem to me if we can avoid it. Is there no kind of categorization that can accompany so many enum fields? With the kind of ordinal example you gave, that wouldn't find much meaningful categorization. I'm hoping it was just an analogical one.

If you can find some kind of meaningful categorization to organize these enum fields, you can potentially use more than one function. Each function (still pretty coarse) could handle a range of cases, with an outer function checking which function to call based on the range.

That's a simple and not-so-intrusive solution, provided you can find some sort of categorization, to still have a fairly centralized and simple solution, but leave behind stable functions for the range of options you've already handled. It'll also reduce the instability of the function implementations involved, since the fewer range of options they handle, the fewer reasons they'll find to individually change.

You can also potentially use look-up table solutions possibly without a subtype involved for all 100 options, which gives you one centralized place to modify but possibly with a possibly more-preferable syntax. It depends on the nature of p1, p2, ..., pn. For example, if they are homogeneous, you might be able to use an array and make a LUT based on the index ordering with option enum fields modeling an index into the LUT. Example (pseudocode):

```<code>lut[o1] = {0, 1, 2};
lut[o2] = {1, 2, 3};
...
lut[on] = {0, 1, 2};

Value method(Option o){
    int i1 = lut[o][0];
    int i2 = lut[o][1];
    int i3 = lut[o][2];
    deriveValue(p[i1], p[i2], p[i3]);
}
</code>```

This doesn't really do much architecturally (though it is open for extension if you allow lut entries to be added from the outside) but you might prefer the syntax. It applies if your solution wasn't a remote analogy and actually just calls deriveValue based on the option with a different ordering of arguments (in which case I'd think there would be at least more than 3 parameters in actuality to call for so many options).

Body:
It's worth noting that, if you have this kind of situation where there's only one single function in the system always which will perform such checks, and it does not have any need whatsoever to be open for external extension, then throwing a polymorphic solution at it may actually leave you finding that your daily maintenance costs have actually increased rather than decreased.

Here, excluding a need for open extension, the practical difference between polymorphic and conditionals might boil down to whether you prefer more centralized, unstable code or more decentralized, stable code.

More decentralized, stable code might actually be favorable in some cases if it leads to less version control toe-stepping, e.g. More centralized, unstable code here would tend to be shorter, if only as a result of avoiding the boilerplate associated with defining new classes for each little simple case.

Currently, we have around 100 enum fields and corresponding number of
  switch cases.e.g. Thus each time a business requirement comes, we add
  enum and corresponding switch case.

This feels like the problem to me if we can avoid it. Is there no kind of categorization that can accompany so many enum fields? With the kind of ordinal example you gave, that wouldn't find much meaningful categorization. I'm hoping it was just an analogical one.

If you can find some kind of meaningful categorization to organize these enum fields, you can potentially use more than one function. Each function (still pretty coarse) could handle a range of cases, with an outer function checking which function to call based on the range.

That's a simple and not-so-intrusive solution, provided you can find some sort of categorization, to still have a fairly centralized and simple solution, but leave behind stable functions for the range of options you've already handled. It'll also reduce the instability of the function implementations involved, since the fewer range of options they handle, the fewer reasons they'll find to individually change.

You can also potentially use look-up table solutions possibly without a subtype involved for all 100 options, which gives you one centralized place to modify but possibly with a possibly more-preferable syntax. It depends on the nature of p1, p2, ..., pn. For example, if they are homogeneous, you might be able to use an array and make a LUT based on the index ordering with option enum fields modeling an index into the LUT. Example (pseudocode):

lut[o1] = {0, 1, 2};
lut[o2] = {1, 2, 3};
...
lut[on] = {0, 1, 2};

Value method(Option o){
    int i1 = lut[o][0];
    int i2 = lut[o][1];
    int i3 = lut[o][2];
    deriveValue(p[i1], p[i2], p[i3]);
}

This doesn't really do much architecturally (though it is open for extension if you allow lut entries to be added from the outside) but you might prefer the syntax. It applies if your solution wasn't a remote analogy and actually just calls deriveValue based on the option with a different ordering of arguments (in which case I'd think there would be at least more than 3 parameters in actuality to call for so many options).


问:
Why use an OO approach instead of a giant "switch" statement?
I am working in a .Net, C# shop and I have a coworker that keeps insisting that we should use giant Switch statements in our code with lots of "Cases" rather than more object oriented approaches.  His argument consistently goes back to the fact that a Switch statement compiles to a "cpu jump table" and is therefore the fastest option (even though in other things our team is told that we don't care about speed).

I honestly don't have an argument against this...because I don't know what the heck he's talking about.

Is he right?

Is he just talking out his ass?

Just trying to learn here.

答:
Normally I hate the word, "premature optimization", but this reeks of it. It's worth noting that Knuth used this famous quote in the context of pushing to use goto statements in order to speed up code in critical areas. That's the key: critical paths.

He was suggesting to use goto for speeding up code but warning against those programmers who would want to do these types of things based on hunches and superstitions for code that isn't even critical.

To favor switch statements as much as possible uniformly throughout a codebase (whether or not any heavy load is handled) is the classic example of what Knuth calls the "penny-wise and pound-foolish" programmer who spends all day struggling to maintain their "optimized" code which turned into a debugging nightmare as a result of trying to save pennies over pounds. Such code is rarely maintainable let alone even efficient in the first place.

Is he right?

He is correct from the very basic efficiency perspective. No compiler to my knowledge can optimize polymorphic code involving objects and dynamic dispatch better than a switch statement. You'll never end up with a LUT or jump table to inlined code from polymorphic code, since such code tends to serve as an optimizer barrier for the compiler (it won't know which function to call until the time at which the dynamic dispatch occurs).

It's more useful not to think of this cost in terms of jump tables but more in terms of the optimization barrier. For polymorphism, calling Base.method() doesn't allow the compiler to know which function will actually end up being called if method is virtual, not sealed, and can be overridden. Since it doesn't know which function is actually going to be called in advance, it can't optimize away the function call and utilize more information in making optimization decisions, since it doesn't actually know which function is going to be called at the time the code is being compiled.

Optimizers are at their best when they can peer into a function call and make optimizations that either completely flatten the caller and callee, or at least optimize the caller to most efficiently work with the callee. They can't do that if they don't know which function is actually going to be called in advance.

Is he just talking out his ass?

Using this cost, which often amounts to pennies, to justify turning this into a coding standard applied uniformly is generally very foolish, especially for places that have an extensibility need. That's the main thing you want to watch out for with genuine premature optimizers: they want to turn minor performance concerns into coding standards applied uniformly throughout a codebase with no regard for maintainability whatsoever.

I take a little offense to the "old C hacker" quote used in the accepted answer though, since I'm one of those. Not everyone who has been coding for decades starting from very limited hardware has turned into a premature optimizer. Yet I've encountered and worked with those too. But those types never measure things like branch misprediction or cache misses, they think they know better, and base their notions of inefficiency in a complex production codebase based on superstitions which don't hold true today and sometimes never held true. People who have genuinely worked in performance-critical fields often understand that effective optimization is effective prioritization, and trying to generalize a maintainability-degrading coding standard out to save pennies is very ineffective prioritization.

Pennies are important when you have a cheap function that doesn't do so much work which is called a billion times in a very tight, performance-critical loop. In that case, we end up saving 10 million dollars. It's not worth shaving pennies when you have a function called two times for which the body alone costs thousands of dollars. It's not wise to spend your time haggling over pennies during the purchase of a car. It is worth haggling over pennies if you are purchasing a million cans of soda from a manufacturer. The key to effective optimization is to understand these costs in their proper context. Someone who tries to save pennies on every single purchase and suggests that everyone else tries to haggle over pennies no matter what they're purchasing isn't a skilled optimizer.


问:
From a set of rules, derive the implications?
I've only just become interested in this domain, so sorry if I'm not using the correct terminologies.

What I want is the following: Say I have a set of rules (or constraints), I want to derive some implications of those rules.

For example, in Conway's Game of Life, there are 4 basic rules. From these rules, we can see a few patterns emerge. I want a system in which I can input the rules (in some formal language), and it would output at least some of these patterns. Also, if I make a change in any rule, or add a new rule, it should show me the implication of this change (or I should be able to derive it myself from comparing the two outputs).

This should ideally apply to any game that has a set of rules. For example in chess, it should say that the knight can move two squares in front of it by performing two L moves. In checkers it could be that having a piece behind another prevents the other playing from taking that piece.

Has anything like this ever be made? Is it even feasible? Can you recommend any courses or books where I could start my (re)search?

All I've found so far are Automated Theorem Provers, but from what I can tell so far, they are way too generic and mathematically oriented (they aim to solve any theory in maths, which has a lot of rules, I just want it for simple games with a small number of rules).

答:
To me it sounds like you have a state of a system (a chess board, a game of life board), set of state transformations (given state A, state B is an output). Then you have a state (starting chess board) and want to get all following states for each state transformation, possibly for multiple steps.

Bruteforcing this is primitive. But the amount of states that you get as output gets overwhelming really fast, so I fail to see any useful application.

AI for those games is usually about searching the state space.


问:
An elegant way to store build counter
We use Git and Jenkins as our build&release system and each build is assigned a version number which looks like this: 6.0.12345. Here, the 12345 part is a counter which increments with each build. The counter is stored on the build server in a simple text file, which is rewritten every time a build occurs. It's worth noticing that this file is not under version control.

I was thinking about more elegant way to manage build numbers, which will allow for easier portability (say move build server to a new machine) and scalability (adding another build server would be very tricky with the current setup).

The only thing I can think of now is using commit hash instead of number, but this will make it difficult to say which of two builds came first, also, I don't think customers will be willing to accept such a change.

Any ideas are appreciated.

答:
When you say that each build is "assigned a version number", do you mean that the commit associated with that build is also tagged with that build number in git? If you aren't doing that, you should:

```<code>git tag -a 6.0.12345 -m "Successful build on 2016-01-04 23:09:01.441"
git push --tags
</code>```

If you are, then you could just use git to tell you what the latest tag is instead of tracking it separately:

```<code>git describe --tags $(git rev-list --tags --max-count=1)
</code>```

Your build system would then parse that tag to get the last build number, increment it, and create the new version tag.

Title: An elegant way to store build counter

Body:
When you say that each build is "assigned a version number", do you mean that the commit associated with that build is also tagged with that build number in git? If you aren't doing that, you should:

git tag -a 6.0.12345 -m "Successful build on 2016-01-04 23:09:01.441"
git push --tags

If you are, then you could just use git to tell you what the latest tag is instead of tracking it separately:

git describe --tags $(git rev-list --tags --max-count=1)

Your build system would then parse that tag to get the last build number, increment it, and create the new version tag.


答:
Your colleague is not talking out of his backside, as far as the comment regarding jump tables goes. However, using that to justify writing bad code is where he goes wrong.

The C# compiler converts switch statements with just a few cases into a series of if/else's, so is no faster than using if/else. The compiler converts larger switch statements into a Dictionary (the jump table that your colleague is referring to). Please see this answer to a Stack Overflow question on the topic for more details.

A large switch statement is hard to read and maintain. A dictionary of "cases" and functions is a lot easier to read. As that is what the switch gets turned into, you and your colleague would be well advised to use dictionaries directly.

Title: Why use an OO approach instead of a giant "switch" statement?

Body:
I am working in a .Net, C# shop and I have a coworker that keeps insisting that we should use giant Switch statements in our code with lots of "Cases" rather than more object oriented approaches.  His argument consistently goes back to the fact that a Switch statement compiles to a "cpu jump table" and is therefore the fastest option (even though in other things our team is told that we don't care about speed).

I honestly don't have an argument against this...because I don't know what the heck he's talking about.

Is he right?

Is he just talking out his ass?

Just trying to learn here.


答:
Consider General Game Playing. The idea is to write a program that performs well playing any game that can be formally described. So you don't create a tic-tac-toe program or checkers program or Reversi program. You create a program/agent that can play all of these games with the best general performance.

The steps of a GGP session are:

The game server delivers a formal description of the game. This includes number of players, initial state, and valid rules.

Each player has some time to process the rules.

The game starts. The game server calls for a move from each player in proper turn order.

As players submit moves, the game server validates that the moves are legal and either accepts or rejects them.

Play continues until a player wins.

Obviously, this doesn't all apply directly to the problem you describe, but the underlying principles are the same. Given some set of rules and an initial state, decide the valid transitions from state to state to state. Michael Genesereth and Michael Thielscher go into detail in their General Game Playing online book.

Your Conway's Game of Life example could be described as a single player game that has no winning state. It runs forever. You'd have to decide when to stop "playing". Chess fits very well in the GGP model, though it's obviously a game with a fairly large state space. Take a look at Tiltyard's game list to get an idea of the variety of games being played by the GGP community.

There's also a very fun General Game Playing Stanford course and an online community that includes starter kits for developing players and a free General Game Playing server.

Title: From a set of rules, derive the implications?

Body:
I've only just become interested in this domain, so sorry if I'm not using the correct terminologies.

What I want is the following: Say I have a set of rules (or constraints), I want to derive some implications of those rules.

For example, in Conway's Game of Life, there are 4 basic rules. From these rules, we can see a few patterns emerge. I want a system in which I can input the rules (in some formal language), and it would output at least some of these patterns. Also, if I make a change in any rule, or add a new rule, it should show me the implication of this change (or I should be able to derive it myself from comparing the two outputs).

This should ideally apply to any game that has a set of rules. For example in chess, it should say that the knight can move two squares in front of it by performing two L moves. In checkers it could be that having a piece behind another prevents the other playing from taking that piece.

Has anything like this ever be made? Is it even feasible? Can you recommend any courses or books where I could start my (re)search?

All I've found so far are Automated Theorem Provers, but from what I can tell so far, they are way too generic and mathematically oriented (they aim to solve any theory in maths, which has a lot of rules, I just want it for simple games with a small number of rules).


问:
What are the advantages of a 3 stage environment for SAP/BI development?
I have recently joined a team in which the developers were able to work completely in live production to produce business intelligence products for customers.

Recently, management, at the advice of SAP, have moved the team to a 3 stage environment

Develop

Test

Production

which require transports between each layer.

The team have accepted this new constraint begrudgingly and no firm business case has been explained to them as to the benefits.

Can anyone articulate the advantages of the 3 stage environment or point me to a white paper/case study they recommend?

答:
There are some benefits of doing maintenance in Production:

Spares time for the developers.

Data for testing is available in production.

The management will not care about the first one, the cost of your (and your colleague's) time is usually insignificant compared to the cost of a production down.

Creating a Development system helps mitigating the risk, but will not provide you with test data. Copying it from Production is long and tedious. Developers usually create 3-5 lines of test data, so only functionality is tested, but not performance.

This is why a Test system is useful, even the creation process is beneficial.

Create a backup from the Production system (the basis team gets some practice)

Restore it to another hardware (many SAP customers only do backups, and have no idea how to restore, so this practice is even more important)

Now you have a system with actual production data, repeat it every quarter

So all three systems have a different purpose. Production is obvious, Development is the one where developers can experiment and fail without consequences, and Test is where you find out why HASHED tables were developed.


问:
Double-check locking around initialisation which performs a web request
I've just refactored some code that managed a global state cache of values that didn't have locking to use double check locking. Other than moving the initialisation to a single source (the cache was being set in a controller method), the main reason was to make sure only one thread was performing the initialisation at a time.

The lifetime of the cache is for the lifetime of the application, so once it is set, it should be fine.

I've locked around the code that does the web request.

My colleague, who wrote the initial code, said he was concerned about the queued threads waiting for because the initialisation makes a web request, and if the request is failing, the last thread has to wait for all the rest to go through so may have to wait awhile and so the request should be outside of the lock, but my thoughts are that a request is expensive and unreliable, and should be minimised.

As this cache doesn't expire, it's not a big deal, but what are your thoughts? I suggested that if we wanted the correct solution, we could test the locks, using Monitor.TryLock etc for some more sophisticated locking, but it's probably overkill.

答:
If the web request is only required to initialize the cache, then you're correct. Throw a lock around the whole thing. Why bother making N requests when you're going to ignore all but the first one that returns?

If the request is required regardless of the cache, then your colleague is correct. Allow all N requests through without constraint. The first one back initializes the cache. Throw a lock around cache initialization but not the request.

Depending on use, you could also eagerly initialize the cache on application start up. I wouldn't bother with data that's seldom accessed. But if it's absolutely necessary, why not just grab it early and be done with it?


问:
Hardware under MIT license?
I have some VHDL design that I plan to release as open source.

I like MIT License for its simplicity and would like to use it for my code.
However, I am not sure if it is OK to use it with hardware design, as it says "software" pretty much in every sentence.

Are there any particular things I should consider when releasing VHDL under MIT license? And can I do it at all?

答:
VHDL is in a grey area between software and hardware.
Some people classify it as hardware, because ultimately the VHDL design gets translated into an actual IC and it doesn't get executed on a microprocessor.
Others classify it as software, because a single FPGA can be given different behaviour by programming it with different VHDL designs.

For licensing your VHDL design, this difference isn't really relevant.

The relevant commonality between "pure" software source and a VHDL design is that both are textual files specifying the behaviour of a system and both have to be processed by some tooling to realize the described behaviour.

This makes VHDL to be close enough to be covered by the term "Software" as it is used by the MIT license. Thus you can use the MIT license without further consideration for your VHDL designs.

Title: Hardware under MIT license?


问:
What makes a large and complex software product slow?
For a reason that is largely irrelevant, I installed Delphi 7 once again in such a long time. I have to say, I was completely blown away - in a way I haven't been for rather a while. This is not how I remember things at all. The installation took around 30 seconds. Launching it took 2 seconds, and it was immediately usable. I can press "Run" the second after it started, and less than a second later the blank program is already visible and running. Hurray for computers getting so much faster!

But the reason I've been blown away like this is because usually I use Visual Studio 2010, that doesn't feel snappy like this at all. Granted, Delphi 7 is a much smaller system than Visual Studio 2010, but it does have the appearance of having all the really necessary things there: a control palette, a form designer, a code editor with code completion. I realise that the language might be simpler, and the code completion might be a lot less powerful, and the IDE might not be nearly as extensible and feature-rich, but still: I do not understand how (i.e. through what mechanism) does having a lot of extra features (that I might not have even triggered yet) cause a system like Visual Studio to always feel sluggish in comparison.

I would like to ask people experienced in working with systems the scale of Visual Studio: what is it that makes them slow? Is it the layers upon layers of abstractions required to keep the codebase within the human comprehension capabilities? Is it the sheer amount of code that needs to be run through? Is it the modern tendency towards programmer-time-saving approaches at the (mindbogglingly huge) expense in the clock cycles / memory usage department?

答:
I would like to ask people experienced in working with systems the
  scale of Visual Studio: what is it that makes them slow? Is it the
  layers upon layers of abstractions required to keep the codebase
  within the human comprehension capabilities? Is it the sheer amount of
  code that needs to be run through? Is it the modern tendency towards
  programmer-time-saving approaches at the (mindbogglingly huge) expense
  in the clock cycles / memory usage department?

I think you guessed a number of them but I would like to offer what I consider to be the biggest factor, having worked on a reasonably large codebase (not sure if it's as big as Visual Studio -- was in the millions of lines of code category and around a thousand plugins) for about 10 years and observing phenomena occur.

It's also a bit less controversial since it doesn't go into APIs or language features or anything like that. Those relate to "costs" which can spawn a debate rather than "spending", and I want to focus on "spending".

Loose Coordination and Legacy

What I observed is that loose coordination and a long legacy tends to lead to a lot of accumulated waste.

For example, I found around one hundred acceleration structures in this codebase, many of them redundant.

We'd have like a K-D tree for accelerating one physics engine, another for a new physics engine that was often running in parallel with the old one, we'd have dozens of implementations of octrees for various mesh algorithms, another K-D tree for rendering, picking, etc. etc. etc. These are all big, bulky tree structures used to accelerate searches. Each individual one can take hundreds of megabytes to gigabytes of memory for a very average-sized input. They weren't always instantiated and used all the time, but at any given time, 4 or 5 of them might be in memory simultaneously.

Now all of these were storing the exact same data to accelerate searches for them. You can imagine it as like the analogical old database which stores all of its fields into 20 different redundant maps/dictionaries/B+ trees at once, organized identically by the same keys, and searches all of them all the time. Now we're taking 20 times the memory and processing.

In addition, because of the redundancy, there's little time to optimize any one of them with the maintenance price tag that comes with that, and even if we did, it would only have 5% of the effect it ideally would.

What causes this phenomena? Loose coordination was the number one cause I saw. A lot of team members often work in their isolated ecosystems, developing or using third party data structures, but not using the same structures other team members were using even if they were outright blatant duplicates of the exact same concerns.

What causes this phenomena to persist? Legacy and compatibility was the number one cause I saw. Since we already paid the cost to implement these data structures and large amounts of code were dependent on these solutions, it was often too risky to try to consolidate them to fewer data structures. Even though many of these data structures were highly redundant conceptually, they weren't always anywhere close to identical in their interface designs. So replacing them would have been a big, risky change as opposed to just letting them consume memory and processing time.

Memory Efficiency

Typically memory use and speed tend to be related at the bulk level at least. You can often spot slow software by how it's hogging up memory. It's not always true that more memory leads to a slowdown, since what matters is "hot" memory (what memory is being accessed all the time -- if a program uses a boatload of memory but only 1 megabyte of it is being used all the time, then it's not such a big deal speed-wise).

So you can spot the potential hogs based on memory usage a lot of the time. If an application takes tens to hundreds of megabytes of memory on startup, it's probably not going to be very efficient. Tens of megabytes might seem small when we have gigabytes of DRAM these days, but the largest and slowest CPU caches are still in the measly megabytes range, and the fastest are still in the kilobytes range. As a result, a program that uses 20 megabytes just to start up and do nothing is actually still using quite "a lot" of memory from the hardware CPU cache point of view, especially if all 20 megabytes of that memory will be accessed repeatedly and frequently as the program is running.

Solution

To me the solution is to seek more coordinated, smaller teams to build products, ones who can kind of keep track of their "spending" and avoid "purchasing" the same items over and over and over.

Cost

I'll dip into the more controversial "cost" side just a teeny bit with a "spending" phenomena I've observed. If a language ends up coming with an inevitable price tag for an object (like one that provides runtime reflection and cannot force contiguous allocation for a series of objects), that price tag is only expensive in the context of a very granular element, like a single Pixel or Boolean.

Yet I see a lot of source code for programs which do handle a heavy load (ex: dealing with hundreds of thousands to millions of Pixel or Boolean instances) paying that cost at such a granular level.

Object-oriented programming can kind of exacerbate that. Yet it's not the cost of "objects" per se or even OOP at fault, it's simply that such costs are being paid at such a granular level of a teeny element that's going to be instantiated by the millions.

So that's the other "cost" and "spending" phenomena I'm observing. The cost is pennies, but pennies add up if we're purchasing a million cans of soda individually instead of negotiating with a manufacturer for a bulk purchase.

The solution here to me is "bulk" purchase. Objects are perfectly fine even in languages that have some price tag of pennies to each one provided that this cost is not being paid individually a million times over for the analogical equivalent of a soda can.

Premature Optimization

I never quite liked the wording Knuth used here, because "premature optimization" rarely makes real-world production programs go faster. Some interpret that as "optimizing early" when Knuth meant more like "optimizing without the proper knowledge/experience to know its true impact on the software." If anything, the practical effect of true premature optimization is often going to make software slower, since the degradation in maintainability means there's little time to optimize the critical paths that really matter.

This is the final phenomena I observed, where developers reaching to save pennies on the purchase of a single can of soda, never again to be bought, or worse, a house, were wasting all their time pinching pennies (or worse, imaginary pennies from failing to understand their compiler or the architecture of the hardware) when there were billions of dollars being wastefully spent elsewhere.

Time is very finite so trying to optimize absolutes without having the proper contextual information is often depriving us the opportunity to optimize the places that genuinely matter, and thus, in terms of practical effect, I would say that "premature optimization makes software much slower."

The problem is that there are developer types who will take what I wrote above about objects and try to establish a coding standard that bans object-oriented programming or something crazy of that sort. Effective optimization is effective prioritization, and it's absolutely worthless if we're drowning in a sea of maintenance problems.

ContentLicense: CC BY-SA 2.5


问:
Designs and practices to guard against erroneous null entries from database
One part of my program fetches data from many tables and columns in my database for processing. Some of the columns might be null, but in the current processing context that is an error.

This should "theoretically" not happen, so if it does it points to bad data or a bug in the code. The errors have different severities, depending which field is null; i.e. for some fields the processing should be stopped and somebody notified, for others the processing should be allowed to continue and just notify somebody.

Are there any good architecture or design principles to handle the rare but possible null entries?

The solutions should be possible to implement with Java but I didn't use the tag because I think the problem is somewhat language-agnostic.

Some thoughts that I had myself:

Using NOT NULL

Easiest would be to use a NOT NULL constraint in the database.

But what if the original inserting of the data is more important that this later processing step? So in case the insert would put a null into the table (either because of bugs or maybe even some valid reason), I wouldn't want the insert to fail. Let's say that many more parts of the program depend on the inserted data, but not on this particular column. So I would rather risk the error in the current processing step instead of the insert step. That's why I don't want to use a NOT NULL constraint.

Naively depending on NullPointerException

I could just use the data as if I expect it to be always there (and that should really be the case), and catch resulting NPEs at an appropriate level (e.g. so that the processing of the current entry stops but not the whole processing progress). This is the "fail fast" principle and I often prefer it. If it is a bug at least I get a logged NPE.

But then I lose the ability to differentiate between various kinds of missing data. E.g. for some missing data I could leave it out, but for others the processing should be stopped and an admin notified.

Checking for null before each access and throwing custom exceptions

Custom exceptions would let me decide the correct action based on the exception, so this seems like the way to go.

But what if I forget to check it somewhere? Also I then clutter my code with null checks which are never or rarely expected (and so definitely not part of the business logic flow).

If I choose to go this way, what patterns are best suited for the approach?

Any thoughts and comments on my approaches are welcomed. Also better solutions of any kind (patterns, principles, better architecture of my code or models etc.).

Edit:

There is another constraint, in that i am using an ORM to do the mapping from DB to persistence object, so doing null checks on that level would not work (as the same objects are used in parts where the null does not do any harm). I added this because the answers provided so far both mentioned this option.

答:
I would put the null checks in your mapping code, where you build your object from the result set. That puts the checking in one place, and won't allow your code to get halfway through processing a record before hitting an error. Depending on how your application flow works, you might want to perform the mapping of all the results as a pre-processing step instead of mapping and processing each record one at a time.

If you're using an ORM then you'll have to perform all your null checks before processing each record. I'd recommend a recordIsValid(recordData)-type method, that way you can (again) keep all the null-checking and other validation logic in one place. I definitely wouldn't intermingle the null checks with the rest of your processing logic.


答:
Nulls can be handled in the translation or mapping of database types to language types.  For example in C#, here's a generic method that handles null for you for any type:

```<code>public static T Convert<T>(object obj)
        {
            if (obj == DBNull.Value)
            {
                return default(T);
            }

            return (T) obj;
        }

public static T Convert<T>(object obj, T defaultValue)
        {
            if (obj == DBNull.Value)
            {
                T t = defaultValue;
                return t;
            }

            return (T) obj;
        }
</code>```

Or, if you want to perform action...

```<code> public static T Convert<T>(object obj, T defaultValue)
        {
            if (obj == DBNull.Value)
            {
                //Send an Alert, we might want pass in the name
                //of column or other details as well
                SendNullAlert();
                //Set it to default so we can keep processing
                T t = defaultValue;
                return t;
            }

            return (T) obj;
        }
</code>```

And then in the mapping, in this case to an object of type "Sample", we will handle null for any of the columns:

```<code>public class SampleMapper : MapperBase<Sample>
    {
        private const string Id = "Id";
        private const string Name = "Name";
        private const string DataValue = "DataValue";
        private const string Created = "Created";

        protected override Sample Map(IDataRecord record)
        {
            return new Sample(
                Utility.Convert<Int64>(record[Id]),
                Utility.Convert<String>(record[Name]),
                Utility.Convert<Int32>(record[DataValue]),
                Utility.Convert<DateTime>(record[Created])
                );
        }
    }
</code>```

Finally, all the mapping classes can be automatically generated based on the SQL query or tables involved by looking at the SQL data types and translating them to the language specific data types.   This is what many ORMs do for you automatically.  Note that some database types may not have a direct mapping (Geo-spatial colunms etc.) and may need special handling.

Title: Designs and practices to guard against erroneous null entries from database

Body:
One part of my program fetches data from many tables and columns in my database for processing. Some of the columns might be null, but in the current processing context that is an error.

This should "theoretically" not happen, so if it does it points to bad data or a bug in the code. The errors have different severities, depending which field is null; i.e. for some fields the processing should be stopped and somebody notified, for others the processing should be allowed to continue and just notify somebody.

Are there any good architecture or design principles to handle the rare but possible null entries?

The solutions should be possible to implement with Java but I didn't use the tag because I think the problem is somewhat language-agnostic.

Some thoughts that I had myself:

Using NOT NULL

Easiest would be to use a NOT NULL constraint in the database.

But what if the original inserting of the data is more important that this later processing step? So in case the insert would put a null into the table (either because of bugs or maybe even some valid reason), I wouldn't want the insert to fail. Let's say that many more parts of the program depend on the inserted data, but not on this particular column. So I would rather risk the error in the current processing step instead of the insert step. That's why I don't want to use a NOT NULL constraint.

Naively depending on NullPointerException

I could just use the data as if I expect it to be always there (and that should really be the case), and catch resulting NPEs at an appropriate level (e.g. so that the processing of the current entry stops but not the whole processing progress). This is the "fail fast" principle and I often prefer it. If it is a bug at least I get a logged NPE.

But then I lose the ability to differentiate between various kinds of missing data. E.g. for some missing data I could leave it out, but for others the processing should be stopped and an admin notified.

Checking for null before each access and throwing custom exceptions

Custom exceptions would let me decide the correct action based on the exception, so this seems like the way to go.

But what if I forget to check it somewhere? Also I then clutter my code with null checks which are never or rarely expected (and so definitely not part of the business logic flow).

If I choose to go this way, what patterns are best suited for the approach?

Any thoughts and comments on my approaches are welcomed. Also better solutions of any kind (patterns, principles, better architecture of my code or models etc.).

Edit:

There is another constraint, in that i am using an ORM to do the mapping from DB to persistence object, so doing null checks on that level would not work (as the same objects are used in parts where the null does not do any harm). I added this because the answers provided so far both mentioned this option.

Body:
Nulls can be handled in the translation or mapping of database types to language types.  For example in C#, here's a generic method that handles null for you for any type:

public static T Convert<T>(object obj)
        {
            if (obj == DBNull.Value)
            {
                return default(T);
            }

            return (T) obj;
        }

public static T Convert<T>(object obj, T defaultValue)
        {
            if (obj == DBNull.Value)
            {
                T t = defaultValue;
                return t;
            }

            return (T) obj;
        }

Or, if you want to perform action...

 public static T Convert<T>(object obj, T defaultValue)
        {
            if (obj == DBNull.Value)
            {
                //Send an Alert, we might want pass in the name
                //of column or other details as well
                SendNullAlert();
                //Set it to default so we can keep processing
                T t = defaultValue;
                return t;
            }

            return (T) obj;
        }

And then in the mapping, in this case to an object of type "Sample", we will handle null for any of the columns:

public class SampleMapper : MapperBase<Sample>
    {
        private const string Id = "Id";
        private const string Name = "Name";
        private const string DataValue = "DataValue";
        private const string Created = "Created";

        protected override Sample Map(IDataRecord record)
        {
            return new Sample(
                Utility.Convert<Int64>(record[Id]),
                Utility.Convert<String>(record[Name]),
                Utility.Convert<Int32>(record[DataValue]),
                Utility.Convert<DateTime>(record[Created])
                );
        }
    }

Finally, all the mapping classes can be automatically generated based on the SQL query or tables involved by looking at the SQL data types and translating them to the language specific data types.   This is what many ORMs do for you automatically.  Note that some database types may not have a direct mapping (Geo-spatial colunms etc.) and may need special handling.


问:
How can attributes/annotations be illustrated on a UML class diagram?
Many programming languages offer a way to annotate types.

For instance, in .NET this can be achieved by deriving a custom attribute class from System.Attribute and then annotating another type with this:

```<code>[AttributeUsage(AttributeTargets.Class, AllowMultiple = false)]
public sealed class SuperSpecialAttribute : Attribute { ... }

[SuperSpecial]
public class MyClass { ... }
</code>```

My question is; how can I show that MyClass has the attribute SuperSpecialAttribute on a UML class diagram?

答:
You could represent the annotation as a stereotype, and then apply the stereotype to the class.

```<code>    +----------------------+
    |   <<SuperSpecial>>   |
    |       MyClass        |
    +----------------------+
</code>```

If your annotation has attributes, you can represent them as tagged values on the class. Tagged attributes have different syntax in UML 1.x and UML 2.x

In UML 1.x you put tagged values between {}, like this:

```<code>    +----------------------+
    |   <<SuperSpecial>>   |
    | {AllowMultiple=true} |
    |       MyClass        |
    +----------------------+
</code>```

In UML 2.x you can use a section for the stereotype attributes:

```<code>    +----------------------+
    |   <<SuperSpecial>>   |
    |       MyClass        |
    +----------------------+
    | <<SuperSpecial>>     |
    | AllowMultiple=true   |
    +----------------------+
</code>```

Body:
You could represent the annotation as a stereotype, and then apply the stereotype to the class.

    +----------------------+
    |   <<SuperSpecial>>   |
    |       MyClass        |
    +----------------------+

If your annotation has attributes, you can represent them as tagged values on the class. Tagged attributes have different syntax in UML 1.x and UML 2.x

In UML 1.x you put tagged values between {}, like this:

    +----------------------+
    |   <<SuperSpecial>>   |
    | {AllowMultiple=true} |
    |       MyClass        |
    +----------------------+

In UML 2.x you can use a section for the stereotype attributes:

    +----------------------+
    |   <<SuperSpecial>>   |
    |       MyClass        |
    +----------------------+
    | <<SuperSpecial>>     |
    | AllowMultiple=true   |
    +----------------------+


答:
Whenever your use case allows to replace NULL safely by a good default value, you can do the conversion in the SELECT Sql statements using ISNULL or COALESCE. So instead of

```<code> SELECT MyColumn FROM MyTable
</code>```

one can write

```<code> SELECT ISNULL(MyColumn,DefaultValueForMyColumn) FROM MyTable
</code>```

Of course, that will only work when the ORM allows to manipulate the select statements directly, or provide changeable templates for the generation. One should make sure that no "real" errors are masked this way, so apply it only if replacing by a default value is exactly what you want in case of NULL.

If you are able to change the database and the schema, and your db system supports this, you may consider to add a default value clause to the specific columns, as suggested by @RobbieDee. However, this will also require to modify the existing data in the database to remove any previously inserted NULL values, and it will remove the ability to distinguish between correct and incomplete import data afterwards.

From my own experience, I know that using ISNULL can work surprisingly well - in the past I had to maintain a legacy application where the original devs had forgotten to add NOT NULL constraints to lots of columns, and we could not easily add those constraints later for some reasons. But in 99% of all cases, 0 as default for number columns and the empty string as default for a text columns was fully acceptable.

Body:
Whenever your use case allows to replace NULL safely by a good default value, you can do the conversion in the SELECT Sql statements using ISNULL or COALESCE. So instead of

 SELECT MyColumn FROM MyTable

one can write

 SELECT ISNULL(MyColumn,DefaultValueForMyColumn) FROM MyTable

Of course, that will only work when the ORM allows to manipulate the select statements directly, or provide changeable templates for the generation. One should make sure that no "real" errors are masked this way, so apply it only if replacing by a default value is exactly what you want in case of NULL.

If you are able to change the database and the schema, and your db system supports this, you may consider to add a default value clause to the specific columns, as suggested by @RobbieDee. However, this will also require to modify the existing data in the database to remove any previously inserted NULL values, and it will remove the ability to distinguish between correct and incomplete import data afterwards.

From my own experience, I know that using ISNULL can work surprisingly well - in the past I had to maintain a legacy application where the original devs had forgotten to add NOT NULL constraints to lots of columns, and we could not easily add those constraints later for some reasons. But in 99% of all cases, 0 as default for number columns and the empty string as default for a text columns was fully acceptable.


问:
When is a circular reference to a parent pointer acceptable?
This Stack Overflow question is about a child having reference to its parent, through a pointer.

Comments were pretty critical initially of the design being a horrible idea.

I understand this is probably not the best idea in general. From a general rule of thumb it seems fair to say, "don't do this!"

However, I am wondering what sorts of conditions would exist where you would need to do something like this. This question here and associated answers/commentary suggests even for graphs to not do something like this.

答:
Doom 3 has an example of a child object with a pointer to a parent object.  Specifically it uses intrusive lists.  To summarize, an intrusive list is like a linked list except each node contains a pointer to the list itself.

Advantages:

When objects can exist in several lists simultaneously, the memory for the list nodes needs to be allocated and deallocated only once.

When a object needs to be destroyed, you can easily remove it from all of the lists it's in without searching through each list linearly.

I think this is a pretty specific scenario, but if I understand your question, it's an example of an acceptable use of a child object containing a pointer to its parent object.


答:
You could apply the general strategy used to deal with big numbers -- structure. Perhaps the prolific number of options handled uniformly is a sign of too little hierarchy? Perhaps the cases fall neatly in groups which seem to belong together (like I/O, manipulation, computation etc.)? (Perhap not -- then please ignore ;-) ).

This could be reflected with a a hierarchical enum system. One enum characterizes the group, the next one the specific operation.

The groups could be coded separately and would each be more managable. Having them in different source files/classes also makes editing errors in unrelated places less likely, may speed up compilation and should reduce dependencies (from system libraries as well as from other custom classes).

Of course this approach is also conceivable as a design idea for polymorhpism: One could have specific option/action classes deriving from group classes deriving from a generic option class), with an opportunity to bundle common code further up in the hierarchy.


问:
Algorithm to merge two sorted arrays with minimum number of comparisons
Given are two sorted arrays a, b of type T with size n and m. I am looking for an algorithm that merges the two arrays into a new array (of maximum size n+m).

If you have a cheap comparison operation, this is pretty simple. Just take from the array with the lowest first element until one or both arrays are completely traversed, then add the remaining elements. Something like this https://stackoverflow.com/questions/5958169/how-to-merge-two-sorted-arrays-into-a-sorted-array

However, the situation changes when comparing two elements is much more expensive than copying an element from the source array to the target array. For example you might have an array of large arbitrary precision integers, or strings, where a comparison can be quite expensive. Just assume that creating arrays and copying elements is free, and the only thing that costs is comparing elements.

In this case, you want to merge the two arrays with a minimum number of element comparisons. Here are some examples where you should be able to do much better than the simple merge algorithm:

```<code>a = [1,2,3,4, ... 1000]
b = [1001,1002,1003,1004, ... 2000]
</code>```

Or

```<code>a = [1,2,3,4, ... 1000]
b = [0,100,200, ... 1000]
</code>```

There are some cases where the simple merge algorithm will be optimal, like

```<code>a = [1,3,5,7,9,....,999]
b = [2,4,6,8,10,....,1000]
</code>```

So the algorithm should ideally gracefully degrade and perform a maximum of n+m-1 comparisons in case the arrays are interleaved, or at least not be significantly worse.

One thing that should do pretty well for lists with a large size difference would be to use binary search to insert the elements of the smaller array into the larger array. But that won't degrade gracefully in case both lists are of the same size and interleaved.

The only thing available for the elements is a (total) ordering function, so any scheme that makes comparisons cheaper is not possible.

Any ideas?

I have come up with this bit in Scala. I believe it is optimal regarding the number of comparisons, but it is beyond my ability to prove it. At least it is much simpler than the things I have found in the literature.

And since the original posting, I wrote a blog post about how this works.

答:
Assume the two arrays have N and M elements, N ≥ M, and all elements are different.

If the sorted array contains an element x of N followed by an element y of M or vice versa then x and y must have been compared, otherwise we wouldn't know in which order they belong. (There cannot be a chain of other elements say a, b, c where we know that x < a < b < c < y, for example, because there are no elements between x and y. So x and y must have been compared directly.

If N > M then it is possible to have an array where each element of M is both preceded and followed by an element of N, which means at least 2M comparisons are needed - even if you use a non-deterministic sorting algorithm that can make a perfect guess which numbers to compare. (What that means: Assume you have N large, M = 1. Binary search takes O (log2 N) steps; a non-deterministic algorithm would guess between which two elements the one element of the second array belongs, and make two comparisons to confirm the guess).


问:
Overriding Equals in an unbrittle way
I have a class WorkflowItemSearchCriteria whose instances represent a set of values and references that will be used to return a filtered result set of workflow items based on the given search criteria.

```<code>class WorkflowItemSearchCriteria
{
    string ID { get; set; }
    Client TargetClient { get; set; }
    Person Supervisor { get; set; }
    Person WorkingAttorney { get; set; }
    SearchDateKind? TargetSearchDateKind { get; set; }
    DateTime? FromDate { get; set; }
    DateTime? ToDate { get; set; }

    public override bool Equals(object obj)
    {
        if (!(obj is WorkflowItemSearchCriteria)) return false;
        var target = (WorkflowItemSearchCriteria) obj;
        if (!target.ID.Equals(this.ID)) return false;
        if (target.TargetClient != this.TargetClient) return false;
        if (target.Supervisor != this.Supervisor) return false;
        if (target.WorkingAttorney != this.WorkingAttorney) return false;
        if (!target.TargetSearchDateKind.Equals(this.TargetSearchDateKind)) return false;
        if (!target.FromDate.Equals(this.FromDate)) return false;
        if (!target.ToDate.Equals(this.ToDate)) return false;
        return true;
    }

    public override int GetHashCode()
    {
        // According to MSDN, two objects that are equal must return the same hash code,
        // but the same hash code does not mean two objects are equal.  If two value
        // objects are equal, then all property and field values must be equal,
        // so it's sufficient to return the hash code from just ONE field or property
        // because it will equal the hash code from another object that is equal to it.
        return ID.GetHashCode();
    }

    enum SearchDateKind
    {
        DateCreated,
        DateCompleted
    }
}
</code>```

At some point, I want to compare two instances of this class for value-wise equality, so I'm overriding Equals as shown.  I've been reading sources such as this to help me along, but I still feel that doing it the way I have results in brittle code because future developers would have to remember to update the Equals overload if there were ever properties added to WorkflowItemSearchCriteria.

Is there a better approach to this?

答:
You can also include Fody Equals, which injects the Equals logic (including IEquatable<T> and overloading operators == and !=) at compile time.

You only need to add the Equals attribute:

```<code>[Equals]
class WorkflowItemSearchCriteria
{
    //...
}
</code>```

Title: Overriding Equals in an unbrittle way

Body:
You can also include Fody Equals, which injects the Equals logic (including IEquatable<T> and overloading operators == and !=) at compile time.

You only need to add the Equals attribute:

[Equals]
class WorkflowItemSearchCriteria
{
    //...
}


问:
Broken Old/Legacy Unit Tests
I work for a big company and I'm responsible for a large java application with thousands of junit tests.  Since I moved to this role, there have been 200-300 broken tests (likely broken for years).  The tests are old and fragile and they're a mess of spaghetti dependencies that typically end with live sandbox data.

My goal is 100% passing tests so we can break the build on unit test failures, but I can't do it until I address the broken tests. I have very little budget because the maintenance budget is primarily for support, but my team has identified and fixed the low-hanging fruit tests (mostly config/local resource issues) and we're down to 30-40 really ugly tests.

What are some opinions on best practice?  I don't think the tests are valuable, but I also don't know what they're testing or why they don't work without digging in, which takes time and money we probably don't have.

I'm thinking we should document the statuses of the broken tests with anything we know, then either delete or ignore the broken tests completely and enter a lower-priority bug/work item to investigate and fix them.  We'll then be at 100% and start to get real value out of the other tests and if we have a maintenance/refactoring windfall we'll be able to pick them up again.

What would be the best approach?

Edit: I think this is a different question than this question because I have a clear direction for the tests that we should be writing going forward, but I inherited legacy failing tests to address before the large current set of tests becomes meaningful.

答:
What I would do is first disable the tests which are failing-and-have-always-failed.

Make it so a test failing matters.

As you investigate you might be able to ask people who have been with your company longer about them, there might be a lot of tribal knowledge about them which you can document/capture. Maybe from your VCS logs. "Oh, that test has always failed since we upgraded to X" or other information can be useful.

Once you know what the functionality being tested was you can determine:

Do we care about this being tested

How important is this to be tested

And then make a priority list.

Likely nothing on this list is important enough to get more time later since it's been ignored for years already. So I wouldn't spend too much time/resources documenting and analyzing all these broken tests.

Title: Broken Old/Legacy Unit Tests


答:
I would do the following:

Attempt to determine exactly what the failing tests are trying to validate.

Triage - if some tests are trying to test unimportant things like (an old) state of the world, delete them. If you realize that some of them are trying to validate something important, try to determine if those tests are doing so correctly. If they're testing incorrectly, make them test correctly.

Fix whatever is wrong with your production code now that you have good tests.

Remember accounting, every line of code is a liability, but may be incorrectly valued as an asset. The delete key can create a lot of value for your firm.

Title: Broken Old/Legacy Unit Tests

Body:
I work for a big company and I'm responsible for a large java application with thousands of junit tests.  Since I moved to this role, there have been 200-300 broken tests (likely broken for years).  The tests are old and fragile and they're a mess of spaghetti dependencies that typically end with live sandbox data.

My goal is 100% passing tests so we can break the build on unit test failures, but I can't do it until I address the broken tests. I have very little budget because the maintenance budget is primarily for support, but my team has identified and fixed the low-hanging fruit tests (mostly config/local resource issues) and we're down to 30-40 really ugly tests.

What are some opinions on best practice?  I don't think the tests are valuable, but I also don't know what they're testing or why they don't work without digging in, which takes time and money we probably don't have.

I'm thinking we should document the statuses of the broken tests with anything we know, then either delete or ignore the broken tests completely and enter a lower-priority bug/work item to investigate and fix them.  We'll then be at 100% and start to get real value out of the other tests and if we have a maintenance/refactoring windfall we'll be able to pick them up again.

What would be the best approach?

Edit: I think this is a different question than this question because I have a clear direction for the tests that we should be writing going forward, but I inherited legacy failing tests to address before the large current set of tests becomes meaningful.


答:
200-300 broken tests (likely broken for years).

Ouch! I faced a similar situation once but with like 7 tests failing where the team started ignoring the fact that they failed for months due to "always-crunch" mentality.

My goal is 100% passing tests so we can break the build on unit test
  failures, but I can't do it until I address the broken tests.

I was obsessed with a similar goal even though I was just a junior developer in the team because I was noticing a pile-up where more tests were failing over the months. I wanted us to turn those from "warnings" into build errors (perhaps somewhat obnoxiously to the rest of the team).

I'm thinking we should document the statuses of the broken tests with
  anything we know, then either delete or ignore the broken tests
  completely and enter a lower-priority bug/work item to investigate and
  fix them. We'll then be at 100% and start to get real value out of the
  other tests and if we have a maintenance/refactoring windfall we'll be
  able to pick them up again.

These are my thoughts as well. You can temporarily disable all these faulty tests and slowly visit them and fix them over time. It's important to schedule those fixes if you consider them really important though, even if they're low priority, since it's easy for such items to simply go unfixed. The priority to me is to make sure that no new tests are introduced that fail.

Like any kind of warnings, if they don't break the build, they tend to pile up quickly. This is assuming that kind of team dynamic where the habit of ignoring warnings (failed tests in this case) can quickly lead to more warnings being introduced, and reduce the temptation to keep those warnings to zero.

A very conscientious team might not suffer these problems and avoid introducing new warnings (new failures in tests), but it's definitely safer to go a bit more heavy-handed and exercise a prevention strategy by turning these into errors that must be fixed prior to a merge process.

So my suggestion is the same as yours (albeit only a strong opinion -- maybe somewhat can back this up with metrics and a more scientific answer). Disable those old tests, and put it on the schedule to eventually fix them. The first priority is to make sure that this problem doesn't pile up and start getting worse by making sure the tests that currently succeed won't ultimately end up being ignored if they start failing.

Title: Broken Old/Legacy Unit Tests