问:
What is the correct approach for creating an application around a library?
I have a library that defines messages. In my particular instance, it's autogenerated code from an XSD using JAXB (Java). The messages can be very complex, with some members as objects and those objects containing other objects. The objects are both classes and enumerations. The library is a single JAR file with no dependencies, billed as a reusable component for Java applications. The idea is that any Java application that needs to send or receive these messages would use this library to do so. At present, there are 10 messages that are defined, but there are a large number of data types (classes and enumerations) that are also defined in the library that are contained within the messages.

The immediate need is for a graphical application that lets the user construct the messages in the specified format and then publish them appropriately over a network connection. This means that the user would be able to create the 10 messages and populate the various fields that go into them. The result is an XML string that conforms to the schema that can be published to a listening application. The listening application is also generating messages that are also defined in the XSD and the library in accordance with the schema and those must be received.

It is desirable to also be able to create a command-line version of this application that can be scripted. This command-line version is not well-defined, but it could be interactive (with prompts) or simply command-line argument driven to publish the messages.

What would the best approach to building my application around this library?

Two approaches come to mind:

The first approach is a a wrapper around the library that is part of the application build. This wrapper would be almost a duplication of the JAXB library, but it wouldn't invoke the generated data objects until requested. It would be a builder that would also provide validation and perhaps a simplified interface for some of the components of the messages. This could be useful, since the JAXB library doesn't have a concept of validation until the elements are marshalled. Although you can marshall to a default handler as opposed to your output format like a file or stream, but that seems not very useful since the messages you get from validation errors aren't exactly user-friendly. In this approach, the GUI, the text user interface, and the command-line interface would all interact with this application data model.

The second approach would be to organize my user interfaces into different packages. The GUI packages would provide user interface elements that correspond to the fields of each message. The validation rules from the XSD would be applied as validation rules to each field, using things like InputVerifiers, combo boxes, check boxes, and other things that prevent invalid data before it's even passed off to the data model. The text user interface would essentially provide the same thing, but by reading from and writing to the standard input, output, and error streams. I suspect in these implementations, the input classes would correspond to the data model classes 1-to-1 - each input class would be responsible for gathering and validating the user inputs needed to construct a single data element (one class in the data model provided by the JAXB library). I suspect that the textual UI and GUI can share base classes or interfaces that each of the UI classes can inherit from to allow them to be passed into the same controller to read from and produce the appropriate object from the data model. A command-line interface would likely use a command-line argument parser to gather the input, ensure everything is there, validate all of the fields, and then carry out the transmissions or receptions. It would be implemented very differently than the interactive user interfaces.

I'm personally leaning toward the second approach, but I suspect that there may be another, better strategy out there.

答:
I recommend that you make your application in such a way that it is easy to change if the library changes. If you use library components in different parts of your application, and the library changes, you will need to change many parts of the application. In work, we usually use a dedicated layer for performing request to external services (consumer layer) which is strictly isolated from other parts of the application.

What about a mix of your both approaches? Your first approach is somewhat similar to the layered approach I described above, while your second approach is more concerned with validation and reuseability, which seems reasonable. Can you do both while thinking how much work will it require to change your stuff if the library changes? I think this will give you clear separatation of concerns and make your application easier to maintain, and provide appropriate levels of reuseability.

You write that you want to make GUI packages for each message which sums up to 10 packages. I assume you mean Java package. I would have combined packages so that similar stuff was in same package. Eg. GUI in one, command line UI in one and the consumer layer in one.

Maybe all of these suggestions are too complex for your use case. I suggest you take a top-down approach, making skeletons or quickly prototyping both the GUI, command line UI and the library usage (for example just for one message) to see what is beneficial.

UPDATE

Consider if the changes in library are only technical while the library still provides the same functionality – e.g. someone performed refactoring like changing names of classes, rearranged code by moving fields from one class to another, or introduced new classes. You will still be hit by the changes but if you use a wrapper/decorator, you only need to change stuff there. You can still use the existing UIs as they are, you just have to map the data differently and this can be the wrappers job.

If the UIs are 100% reflection of the library, then you also must change the UIs and you get no benefit of using the wrapper, as you have stated. But then, you may use some existing GUI client for calling web services e.g. SoapUI which generates a client for you based on the web service contract (XSD in your case). This requires that the contract follow some standard like WSDL


问:
Does the phrase "range for R1-1 through R10H" actually have an agreed-upon meaning?
I was recently sent a code test by a potential employer to complete on my own time. The first question has to do with identifying whether a given zoning code falls within a range of codes. I'm not looking for the actual solution (someone else already tried that on SO) and I already declined to take the test since I was frustrated by this question as well as another one that was about screen scraping where the site in question was down for maintenance.

My question is just whether the "ranges" described are ambiguous or not. Is there some standard that clearly defines what is meant by "M1-1/R5 – M1-6/R10"? There is a hint for the first range that seems to suggest that this could be a self-contained question that doesn't require any domain-specific knowledge, except many of the test inputs, like C4-4A (no letters mentioned in the range with C codes), seem like they wouldn't be in any of the ranges until you research the zoning codes in New York and discover that they are actually valid, as shown in this document. Here is the original question:

Question 1 Description: Write a method which accepts a list/array of
  codes and returns the code along with its description as JSON.  Note
  that the range for R1-1 - R10H encompasses R1-1 to R10-10 and R1A to
  R10H.

Bonus: Write a unit test.

```
Description Data:
Codes               Descriptions
---------------------------------------------------------------
R1-1 - R10H         General Residence Districts
C1-6 - C8-4         Commercial Districts
M1-1 - M3-2         Manufacturing Districts
M1-1/R5 – M1-6/R10  Mixed Manufacturing & Residential Districts
BPC                 Battery Park City
PARK                New York City Parks
PARKNYS             New York State Parks
PARKUS              United States Parks
ZNA                 Zoning Not Applicable
ZR 11-151           Special Zoning District

Example output:
Given ['M3', 'R3-2', 'PARKNYS', 'M1-3/R9']
{
    "codes":[
        {"code":"M3", "description”:”Not found“},
        {"code":"R3-2", "description":"Residential Districts"},
        {"code":"PARKNYS", "description":"New York State Parks"},
        {"code":"M1-3/R9", "description":"Mixed Manufacturing & Residential Districts"}
    ]
}

Input:
The list is:
    R7A
    R8A
    C4-4A
    M3-2
    R8B
    C1-6A
    R7B
    R8X
    C1-7A
    PARK
    C1-9A
    R6
    C1-7
    C2-6
    R10
    C4-5
    C6-3X
    C1-6
    C6-2M
    C6-4M
    M2-4
    M1-5/R7X
```

UPDATE: The answer from MichaelT makes some good points, but I disagree that the hint in the first question gives you the complete list of zoning codes. It only does so for the first range.

答:
The description of the range is poor.  However, there is a clearly defined in the zoning codes.

From section 11-122 of Zoning Resolution (Web Version) (the linked pdf)

```
R1-1   Single-Family Detached Residence District
R1-2   Single-Family Detached Residence District
R1-2A   Single-Family Detached Residence District

R2     Single-Family Detached Residence District
R2A    Single-Family Detached Residence District
R2X    Single-Family Detached Residence District

...

R9     General Residence District
R9-1   General Residence District
R9A    General Residence District
R9D    General Residence District
R9X    General Residence District

R10    General Residence District
R10A   General Residence District
R10H   General Residence District
R10X   General Residence District
```

Thus, R1-1 - R10H is intended to convey this range.

Lets face it, often in the real world business logic is poorly defined. Summaries of the logic are even worse.  It is the responsibility of the programmer to seek clarity on the poorly defined requirements.  In this case, that is done by looking at the associated comment which gives you the complete list of the acceptable zoning codes.

At this point, you could stick every zoning code into a dictionary and not worry about ranges at all if you wanted to (I'd read the dictionary from an associated data file, but thats just me).  Thats probably best as there isn't a range for these.  Any regex to try to match just those described would be hopelessly complex.

It should be perfectly acceptable to return an error when someone passes the information R9-2 since that isn't a valid zoning code.  But that's my take on the requirements.

When in doubt what the requirements are, clarify the requirements.  Don't guess about requirements - you can often find yourself coding for a period of time just to find out you misunderstood them on day 1 and have been going down the wrong path.

Yes, this is apparently part of a job interview test. Remember that programming isn't just writing code, its also gathering requirements (and many other things).

ContentLicense: CC BY-SA 3.0

Body:
The description of the range is poor.  However, there is a clearly defined in the zoning codes.

From section 11-122 of Zoning Resolution (Web Version) (the linked pdf)

R1-1   Single-Family Detached Residence District
R1-2   Single-Family Detached Residence District
R1-2A   Single-Family Detached Residence District

R2     Single-Family Detached Residence District
R2A    Single-Family Detached Residence District
R2X    Single-Family Detached Residence District

...

R9     General Residence District
R9-1   General Residence District
R9A    General Residence District
R9D    General Residence District
R9X    General Residence District

R10    General Residence District
R10A   General Residence District
R10H   General Residence District
R10X   General Residence District

Thus, R1-1 - R10H is intended to convey this range.

Lets face it, often in the real world business logic is poorly defined. Summaries of the logic are even worse.  It is the responsibility of the programmer to seek clarity on the poorly defined requirements.  In this case, that is done by looking at the associated comment which gives you the complete list of the acceptable zoning codes.

At this point, you could stick every zoning code into a dictionary and not worry about ranges at all if you wanted to (I'd read the dictionary from an associated data file, but thats just me).  Thats probably best as there isn't a range for these.  Any regex to try to match just those described would be hopelessly complex.

It should be perfectly acceptable to return an error when someone passes the information R9-2 since that isn't a valid zoning code.  But that's my take on the requirements.

When in doubt what the requirements are, clarify the requirements.  Don't guess about requirements - you can often find yourself coding for a period of time just to find out you misunderstood them on day 1 and have been going down the wrong path.

Yes, this is apparently part of a job interview test. Remember that programming isn't just writing code, its also gathering requirements (and many other things).


问:
Is there accepted decimal-based Unicode notation for technical audiences?
When writing for technical audiences, there are various ways to type Unicode representations, but they all seem to be Hexadecimal:

\uFFFF - From C# / Java Strings

\U0000FFFF - From C# / Java Strings

However, Unicode can also be specified in decimal, and can be entered both ways in XML Entities:

Hexadecimal:

Decimal:

While I could just (kind of) use \u65535, this does make something that is already defined as specifically hexadecimal and abuse it, and also could cause problems - is \u1111 decimal or hex?

So - are there any programming languages that allow similar ways to denote Unicode characaters by decimal, or common shorthand conventions for specifying Unicode in decimal notation for technical audiences?

答:
There is no commonly accepted decimal notation for Unicode codepoints.

Unicode codepoints are almost universally represented in hexadecimal.  The sole exception I'm aware of is the use of Numeric Character References (NCRs) in languages derived from SGML (e.g., HTML and XML), which can take one of two forms: &#nnn; in decimal or &#xnnn; in hexadecimal.

In other contexts, some languages have attempted to intermingle differing numeric bases - the best-known example being C's use of nnn for decimal, 0nnn for octal, and 0xnnn for hexadecimal.  Even with this well-known usage, it trips up beginning C programmers all the time that 012 and 12 are different numbers.


问:
Design pattern for multiple companies in AngularJS
We are on the verge of launching v3 of our in-house product. v2 was written completely in AngularJS and V3 will be the same, but written completely in Typescript.

We have a few customers that have very specific needs for the product and each have their own version (hosted on our own platform), and have their own deployment. There are common shared parts which is the same for everybody.

At the moment everything is in 1 solution (Visual Studio) which we prefer due to ease of maintaining it, but it made a big mess of mixing of settings, company specific options etc. There are a lot of if-else statements in the controllers, directives and even views (still, all angularJS) to accompany for multiple companies.

Now in V3 we are rethinking this setup and want to become more flexible, every company will have its own controllers, directives and views, with the option of some parts being shared and using a service/data layer to handle common shared tasks.

Now I am thinking how can I make this manageable? At the moment we have 6 customers but this is expected to grow with 10-15 each year. I am afraid we will again end up with a big pile of files scattered through the project which makes it hard to maintain and test.

I want to avoid this and keep the project flexible and manageable.

Should I figure out a way to split every company in separate solutions with the common Javascript/Typescript files for app startup and routing in a common project? Maybe just have a single folder for every company with custom controllers, directives and views, but how would the routing be handled?

How do we approach managing enterprise apps with multiple companies?

PS. Our web application is angular because almost everything is client side, it has the ability to work 100% offline for longer periods of time, the backend side is not important for this problem.

答:
Perhaps it may make sense to factor everything into services first and then break down each of the company specific things into its own application.  If memory serves me you can have multiple angular applications on a single page so it should be possible to set things up to route each of your clients to the right place.

I assume that you could then setup your grunt or gulp file with a kind of master task to handle building and running the tests for each project.


问:
Use interface or abstract class?
Now the title might seem like the question has been asked before but let me explain my situation and you'll see why I am asking this question.

Let's consider this interface :

```<code>public interface IListChangedListener {
    public event EventHandler ItemAdded;
    public event EventHandler ItemRemoved;
}
</code>```

If someone wants to implement this interface, they would have to check if ItemAdded is null and only if it is not null, they would call it. Well, I want to simplify their job by giving an in built function which will do the same. Just calling that function will check if the corresponding event is null, and if it isn't, it'll fire its corresponding event.

The problem is, this is an interface. So I can't define a method. So I thought I might go with an abstract class. But considering the situation, it would be best suited to make this as an interface rather than an abstract class.

So do I just drop the idea of giving an in built function? Or do I make it an abstract class? Which better suits the situation? I am having a tough time deciding which one to use. Some help please?

答:
You should have both the interface and a class with a default implementation. For example, if you have an IDog interface, it could have a Bark() method defined. A separate Voicebox class could provide the default implementation. This could look like this:

```<code>public interface IDog
{
    public void Bark();
}

public class VoiceBox
{
    public void Bark()
    {
        // Do bark
    }
}

class Poodle : IDog
{
    private Voicebox voicebox = Voicebox.GetDefault();

    public Bark()
    {
        voicebox.Bark();
    }
}
</code>```

This way, the client is not required to use your Voicebox class, but they are still required to make their dog bark somehow. If they want to use your default implementation, then they can use the Voicebox class. Of course, you can make the bark method virtual in the Voicebox class if it needs to be overridden by subclasses.

Title: Use interface or abstract class?

Body:
You should have both the interface and a class with a default implementation. For example, if you have an IDog interface, it could have a Bark() method defined. A separate Voicebox class could provide the default implementation. This could look like this:

public interface IDog
{
    public void Bark();
}

public class VoiceBox
{
    public void Bark()
    {
        // Do bark
    }
}

class Poodle : IDog
{
    private Voicebox voicebox = Voicebox.GetDefault();

    public Bark()
    {
        voicebox.Bark();
    }
}

This way, the client is not required to use your Voicebox class, but they are still required to make their dog bark somehow. If they want to use your default implementation, then they can use the Voicebox class. Of course, you can make the bark method virtual in the Voicebox class if it needs to be overridden by subclasses.


问:
How do you maintain protocol of function return value in dynamic language?
In dynamic programming language like Python, Javascript,
it's very easy to make a function return an object:

```<code>def make_vector2(a, b):
    return {"x": a, "y": b}
</code>```

The 'signature' of the returned object from make_vector2 acts as a protocol or interface to other functions.
For example:

```<code>def norm(v):
    return math.sqrt(v["x"]**2 + v["y"]**2)
</code>```

During the evolution of a program, there may be changes on the signature of the returned object.
For example, I would change above vector definition to {"x0": a, "x1":b}.

This protocol change may invalidate its related functions(i.e. norm in above example).

In static programming language, I have compiler help me to keep the protocol consistent.
While in dynamic programming language I cannot detect the problems unless I actually run the code.

Is there any suggestions/good tools for this problem?

答:
People who prefer dynamic languages don't consider this to be a problem in the first place.  the basic tools used to deal with this are simply good testing.  Yes, you cannot detect a problem such as you describe without running the code, so the key is to run the code through a good test-suite, probably as often as you compile.

There are disadvantages to this, of course.  If code doesn't get tested, the error goes undetected whereas a compiler would have caught it.  But what you lose there, you gain in speed of development, and the ability to use looser coupling.

Your code will of course throw when norm is called.  So if you run a basic test suite that calls norm with the results of make_vector2 at least once, you'll catch the error.  If you run those tests as often as you would compile a C++ or Java program, then it really makes no difference.  Either way, the bug gets caught.

The other part of the answer is to try to design the code in a way that the contract is more clear.  For example:

```<code>class Vector:
    def __init__(self,x,y):
        self.x = x
        self.y = y

def make_vector2(a, b):
    return Vector(a,b)

def norm(v):
    return math.sqrt(v.x**2 + v.y**2)
</code>```

That still has to be tested the same way as the original code, of course, but this documents the protocol in a way that the earlier version doesn't.  It means that you can do a help(make_vector2) and see that it returns a Vector object with an x and a y attribute.  This makes the error less likely in the first place.

In general, returning raw dictionaries is best when you are intentionally keeping things loosely coupled and therefore can handle more variant contents.

Body:
People who prefer dynamic languages don't consider this to be a problem in the first place.  the basic tools used to deal with this are simply good testing.  Yes, you cannot detect a problem such as you describe without running the code, so the key is to run the code through a good test-suite, probably as often as you compile.

There are disadvantages to this, of course.  If code doesn't get tested, the error goes undetected whereas a compiler would have caught it.  But what you lose there, you gain in speed of development, and the ability to use looser coupling.

Your code will of course throw when norm is called.  So if you run a basic test suite that calls norm with the results of make_vector2 at least once, you'll catch the error.  If you run those tests as often as you would compile a C++ or Java program, then it really makes no difference.  Either way, the bug gets caught.

The other part of the answer is to try to design the code in a way that the contract is more clear.  For example:

class Vector:
    def __init__(self,x,y):
        self.x = x
        self.y = y

def make_vector2(a, b):
    return Vector(a,b)

def norm(v):
    return math.sqrt(v.x**2 + v.y**2)

That still has to be tested the same way as the original code, of course, but this documents the protocol in a way that the earlier version doesn't.  It means that you can do a help(make_vector2) and see that it returns a Vector object with an x and a y attribute.  This makes the error less likely in the first place.

In general, returning raw dictionaries is best when you are intentionally keeping things loosely coupled and therefore can handle more variant contents.


答: One approach is to use a software contract system, specifically one with support for function contracts and higher-order function contracts. The general idea is you use contract-creation tools to wrap a function with rules that specify "I accept these kinds of values and return this kind of value." It's similar to a type system, except checked dynamically. The most prominent implementation of such a system can be found in the language Racket, though implementations exist for other dynamic languages including JavaScript and Python.</p>

Title: How do you maintain protocol of function return value in dynamic language?

Body:
In dynamic programming language like Python, Javascript,
it's very easy to make a function return an object:

```<code>def make_vector2(a, b):
    return {"x": a, "y": b}
</code>```

The 'signature' of the returned object from make_vector2 acts as a protocol or interface to other functions.
For example:

```<code>def norm(v):
    return math.sqrt(v["x"]**2 + v["y"]**2)
</code>```

During the evolution of a program, there may be changes on the signature of the returned object.
For example, I would change above vector definition to {"x0": a, "x1":b}.

This protocol change may invalidate its related functions(i.e. norm in above example).

In static programming language, I have compiler help me to keep the protocol consistent.
While in dynamic programming language I cannot detect the problems unless I actually run the code.

Is there any suggestions/good tools for this problem?


问:
How important are code formatting guidelines?
Coding standards are common in any software development organization, but how important are they to follow?  I can understand the need for some consistency, but when dealing with simple things like the position of braces, line length, etc., I'm not sure excessively strict standards contribute much to software development.

Isn't it more important that your code is readable, not that it conforms to a predefined standard?  It seems they're more like... guidelines anyway.

答: For formatting standards, I follow what everybody else is doing. If they are using PascalCase for everything, then I use PascalCase. If they use _camelCase, then I use _camelCase. Why? Because it limits the amount of reformatting I do, and limits what others have to do to make it "look good". Formatting standards are usually there to make things easy for everybody.</p>

ContentLicense: CC BY-SA 2.5


问:
Two Database Architecture : Operational and Historical
I thought about an uncommon database structure and wonder if anyone has seen it in use before. It's basically using 2 databases:

First database hold only the data that is currently valid

Second database hold the history of everything that has ever been entered, updated or
deleted in the first database

Scenario

I'm working on a project where I'm required to log everything that happens and where the data changes frequently.

Example (not the real one)

You have to do the database design for a soccer league. In this league there are players and teams. The players often switch teams.

First requirement : The database must hold the information necessary to play the   next match. This means a list of every players, teams and in which team each player is currently.

Second requirement : The database must hold historical values that we will use to generate statistics. This means the list of all the players that have been part of a team or the list of all the teams a player has been a part of.

The problem

These two requirements are kinda the opposite of each other. I have tried to do everything in the same database but it doesn't make sense. The first requirement only cares about "playing the next match" while the second requirement only cares about "generating statistics".

To do everything in the same database, I went with a sort of "insert only" database using the obvious soft delete to delete/update information...

What initially seemed like an easy task, holding a list of players, teams and the current team of each players, suddenly becomes a lot harder. The application logic required to play the next match is already complicated enough but now the database has a very unhelpful design where the application is required to add "is deleted" check on every single query just to play the next match.

Would you want to be that coach that yell "all players in the team, come to me" and then 2000 players come at you. At which point, you will probably yell "all players that are not deleted in the team, come to me" (while swearing about this stupid design).

My conclusion

I came to wonder why you need to put everything in the same database. Not only does the soft delete do a poor job at logging everything unless you add many columns (time_created, who_created_it, time_deleted, who_deleted_it) but it also complicate everything. It complicates the database design and it complicates the application design.

Also, I receive these 2 requirements as part of one single application that cannot be split but I keep thinking : this is 2 completely distinct applications. Why am I trying to do everything together.

That's when I thought about splitting the database in two. An operational database that is used only to play the next match and only contain the information that is currently valid and an historical database that hold all the information that ever existed, when it was created, deleted and who did it.

The goal is to keep the first database (operational) and the application as simple as possible while having as much information as possible in the second database (historical).

Questions

Have you seen that design before? Does it have name?

Are there any obvious pitfalls that I am missing?

EDIT 2015-03-16

Current architecture

You can basically think about the whole architecture as a 2 steps process.

Step 1 :

Application is running and users are doing some actions

Each time an event happen, it is recorded automatically (audit solution) in an event table

Then the correct row, in the operational database is updated

Step 2 :

A job read the latest insertion in the event table and insert this new data in the
historical database.

Users query the historical database to retrieve the information that they need.

Just from the event table, you can reconstruct the information to any point in time. The problem is that this event table is not easily queryable. This is where the historical database kicks in; to present the data into a way that it is easy to retrieve exactly what we want.

Additional problems when putting everything in the same tables

I have already expressed my concern about the added complexity of checking "is deleted" on each query. But there is another issue : integrity.

I make heavy use of foreign key and constraint to make sure that at any point in time, the data that is in my database is valid.

Let's look at an example :

Constraint : There can only be one goal keeper per team.

It's easy to add a unique index that check if there is only one goal keeper per team. But then what happen when you change the goal keeper. You still need to preserve the information about the previous one but now you have 2 goal keepers in the same teams, one active and one inactive, which contradicts your constraint.

Sure it's easy to add a check to your constraint, but it's another thing to manage and think about.

答:
This is actually similar to the way database transactions are generally implemented, except the historical data is usually thrown away after being written to the operational database. The closest programming pattern I can think of is event sourcing.

I think splitting these two databases up is the right move. More specifically, I would look at the "operational" database as a cache, since the historical data will be sufficient to re-construct the operational database at any time. Depending on the nature of your application and performance requirements, it may be unnecessary to maintain this cache as a separate database if it is reasonable to reconstruct the current state from the historical data in memory each time the program is started.

As far as pitfalls, the main issue you could run into is if you need any kind of concurrency (either in the same program or by having multiple clients use the database at the same time). In that case you would want to make sure that the modifications to the historical and operational databases is done atomically. In the case of concurrency within the same program, your best bet is probably some sort of locking mechanism. For multiple clients interacting with the same database, the easiest way would be to keep both as tables in the same database, and use transactions to keep the database consistent.


问:
Data padding for different hardware support?
Some hardware cannot read non-aligned data. For example Bitmap images are aligned to 4-bytes from the header and with each scan line in order to maintain device independence.

For example you can cause an alignment fault by accessing unaligned memory as per the example in the wiki. However, does this actually hold true for files?

If I tried to read data from a binary file(like Bitmaps) at an non-aligned index will this cause an alignment fault on certain hardware? My guess is that it will because Bitmaps safeguard against this, however is it entirely necessary to take this precaution if I want to make a binary file or even a piece of code to load that file device independent?

答:
If I tried to read data from a binary file(like Bitmaps) at an
  non-aligned index will this cause an alignment fault on certain
  hardware?

Yes, it will. This depends on the hardware, some are more and some less tolerant, but it can cause faults. In particular, in C reading from improperly aligned address is undefined behaviour, meaning anything (crash, data corruption, nothing) may happen.

The only safe and platform-independent way is to always read byte-by-byte. If that is too inefficient, you'll need a platform-dependent solution, which can then perform only the alignment truly needed by this platform.


答:
First things first, does your codebase already offer a clean separation of concerns where the business logic (of picking players to play in the next match) is distinguished from your database access logic (a layer that simply connects to the database and maps your data structures into database rows and vice versa)? The answer for this will go a great deal in explaining why you are dealing with this:

It complicates the database design and it complicates the application design.

Now...

The application logic required to play the next match is already complicated enough but now the database has a very unhelpful design where the application is required to add "is deleted" check on every single query just to play the next match.

Assuming you are talking about RDBMS, you can still have a bitemporal database that captures all valid data past, present and possibly future, and then use a robust-enough database access library/ORM framework to handle the database querying logic for you. You may even use a database view to aid in your selection. Then, the business logic parts of your code should not need to know the underlying temporal fields, which will eliminate the problem you described above.

For example, instead of having to hard-code a SQL query in your application:

```<code>SELECT * FROM team_players WHERE team_id = ? AND valid_from >= ? AND valid_to <= ? AND ...
</code>```

(using ? as parameter bindings)

A hypothetical database access library might let you query as (pseudo-code):

```<code>dbConnection.select(Table.TEAM_PLAYERS).match(Field.TEAM_ID, <value>).during(Season.NOW)
</code>```

Or using the database view approach:

```<code>-- Using MySQL dialect for illustration
CREATE VIEW seasonal_players AS SELECT * FROM team_players
    WHERE valid_from >= ... AND valid_to <= ... AND ...
</code>```

You can then query for players during that period as:

```<code>SELECT * FROM seasonal_players WHERE team_id = ?
</code>```

Back to your suggested database model, how do you intend to handle the removal of historical data from your operational database? Do you simply INSERT two similar rows into your operational and historical databases each, and then run a DELETE on the operational database whenever there is a user action to delete historical data?

Think Big

If you're talking about large-scale data processing where your 'database' is a scaled-out distributed database-clustering/stream-processing solution, your approach will sound vaguely similar (probably just on some of the defined terms) to the Lambda Architecture, in which your 'historical' (i.e. past-real-time) data is batch processed separately to perform the kind of statistics you are looking for, and your 'operational' (i.e. streaming real-time) data remains query-able for a predefined limit before the batch processing persists them. However, the basis of this approach is driven more by the advantages and limitations of current Big Data implementations, than mere simplification of one's application logic.

edit (after OP's edit)

I should have replied to this earlier, but anyways:

Also, I receive these 2 requirements as part of one single application that cannot be split but I keep thinking : this is 2 completely distinct applications. Why am I trying to do everything together.

That's generally because end-users tend to think in terms of features, not number of databases required.

You also mention that:

Step 1:

Each time an event happen, it is recorded automatically (audit solution) in an event table.

Then the correct row, in the operational database is updated.

Step 2:

A job reads the latest insertion in the event table and insert this new data in the historical database.

Great! So now you have an event table, which I presume really is the event sourcing concept mentioned by the other answers. However, what is it that reads your event table and update the correct row in the operational database? Is that the same as the job that reads the last event and inserts it into the historical database?

One more point regarding your constraint example:

I make heavy use of foreign key and constraint to make sure that at any point in time, the data that is in my database is valid.

Let's look at an example :

Constraint : There can only be one goal keeper per team. (active on the pitch during a game, just a side-note)

Is "any point in time" referring to valid time or transaction time?

What happens when we have a third new goalkeeper? Do you create an unique constraint on temporal fields in the historical database to keep the data for two "old" goalkeepers valid?

Title: Two Database Architecture : Operational and Historical

Body:
I thought about an uncommon database structure and wonder if anyone has seen it in use before. It's basically using 2 databases:

First database hold only the data that is currently valid

Second database hold the history of everything that has ever been entered, updated or
deleted in the first database

Scenario

I'm working on a project where I'm required to log everything that happens and where the data changes frequently.

Example (not the real one)

You have to do the database design for a soccer league. In this league there are players and teams. The players often switch teams.

First requirement : The database must hold the information necessary to play the   next match. This means a list of every players, teams and in which team each player is currently.

Second requirement : The database must hold historical values that we will use to generate statistics. This means the list of all the players that have been part of a team or the list of all the teams a player has been a part of.

The problem

These two requirements are kinda the opposite of each other. I have tried to do everything in the same database but it doesn't make sense. The first requirement only cares about "playing the next match" while the second requirement only cares about "generating statistics".

To do everything in the same database, I went with a sort of "insert only" database using the obvious soft delete to delete/update information...

What initially seemed like an easy task, holding a list of players, teams and the current team of each players, suddenly becomes a lot harder. The application logic required to play the next match is already complicated enough but now the database has a very unhelpful design where the application is required to add "is deleted" check on every single query just to play the next match.

Would you want to be that coach that yell "all players in the team, come to me" and then 2000 players come at you. At which point, you will probably yell "all players that are not deleted in the team, come to me" (while swearing about this stupid design).

My conclusion

I came to wonder why you need to put everything in the same database. Not only does the soft delete do a poor job at logging everything unless you add many columns (time_created, who_created_it, time_deleted, who_deleted_it) but it also complicate everything. It complicates the database design and it complicates the application design.

Also, I receive these 2 requirements as part of one single application that cannot be split but I keep thinking : this is 2 completely distinct applications. Why am I trying to do everything together.

That's when I thought about splitting the database in two. An operational database that is used only to play the next match and only contain the information that is currently valid and an historical database that hold all the information that ever existed, when it was created, deleted and who did it.

The goal is to keep the first database (operational) and the application as simple as possible while having as much information as possible in the second database (historical).

Questions

Have you seen that design before? Does it have name?

Are there any obvious pitfalls that I am missing?

EDIT 2015-03-16

Current architecture

You can basically think about the whole architecture as a 2 steps process.

Step 1 :

Application is running and users are doing some actions

Each time an event happen, it is recorded automatically (audit solution) in an event table

Then the correct row, in the operational database is updated

Step 2 :

A job read the latest insertion in the event table and insert this new data in the
historical database.

Users query the historical database to retrieve the information that they need.

Just from the event table, you can reconstruct the information to any point in time. The problem is that this event table is not easily queryable. This is where the historical database kicks in; to present the data into a way that it is easy to retrieve exactly what we want.

Additional problems when putting everything in the same tables

I have already expressed my concern about the added complexity of checking "is deleted" on each query. But there is another issue : integrity.

I make heavy use of foreign key and constraint to make sure that at any point in time, the data that is in my database is valid.

Let's look at an example :

Constraint : There can only be one goal keeper per team.

It's easy to add a unique index that check if there is only one goal keeper per team. But then what happen when you change the goal keeper. You still need to preserve the information about the previous one but now you have 2 goal keepers in the same teams, one active and one inactive, which contradicts your constraint.

Sure it's easy to add a check to your constraint, but it's another thing to manage and think about.

Body:
First things first, does your codebase already offer a clean separation of concerns where the business logic (of picking players to play in the next match) is distinguished from your database access logic (a layer that simply connects to the database and maps your data structures into database rows and vice versa)? The answer for this will go a great deal in explaining why you are dealing with this:

It complicates the database design and it complicates the application design.

Now...

The application logic required to play the next match is already complicated enough but now the database has a very unhelpful design where the application is required to add "is deleted" check on every single query just to play the next match.

Assuming you are talking about RDBMS, you can still have a bitemporal database that captures all valid data past, present and possibly future, and then use a robust-enough database access library/ORM framework to handle the database querying logic for you. You may even use a database view to aid in your selection. Then, the business logic parts of your code should not need to know the underlying temporal fields, which will eliminate the problem you described above.

For example, instead of having to hard-code a SQL query in your application:

SELECT * FROM team_players WHERE team_id = ? AND valid_from >= ? AND valid_to <= ? AND ...

(using ? as parameter bindings)

A hypothetical database access library might let you query as (pseudo-code):

dbConnection.select(Table.TEAM_PLAYERS).match(Field.TEAM_ID, <value>).during(Season.NOW)

Or using the database view approach:

-- Using MySQL dialect for illustration
CREATE VIEW seasonal_players AS SELECT * FROM team_players
    WHERE valid_from >= ... AND valid_to <= ... AND ...

You can then query for players during that period as:

SELECT * FROM seasonal_players WHERE team_id = ?

Back to your suggested database model, how do you intend to handle the removal of historical data from your operational database? Do you simply INSERT two similar rows into your operational and historical databases each, and then run a DELETE on the operational database whenever there is a user action to delete historical data?

Think Big

If you're talking about large-scale data processing where your 'database' is a scaled-out distributed database-clustering/stream-processing solution, your approach will sound vaguely similar (probably just on some of the defined terms) to the Lambda Architecture, in which your 'historical' (i.e. past-real-time) data is batch processed separately to perform the kind of statistics you are looking for, and your 'operational' (i.e. streaming real-time) data remains query-able for a predefined limit before the batch processing persists them. However, the basis of this approach is driven more by the advantages and limitations of current Big Data implementations, than mere simplification of one's application logic.

edit (after OP's edit)

I should have replied to this earlier, but anyways:

Also, I receive these 2 requirements as part of one single application that cannot be split but I keep thinking : this is 2 completely distinct applications. Why am I trying to do everything together.

That's generally because end-users tend to think in terms of features, not number of databases required.

You also mention that:

Step 1:

Each time an event happen, it is recorded automatically (audit solution) in an event table.

Then the correct row, in the operational database is updated.

Step 2:

A job reads the latest insertion in the event table and insert this new data in the historical database.

Great! So now you have an event table, which I presume really is the event sourcing concept mentioned by the other answers. However, what is it that reads your event table and update the correct row in the operational database? Is that the same as the job that reads the last event and inserts it into the historical database?

One more point regarding your constraint example:

I make heavy use of foreign key and constraint to make sure that at any point in time, the data that is in my database is valid.

Let's look at an example :

Constraint : There can only be one goal keeper per team. (active on the pitch during a game, just a side-note)

Is "any point in time" referring to valid time or transaction time?

What happens when we have a third new goalkeeper? Do you create an unique constraint on temporal fields in the historical database to keep the data for two "old" goalkeepers valid?


问:
Should I avoid using style like `for k, v in dict_sample.items()`?
Today I was viewing my colleague's code and I saw a function like this:

```<code>def manager_skill_tree_func(*args, **kwargs):
    """# manage_skill_tree: Initialize the manage skill tree
    """
    skill_tree0 = {}
    head = []
    tree_skill_list = {}
    manage_skill = globals()['manage_skill']

    for k, v in manage_skill.iteritems():
        if not k in skill_tree0:
            skill_tree0[k] = {}
        if v['tree'] not in tree_skill_list:
            tree_skill_list[v['tree']] = set([k])
        else:
            tree_skill_list[v['tree']].add(k)
        pre_k_list = v['pre_skill']
        if not pre_k_list:
            head.append(k)
        for pre_k in pre_k_list:
            pre_k = int(pre_k)
            if not pre_k in skill_tree0:
                skill_tree0[pre_k] = {}
            skill_tree0[pre_k][k] = skill_tree0[k]

    # Rest of the function is omitted here
</code>```

There are lots of readability issues in this piece of code, but my attentions was mostly draw by the for k, v in manage_skill.itermitems() statement. This kind of statement was heavily used in the project, it seems that my colleague doesn't like to specify what kind of keys and values they are obtaining from the dict.

For me, I'll use something like for index, arg_dict in dict_sample.iteritems() if the key is a number and the value is another dict, and for employee_tuple in employee_info to specify the item type I'll get tuple from the list.

I think names should offer some meaning, in the statement for pre_k in pre_k_list: of above code, I got no information about what kind of data this pre_k is, except that it's not called k again.

I would use for k, v in sample_dict.items() or for i in sample_list only when the structure was already mentioned and is really obvious, like:

```<code>import random
sample_dict = {}
for i in xrange(10):
    sample_dict[i] = random.choice([1,2,3])

for k,v in sample_dict.items():
    sample_dict[k] = k/v
</code>```

What's your opinion on for k,v in unknown_structure_dict.iteritems()?

PS: As an extra gift, here is a pylint evaluation of a large file for @jonrsharpe.

A weak defend for this disaster score:

Project lasts for years

Written by plenty of programmers(from novice to veteran)

Lots of requirement changes, lots of hot-fix.

Some modules or classes are loaded at running time.

Anyway, enjoy this:

```<code>+-----------------------------+------------+
|message id                   |occurrences |
+=============================+============+
|bad-whitespace               |1572        |
+-----------------------------+------------+
|bad-continuation             |479         |
+-----------------------------+------------+
|line-too-long                |303         |
+-----------------------------+------------+
|unused-argument              |99          |
+-----------------------------+------------+
|invalid-name                 |95          |
+-----------------------------+------------+
|no-member                    |65          |
+-----------------------------+------------+
|missing-docstring            |32          |
+-----------------------------+------------+
|redefined-outer-name         |31          |
+-----------------------------+------------+
|trailing-whitespace          |14          |
+-----------------------------+------------+
|unused-variable              |8           |
+-----------------------------+------------+
|global-variable-not-assigned |6           |
+-----------------------------+------------+
|fixme                        |6           |
+-----------------------------+------------+
|unnecessary-semicolon        |5           |
+-----------------------------+------------+
|bare-except                  |3           |
+-----------------------------+------------+
|eval-used                    |2           |
+-----------------------------+------------+
|too-many-locals              |1           |
+-----------------------------+------------+
|too-many-lines               |1           |
+-----------------------------+------------+
|superfluous-parens           |1           |
+-----------------------------+------------+
|multiple-statements          |1           |
+-----------------------------+------------+
|empty-docstring              |1           |
+-----------------------------+------------+

Global evaluation
-----------------
Your code has been rated at -47.74/10
</code>```

答:
I agree with you, and e.g. pylint would complain about those names too (albeit purely on a length basis).

for k, v in ... gives the reader no helpful information about what they should be expecting to get from the dictionary, which makes the subsequent code harder to follow. For example, it's only at if v['tree']... that you find out that the value is (presumably!) a dictionary. This requires you to look at the code where the dictionary is built to figure out what should be in it, and means you cannot just read and understand the function on its own (especially as the docstring is useless).

Even something as simple as:

```<code>for skill_name, skill_dict in manage_skill.iteritems():
</code>```

would make the code clearer.

Body:
I agree with you, and e.g. pylint would complain about those names too (albeit purely on a length basis).

for k, v in ... gives the reader no helpful information about what they should be expecting to get from the dictionary, which makes the subsequent code harder to follow. For example, it's only at if v['tree']... that you find out that the value is (presumably!) a dictionary. This requires you to look at the code where the dictionary is built to figure out what should be in it, and means you cannot just read and understand the function on its own (especially as the docstring is useless).

Even something as simple as:

for skill_name, skill_dict in manage_skill.iteritems():

would make the code clearer.


答:
It happens quite often, though the history (sometimes known as audit records) is kept either in the same table, or in a separate one in the same database.

For example, I used to work with a system where any updates to a table would be implemented as an insert, the old 'current' record would have a flag set on it saying it was a historical record, and the timestamp when it was updated written to a column.

Today I work on a system where every change is written to a dedicated audit table and then the update occurs on the table.

The latter is more scalable, but not as easy to implement in a generic manner.

The easiest way to achieve your goal of making queries simple and not requiring adding the 'is current' flag is to only allow read queries via a view or stored procedure. Then you make a call to say "get all players" and the stored proc will return only current players (you can implement a 2nd procedure to return players with more control over which ones are returned). This works well for writing too. A stored procedure for updating a player can then write whatever history details are needed and update the player - without the client ever knowing what the history mechanism is. For this reason, stored procedures are better than a view which returns only current players, as it keeps the entire DB access mechanism the same for reading and writing - everything goes through a sproc.


答:
Some hardware cannot read non-aligned data.

True.

For example Bitmap images
  are aligned to 4-bytes from the header and with each scan line in
  order to maintain device independence.

I have a number of points to make on this sentence:

The internal organization of bitmap files is not an example of hardware being or not being able to read non-aligned data.

The word "device" in the term "device independence" for bitmaps refers to the display, not to the CPU.  Data alignment is an issue that the CPU has to deal with.

The reason for this organization is not in order to maintain device independence. Windows .BMP files (I suppose that's what you mean by "Bitmap images") are 4-byte aligned because the x86 architecture will perform best when reading aligned data.  It can, however, read non-aligned data. It just won't perform as fast.

For example you can cause an alignment fault by accessing unaligned
  memory as per the example in the wiki. However, does this actually
  hold true for files?

Yes and no.  Obviously, a CPU could not care less whether a certain file sitting on the disk is or is not aligned.  But in order for the file to be of any use at all, it will need to be loaded into memory at some point, right?  That's when proper alignment is useful.

If I tried to read data from a binary file(like Bitmaps) at an
  non-aligned index will this cause an alignment fault on certain
  hardware?

Yes, it will.  But not on any x86 or x64.  On a x86 & x64 you will just get sub-optimal performance.

My guess is that it will because Bitmaps safeguard against
  this,

Taking file format organizations as hints about potential CPU behaviors is as misguided as taking football team colors as hints about stock market investment decisions.

however is it entirely necessary to take this precaution if I
  want to make a binary file or even a piece of code to load that file
  device independent?

Again, regarding "Device Independent",

I presume that by "device independent" you mean "hardware architecture independent", (which actually means "CPU independent",) so the answer here is that it depends on how independent you want to be.

If you want to be completely independent, then you cannot go with binary files at all.

If you want to be moderately independent, then go ahead with your binary file but make sure to use an alignment of 4 or better yet of 8.

If you want to be just reasonably independent, ignore alignment.  It is just a file.  If someone has to read it on some weird hardware, they can read it byte by byte.

Title: Data padding for different hardware support?

Body:
Some hardware cannot read non-aligned data. For example Bitmap images are aligned to 4-bytes from the header and with each scan line in order to maintain device independence.

For example you can cause an alignment fault by accessing unaligned memory as per the example in the wiki. However, does this actually hold true for files?

If I tried to read data from a binary file(like Bitmaps) at an non-aligned index will this cause an alignment fault on certain hardware? My guess is that it will because Bitmaps safeguard against this, however is it entirely necessary to take this precaution if I want to make a binary file or even a piece of code to load that file device independent?


问:
What is the difference between data retention and data archival?
The article at http://www.complianceguru.com/2010/11/archiving-vs-retention-of-email-and-other-electronic-data/ seems to suggest there is a difference. It isn’t very clear though what if the difference is apart for the length of time it may be stored.

The description below also suggests that other than archived data e.g. retained data is mutable or changeable.

Now, retention vs. archiving: Think of an archive as a non-alterable backup.  Some archive solutions also add a search feature, but the key is that the data cannot be deleted or modified in any way.  So why consider archiving instead of simple retention?

Is there a difference and if so what are the differences?

答:
"Archiving" is a bit like an overflow car park.  When the main one gets "full", you need somewhere else to park new cars as they arrive.  The difference with an archive is that you move the cars that are already parked but that haven't moved for a while; you shunt these off to the overflow car park (archive) so that the new arrivals can be more easily accessed, nearer the front gate.

Cars (data) parked in the overflow (archive) are generally ignored, except when you want to search them.

"Retention" applies to both car parks - regular and overflow - and is when you start issuing parking tickets (or deleting data).  This may be simple housekeeping (Does it really matter anymore that "Fred" accessed a particular web page on your site three years ago?) or may be influenced by accountancy or even legal requirements.

You don't have to use archiving; if you have enough storage (parking space) and you still get reasonable performance from your queries even with many years of accumulated data, then you don't need archiving (and well done; you database structure must be pretty good).  It is, however, a very common method used to streamline systems by moving the bulk of records, that aren't used every day, into secondary storage.  For example, taking new orders from a customer today probably doesn't need to go through their entire, twenty year transaction history but a once-a-year, Management Information exercise to identify long-term customers would.


答:
By splitting a database into two databases you will lose all the benefits of relational references and referential integrity checking.  I have never tried such a thing, but my guess is that it would become a major nightmare.

I believe that the entire data set which describes a certain system belongs to a single database.  Issues of convenience in accessing the data are almost never a good reason for making data organization decisions.

Issues of convenience in accessing your data should be handled by making use of whatever convenience features are offered by your RDBMS.

So, instead of having a 'current' database and a 'historical' database, you should only have one database, and all tables in it should be prefixed with 'historical'.  Then, you should create a set of views, one for each table that you want to see as 'current', and have each view filter out the historical rows that you do not want to see, and let only the current ones pass through.

This is a suitable solution to your problem because it makes use of a convenience feature of the RDBMS to address a convenience concern of the programmer, leaving the database design intact.

An example of a problem you are likely to run into (too long for a comment)

Suppose you are looking at a screen which is showing you current information about a team, say team.id=10, team.name="Manchester United", and you click on the button which says "show history".  At that point, you will want to switch to a screen which shows historical information about that same team.  So, you are going to take the id of 10 which you know that in the "current" database stands for "Manchester United", and you are going to have to hope that this id number of 10 also stands for "Manchester United" in the historical database.  There is no referential integrity rule which enforces that the id refers to the exact same entity in both databases, so, essentially, you will have two entirely disjoint data sets with implicit connections which are only known by, honored by, and promised to be maintained by, code outside of the database.

And of course this holds true not only for major tables, like the "Teams" table, but even for the tiniest little table that you will have on the side, like "Player positions: Forward, Midfielder, Goalkeeper, etc."

Achieving historicity within the same database

There are various methods of maintaining historicity, and even though they are beyond the scope of this question, which is basically what pitfalls this particular idea of yours might have, here is an idea:

You can maintain a log table containing one entry for every single change that was ever made to the database. This way, all the "current" tables can be completely cleared of data and fully reconstructed by replaying the logged changes. Of course, if you can reconstruct the "current" tables by replaying the changes from the beginning of time until now, then you can also construct a temporary set of tables in order to get a view of the database at a specific time coordinate by replaying the changes from the beginning of time until that specific time coordinate.

This is known as "Event Sourcing" (Article by Martin Fowler.)


问:
Why is an anemic domain model considered bad in C#/OOP, but very important in F#/FP?
In a blog post on F# for fun and profit, it says:

In a functional design, it is very important to separate behavior from
  data. The data types are simple and "dumb". And then separately, you
  have a number of functions that act on those data types.

This is the exact opposite of an object-oriented design, where
  behavior and data are meant to be combined. After all, that's exactly
  what a class is. In a truly object-oriented design in fact, you should
  have nothing but behavior -- the data is private and can only be
  accessed via methods.

In fact, in OOD, not having enough behavior around a data type is
  considered a Bad Thing, and even has a name: the "anemic domain
  model".

Given that in C# we seem to keep borrowing from F#, and trying to write more functional-style code; how come we're not borrowing the idea of separating data/behavior, and even consider it bad? Is it simply that the definition doesn't with with OOP, or is there a concrete reason that it's bad in C# that for some reason doesn't apply in F# (and in fact, is reversed)?

(Note: I'm specifically interested in the differences in C#/F# that could change the opinion of what is good/bad, rather than individuals that may disagree with either opinion in the blog post).

答:
Why is an anemic domain model considered bad in C#/OOP, but very important in F#/FP?

Your question has a big problem that will limit the utility of the answers you get: you are implying/assuming that F# and FP are similar. FP is a huge family of languages including symbolic term rewriting, dynamic and static. Even among statically-typed FP languages there are many different technologies for expressing domain models such as higher-order modules in OCaml and SML (that don't exist in F#). F# is one of these functional languages but it is particularly notable for being lean and, in particular, does not provide either higher-order modules or higher-kinded types.

In fact, I could not begin to tell you how domain models are expressed in FP. The other answer here talks very specifically about how it is done in Haskell and is not at all applicable to Lisp (the mother of all FP languages), the ML family of languages or any other functional languages.

how come we're not borrowing the idea of separating data/behavior, and even consider it bad?

Generics might be considered a way of separating data and behaviour. Generics come from the ML family of functional programming languages are are not part of OOP. C# has generics, of course. So one could argue that C# is slowly borrowing the idea of separating data and behaviour.

Is it simply that the definition doesn't fit with OOP,

I believe OOP is based upon a fundamentally different premise and, consequently, does not give you the tools you need to separate data and behaviour. For all practical purposes you need product and sum datatypes and dispatch over them. In ML this means union and record types and pattern matching.

Check out the example I gave here.

or is there a concrete reason that it's bad in C# that for some reason doesn't apply in F# (and in fact, is reversed)?

Be careful about jumping from OOP to C#. C# is nowhere near as puritanical about OOP as other languages. The .NET Framework is now full of generics, static methods and even lambdas.

(Note: I'm specifically interested in the differences in C#/F# that could change the opinion of what is good/bad, rather than individuals that may disagree with either opinion in the blog post).

The lack of union types and pattern matching in C# makes it almost impossible to do. When all you have is a hammer, everything looks like a nail...


问:
Lucene vs Solr - Indexing pdf/word documents reisiding on a NAS drive using .Net
Using ASP.Net, I want to implement full text search using Lucene/Solr on a LARGE number of docs (word, pdf etc.) residing in a directory on a NAS drive. The NAS drive would be mapped as a network drive on the server. The list of documents get changed frequently. As per my research, Lucene doesnot index pdf/word docs directly. The raw data from the docs need to be extracted and then passed to the Lucene indexer. Is it advisable to use PDFBox and other third party tools to extract binary data and pass to Lucence indexer. What would be the impact on the performance on Lucene search? Shall I use Solr instead of Lucene as it supports indexing of pdf/word docs?</p>

答:
Yes, Solr supports out-of-the box (well, after a bit of configuration, see the examples from version 4.9 onwards) PDF and Word documents. The thing to note is that Solr != Lucene. Solr is a higher level abstraction over Lucene, and as such it has a different API, features and behaviour.

IMHO, the difference between Solr and Lucene utilisation can briefly be summarized as follows: Solr needs less configuration / set-up, makes for a quicker implementation, but will require more resources to run than Lucene. To detail: Solr comes with a REST API, a ton of caches and support for fancy features such as clustering. Some of these things are enabled by default. As such, Solr will need roughly more memory to run properly, and possibly more CPU. You should take all these into account when configuring it, because otherwise, if you do a trivial Solr implementation, you'll get what seems to be the same behaviour as with the Lucene implementation, but requiring a lot more resources to run with similar performance.


问:
How granular should we get with repositories?
Is it better to have a repository per model (this could be a class that represents a table from the database) or should just have one big repository for all data access methods?

I have done the second approach and have found it to get way out of hand, but maybe somewhere in between is a solution as well.

答:
Why are repositories any different from other classes?

A class should have a single responsibility - a single reason to change. If you have a bundle of related data sources that are (necessarily) tightly coupled, then it can make sense to have a single repo to deal with all of them. If you have some unrelated tables, then it's probably better to have a single repository per table.


答:
Implementing repositories "by the book" will typically mean to create one per entity or table (see here, for example). However, when your entities are uniform, and all of your repositories look very similar, it can make sense to implement a generic repository class (just as shown in the former link). That will probably be the better alternative for avoiding too much boiler plate code than the "big-ball-of-mud" repo.

Nevertheless, Fowler's description of the repository pattern does not explicitly require to have one repo per entity. So I agree to Telastyn: when there is bundled data, which has to be pulled and pushed always together in one query or transaction, it can make more sense to have one repo for more than one table.

Title: How granular should we get with repositories?

Body:
Is it better to have a repository per model (this could be a class that represents a table from the database) or should just have one big repository for all data access methods?

I have done the second approach and have found it to get way out of hand, but maybe somewhere in between is a solution as well.


问:
What is the reason of using an interface versus a generically constrained type
In object-oriented languages that support generic type parameters (also known as class templates, and parametric polymorphism, though of course each name carries different connotations), it is often possible to specify a type constraint on the type parameter, such that it be descended from another type. For example, this is the syntax in C#:

```<code>//for classes:
class ExampleClass<T> where T : I1 {

}
//for methods:
S ExampleMethod<S>(S value) where S : I2 {
        ...
}
</code>```

What are the reasons to use actual interface types over types constrained by those interfaces? For example, what are the reasons for making the method signature I2 ExampleMethod(I2 value)?

答:
What are the reasons to use actual interface types over types constrained by those interfaces?

Because that's what you need...

```<code>IFoo Fn(IFoo x);
T Fn<T>(T x) where T: IFoo;
</code>```

are two decidedly different signatures. The first takes any type implementing the interface and the only guarantee it makes is that the return value satisfies the interface.

The second takes any type implementing the interface and guarantees that it will return at least that type again (rather than something that satisfies the less restrictive interface).

Sometimes, you want the weaker guarantee. Sometimes you want the stronger one.

Body:
What are the reasons to use actual interface types over types constrained by those interfaces?

Because that's what you need...

IFoo Fn(IFoo x);
T Fn<T>(T x) where T: IFoo;

are two decidedly different signatures. The first takes any type implementing the interface and the only guarantee it makes is that the return value satisfies the interface.

The second takes any type implementing the interface and guarantees that it will return at least that type again (rather than something that satisfies the less restrictive interface).

Sometimes, you want the weaker guarantee. Sometimes you want the stronger one.


答:
Using the parametric version gives

More information to the users of the function

Constrains the number of programs you can write (free bug checking)

As a random example, suppose we have a method which calculates the roots of a quadratic equation

```<code>int solve(int a, int b, int c) {
  // My 7th grade math teacher is laughing somewhere
}
</code>```

And then you want it to work on other sorts of number like things besides int. You can write something like

```<code>Num solve(Num a, Num b, Num c){
  ...
}
</code>```

The issue is that this doesn't say what you want it to. It says

Give me any 3 things that are number like (not necessarily in the same way)
  and I'll give you back some sort of number

We can't do something like int sol = solve(a, b, c) if a, b, and c are ints because we don't know that the method is going to return an int in the end! This leads to some awkward dancing with downcasting and praying if we want to use the solution in a larger expression.

Inside the function, someone might hand us a float, a bigint, and degrees and we'd have to add and multiply them together. We'd like to statically reject this because the operations between these 3 classes is going to be gibberish. Degrees are mod 360 so it won't be the case that a.plus(b) = b.plus(a) and similar hilarities will arise.

If we use the parametric polymorphism with subtyping we can rule all this out because our type actually says what we mean

```<code><T : Num> T solve(T a, T b, T c)
</code>```

Or in words "If you give me some type which is a number, I can solve equations with those coefficients".

This comes up in a lot of other places as well. Another good source of examples are functions which abstract over some sort of container, ala reverse, sort, map, etc.

Title: What is the reason of using an interface versus a generically constrained type

Body:
In object-oriented languages that support generic type parameters (also known as class templates, and parametric polymorphism, though of course each name carries different connotations), it is often possible to specify a type constraint on the type parameter, such that it be descended from another type. For example, this is the syntax in C#:

```<code>//for classes:
class ExampleClass<T> where T : I1 {

}
//for methods:
S ExampleMethod<S>(S value) where S : I2 {
        ...
}
</code>```

What are the reasons to use actual interface types over types constrained by those interfaces? For example, what are the reasons for making the method signature I2 ExampleMethod(I2 value)?

Body:
Using the parametric version gives

More information to the users of the function

Constrains the number of programs you can write (free bug checking)

As a random example, suppose we have a method which calculates the roots of a quadratic equation

int solve(int a, int b, int c) {
  // My 7th grade math teacher is laughing somewhere
}

And then you want it to work on other sorts of number like things besides int. You can write something like

Num solve(Num a, Num b, Num c){
  ...
}

The issue is that this doesn't say what you want it to. It says

Give me any 3 things that are number like (not necessarily in the same way)
  and I'll give you back some sort of number

We can't do something like int sol = solve(a, b, c) if a, b, and c are ints because we don't know that the method is going to return an int in the end! This leads to some awkward dancing with downcasting and praying if we want to use the solution in a larger expression.

Inside the function, someone might hand us a float, a bigint, and degrees and we'd have to add and multiply them together. We'd like to statically reject this because the operations between these 3 classes is going to be gibberish. Degrees are mod 360 so it won't be the case that a.plus(b) = b.plus(a) and similar hilarities will arise.

If we use the parametric polymorphism with subtyping we can rule all this out because our type actually says what we mean

<T : Num> T solve(T a, T b, T c)

Or in words "If you give me some type which is a number, I can solve equations with those coefficients".

This comes up in a lot of other places as well. Another good source of examples are functions which abstract over some sort of container, ala reverse, sort, map, etc.


问:
Security scheme that prevents duplicate message attacks
I am using custom security scheme to verify communication between client and server.

Both client and server have same secret hash phrase.

Client builds a message, combines it with hash phrase and calculates
SHA512 hash, and sends both message and hash to server

Server receives a message, combines it with hash phrase and
calculates SHA512 hash, and then verifies that hash is the same as
the one sent by client

This scheme works fine, but allows man in the middle to sniff the message and send it again, again and again to server.

Now, I know how to solve this, for example, by assigning unique id to each message and rejecting duplicates - but these unique ids must be stored somewhere (database, session, ...)

Is there some more stateless approach that can solve this problem?

答:
Presuming the "hash phrase" is client-unique, your integration already entails a level of statefulness. It's not much more of a leap to provide the client with unique request tokens. The server can issue replacement token(s) as part of the response for each request.

The unique request tokens can either be/replace the "hash phrase" itself, or simply operate as an additional request identifier/authorization.


问:
Why aren't field-like events implemented as a list of delegates?
tl;dr: Why are field-like events implemented as a single delegate field? Wouldn't it be more straight-forward to use a list of delegates, thereby eliminating the null special case and avoiding all the MulticastDelegate magic?

The usual pattern to raise field-like events (ignoring synchronization issues for the moment) is

```<code>if (MyEvent != null)
    MyEvent(this, e);
</code>```

which can be quite confusing (Why do I have a null check for 0 handlers, but no loop for multiple handlers?), until you understand how field-like events are implemented and that MyEvent refers to an automatic backing field which can contain either null, a single delegate or a MulticastDelegate holding references to multiple handlers.

On a first glance, this seems to be a strange design choice. In particular, the obvious alternative would be to use a list of simple delegates, which is how the Observer pattern is commonly implemented in languages such as Java. True, the invocation would be slightly longer:

```<code>foreach (var handler in MyEvent)
    handler(this, e);
</code>```

but it would be easier to understand (in my subjective opinion) and probably easier to implement (no need for all the delegate multicasting logic).

Obviously, the C# designers thought otherwise. Since they are a bunch of very smart people, there must have been a good reason for it. What is it?

答:
All delegates are effectively lists. The empty list is null, but you can still append to it with +=.

The need to check for null before calling is very inconvenient, yes. But does not occur because they are not lists! It occurs because of how the empty list is represented.

However you can avoid this by initialising the event:

```<code>public event EventType MyEvent = delegate {};
</code>```

See also: https://stackoverflow.com/questions/170907/is-there-a-downside-to-adding-an-anonymous-empty-delegate-on-event-declaration?rq=1

OwnerDisplayName: Heinzi

Body:
All delegates are effectively lists. The empty list is null, but you can still append to it with +=.

The need to check for null before calling is very inconvenient, yes. But does not occur because they are not lists! It occurs because of how the empty list is represented.

However you can avoid this by initialising the event:

public event EventType MyEvent = delegate {};

See also: https://stackoverflow.com/questions/170907/is-there-a-downside-to-adding-an-anonymous-empty-delegate-on-event-declaration?rq=1

OwnerDisplayName: Daniel Earwicker


答:
A common use for events is in form controls. A form control can have a lot of events, a ListView for example has 79 different events.

Having to create a delegate list for every event would mean that there would be a lot of those. With just a dozen controls there would be around a thousand delegate lists to create, but most of those would end up unused and only taking up space.

There is a clear advantage to have a more light-weight way of representing an unused event. By using a null reference for that there is no extra initialisation needed for the events, as objects fields are automatically set to their default value.

Similarly there is an advantage of having separate single-cast and multi-cast events, and move the responsibility for calling the delegates into those classes. An event doesn't have to be a list when there is only one subscriber to it (which is the most common case), which makes the object smaller and the calling process simpler.

Title: Why aren't field-like events implemented as a list of delegates?

Body:
tl;dr: Why are field-like events implemented as a single delegate field? Wouldn't it be more straight-forward to use a list of delegates, thereby eliminating the null special case and avoiding all the MulticastDelegate magic?

The usual pattern to raise field-like events (ignoring synchronization issues for the moment) is

```<code>if (MyEvent != null)
    MyEvent(this, e);
</code>```

which can be quite confusing (Why do I have a null check for 0 handlers, but no loop for multiple handlers?), until you understand how field-like events are implemented and that MyEvent refers to an automatic backing field which can contain either null, a single delegate or a MulticastDelegate holding references to multiple handlers.

On a first glance, this seems to be a strange design choice. In particular, the obvious alternative would be to use a list of simple delegates, which is how the Observer pattern is commonly implemented in languages such as Java. True, the invocation would be slightly longer:

```<code>foreach (var handler in MyEvent)
    handler(this, e);
</code>```

but it would be easier to understand (in my subjective opinion) and probably easier to implement (no need for all the delegate multicasting logic).

Obviously, the C# designers thought otherwise. Since they are a bunch of very smart people, there must have been a good reason for it. What is it?

OwnerDisplayName: Guffa


答:
The code which reads a file just has to respect the file rules. A zip file often contains data which isn't even byte-aligned (basically that's how the format achieve compression - use less than 8 bits for common bytes in the input). Unzipping code must therefore deal with individual bits.

For bitmaps, it's often easier. Having 4-byte aligned structures means that you can memory map the file, and then safely access the header fields as integers. If they weren't aligned, you'd need to copy the 4 unaligned bytes to 4 aligned bytes. This can often be done transparently, e.g. by fread. It copies bytes from its internal buffer to the address you specified.


答: Eric Evans Domain Driven Design specifies that you should provide a repository for 'aggregate roots' that need direct access. So this is not one repository per model but one repository per cluster of models, if you like. This assumes you are doing DDD and therefore understand the business context, though.</p>


答:
So far here are some ways decimal Unicode can be specified enough to convey the meaning to anyone who searches:

XML Entity Notation: &#nnn;

C / C++ Wide Character: wchar_t(nnn)

SQL National Character: NCHAR(nnn) (Available in MSSQL, MySQL, Oracle)

Windows ALT Notation: ALT+nnn (although it can be easily confused with the hex version with the plus key, ALT++01bd)

I'm interested in any others to add to this list.

Title: Is there accepted decimal-based Unicode notation for technical audiences?

Body:
When writing for technical audiences, there are various ways to type Unicode representations, but they all seem to be Hexadecimal:

\uFFFF - From C# / Java Strings

\U0000FFFF - From C# / Java Strings

However, Unicode can also be specified in decimal, and can be entered both ways in XML Entities:

Hexadecimal:

Decimal:

While I could just (kind of) use \u65535, this does make something that is already defined as specifically hexadecimal and abuse it, and also could cause problems - is \u1111 decimal or hex?

So - are there any programming languages that allow similar ways to denote Unicode characaters by decimal, or common shorthand conventions for specifying Unicode in decimal notation for technical audiences?


问:
Finding the first subarray that sums to a given total
Here is the question:

Given and unordered array of positive and negative integers. How can you find the first subarray to sum to a given value?

For example. Given the array...

```<code>[1, -3, 4, 8, 2, -14, 3, -1, 10, 6]
</code>```

Find the first subarray to sum to 9.

In this example the sub array would be...

```<code>[-3, 4, 8]
</code>```

I found a solution that is O(n^2) by iterating the array for start points and then iterating every end point until you find the value.

But is there a way to do this better than O(n^2)?

答:
I think I came up with a solution. Suppose the following numbers, and you want to find the first subarray to sum to 4:

```<code>2 -3 7 1 5 -1
</code>```

First to get rid of negative numbers, I suggest find the smallest number (-3) and add its absolute value to all the numbers, that takes O(n), we have:

```<code>5  0  10 4  8 2
</code>```

Then for any sub-array with size n, the sum should be n*3 + 4, if you find that sum, then you have found the answer

I wrote the target for the sub-arrays with size 1, 2, 3 ...

```<code>7   10  13  16   19
</code>```

Now start with 5, it is less than 7 then you can continue, add it to 0, it is less than 10, and you can continue, add it to 10, it is more than 13, then ignore 5, and regard 0 + 10, it is less than 10, actually it is equal to 10, it means you have found the answer, the answer is -3 + 7 = 4

I try to write the algorithm down! ,

```<code> SubArraySumTo(A[], y)
 {
     x = abs(min(A));  // a loop over the array
     foreach (var a in A)
     {
         a+=x;
     }
     int start =0;
     int sum =A[0];
     int i=0;
     while (i < n)
     {
         target = (i - start +1)*x + y;
         if (sum == target)
         {
              return A[start..i];
         }
         else if (sum < target)
         {
            i++;
            sum += A[i];
         }
         else if (sum > target)
         {
             start++;
             sum -= A[start];
         }
     }
 }
</code>```

It's an iteration with O(2n) in worst case

Body:
I think I came up with a solution. Suppose the following numbers, and you want to find the first subarray to sum to 4:

2 -3 7 1 5 -1

First to get rid of negative numbers, I suggest find the smallest number (-3) and add its absolute value to all the numbers, that takes O(n), we have:

5  0  10 4  8 2

Then for any sub-array with size n, the sum should be n*3 + 4, if you find that sum, then you have found the answer

I wrote the target for the sub-arrays with size 1, 2, 3 ...

7   10  13  16   19

Now start with 5, it is less than 7 then you can continue, add it to 0, it is less than 10, and you can continue, add it to 10, it is more than 13, then ignore 5, and regard 0 + 10, it is less than 10, actually it is equal to 10, it means you have found the answer, the answer is -3 + 7 = 4

I try to write the algorithm down! ,

 SubArraySumTo(A[], y)
 {
     x = abs(min(A));  // a loop over the array
     foreach (var a in A)
     {
         a+=x;
     }
     int start =0;
     int sum =A[0];
     int i=0;
     while (i < n)
     {
         target = (i - start +1)*x + y;
         if (sum == target)
         {
              return A[start..i];
         }
         else if (sum < target)
         {
            i++;
            sum += A[i];
         }
         else if (sum > target)
         {
             start++;
             sum -= A[start];
         }
     }
 }

It's an iteration with O(2n) in worst case


问:
Should we estimate tasks during backlog grooming?
I was just taken by surprise in a backlog grooming meeting when we assigned time estimates to BA, dev, and QA tasks...but the story is not yet scheduled in any sprint and we are not yet assigning resources to the tasks.

This feels backwards to me, since:

I can estimate my own darn tasks, thankyouverymuch.

We're going to review/revise the estimates during sprint planning anyway.

Each task should be estimated by the person who's going to do the work. (developers are nonfungible)

Should we really be trying to estimate to the work-hours level of precision at this point?

This question's discussion says no, since the backlog is made of stories, not tasks. And this one addresses time scales. Both relevant, but I'd particularly like to address the topic of estimating unassigned tasks.

[Side note: <3 <3 <3 your comments and answers and I wish I could upvote most of them, but my account is too new :s ]

答:
Should we really be trying to estimate to the work-hours level of precision at this point?

Hell no, but it happens all the time.

Partly it's because most places do SortaAgile. SortaAgile doesn't believe in story points. It doesn't believe that individuals have different velocities. And it certainly doesn't believe in team involvement.

Partly it's because Agile does a really bad job at answering a key business question: "when is this going to be done?". Well - that's not Agile's fault really. Developers need to have enough soft skills to push back on business to get clear scope and negotiate a tentative done date for that scope. You don't need detailed estimates for that, you need the ability to set expectations. Agile perhaps places more of that burden on developers since we're the ones pushing Agile, not project managers. Ideally, your project manager would be on board with Agile and competent. That is... uncommon though.

So push back where you can. But remember that estimation doesn't pay the bills - arguing with management doesn't pay the bills. Do what you can and then get back to making software.


答:
Use of constrained generics for method parameters can allow a method to very its return type based upon that of the thing passed in.  In .NET they can have additional advantages as well.  Among them:

A method which accepts a constrained generic as a ref or out parameter may be passed a variable which satisfies the constraint; by contrast, a non-generic method with an interface-type parameter would be limited to accepting variables of that exact interface type.

A method with generic type parameter T can accept generic collections of T.  A method that accepts an IList<T> where T:IAnimal will be able to accept a List<SiameseCat>, but a method which wanted an IList<Animal> would not be able to do so.

A constraint can sometimes specify an interface in terms of the generic type, e.g. where T:IComparable<T>.

A structure which implements an interface may be kept as a value type when passed to a method accepting a constrained generic parameter, but must be boxed when passed as an interface type.  This can have a huge effect on speed.

A generic parameter can have multiple constraints, while there is no other way to specify a parameter of "some type that implements both IFoo and IBar".  Sometimes this can be a double-edged sword, since code which has received a parameter of type IFoo will find it very hard to pass it to such a method expecting a double-constrained generic, even if the instance in question would satisfy all constraints.

If in a particular situation there would be no advantage to using a generic, then simply accept a parameter of the interface type.  Use of a generic will force the type system and JITter to do extra work, so if there's no benefit one shouldn't do it.  On the other hand, it's very common that at least one of the above advantages will apply.


问:
Storm topology to handle "dating website"-like workloads
suppose im writing a dating website, similar to okcupid. there are profiles, and i need to compute the (N^2) "match" table - given every 2 profiles whats the match between them?

I was thinking this could be done by creating a spout to listen on a "new/updated profiles" queue (say kafka, doesnt matter), but then how do I break down the matching to achieve any degree of parallelism?

If I have a single bolt that compares the profile vs the entire DB that wont scale.

If I create another spout, for "all profiles" it will run in a continous loop and never stop (?)

Obviously the assumption is that the "churn rate" (rate of new/updated profiles) is less of an issue than the sheer size of the database.

Any suggestions on how to design the topology would be very welcome.

答:
The obvious answer to this problem is "don't do N^2", at least not for the full value of N - you can very rapidly reject large parts of your data set for all sorts of reasons:

Wrong sex. While I'm a liberal kind of guy, I'm still not interested in dating other guys. There may obviously be some exceptions to this - e.g. bisexuals.

Wrong country. It doesn't matter how hot they are, I'm not going to want to date someone in Australia if I'm based in the UK. Again, there may be exceptions but most people aren't looking for a long-distance relationship. (More generally, get people to say how far they're prepared to travel for a relationship. New York to California probably isn't going to work for most people).

There's probably lots more criteria I haven't thought of yet - by doing this you've hopefuly solved the "sheer size of the database" problem and got it down to something tractable.

As an aside, I suspect you'll find Data Science of Love interesting viewing. Disclaimer: I know the presenter personally, but have no professional relationship with either him or eHarmony.


答:
It doesn't sounds like you're doing any kind of methodology that I'm familiar with. You wouldnt normally have tasks in a backlog. How do you prioritize a task? You can't complete a story without implementing all tasks so it makes no sense to have them in the backlog. Stories should follow INVEST

Should we really be trying to estimate to the work-hours level of precision at this point?

No, this should be deferred to the "last responsible moment". By the time you actually come round to implement it, things may have changed and you might not need to do it (and in fact it sounds like this is the case in your sprint planning). People usually recommend you do "just enough"

I can estimate my own darn tasks, thankyouverymuch.

Programmers are usually quite bad at estimating tasks. This is why people often use "ideal man hours" or just points.

Each task should be estimated by the person who's going to do the work. (developers are nonfungible)

While it is true that developers are non-fungible, Agile teams usually practice collective code ownership; it's one of the rules of XP. Just because the developers have different skills, it doesnt mean that anyone can't work on any particular task.

Sounds like you have a high Bus Factor.

Title: Should we estimate tasks during backlog grooming?

Body:
I was just taken by surprise in a backlog grooming meeting when we assigned time estimates to BA, dev, and QA tasks...but the story is not yet scheduled in any sprint and we are not yet assigning resources to the tasks.

This feels backwards to me, since:

I can estimate my own darn tasks, thankyouverymuch.

We're going to review/revise the estimates during sprint planning anyway.

Each task should be estimated by the person who's going to do the work. (developers are nonfungible)

Should we really be trying to estimate to the work-hours level of precision at this point?

This question's discussion says no, since the backlog is made of stories, not tasks. And this one addresses time scales. Both relevant, but I'd particularly like to address the topic of estimating unassigned tasks.

[Side note: <3 <3 <3 your comments and answers and I wish I could upvote most of them, but my account is too new :s ]