答:
This is the intended usage.

What are the pro's/con's to putting all your applications in a single solution rather than having separate solutions for each application?

pros:

easer references between projects

easier debugging/stepping through

easier to attach to processes (ie. you're stepping through a windows service, when it jumps to shared library code, and it just works because all of the symbols are already loaded and accounted for)

code search within the ide works better (you don't necessarily have to know which project the code you're looking for resides in)

cross-project changelists are easier to manage (ie. you changed library code, so you have to update client code at the same time, with 1 checkin)

cons:

build speed: if you're in the habit of "rebuild all" then builds take longer. Much longer, and incremental builds sometimes have funky behaviour.

ide speed: see next

How fast is Visual Studio with a solution of 20+ projects?

With 20+ projects... might not be so bad. I've seen more without VS having problems. It's pretty stable/quick. HOWEVER... if it is a problem, it can be mitigated: open the .csproj in VS and work from there. You don't get the benefits of having everything in one solution, but you can always re-open the sln when you need to and work in just the .csproj when you need to (like you do now).

And how stable is the solution file with source-controlled concurrent development?

The solution file only changes when you add/remove projects or solution-level objects, so it rarely changes. It's xml, so you can see what's in it. They change rarely.

Title: Multiple applications in a single Visual Studio solution

Body:
I am doing work for a company that is wanting to put all their .Net applications (web applications, Windows applications and console applications) together in a single Visual Studio solution.

I am looking for experience from developers who either structure their own applications this way or who work in a company that does this. Is this a good idea?

What are the pro's/con's to putting all your applications in a single solution rather than having separate solutions for each application?

How fast is Visual Studio with a solution of 20+ projects?

And how stable is the solution file with source-controlled concurrent development?


问:
How to approach scrum task burn down when tasks have multiple peoples involvement?
In my company, a single task can never be completed by one individual. There is going to be a separate person to QA and Code Review each task. What this means is that each individual will give their estimates, per task, as to how much time it will take to complete.

The problem is, how should I approach burn down? If I aggregate the hours together, assume the following estimate:

10 hrs - Dev time

4  hrs - QA

4  hrs - Code Review.

Task Estimate = 18hrs

At the end of each day I ask that the task be updated with "how much time is left until it is done". However, each person generally just thinks about their part of it. Should they mark the effort remaining, and then ADD the effort estimates to that? How are you guys doing this?

UPDATE

To help clarify a few things, at my organization each Task within a story requires 3 people.

Someone to develop the task. (do unit tests, ect...)

A QA specialist to review task (they primarily do integration and regression tests)

A Tech lead to do code review.

I don't think there is a wrong way or a right way, but this is our way ... and that won't be changing. We work as a team to complete even the smallest level of a story whenever possible. You cannot actually test if something works until it is dev complete, and you cannot review the quality of the code either ... so the best you can do is split things up into small logical slices so that the bare minimum functionality can be tested and reviewed as early into the process as possible.

My question to those that work this way would be how to burn down a "task" when they are setup this way. Unless a Task has it's own sub-tasks (which JIRA doesn't allow) ... I'm not sure the best way to accomplish tracking "what's left" on a daily basis.

答:
That is 3 tasks, not one.

It may be one feature/story, but it is three tasks. A single task can be completed by a single person in finite time.

ContentLicense: CC BY-SA 3.0


问:
If the user declines a "continue?" prompt, what exit code should be returned?
In terminal based scripts (Shell, Bash, Python, etc), if you prompt the user "Do you wish to continue?" and the user chooses "No", what exit code should the script ideally return?

On one hand, the script executed as the user expected it to (the user wanted to quit, so it quit), so it should give an exit code of 0. On the other hand, the script was unable to complete its main task, which should be a non-zero exit code.

答:
There is no real standard on this, other than "zero means keep going".

So it depends on what should typically happen in a situation where you do this:

```<code>$ your-script && do-something-else
</code>```

...and the user cancels out in your-script. Is the desired behavior to skip that step and continue (i.o.w., is your-script completely optional)? Then a zero exit code is appropriate. But if, typically, continuing after the user has cancelled your script is inappropriate, then you should return non-zero. If in doubt, use non-zero: otherwise whoever calls your script won't be able to distinguish between "user cancelled out" and "user said yes and everything worked as intended" - discarding a non-zero exit code is easy, but figuring out what happened if the code is zero either way is not.

Just make sure that you use distinct exit codes for "user cancelled out" and "something went horribly wrong", and of course document them.

Body:
There is no real standard on this, other than "zero means keep going".

So it depends on what should typically happen in a situation where you do this:

$ your-script && do-something-else

...and the user cancels out in your-script. Is the desired behavior to skip that step and continue (i.o.w., is your-script completely optional)? Then a zero exit code is appropriate. But if, typically, continuing after the user has cancelled your script is inappropriate, then you should return non-zero. If in doubt, use non-zero: otherwise whoever calls your script won't be able to distinguish between "user cancelled out" and "user said yes and everything worked as intended" - discarding a non-zero exit code is easy, but figuring out what happened if the code is zero either way is not.

Just make sure that you use distinct exit codes for "user cancelled out" and "something went horribly wrong", and of course document them.


问:
Is it poor practice to name a property/member the same as the declaring type in C#?
For instance, a class like:

```<code>class Dog { } //never mind that there's nothing in it...
</code>```

and then a property like:

```<code>Dog Dog { get; set; }
</code>```

I've been told that if I can't come up with a more imaginative name for it, then I must use:

```<code>Dog DogObject { get; set; }
</code>```

Any thoughts on how to name these better?

答:
Not only is this a reasonable practice, the language was specifically designed to permit this. Search the C# specification for "Color Color" for the rules and a justification, and see
the Microsoft doc on Color Color
for some interesting corner cases that arise from this decision.

Under no circumstances should you name a property "DogObject" in order to avoid calling it the same as its type; that directly contradicts the framework design guidelines.

ContentLicense: CC BY-SA 4.0


问:
REST Relationships Besides Parent/Child
This would seem to be a pretty basic question, but I can't seem to find appropraite guidelines for it. What is the "restful" way of POSTing a relationship between two entities that are not necessarily parent/child. As an example, consider customers and orders.

One could create a customer with a POST to /customer. However, what is the best way to create the order? One example would be a POST to /customer/123/order, but what if the order isn't assigned to a customer directly, and could exist on its own (in the case of a web-cart that allows for "anonymous" checkout)? Is the right way, then, to do POST /order?customerId=123

Further, consider the case that an order can be submitted separately from payments (let's say we are invoicing a customer). You could create the order (per either of the methods above), but how would we make the payment? POST /order/123/payment or POST /payment?orderId=123 (if we assume that a payment can be made for something other than an order, and one might want to get information about an order/payment without knowing the specific customer).

答: I would say that if an order is a first class object, use /order. If it needs to be associated with one or more customers, the customer ids should be an attribute of the order object.

Title: REST Relationships Besides Parent/Child


问:
What is the big-O cpu time of Euclid's Algorithm of "Greatest common divisor of two numbers"
Looking at Euclid's algorithm for the "Greatest common divisor of two numbers", I'm trying to divine the big-O cpu time for numbers K, and N. Can anybody help?

This is the algorithm as I understand it..

Where:

max(A,B) = the greater of A or B such that: min(10,3) = 10

min(A,B) = the smaller of A or B such that: min(10,3) = 3

modulus(A,B) = the remainder of A divided by B such that: modulus(10, 3) = 1

The algorithm is:

r = modulus(max(K,N), min(K,N))

if r = 0

then GCD is max(K,N)

else:

max(K,N) = r

go to step 1

It appears to me that due to the division occurring, it can't be a linear algorithm which is why it's useful as something more efficient than a naive implementation that just tries all the possibilities from 0 to min(K,N). But I can't quite figure out just what the runtime is.

Any pointers would be very helpful!

答:
This answer draws from the answers within the SO question Algorithm complexity with input is fix-sized

The first thing to do is to rewrite this as pseudo code with line numbers so that we can talk about specific loops.

```<code>1) def GCD(x, y):
2)    while x != y:
3)       if x < y:
4)          x, y = y - x, x
5)       else:
6)          x, y = x - y, x
7)     return x
</code>```

This is a form of the subtraction based call of the euclidean algorithm rather than the recursive form / modulus form (which the question alludes to).  That one, for the sake of completeness is (different numbering so that the two are not confused):

```<code>a) def GCD(x, y):
b)   if y = 0
c)       return x
d)   else
e)       return GCD(y, x % y)
</code>```

For the subtractive method, the worst case is where the two numbers are coprime which has the property of GCD(x,y) = 1.  An example of coprime integers is 15 and 8.

None of the statements of lines 2 .. 6 are anything other than O(1).  There is no multiple loop.  The question is one of what the the loop count is.

For any call gcd(n,1), this will take n-1 loops.  Each step, the path will be 1, 2, 5, 6.  Line 6 will decrement n by 1, until n = 1.  As stated, this is n-1 loops.  As this is the worst case, one has gcd(n,1) = O(n).

So, for GCD(15,1), there are 14 loops.  This is also a very boring function that only has one variable.  What if we make it interesting by doing, as asked, GCD(x,y) where they are coprime?

As mentioned above, GCD(15,8) = 1.  How many loops does it do then?

```<code>GCD(15,8)
  x = 15, y = 8
    x, y = 15 - 8, 8
  x = 7, y = 8
    x, y = 8 - 7, 8
  x = 1, y = 8
  (7 more steps)
</code>```

The important thing to notice here is that this is more steps than 8.  The asymptotic worst case is related to the max of the two parameters.  The actual running time may have other factors mixed in - GCD(15,2) is twice as 'fast' as GCD(15,1), but that's a constant factor which still keeps at at O(n).

The asymptotic worst case for this form of GCD(x,y) is O(max(x,y)).

But wait, you say... the other form of GCD, the recursive one... GCD(15,1) is the best case, you get GCD(1, 15 % 1 = 0) after one step.

This area has a special place in the history of computation.  In 1844 a proof was published by Gabriel Lamé on the running time of the Euclidean algorithm. This marks the beginnings of computation complexity theory.  For this, the worst case is pairs of Fibonacci numbers and it runs in no more than 5k steps where  k = log10(min(x,y)).  This is known as Lamé's Theorem.  Actually, the 5 is a simplification - the actual number is ln(10)/lnϕ ≈ 4.785 where ϕ is the golden ratio.  And yes, the math for this is getting over my head.

Some reading on Lamé's Theorem:

Help understanding the proof of Lame's Theorem. from Math.SE

Lamé's Theorem - the Very First Application of Fibonacci Numbers

Lamé's Theorem from The Primes

Read the Math.SE one - there's no way I can get all of that in here to explain that part.

Suffice to say, this form of GCD(x,y) is O(log(min(x,y)))

Of special note, there's also a binary form of GCD that works with bit manipulations.  Read about it at Binary GCD algorithm on wikipedia.  It potentially runs a bit faster on computers because the operations are all bitwise which run faster than subtraction or division.  This one is O(n2) where n is the number of bits in the larger of the two numbers - please note the change in the definition of n between this and the previous forms of the analysis.  Despite the larger O, it is between 15%  to 60% faster (Knuth 1998 - TAOCP vol 2, 3rd edition) depending on architecture because of the simpler instructions (if I read that right).

Body:
This answer draws from the answers within the SO question Algorithm complexity with input is fix-sized

The first thing to do is to rewrite this as pseudo code with line numbers so that we can talk about specific loops.

1) def GCD(x, y):
2)    while x != y:
3)       if x < y:
4)          x, y = y - x, x
5)       else:
6)          x, y = x - y, x
7)     return x

This is a form of the subtraction based call of the euclidean algorithm rather than the recursive form / modulus form (which the question alludes to).  That one, for the sake of completeness is (different numbering so that the two are not confused):

a) def GCD(x, y):
b)   if y = 0
c)       return x
d)   else
e)       return GCD(y, x % y)

For the subtractive method, the worst case is where the two numbers are coprime which has the property of GCD(x,y) = 1.  An example of coprime integers is 15 and 8.

None of the statements of lines 2 .. 6 are anything other than O(1).  There is no multiple loop.  The question is one of what the the loop count is.

For any call gcd(n,1), this will take n-1 loops.  Each step, the path will be 1, 2, 5, 6.  Line 6 will decrement n by 1, until n = 1.  As stated, this is n-1 loops.  As this is the worst case, one has gcd(n,1) = O(n).

So, for GCD(15,1), there are 14 loops.  This is also a very boring function that only has one variable.  What if we make it interesting by doing, as asked, GCD(x,y) where they are coprime?

As mentioned above, GCD(15,8) = 1.  How many loops does it do then?

GCD(15,8)
  x = 15, y = 8
    x, y = 15 - 8, 8
  x = 7, y = 8
    x, y = 8 - 7, 8
  x = 1, y = 8
  (7 more steps)

The important thing to notice here is that this is more steps than 8.  The asymptotic worst case is related to the max of the two parameters.  The actual running time may have other factors mixed in - GCD(15,2) is twice as 'fast' as GCD(15,1), but that's a constant factor which still keeps at at O(n).

The asymptotic worst case for this form of GCD(x,y) is O(max(x,y)).

But wait, you say... the other form of GCD, the recursive one... GCD(15,1) is the best case, you get GCD(1, 15 % 1 = 0) after one step.

This area has a special place in the history of computation.  In 1844 a proof was published by Gabriel Lamé on the running time of the Euclidean algorithm. This marks the beginnings of computation complexity theory.  For this, the worst case is pairs of Fibonacci numbers and it runs in no more than 5k steps where  k = log10(min(x,y)).  This is known as Lamé's Theorem.  Actually, the 5 is a simplification - the actual number is ln(10)/lnϕ ≈ 4.785 where ϕ is the golden ratio.  And yes, the math for this is getting over my head.

Some reading on Lamé's Theorem:

Help understanding the proof of Lame's Theorem. from Math.SE

Lamé's Theorem - the Very First Application of Fibonacci Numbers

Lamé's Theorem from The Primes

Read the Math.SE one - there's no way I can get all of that in here to explain that part.

Suffice to say, this form of GCD(x,y) is O(log(min(x,y)))

Of special note, there's also a binary form of GCD that works with bit manipulations.  Read about it at Binary GCD algorithm on wikipedia.  It potentially runs a bit faster on computers because the operations are all bitwise which run faster than subtraction or division.  This one is O(n2) where n is the number of bits in the larger of the two numbers - please note the change in the definition of n between this and the previous forms of the analysis.  Despite the larger O, it is between 15%  to 60% faster (Knuth 1998 - TAOCP vol 2, 3rd edition) depending on architecture because of the simpler instructions (if I read that right).


问:
Good Procedure or Function Design
This is in reference to the question posted here.  As I would judge it, the question there should be closed simply because it seems to ask why the Borland developers made one thing a function and another a procedure (function that returns a void result).

Now the example was given there of some string manipulation functions that exist in Borland Pascal or Delphi:

```<code>S2 := Copy(S1,3,2);
</code>```

Copies into S2 from S1 the 2 characters starting from the 3rd
  position.

```<code>Delete(S,3,2);
</code>```

Deletes from S the 2 characters starting from the 3rd position.

Now I thought it might be a good discussion on what would be good design decisions regarding devising these things.  I picked up a few rules of good design on what a function prototype should look like beyond the obvious (name reflects function, function result should be atomic, no extraneous variables), but are there any others that should govern a situation such as this one or others?   I really didn't see a good case in Atwood's code book where it was discussed.

Anyway, are there some good rules on how functions should be designed to work?  Barring the specious naming, would it be better to have

```<code>S2 := Delete(S1,3,2);
</code>```

as opposed to

```<code>Delete(S,3,2);
</code>```

?  Are there any other useful considerations or examples to discuss of good and bad function design?

答:
As long as some basic language rules hold, either notation can be translated to the other.

```<code>S2 := Delete(S1,3,2);
</code>```

can be rewritten as

```<code>Delete(S2,S1,3,2);
</code>```

if S2 is declared to be pass-by-reference.

And

```<code>Delete(S,3,2);
</code>```

can be rewritten as

```<code>S := Delete(S,3,2);
</code>```

In most languages I can think of, this is the case.  If it's not the case, the argument for notation is clear:  use the one that works.

But assuming one can use either, you asked for useful considerations.  To me, the two guiding ones should be clarity and efficiency.

On clarity, take the first example above.  Personally, from a function, I expect one of two behaviors.  Either it returns all of the work it does in S2 (with no pass-by-reference arguments at all), or it returns all of its work in pass-by-reference args and then returns only a code indicating success or failure.  To do otherwise -- to manipulate some args as work product and then return some other work product -- makes the code impossible to understand.  Now, if there is no question about the success or failure of the function, and if pass-by-reference is more appropriate for the work being done, then procedural syntax is fine.  But in my experience, 90+% of the time, I either want a success/failure indication, or don't want to pass by reference, or both.

On efficiency, take the second example.  Obviously if S or the operations on it are large and complex, pass-by-reference can be more efficient than working on a copy of data and returning only to assign to the same original variable.  And as above, if pass-by-reference is appropriate, and if there is no question about the function/procedure succeeding or failing, then no return is necessary.

Title: Good Procedure or Function Design

Body:
As long as some basic language rules hold, either notation can be translated to the other.

S2 := Delete(S1,3,2);

can be rewritten as

Delete(S2,S1,3,2);

if S2 is declared to be pass-by-reference.

And

Delete(S,3,2);

can be rewritten as

S := Delete(S,3,2);

In most languages I can think of, this is the case.  If it's not the case, the argument for notation is clear:  use the one that works.

But assuming one can use either, you asked for useful considerations.  To me, the two guiding ones should be clarity and efficiency.

On clarity, take the first example above.  Personally, from a function, I expect one of two behaviors.  Either it returns all of the work it does in S2 (with no pass-by-reference arguments at all), or it returns all of its work in pass-by-reference args and then returns only a code indicating success or failure.  To do otherwise -- to manipulate some args as work product and then return some other work product -- makes the code impossible to understand.  Now, if there is no question about the success or failure of the function, and if pass-by-reference is more appropriate for the work being done, then procedural syntax is fine.  But in my experience, 90+% of the time, I either want a success/failure indication, or don't want to pass by reference, or both.

On efficiency, take the second example.  Obviously if S or the operations on it are large and complex, pass-by-reference can be more efficient than working on a copy of data and returning only to assign to the same original variable.  And as above, if pass-by-reference is appropriate, and if there is no question about the function/procedure succeeding or failing, then no return is necessary.


问:
What's the benefit of object-oriented programming over procedural programming?
I'm trying to understand the difference between procedural languages like C and object-oriented languages like C++. I've never used C++, but I've been discussing with my friends on how to differentiate the two.

I've been told C++ has object-oriented concepts as well as public and private modes for definition of variables: things C does not have. I've never had to use these for while developing programs in Visual Basic.NET: what are the benefits of these?

I've also been told that if a variable is public, it can be accessed anywhere, but it's not clear how that's different from a global variable in a language like C. It's also not clear how a private variable differs from a local variable.

Another thing I've heard is that, for security reasons, if a function needs to be accessed it should be inherited first. The use-case is that an administrator should only have as much rights as they need and not everything, but it seems a conditional would work as well:

```<code>if ( login == "admin") {
    // invoke the function
}
</code>```

Why is this not ideal?

Given that there seems to be a procedural way to do everything object-oriented, why should I care about object-oriented programming?

答:
Object Oriented Programming is Procedural Programming, with boxes.

In PP, you have one box, one state, that becomes incredibly large as the project grows, causing side-effects to appear every time you forget a tiny bit of that large state.

In OO, you have many boxes, many states, and as the project grows, the boxes grow a little, and the number of boxes grows a lot.

It remains easier to look at smaller boxes, easy to have the illusion of a whole picture, but actually almost impossible, as looking at classes and interfaces hides the implementation details that can have important ramifications.

In functional programming, you have many function boxes, and decide that every function has one entry (parameters) and one exit (returns), with strictly no other access to the outside context.

Because there is no state and no side-effects (by design), you can safely analyze any function separately from the whole and know 100% how it will behave in any circumstance.

Because you are boxing code by logical units which represent actions, it also becomes possible to only have one box per typical action.

This will shrink the code of any large scale project by a huge factor compared to OOP which promotes hiding multiple analog functions all over the code base in different classes.

This will also beat PP by very far because you can grow the project for much longer since there is no more XXXXXXXL state to keep track of.

In summary, PP is probably the simplest way to approach a simple program and FP is probably the simplest way to approach a complex one.

If you factor in the goal of unifying all code bases and improving code reuse, FP should always be used, as it's the only paradigm that makes sense at a very large scale, as well as the only paradigm that has 100% reusability (you can simply copy paste a function and use it somewhere else, with no overhead whatsoever).

And you get 100% reliable unit testing for free.

And you don't have to write "private static final of_doom genius awesome string_1".

And you get parallelism for free.


问:
Can compilers check whether certain axioms hold for a user defined type?
Suppose the following type is defined (C++ syntax, can be conceptually applied to any statically typed language.)

```<code>class T {
   int val;
   friend bool operator<(const T& lhs, const T& rhs) {
      return lhs.val < rhs.val;
   }
   // Ideally, if we should make them "friends"
   // Also, this doesn't make them belong to the type itself as they're not member functions
   friend bool operator>=(const T& lhs, const T& rhs) {
      return !(lhs < rhs);
   }
   // Also the following implementation is interesting
   friend bool operator>=(const T& lhs, const T& rhs) {
      return lsh.val >= rhs.val;
   }
};
</code>```

So, the compiler can definitely check whether T supports operator< or operator>=, it can just look up the declaration.

This gets more complicated should these be defined as "friends" instead of member functions, but it can be done as far as I'm concerned.

However, can the compiler check whether the "common axiom of sorts" hold? (Can as in is it possible to check this in all scenarios? Doesn't matter whether or not this would be standard-compliant.) Say:

not (a = b

Should we define one in terms of the other, it is definitely possible in this particular scenario, but this is naturally not the only axiom that needs to be checked, what about:

(a > b) => (b < a)

and it's also not necessary (unfortunately) to define our behaviour in terms of something that has already been defined. And sometimes it wouldn't be even possible.

So, if the compiler can check whether certain axioms hold, how? Generating random values from certain ranges? Is it possible to devise and proof correctness of such ranges?

This is mainly related to C++ concepts that unlike C++ concepts lite, should also be able to "support axioms", whatever that means, I just can't see a clear way as to how they can be enforced. Or will the creator of the type have to state what axioms his class obeys? C++ dropped the "concept maps" as far as I know.

答: No. Since the operators can hold arbitrary code, it is impossible to even determine whether they will terminate in all instances. For more info on this, look up the halting problem.</p>


答:
It doesn't matter.  As long as it's relatively consistent across stories, your burndown chart will still work either way.  Use whichever way is most natural for your team to report.

My team actually does a sort of hybrid, although not by formal agreement.  We put 16 hours if we think a task will take one person two days, but if two people end up working on it together, we don't change it.

After the initial estimate, it informally becomes more of a percent complete for our team than an hours remaining.  If we originally thought it would take two days, but after one day, we think it's only 25% complete, we take 4 of the original 16 hours off.  That leaves 12 hours where technically we're estimating 24, because we'll probably deduct 4 hours for the remaining 3 days.

That annoyed me as a scrum master at first, but strange as it seems, it's a very natural way to give estimates, because developers really hate to add hours to an estimate.  It all averages out to make the burndown still useful, and that's what's important.

Title: How to approach scrum task burn down when tasks have multiple peoples involvement?

Body:
In my company, a single task can never be completed by one individual. There is going to be a separate person to QA and Code Review each task. What this means is that each individual will give their estimates, per task, as to how much time it will take to complete.

The problem is, how should I approach burn down? If I aggregate the hours together, assume the following estimate:

10 hrs - Dev time

4  hrs - QA

4  hrs - Code Review.

Task Estimate = 18hrs

At the end of each day I ask that the task be updated with "how much time is left until it is done". However, each person generally just thinks about their part of it. Should they mark the effort remaining, and then ADD the effort estimates to that? How are you guys doing this?

UPDATE

To help clarify a few things, at my organization each Task within a story requires 3 people.

Someone to develop the task. (do unit tests, ect...)

A QA specialist to review task (they primarily do integration and regression tests)

A Tech lead to do code review.

I don't think there is a wrong way or a right way, but this is our way ... and that won't be changing. We work as a team to complete even the smallest level of a story whenever possible. You cannot actually test if something works until it is dev complete, and you cannot review the quality of the code either ... so the best you can do is split things up into small logical slices so that the bare minimum functionality can be tested and reviewed as early into the process as possible.

My question to those that work this way would be how to burn down a "task" when they are setup this way. Unless a Task has it's own sub-tasks (which JIRA doesn't allow) ... I'm not sure the best way to accomplish tracking "what's left" on a daily basis.


问:
Obscurity of Lisp in collaborative projects
I'm playing with the idea of learning Scheme but I have a few misgivings.

From what I understand Lisp makes heavy use of macros that allow programmers to drastically change the language itself. I understand why that's useful, but how does this affect collaborative projects? How much confusion can this cause if other developers implement obscure macros, with or without documentation? Are there well defined guidelines for avoiding this confusion or are there language features that help avoid it?

I ask because my current job is C++ and we have to use an in house framework. Our senior developer has tried to extend C++ to make it as Java-ish as possible. We have to spatter our code with obscure macros to make everything a smartpointer (in house of course, not the standard library kind) and homebrew exceptions etc. As a result, everything we write turns into a heaping mess and when these magic macros break we have no idea how to fix them. Obviously job security comes into play, but let's forget about politics and assume everyone is here for the right reason.

From what I've read, the C preprocessor is very primitive compared to Lisp. So would a situation like this be 1000 times worse in Lisp or would it be better somehow?

答:
Having worked on a few Lisp projects (Mostly Scheme, some CL) here are my thoughts.

With macros you can make code hell to maintain. But this is only when you don't document. Frankly I think the same can be said for any language feature. When was the last time you read 500k lines of OO ravioli and went "Gee there was no docs, but this code is so understandable anyways".

The best way to make sure everything stays readable is simply to enforce documentation. In the projects I've worked on the documentation had to have something like

```<code>;; sommac, Used to do foo.
;; Translation:
;;   (sommac (a b c)
;;       d e f)
;; to
;;    (loop for a from b to c
;;          do (begin d e f))
</code>```

The translation isn't exactly what the macro will do. Sometimes they were in pseudo-code. But it was a simple effective way to make sure everyone knew what was going on.

Body:
Having worked on a few Lisp projects (Mostly Scheme, some CL) here are my thoughts.

With macros you can make code hell to maintain. But this is only when you don't document. Frankly I think the same can be said for any language feature. When was the last time you read 500k lines of OO ravioli and went "Gee there was no docs, but this code is so understandable anyways".

The best way to make sure everything stays readable is simply to enforce documentation. In the projects I've worked on the documentation had to have something like

;; sommac, Used to do foo.
;; Translation:
;;   (sommac (a b c)
;;       d e f)
;; to
;;    (loop for a from b to c
;;          do (begin d e f))

The translation isn't exactly what the macro will do. Sometimes they were in pseudo-code. But it was a simple effective way to make sure everyone knew what was going on.


问:
which directory to initialize git repo
I'm just getting started with Git.  I'll be doing PHP development and was wondering if I should initialize my repo within my server root folder, or any directory on my hard drive and publish my changes to the server root.  Does it really matter, any pros/cons?

For now, this is just to keep track of personal projects I'm doing for fun, but any feedback on best practice, personal preference/experience, do's and don'ts will be greatly appreciated.

答:
Putting your source control files in the deployed directory runs the risk of exposing those files and their complete history.  This is potentially a bad thing (do you have passwords in these files? Are the files themselves important to not let leak?)

One could try to staunch this by putting a .htaccess to restrict raw access to .git files from a web browser:

RedirectMatch 404 "(?:.*)/(?:\.git|file_or_dir)(?:/.*)?$"
```

However, this only restricts files that way - it doesn't prevent someone from passing in something to poorly coded php that accesses those files.

The best way to avoid this is to keep the git repository well away from the production code itself.

If you specify a working tree (your production area) to be different then the git files are well away from prying eyes.  This can be done with the environment variable GIT_WORK_TREE.

Related:

P.SE - How to make a “git push” update files on your web host?

P.SE - how safe/sane is it to use git for deployment on my webapp production server?

Blog - Using Git to manage a web site

Blog - Managing a Website Using git

Blog - Automatically Deploying Website From Remote Git Repository

Title: which directory to initialize git repo

Body:
Putting your source control files in the deployed directory runs the risk of exposing those files and their complete history.  This is potentially a bad thing (do you have passwords in these files? Are the files themselves important to not let leak?)

One could try to staunch this by putting a .htaccess to restrict raw access to .git files from a web browser:

RedirectMatch 404 "(?:.*)/(?:\.git|file_or_dir)(?:/.*)?$"

However, this only restricts files that way - it doesn't prevent someone from passing in something to poorly coded php that accesses those files.

The best way to avoid this is to keep the git repository well away from the production code itself.

If you specify a working tree (your production area) to be different then the git files are well away from prying eyes.  This can be done with the environment variable GIT_WORK_TREE.

Related:

P.SE - How to make a “git push” update files on your web host?

P.SE - how safe/sane is it to use git for deployment on my webapp production server?

Blog - Using Git to manage a web site

Blog - Managing a Website Using git

Blog - Automatically Deploying Website From Remote Git Repository


问:
Implementation of a serial communication protocol
I need to implement a serial protocol to communicate with a device using .NET (C#). This implementation should be a library (.dll) to be used in different projects.

I have the datasheet that describe the protocol (baudrate, stop bits, message protocol, commands, etc.) and I know that I need to use SerialPort class from System.IO.Ports namespace.

But I have some doubts on how to struture/organize the code.

Here are some questions:

Should I be concern about thread management or this aspect should be managed by how is using the library?

Can I manage received/sent data using strings or should I use bytes?

Commands and fixed contents should be stored using enums, constants or something else?

Maybe those questions could be subjectives, but I would like to have some feedback from someone with more experience than me.

If someone has some examples, tips, best pratices, etc. on this subject, I will be very grateful.

答:
Virtually all of your choices are going to be driven by the device itself and the traffic required to make use of the device.  Your design will, necessarily, be driven by what the device is, how it works, how you want developers to interact with it, and how users want to use the device.

1) The question of thread management is the same everywhere--use a separate thread to keep other things from blocking if that's going to be a problem. Of course, that doesn't make the problem go away, it just changes the problem from blocking to concurrency/synchronization.  But that's a different problem.  All other things being equal, a separate thread for reads and writes dramatically simplifies how you decouple the IO from how you process the IO.

2) That depends entirely on what the device sends/recieves, both at the low level and at the design level.

3) Would the commands, fixed constants, or other aspects be subject to change?  If the device is a work in progress, it certainly will.  The most successful implimentation I've ever done was to create the IO as a read/write layer, connected to an interpreter layer that actually read the grammar from a resource, topped by an API layer for use by applications.  That was for a device that was in development for a very long time and the command set changed frequently; that design made it possible to use the same code for multiple devices.  Whether or not that works for you depends, again, on the device.


答:
(Note This answer is based on experience in c#. I have never tried it in c/c++)

I don-t think, pre- or postcoditions (like sorting) can be checked at compiletime at all.

But a compiler (or a compiler-postprocesser) can automatically insert code to do runtimechecks for pre- and postconditions. (That is the way it is done in c# code contracts which make shure that contracts are also enforced in overwritten virtual functions, too.

Wikipedia lists under Design_by_contract some c/c++ libraries to support contracts.

UML uses OCL=Object_Constraint_Language for this.

Title: Can compilers check whether certain axioms hold for a user defined type?

Body:
Suppose the following type is defined (C++ syntax, can be conceptually applied to any statically typed language.)

```<code>class T {
   int val;
   friend bool operator<(const T& lhs, const T& rhs) {
      return lhs.val < rhs.val;
   }
   // Ideally, if we should make them "friends"
   // Also, this doesn't make them belong to the type itself as they're not member functions
   friend bool operator>=(const T& lhs, const T& rhs) {
      return !(lhs < rhs);
   }
   // Also the following implementation is interesting
   friend bool operator>=(const T& lhs, const T& rhs) {
      return lsh.val >= rhs.val;
   }
};
</code>```

So, the compiler can definitely check whether T supports operator< or operator>=, it can just look up the declaration.

This gets more complicated should these be defined as "friends" instead of member functions, but it can be done as far as I'm concerned.

However, can the compiler check whether the "common axiom of sorts" hold? (Can as in is it possible to check this in all scenarios? Doesn't matter whether or not this would be standard-compliant.) Say:

not (a = b

Should we define one in terms of the other, it is definitely possible in this particular scenario, but this is naturally not the only axiom that needs to be checked, what about:

(a > b) => (b < a)

and it's also not necessary (unfortunately) to define our behaviour in terms of something that has already been defined. And sometimes it wouldn't be even possible.

So, if the compiler can check whether certain axioms hold, how? Generating random values from certain ranges? Is it possible to devise and proof correctness of such ranges?

This is mainly related to C++ concepts that unlike C++ concepts lite, should also be able to "support axioms", whatever that means, I just can't see a clear way as to how they can be enforced. Or will the creator of the type have to state what axioms his class obeys? C++ dropped the "concept maps" as far as I know.


答:
It depends. If the device can send messages asynchronously, you need separate threads for sending and receiving; if it only speaks when spoken to, you should use a single thread for simplicity.

It depends on the kind of information you need to send and receive, but typically you should use bytes to allow for non-ASCII control codes.

Use constants of type unsigned char.

Title: Implementation of a serial communication protocol

Body:
I need to implement a serial protocol to communicate with a device using .NET (C#). This implementation should be a library (.dll) to be used in different projects.

I have the datasheet that describe the protocol (baudrate, stop bits, message protocol, commands, etc.) and I know that I need to use SerialPort class from System.IO.Ports namespace.

But I have some doubts on how to struture/organize the code.

Here are some questions:

Should I be concern about thread management or this aspect should be managed by how is using the library?

Can I manage received/sent data using strings or should I use bytes?

Commands and fixed contents should be stored using enums, constants or something else?

Maybe those questions could be subjectives, but I would like to have some feedback from someone with more experience than me.

If someone has some examples, tips, best pratices, etc. on this subject, I will be very grateful.


答:
Lisp macros can be used well or badly.

Simple use of macros can be similar in effect to cpp - replacing something simple looking with related boilerplate for example.  It can make things prettier without changing much if used well, or it can make things awful by doing something unexpected.

The entire lisp language and environment is available to lisp macros. Sophisticated use of lisp macros can implement whole language constructs ("loop" for example is just a lisp macro), or perform complex analysis or build complex data structures; effectively becoming part of the compiler.

Title: Obscurity of Lisp in collaborative projects

Body:
I'm playing with the idea of learning Scheme but I have a few misgivings.

From what I understand Lisp makes heavy use of macros that allow programmers to drastically change the language itself. I understand why that's useful, but how does this affect collaborative projects? How much confusion can this cause if other developers implement obscure macros, with or without documentation? Are there well defined guidelines for avoiding this confusion or are there language features that help avoid it?

I ask because my current job is C++ and we have to use an in house framework. Our senior developer has tried to extend C++ to make it as Java-ish as possible. We have to spatter our code with obscure macros to make everything a smartpointer (in house of course, not the standard library kind) and homebrew exceptions etc. As a result, everything we write turns into a heaping mess and when these magic macros break we have no idea how to fix them. Obviously job security comes into play, but let's forget about politics and assume everyone is here for the right reason.

From what I've read, the C preprocessor is very primitive compared to Lisp. So would a situation like this be 1000 times worse in Lisp or would it be better somehow?


问:
Before you develop a web app, how much of a spec do you need?
If you were to accept a role as a full time web app developer responsible for creating a new application from scratch, how much of a defined specification and design would need and/or expect to start development?

What tools/software is even used for this (for web apps built with something like RoR)? I would expect a mockup/wireframe to be required, but what else is done in the design stage? What is the best method or process to convey the functionality of the application so that developers know what to translate into code.

I'm interested to hear from web app developers who have thought: "If I only had <this>, I could develop this so much quicker/easier/cleaner", or "I have no idea what this customer wants unless they document it in <thing>."

I'm not trying to start a philosophical debate, I am curious from the developers' perspective what is the "right" way to design web apps from scratch. There is probably a book somewhere that someone can tell me to RTFM, but I've only been able to find resources on learning actual coding and development, not the design phase.

答:
It depends.

My favorite project I did started off with the specification of "Make the knowledge base browse-able."  Thats it.  One sentence.  I worked with the knowledge manager (who was responsible for the data and structure of the knowledge base), and a user interface guy.  And we did it.  Turned out to be a very successful project in part because the goal donor (same as gold owner in this case) didn't have any preconceptions of how the final thing worked - just that it worked and trusted us enough to let us make it.

On the other hand, there have been projects with many integration points between different teams where there were dozens of xsds floating around that specified every detail of the interface.  The business came to us with mock ups of what they wanted and all the rules for how each field behaved and what was sent to the other systems.  This worked too - and it was necessary.  If any one team went off and did their own thing, it would have fallen like a house of cards once we tried to put it together.  There were reams of paper that represented various versions of the specification.

Recently, one of the companies that subcontracts work to my current employer commented that they are very happy with us because they give us not much in the way of specifications and we do what they were thinking (our project managers must be telepathic).

There is no right answer, it depends completely on what the expectations of the client (be it another business uint in the company you work for, or an external client).  The more they expect things and the less they trust you do to the right thing and accept what you give them, the more the work will need to specified.

One set of the approaches that was taken up to try to address this are the iterative methodologies.  The idea with these is you build something in a short time period and then ask "is this what you were thinking?" Each iteration you add functionality or change it as the client asks.  The most well known of these approaches is agile (which itself encompasses a range of methodologies).

As to books?  There are dozens of them.  Seriously, dozens.  Maybe dozens of dozens if not more (there's no one book that we can tell you will solve that problem - just go into the project management section of a library or bookstore and start thumbing through them).


问:
Treating a 1D data structure as 2D grid
I am working with a native class that represents a 2D image as a 1D array. If you want to change one pixel, for example, you need to now how to derive the index from the x,y coordinates.

So, let's say we have a 1D array array1d like this:

```<code>array1d = [ a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t, u, v, w, x, y ]
</code>```

In the context of our program, array1d represents a 2D grid:

```<code>a b c d e
f g h i j
k l m n o
p q r s t
u v w x y
</code>```

And we want to perform operations on array1d such as:

Get the value at x,y coordinates (in this example, 1,2 would give l)

Get any sub-grid using x,y,width,height (1,2,2,2 would give [l, m, q, r])

Set the value at any x,y coordinate (etc.)

How do we do these?

答:
2D / 1D - mapping is pretty simple. Given x and y, and 2D array sizes width (for x-direction) and height (for y-direction), you can calculate the according index i in 1D space (zero-based) by

```<code>i = x + width*y;
</code>```

and the reverse operation is

```<code>x = i % width;    // % is the "modulo operator", the remainder of i / width;
y = i / width;    // where "/" is an integer division
</code>```

You can extend this easily to 3 or more dimensions. For example, for a 3D matrix with dimensions "width", "height" and "depth":

```<code>i = x + width*y + width*height*z;
</code>```

and reverse:

```<code>x = i % width;
y = (i / width)%height;
z = i / (width*height);
</code>```

Title: Treating a 1D data structure as 2D grid

Body:
2D / 1D - mapping is pretty simple. Given x and y, and 2D array sizes width (for x-direction) and height (for y-direction), you can calculate the according index i in 1D space (zero-based) by

i = x + width*y;

and the reverse operation is

x = i % width;    // % is the "modulo operator", the remainder of i / width;
y = i / width;    // where "/" is an integer division

You can extend this easily to 3 or more dimensions. For example, for a 3D matrix with dimensions "width", "height" and "depth":

i = x + width*y + width*height*z;

and reverse:

x = i % width;
y = (i / width)%height;
z = i / (width*height);


问:
When is it NOT good to use actors in akka/erlang?
I've been working with akka for 7-8 months now daily.
When I started, I would be working on applications and notice that actors would be used basically anywhere once inside the actor system for communicating between most objects. So I did the same - spin up another actor for x/y/z.

It seems to me that this may be too indiscriminate, adding complexity where it isn't needed - but I can't find any discussions on where actors vs plain synchronous or even async logic via futures should be used. I started pondering my stance after my coworker mentioned something similar. I realized several cases more recently where I have pondered a task and then avoided creating another actor because I could achieve the same result safely in an immutable implementation - eg something like getting configuration values from a db or file somewhere where you access very infrequently and will wait for the result is the actual use case.

In particular, it seems to me that any case where you're playing with immutable state, actors create complexity and limit throughput - a pure function in an object, for example, can be called concurrently with no risk with any level of concurrency, yet an actor can only process one message at a time. The alternate consideration is you'll park the thread if you need to wait for the result unless you start using futures but cases where you don't need to worry about async messaging or scale it seems it may be overkill to employ an actor.

So my question is - is there a bad time to use actors? I'm curious how erlang looks and would really like other people's insight. Or if there are some principles around actor use.

答:
It's worth considering what the actor model is used for: the actor model is

a concurrency model

that avoids concurrent access to mutable state

using asynchronous communications mechanisms to provide concurrency.

This is valuable because using shared state from multiple threads gets really hard, especially when there are relationships among different components of the shared state that must be kept synchronized. However, if you have domain components in which:

You don't allow concurrency, OR

You don't allow mutable state (as in functional programming), OR

You must rely on some synchronous communications mechanism,

then the actor model will not provide much (if any) benefit.

Hope that helps.


答:
Since you have a C background it is best to first look at lisp and C macros, and see what properties they share and which they are not sharing.

In C, macros are compile-time-rules for almost textual replacement of code parts by the preprocessor. C macros have a different syntax than the C programming language, they are executed by the preprocessor and emit C code that is compiled by the C compiler. So it is a preceding step to program compilation.

Lisp is a little different here. In Lisp, every code is a tree-like (directed, acyclic graph) datastructure that is written in S-Expressions (think of S-Expressions like compact XML).

```<code>(case a
   ((10) "a is ten")
   (else "foo"))
</code>```

This datastructure consists of nested list objects that offer a simple API for accessing its elements. The great thing about lisp is, that the same data structures used in a lisp program for storing lists is also used during the compilation of the Lisp code. Therefore one can write small "compiler" extensions in Lisp code that transform branches of the abstract syntax tree into other representations. Again: You use the same language that you use for programming an algorithm to transform the code.

For a moment, I would downplay the differences between a procedure and a macro call. The first real difference between a procedure and a macro is about the time when they are executed not about the nature of their abstraction in the first place.

The overall question is: How can we keep a codebase from becoming an obscure meta-langauge that is not accessible to colleagues?

Keeping programs readable

Readability of programs is a complex topic.

First of all it comes down to the experience and expectancy of the reader. Code that looks foreign is harder to understand for a programmer than code he/she is familiar with.

Then there is the problem of code-scope. When reading code we are quite local and cannot look at all parts of the source at the same time. Therefore code that is compact should be easier to read than related code that is distributed widely over the source file.

However, good abstraction helps code readability since it groups operations into logical chunks. A function composes these abstracted operations is more readable than an equivalent function that concretely implements everything on its own.

Good abstraction means separation of concerns. A function that does several unrelated things at a time is harder to understand since you have to disentangle the functionalities within the procedure.

Consider a function write_file_and_show_results might be a bad function. Especially if it does not compose one write_file and one show_results function but directly opens the file, starts writing to it, shows results, writes some more....

The virtue of lisp is, that it is really good for implementing these good abstractions. This is often compared to "writing an own language for the problem". Yet it is hardly different than programming in other languages, Lisp just has more drastic means for extending syntax.

Commonly if people argue against macro use or abstraction, they point out, that a direct, concrete implementation is more readable than one that involves a few abstractions. In practice, it mostly pays off to introduce some abstraction that compacts the rest of the code. It saves LoC, maintainability is increased since you only have to perform changes once than in several parts of the problem

Architectural design: Bottom up

Paul Graham has written a nice essay on how to work when writing Lisp code. http://www.paulgraham.com/progbot.html

Macro Hygiene

There is one huge problem associated with macros: Hygiene.

Take the following example of a C program to illustrate it.

```<code>#define max_defect(a,b) ((a) > (b) ? (a) : (b))
</code>```

The issue at hand is, that an invocation of max_defect(i++, 3) will increase i twice. Typically this problem is solved by introducing another binding of a and b (something like _a) and evaluate the parameters to it. Then, the bound results are used to perform the actual code.

However, what happens if you then use _a as a parameter to the maximum-determining function? You have a name conflict at hand. In Lisp this problem has been solved by introducing gensyms. Generated symbols that are unique in the program.

However since this still leaves the possibility for errors, other ways to define macros have been developed, so called hygienic macros. Scheme does offer hygienic macros. They always guarantee correct lexical scoping.

Developing a common language

When presenting the idea of a "modifiable" programming language, people sometimes fear, that this might lead to a programmer building up "his/her own world" that is inaccessible to the outsider. While this is certainly possible, as long as you are not working in total isolation it will not be the immediate consequence. A lot of macros are already existing that encapsulate a certain abstraction, their naming schemes can be adapted, and there are also surprisingly readable macro constructs, such as the loop macro.

Generally the lisp communities have slightly different traditions in naming things than other programming communities. make-whatever for creating an object, with-RESOURCE for RAII like resource management, etc. Getting used to this is not an issue when reading established lisp codes and will enable you to write consisten lisp code.

Disclaimer: I have mostly worked with Scheme and Clojure, not very much with Common Lisp. Clojure is probably the most readable dialect for people who have some experience with Python and/or ruby.

Body:
Since you have a C background it is best to first look at lisp and C macros, and see what properties they share and which they are not sharing.

In C, macros are compile-time-rules for almost textual replacement of code parts by the preprocessor. C macros have a different syntax than the C programming language, they are executed by the preprocessor and emit C code that is compiled by the C compiler. So it is a preceding step to program compilation.

Lisp is a little different here. In Lisp, every code is a tree-like (directed, acyclic graph) datastructure that is written in S-Expressions (think of S-Expressions like compact XML).

(case a
   ((10) "a is ten")
   (else "foo"))

This datastructure consists of nested list objects that offer a simple API for accessing its elements. The great thing about lisp is, that the same data structures used in a lisp program for storing lists is also used during the compilation of the Lisp code. Therefore one can write small "compiler" extensions in Lisp code that transform branches of the abstract syntax tree into other representations. Again: You use the same language that you use for programming an algorithm to transform the code.

For a moment, I would downplay the differences between a procedure and a macro call. The first real difference between a procedure and a macro is about the time when they are executed not about the nature of their abstraction in the first place.

The overall question is: How can we keep a codebase from becoming an obscure meta-langauge that is not accessible to colleagues?

Keeping programs readable

Readability of programs is a complex topic.

First of all it comes down to the experience and expectancy of the reader. Code that looks foreign is harder to understand for a programmer than code he/she is familiar with.

Then there is the problem of code-scope. When reading code we are quite local and cannot look at all parts of the source at the same time. Therefore code that is compact should be easier to read than related code that is distributed widely over the source file.

However, good abstraction helps code readability since it groups operations into logical chunks. A function composes these abstracted operations is more readable than an equivalent function that concretely implements everything on its own.

Good abstraction means separation of concerns. A function that does several unrelated things at a time is harder to understand since you have to disentangle the functionalities within the procedure.

Consider a function write_file_and_show_results might be a bad function. Especially if it does not compose one write_file and one show_results function but directly opens the file, starts writing to it, shows results, writes some more....

The virtue of lisp is, that it is really good for implementing these good abstractions. This is often compared to "writing an own language for the problem". Yet it is hardly different than programming in other languages, Lisp just has more drastic means for extending syntax.

Commonly if people argue against macro use or abstraction, they point out, that a direct, concrete implementation is more readable than one that involves a few abstractions. In practice, it mostly pays off to introduce some abstraction that compacts the rest of the code. It saves LoC, maintainability is increased since you only have to perform changes once than in several parts of the problem

Architectural design: Bottom up

Paul Graham has written a nice essay on how to work when writing Lisp code. http://www.paulgraham.com/progbot.html

Macro Hygiene

There is one huge problem associated with macros: Hygiene.

Take the following example of a C program to illustrate it.

#define max_defect(a,b) ((a) > (b) ? (a) : (b))

The issue at hand is, that an invocation of max_defect(i++, 3) will increase i twice. Typically this problem is solved by introducing another binding of a and b (something like _a) and evaluate the parameters to it. Then, the bound results are used to perform the actual code.

However, what happens if you then use _a as a parameter to the maximum-determining function? You have a name conflict at hand. In Lisp this problem has been solved by introducing gensyms. Generated symbols that are unique in the program.

However since this still leaves the possibility for errors, other ways to define macros have been developed, so called hygienic macros. Scheme does offer hygienic macros. They always guarantee correct lexical scoping.

Developing a common language

When presenting the idea of a "modifiable" programming language, people sometimes fear, that this might lead to a programmer building up "his/her own world" that is inaccessible to the outsider. While this is certainly possible, as long as you are not working in total isolation it will not be the immediate consequence. A lot of macros are already existing that encapsulate a certain abstraction, their naming schemes can be adapted, and there are also surprisingly readable macro constructs, such as the loop macro.

Generally the lisp communities have slightly different traditions in naming things than other programming communities. make-whatever for creating an object, with-RESOURCE for RAII like resource management, etc. Getting used to this is not an issue when reading established lisp codes and will enable you to write consisten lisp code.

Disclaimer: I have mostly worked with Scheme and Clojure, not very much with Common Lisp. Clojure is probably the most readable dialect for people who have some experience with Python and/or ruby.


问:
Why many designs ignore normalization in RDBMS?
I got to see many designs that normalization wasn't the first consideration in decision making phase.

In many cases those designs included more than 30 columns, and the main approach was "to put everything in the same place"

According to what I remember normalization is one of the first, most important things, so why is it dropped so easily sometimes?

Edit:

Is it true that good architects and experts choose a denormalized design while non-experienced developers choose the opposite?  What are the arguments against starting your design with normalization in mind?

答:
A lot of developers don't know or care about normalization, or about data modeling or database.

For some jobs it's really not important.

Sometimes there's a really good reason to de-normalize, for example to make a particular difficult workload perform well.

Relational Database concepts are recently less in fashion than they were in the 1990s and 2000s. Developers tend to be influenced by fashion, even if they claim to be very rational. There's no point arguing about taste.

Normalization is also, historically, a territory for near religious argument, so I hesitate to say much more.


答:
In large projects, and specially those in mainframes, this is not the case. In fact if you search job sites you will see several positions for data modelers. Also, having many columns on a single table does not go against normalization. Nevertheless, your observation is valid for some projects.

Database design is one of the skills required to build quality systems. Having said that, some developers don't know enough about database design and still get assigned to the task of data modeling and database design. Some projects even skip data modeling. The focus on many projects is mainly on coding and front-end design.

Another factor for poor database design is the fact that Normalization is not a trivial topic specially when it comes for 4th NF,5th NF, etc. Most books I have seen could not clearly explain those forms well. There is usually bad examples and too much theory. This makes the topic less popular than it should.

Errors in database design are hard to come by unless you look for them or you encounter them during testing. Having no standard for database design quality lets errors happen more likely.

Add to that the fact that some projects don't follow a rigorous development methodology (one that promotes database design), as a result, responsibilities get mixed and tasks get lost between the business analyst, the developers and the DBAs. Developers talk in OO and UML where DBAs talk in DD and some in ERDs and probably many don't get UML or OO. In short, the lack of knowledge, lack of good clear resources, lack of unified language to describe data and lack of methodology are all to blame.

Title: Why many designs ignore normalization in RDBMS?

Body:
I got to see many designs that normalization wasn't the first consideration in decision making phase.

In many cases those designs included more than 30 columns, and the main approach was "to put everything in the same place"

According to what I remember normalization is one of the first, most important things, so why is it dropped so easily sometimes?

Edit:

Is it true that good architects and experts choose a denormalized design while non-experienced developers choose the opposite?  What are the arguments against starting your design with normalization in mind?


问:
Generating a custom widget that users can embed into their external website based on my server data which changes periodically
I'm deciding on how to generate the code to allow users generate an embeddable widget (much like the StackOverflow badge) into external websites.

The content of the embedded widget will periodically change, but it doesn't need to be communicating in real-time with my server after a visitor has loaded the page where it's embedded.

This opens up several possibilities for getting this to work:

On the server, pre-generate the static html content I want rendered in their page (and have a scheduled job which regenerates the static file). My users can then embed it in their page using an iframe reference to the static resource.

Same as above, but instead of the iframe I create a js file which they reference (much like an Google Analytics code) and then the js file served inserts the data in into their DOM. My web server would need to dynamically generate the js file on each request for the file resource.

Give them a js file which creates the element on their page like the above, except the js file payload doesn't include all the data and instead basically generates the DOM element as a template, then calls web services to populate the data (like their name and score) using JSONP requests to my server.

I like 2 for the explicit server-side control, but it will take more time. 3 is good because the services I expose to retrieve this data can be reused for other purposes later. I don't generally like iframes, but they work and are very quick to implement.

Any suggestions of which way to proceed or ideas that I've missed?

答:
I'd go with #3. #2 and #3 have basically the same difficulty (DOM manipulation in Javascript and catering to, or at least investigating, the quirks of different browsers). #1 is surely faster, but IFRAMEs have problems of their own (some platforms, e.g. Joomla!, routinely filter them out; some odd antivirus may balk or strip the iframe altogether; etc.).

Solution #3 decouples data and presentation, and might more easily be further developed in order to let users customize the widget to blend with different designs, if desired. Just my opinion, I like that one best.

A fourth possibility, depending on the widget features and specs, could be to generate a server side image - say, an (animated?) GIF. Of course this assumes the widget is nothing more than an informational passive object (one HREF at the most), so it might be out of the question.


问:
Do we need a weekly project status meeting in Agile?
I am working in a small company with around 15 people (dev and qa).  We are trying to practice Agile in our company. We are doing daily stand-up meetings. We are also doing a weekly project status meeting with our CTO. There we report our CTO regarding last week task and overall project status. My question is whether doing this weekly status meeting  with our CTO supported in Agile or not?</p>

答:
It depends on which Agile methodology you are following.

If you are doing Scrum, the answer would probably be no.  The sprint backlog and conversations with the Product Owner and Scrum Master should be enough to keep your CTO filled in and happy.  If you do not have a Product Owner and a Scrum Master, then you are not doing Scrum, and so the weekly meeting may be necessary.

Of course, Scrum is not the only Agile methodology out there.  It is just the most popular. Others, such as Crystal, do not say anything about whether meetings should occur or not.  A Lean approach would suggest that a weekly meeting with your boss might be a good thing as long as it is a productive meeting.  If it is not productive, it is waste and you should find another way to let your CTO know how you are progressing.


答:
There are many, many practitioners and thought leaders in the Agile space who will tell you to focus on products, not projects. Many will say that the very idea of a "project" flies in the face of everything Agile stands for, and in my experience, they're basically right. "Projects" are synonymous with heavy planning, detailed requirements, and fixed deadlines. They're the anti-Agile.

That's not to say you can't be "lowercase-'a' agile" in a project-based environment, but the idea of a "project status" is utter nonsense in a Scrum or Open Agile setting because it presupposes that you're working toward some enormous Big Bang release or similar milestone. If that's the case, you're simply not doing Agile.

Your team should be doing frequent delivery - at a bare minimum you should have the product showcases/demos every few weeks, and be releasing something to production every quarter at most (preferably once per month, if not more frequently). Status meetings are cheap words and empty promises; if you've got nothing to show, then they're just a waste of your time and your CTO's.

And if the developers have to be involved in these meetings (you don't say, but seem to hint at that) then it is a massive waste of their time.

I'll also point out as a sidebar that non-actionable meetings in general are useless. The purpose of any meeting should be to make a decision. The purpose of the stand-up is to determine if anyone is being blocked or interrupted and allow the managers to take corrective action. It's not clear what the purpose of your status meeting is, but unless there is an actual decision to be made or reevaluated between you and your CTO each week, the meeting has negative value regardless of whether or not you are Agile.

Title: Do we need a weekly project status meeting in Agile?

Body: I am working in a small company with around 15 people (dev and qa).  We are trying to practice Agile in our company. We are doing daily stand-up meetings. We are also doing a weekly project status meeting with our CTO. There we report our CTO regarding last week task and overall project status. My question is whether doing this weekly status meeting  with our CTO supported in Agile or not?</p>


问:
Storing donor addresses in a relational database
I am building a donor database for a non-profit organization and one issue I'm mulling over is how to store some of the donor data.  There are some families for whom we capture the names of both spouses, hence each spouse gets its own record in the DB, and naturally they share the same address.  Because it is likely to happen that multiple people from a family will exist in the database, and they will share the same address (husband, wife, kids), I am considering storing the addresses in a separate table and having Address_ID be a foreign key on the Roster table.  This would reduce the amount of duplicate data, and when it came to address changes, it would be far easier to handle for multiple donors that are part of a family.

```<code>Table: Roster
Columns:
Roster_ID (PK)
PrimaryRoster_ID (FK, references Roster_ID of the head of the household)
Address_ID (FK)
FirstName
LastName
Phone

Table: Address
Columns:
Address_ID
Street
Apt
City
State
ZipCode
</code>```

I haven't build the GUI yet for entering the data - I'm still building the DB and populating it via scripts, but the idea is that when a new donor is created, he/she is assigned the Primary, and then any other donor who is part of the same family will have the primary donor's Roster_ID populated in the PrimaryRoster_ID column so that a link can be made between family members.

People in the same family don't have to have the same address (if they don't, a separate Address record would be made for that donor), but if there are 4 people in a family, all with separate DB records and the user who enters the data assigns the same address record to all of them, only one record will exist in the Address table, with 4 records in the Roster table having that Address record as an FK to associate each individual to that address.

Can anyone think of a better way of handling this, or would this design be sufficient?  The size of the database right now is around 150 records in the Roster table, with maybe a max of 1000 records down the road.

Edit: Changed table design slightly - I realized that Address_ID should really just be an FK in Roster instead of having a Roster_Address table.

答:
With at most a few thousand records, this (part of the) database is not particularly large, so there is no particular need to split off the address information to conserve space.

With space considerations out of the way, the most important reasons that influence how to store the address information are the business rules.
For example, if the organisation does not do anything with the fact that two people live at the same address, and there is no indication that this might change in the foreseeable future, then the fact that two table rows contain the same address information is purely incidental and there is no need to create a separate address table. Just like you wouldn't think of creating a separate table of first and last names just because there might be multiple persons called "John Smith".

The same goes for the relation among family members. If that is of no concern to the business, then why bother putting it in the database.

On the other hand, if the organisation distributes a magazine among its members, and they have the policy of sending only one copy per family living at one address, then your design is certainly appropriate and sufficient to meet the requirement of the business rule.

Regarding the design of the address table itself, you have to ask yourself if all relevant addresses can be represented in that format, or if you might need a different table layout. Especially of the organisation might have to deal with foreign donors, you might need to opt for a much more free format for the addresses, because nothing can vary as widely between countries as how they represent street addresses.


问:
User Story vs Requirement
User Story captures what the user wants to do with the system at a high level. I understand that the user story would further drive a number low level requirements. Is user story same as high level requirement for the system?</p>

答:
Ron Jeffries wrote a long time ago about the 3Cs of user stories (http://xprogramming.com/articles/expcardconversationconfirmation/) with the emphasis on a card (short description), conversation between the customers and the delivery team once a user story becomes actionable, and the agreed confirmation of a story after that conversation.

essentially, before the conversation, user stories are just planned scope - rough ideas about what we might do. after the conversation, you should have a way to confirm that the story is complete. So depending on the time when you look at the story in this timeline, a story might be just a broad idea about scope, or a detailed requirement.

these days, the meaning of "user story" seems to be narrowed to just the "card" part of Jeffries' 3Cs. in that case, a user story is not a requirement but a promise to hold a conversation about the requriements.

You can find a ton of gold nuggets of wisdom about user stories on the c2 wiki (http://xp.c2.com/UserStory.html)

Title: User Story vs Requirement


答:
To me, a critical element of a User Story is that it captures Why and How a user uses the system.  It is especially useful because it does not specify much in the way of how the system delivers the required functionality.  When UI and Usability testing is needed, the User Story may be the most important document.

Sure, you can have selenium verify that certain nodes are present in the HTML or take screen shots, or verify that certain pixels are where you hope that they are.  But if there is misleading text, or it is not apparent how the user should use the system or it is hard for a user to figure out how to achieve their goal, suddenly you don't have a complete system any more.  Now training is required in order to use the system.  Reviewing the completed system against the user scenarios is a critical component of manual testing.

There is a mind set captured in user stories/scenarios which should influence many detailed design decisions about the implementation.  Unless developers talk directly to users or watch them use the system, the user scenario may be the only link to allow them to understand user needs.

Finally, they allow the business people to specify what they need without suggesting how that should be achieved.  It is much easier to describe a solution, than a need, and user scenarios provide a framework for describing needs without proposing a specific solution.  The most common business requirement I've heard is, "I want it to be just like Excel, but on the web" which has never yet been what they actually needed.

So I would say that a good story should not include any specific details about how the system should be implemented.  It should say, "Once a month, system A needs to be updated with any new data from system B.  That data may require corrections.  The client has a history of inputting invalid data and not realizing the problem for weeks."  It should not say, "The system must accept a latin1 CSV file at least once a month and throw a NumberFormatException if column 3 is not a number."  Do you see the difference?  The scenario describes the need, not any specific solution.  Then in testing you circle back to the scenario to make sure that the solution fits the need.  Requirements may mix some needs with some solutions, or even focus entirely on solutions.

Title: User Story vs Requirement

Body: User Story captures what the user wants to do with the system at a high level. I understand that the user story would further drive a number low level requirements. Is user story same as high level requirement for the system?</p>


问:
Datastructure for a factory pattern in practice
I'm implementing what's basically an event log system for a larger system.  I used Single-table inheritance to build out the table.

The problem I'm having is figuring out how to build out the classes from the database.  It's easy enough for me to figure out what they are to load them into the database, but to pull them out and create objects and collections out of them is a little trickier.  The only way I know of is to have a switch statement and hope that if someone implements a new object that they'll update the switch, but that doesn't seem practical.

I'm sure I'm getting something wrong about how I'm thinking of the factory pattern here.

答:
You might want to look at how Hibernate does it. See the SingleTableEntityPersister if you are truly using single-table inheritance.

Or better yet... just use Hibernate/NHibernate or another ORM and don't reinvent the wheel.

To be clear, I'm not a huge fan of ORMs, but phrases like "single-table inheritance" imply that you're already trying to object-relational mapping the "manual" way, and you'd save yourself a lot of time by using a tool that does all the heavy lifting for you.

If you must do it yourself for whatever reason... well, as you can see, there are different methods, but typically you will need to use a discriminator column, and for that you will either need a map, a switch statement, or a consistent convention (e.g. the discriminator value has an obvious relationship to the class name and you can instantiate it directly with reflection). Switch statement is going to be the fastest and is probably the preferred approach for a fairly small number of derived classes.

Tools like Hibernate can't do it that way because, obviously, it would require specialized knowledge about the specific hierarchy, which they don't have. You do have that knowledge, so you can skip all the mapping/reflection and just use switch/case in a factory class somewhere. The fact that someone might forget to update it in the future isn't really an argument... your factory should throw an exception as soon as it encounters an unknown discriminator value.

If you're really that worried, make the discriminator an enum, and write a unit test for the factory that iterates through every known enum value and tries to instantiate it with the factory. That way the future maintenance programmer will find out right away if they missed something.


答:
To be honest, after spending close to two years immersed in Agile development, I still think "user story" is just a fancy term for "functional requirement".

It's different at a superficial level, e.g. it always takes a certain form ("as an X, I want Y so that Z..."), but the key elements - identifying the stakeholder and the rationale - are also inherent in well-written functional requirements. It's just as easy to write a bad user story as it is to write a bad requirement ("as [our company name], I want [vague feature] so that I can [do something that's self-evidently part of my job, like 'sell more to customers']").

What user stories almost never capture, in my experience, are non-functional requirements like performance and security. These kinds of requirements are very difficult to write properly and the format of the user story simply isn't very good for capturing them, because they're more about general product quality and mitigating (but not eliminating) risks rather than meeting a specific user's need.

So, I really think of user stories as a subset of requirements, with a specific formula, and still use the terms pretty much interchangeably.

The one major advantage user stories do have over requirements is that the word "requirement" suggests that a feature is required where it is often just desired. User stories can in theory be prioritized and slotted in for any release, whereas requirements appear to be a prerequisite for every release.

Of course, for the aforementioned distinction to matter, your customers and/or senior management must embrace it; it does you no good whatsoever if you have 30 user stories all grouped into a "project" that must all be completed at the same time. You might as well call them "requirements" in that case because they are in fact required.

Title: User Story vs Requirement


问:
The Lisp in Gnu
Since the GNU project is celebrating its anniversary, and the initial announcement for GNU is linked to (http://www.gnu.org/gnu/initial-announcement.en.html) all over the place, I reread it and I stumbled upon the plan for a lisp-based window system:

and eventually a Lisp-based window system through which several Lisp programs and ordinary Unix programs can share a screen.

It is well known that Stallman had a lisp background and has launched lisp-based projects (Emacs, Guile). As I understand, Stallman's preferred desktop environment for the GNU system is nowadays GNUstep (which seems to be a little dormant).

When looking at demonstrations of Symbolics Lisp machines, I am really impressed by the powerful approach and I think that Stallman had something like this in mind during the announcement.

So I wonder: What happened to this plan of a Lisp-based window system? Has it been actually pursued once?

答:
Nothing happened.

The main reason is that Stallman grossly overestimated the amount of help he was going to get from other developers, so his plans were, correspondingly, grandiose.

In fact, the first generally useful GNU systems did not become available for 10 years (until the Linux kernel became usable for a non-hacker user). This lead to the sarcastic expansion of GNU as "Generally Not Used/Useful/Usable".

The closest were, I guess, two window managers: a lisp-based one gwm and a scheme-based one scwm, neither of which actually GNU projects.

Nothing Will Happen

One has also to take into account that the X Window system made an enormous progress in the last 30 years, and became very much entrenched, so replacing it with something only slightly better - but incompatible - is not really an option.

Title: The Lisp in Gnu


问:
Architecture design for a web application that runs SSH connections to remote servers
This is a followup to my question here.

I have to make a web GUI for a library that manages remote clusters via SSH.

The main point is that this library spawns threads and subprocesses (Popen) internally (for the SSH connections), so I think I am I/O bound by these SSH connections. I want to prevent my server from going down as soon as I get 20-50 simultaneous requests.

I have no idea of how to proceed with the architecture. I don't know if I can solve this at the server level with a more powerful server (gevent??) or if I should do something like using background threads (twisted reactor??)

Some considerations:

I work with django so the server (if I finally have only one) must be WSGI.

I need both blocking and non-blocking calls via SSH.

I will probably need shared state between successive requests as well.

答:
You really want to be looking at separating the concerns here -- have your web app make requests to a message queue, have the execution components process messages from the queue. This also makes security much less nightmarish because you can push the things manging SSH tasks well off the web tier.

The original answer [above] was a bit terse and didn't answer a few questions.

How will this avoid SSH connection threads exhausting the server? It will avoid it because you can easily throttle most message queues to avoid that. And you can easily ad back-end workers if you are exhausing your initial setup without material change in your application.

How can I have shared state between requests? The same way you would normally -- you send the request out to the queue, it completes and brings data back which can be passed to the next request.

How would it appear to users? Could there be status updates? In most cases you can get the data out of the queuing system, or at least have status updates for task completion. Most message queues can easily give indication of failures or failures to start as well.

Would I have to block the process when communicating between my web server in the message queue be blocking? Not unless you wrote it that way. You can easily setup an ajax request to tickle the queue every few seconds for status on a message to update the UI.

Is this over-engineering? Not unless you decide to roll your own message queue.


问:
Deduplication of complex records / Similarity Detection
I'm working on a project that involves records with fairly large numbers of fields (~15-20) and I'm trying to figure out a good way to implement deduplication. Essentially the records are people along with some additional data. For example, the records are likely to include personal information like first name, last name, postal address, email address, etc. but not all records have the same amount of data.

Currently records are stored in a RDBMS (MySQL) and I want to detect duplicates on insertion  and still have them inserted but flagged as a duplicate. It needs to be fast as I need to provide feedback as to if it is a duplicate or not in real time. The dataset is large (millions of records).

I've considered the following options but I'm not sure which is best/if they are better options available:

Use MySQL's built in fulltext search and use fuzzy searching. Major issue with this is that is seems slow, only the latest version supports fulltext indexes with InnoDB (alternative engine is MyISAM which is not good and critically does not support transactions) and fuzzy searching alone does not seem the best method for similarity detection.

Use simhash or similar. Issue with this is that I'd also like to be able to detect synonyms which I don't see how simhash handles this. For example, address might be: "Some Road" or "Some Rd." and names might be: "Mike" or "Michael"

Index the data using an Apache Lucene derivative (elasticsearch/solr/etc) and perform a query that would likely return numerous results.

In terms of using Apache Lucene I've been reading about similarity detection and using cosine similarity to produce a value from 0 to 1 from the term frequency vectors that lucene stores. I could apply this to the results from the lucene query and check to see if any of the results are above a certain threshold. My concern about this is how relevant the cosine similarity would be for the type of data I'm storing, i.e a number of fields with either single or a small number of words compared to calculating the cosine similarity of a comparison of some large text document.

Basically, I'm wondering what is the best way to deduplicate this type of data (or put alternatively, detect similarities with this type of data)?

答:
make the email address as primary key, as always email address is unique. so that redundant data will not be there.

Else if you have address and name of the person, then you can use both to check of duplicates


答:
The assumption built into the question and in some of the answers is that normalization is synonymous good database design.  This is in fact often not the case.  Normalization is one way of achieving a particular set of design goals and a requirement if you are relying heavily on the database to enforce "business rules" about the relationships between data elements.

Normalization gives you a few key benefits:

Minimizes the amount of redundant data.

Maximizes the extent to which the database's built in integrity mechanisms (foreign key constraints, uniqueness constraints) can be leveraged to ensure the integrity of the data.

Reduces the number of columns per row increasing the efficiency of IO in some cases.  Wide rows take longer to retrieve.

That said, there are plenty of valid reasons to denormalize:

Performance, particularly for analytics, can be crippled by normalization. For analysis against relational databases, denormalized dimensional models are the standard approach.

The benefit of enforcing data integrity inside of the database is starting to decline.  As more and more development is focused on the object-oriented middle-tier that is often enforcing business rules, reliance on relational constraints in the database is less important.

As others have mentioned, normalization will complicate the queries required to retrieve relevant data.

It is not clear that normalization is a sign of good design.  In some cases, normalization is an artifact of a time when storage space was at a premium and when much of the responsibility for encoding business rules resided in the database (think about 2-tiered client-server applications with most if not all business logic in stored procedures).  It may well be that many projects veer away from normalization based on good architectural decisions rather than a poor grasp of database design principles.

The article by Jeff Atwood referenced in the comments above provides some good detailed discussion - "Maybe Normalizing Isn't Normal".


答:
There is no silver bullet for de-duplication.  You should concentrate first on normalization (from a pattern sense, not 3NF) and standardization.  This gives you some kind of level playing field from which to start making comparisons.

To achieve this, you need to apply the standardization techniques that work for each type of data.  Standardizing address data is a totally different problem domain than standardizing given names.  Most of these data standardization problem domains are much too complex to attempt to solve yourself.  Consider buying 3rd party software that does postal address validation and standardization and one which does name standardization.

For things like e-mail addresses or phone numbers, you can probably roll your own, since these are relatively straight-forward by comparison.

Once you've got your data components properly standardized, then you can worry about what's better: fuzzy match, Levenshtein distance, or cosine similarity (etc.)

You're best to consider matching like sub-elements rather than trying to take records as a whole.  Then look at how many sub-elements reasonably match.  Two identical names with different email addresses and mailing addresses are a very weak match.  Two nearly identical names with nearly identical mailing addresses with one record missing the email address is probably a pretty strong match.

Title: Deduplication of complex records / Similarity Detection

Body:
I'm working on a project that involves records with fairly large numbers of fields (~15-20) and I'm trying to figure out a good way to implement deduplication. Essentially the records are people along with some additional data. For example, the records are likely to include personal information like first name, last name, postal address, email address, etc. but not all records have the same amount of data.

Currently records are stored in a RDBMS (MySQL) and I want to detect duplicates on insertion  and still have them inserted but flagged as a duplicate. It needs to be fast as I need to provide feedback as to if it is a duplicate or not in real time. The dataset is large (millions of records).

I've considered the following options but I'm not sure which is best/if they are better options available:

Use MySQL's built in fulltext search and use fuzzy searching. Major issue with this is that is seems slow, only the latest version supports fulltext indexes with InnoDB (alternative engine is MyISAM which is not good and critically does not support transactions) and fuzzy searching alone does not seem the best method for similarity detection.

Use simhash or similar. Issue with this is that I'd also like to be able to detect synonyms which I don't see how simhash handles this. For example, address might be: "Some Road" or "Some Rd." and names might be: "Mike" or "Michael"

Index the data using an Apache Lucene derivative (elasticsearch/solr/etc) and perform a query that would likely return numerous results.

In terms of using Apache Lucene I've been reading about similarity detection and using cosine similarity to produce a value from 0 to 1 from the term frequency vectors that lucene stores. I could apply this to the results from the lucene query and check to see if any of the results are above a certain threshold. My concern about this is how relevant the cosine similarity would be for the type of data I'm storing, i.e a number of fields with either single or a small number of words compared to calculating the cosine similarity of a comparison of some large text document.

Basically, I'm wondering what is the best way to deduplicate this type of data (or put alternatively, detect similarities with this type of data)?