问:
How to create single integer index value based on two integers where first is unlimited?
I have table data containing an integer value X ranging from 1.... unknown, and an integer value Y ranging from 1..9

The data need to be presented in order 'X then Y'.

For one visual component I can set multiple index names: X;Y

But for another component I need a one-dimensional integer value as index (sort order).

If X were limited to an upper bound of say 100, the one-dimensional value could simply be X*100 + Y.

If the one-dimensional value could have been a real, it could be X + Y/10.

But if I want to keep X unlimited, is there a way to calculate a single integer 'indexing' value from X and Y?

[Added] Background information:

I have a Gantt/TreeList component where the tasks are ordered on a TaskIndex integer. This does not need to be a real database field, I can make it a calculated field in the underlying client dataset.

My table data is e.g. as follows:

```<code>ID    Baseline   ParentID
 1       0          0     (task)
 5       2          1     (baseline)
 8       1          1     (baseline)
 9       0          0     (task)
12       0          0     (task)
16       1         12     (baseline)
</code>```

Task 1 has two baselines numbered 1 and 2 (IDs 8 and 5)

Task 9 has no baselines

Task 12 has one baseline numbered 1 (ID 16)

Baselines number 1-9 (the Y variable from my question); 0 or null identify the tasks

ID's are unlimited (the X variable)

The user plays with visibility of baselines, e.g. he wants to see all tasks and all baselines labeled 1. This is done by updating a filter on the table.

Right now I constantly have to recalculate TaskIndex after changing the filter (looping through records with a counter). It would be nice if TaskIndex could be calculated on the fly for each record knowing only the ID and Baseline data in the current record (I work in Delphi where a client dataset has an OnCalcFields event handler, that is triggered for each record when necessary).

I have no control over the inner workings of the visual component.

答:
As your least-significant index (Y) is bounded, to get the right sort order, you can just do index = X*10 + Y and order index on its numerical value.

If the overall index is too large to fit into an integer value, you can simulate numerical ordering by padding the shorter numbers with leading zeros until all values in the comparison are the same length.


问:
Looking for monotonically increasing (integer) hash function
I'm looking for a HashFunction(X,Y: Integer): Integer that is monotonically increasing on X, then Y.

So:

HashFunction(x1,y1) > HashFunction(x2,y2) if x1>x2

HashFunction(x,y1) > HashFunction(x,y2) if y1>y2

Does such an animal exist?

Background: This question arises from a comment on

How to create single integer index value based on two integers where first is unlimited?

答:
Without trying to go into proofs, I'd say that such a beast does not exist for generic integers. Your problem would ultimately boil down to finding a hash function H with x>y → H(x) > H(y); with the constraint that the output of H is a finite set, and assuming that x∊ℕ (or in another set S with |S|>|image(H)|), there would necessarily be x₁>x₂∊ℕ with H(x₁)≤H(x₂).

If, however, you have both variables from a finite set, e.g. N-bit integer representations r(x). you could trivially define H(x₁,x₂)=r(x₁)||r(x₂) and satisfy your requirement.

ContentLicense: CC BY-SA 3.0


答:
Are you looking for coordinate hashing? The normal equation is:

hash = y * width + x
(in your case it would probably be x * height + y)

So if your hash size is a signed 32 bits then sqrt(2,147,483,647) would give the width value, in this case 46340. This defines are min as (-46340, -46340). The max would be (46340, 46340).

-46340 * 46340 + -46340 = -2,147,441,940, and minimum signed int in 32 bits is -2,147,483,648
46340 * 46340 + 46340 = 2,147,441,940 and maximum signed int in 32 bits is 2147483648

For large ranges of x and y you can usually just use a 64 bit.

If you know bitwise operators you can just assign low and high bits to x and y. So in a 64-bit number you'll store the x in the top 32 and y in the lower 32. This will generate a hash that follows your rules.

Title: Looking for monotonically increasing (integer) hash function

Body:
I'm looking for a HashFunction(X,Y: Integer): Integer that is monotonically increasing on X, then Y.

So:

HashFunction(x1,y1) > HashFunction(x2,y2) if x1>x2

HashFunction(x,y1) > HashFunction(x,y2) if y1>y2

Does such an animal exist?

Background: This question arises from a comment on

How to create single integer index value based on two integers where first is unlimited?


问:
What's the reason exceptions are heavily used in managed (C# and Java) languages but not in C++?
AFAIK, a lot of C++ projects don't allow exceptions and deny them in coding guidelines. I have a lot of reasons, for example, exception is hard to handle correctly if your binary needs to be compiled by separate and different compilers.

But it doesn't fully convince me, there is a lot of projects which are just using one compiler. Compared to C++, exceptions are heavily used in C# and Java and the reason can only be that exception are not bringing enough benefit.

One point is debugbility in practice. Exception can not get the call stack in C++ code, but in C# and Java you can get the call stack from exception, it is significant and makes debugging easier. No-callstack is not the fault of the exception, it is the language difference, but it impacts the exception usage.

So what's the reason that exceptions are frowned upon in c++ programs?

答:
The google C++ style guidelines say:

We do not use Exceptions.

In that guide, you can find a list of pro's and con's of exceptions in C++, and a discussion of why they decided against using them. I feel these apply to many projects, not just google-originated ones.

Basically, the reason is "compatibility with legacy code".


答:
There are two main reasons why exceptions in C++ are often shunned.

Legacy code

Fear/uncertainty/doubt (FUD)

Exceptions require awareness that, even if you do not throw an exception yourself, any function you call could throw an exception and you should be able to deal with that gracefully (at the very least, without leaking resources).
When exceptions were first introduced in C++, this awareness did not yet exist by many of the C++ developers. If you add to that that a lot of existing C code has been ported to C++ and that the C++ runtime environment did not offer that many aids to help you avoid memory leaks, you get a vast amount of legacy C++ code that is not exception safe.

The initial lack of awareness about how to write exception-safe code also cause a lot of fear, uncertainty and doubt, because exceptions worked so differently from the familiar flow control mechanisms that a lot of the common wisdom for writing robust applications had to be critically re-evaluated.

None of these problems exist for C# and Java because those languages only came along when exceptions were already much better understood, they have garbage collection so there is one major resource you don't have to look after that carefully and they don't have pre-exception legacy code.

Title: What's the reason exceptions are heavily used in managed (C# and Java) languages but not in C++?

Body:
AFAIK, a lot of C++ projects don't allow exceptions and deny them in coding guidelines. I have a lot of reasons, for example, exception is hard to handle correctly if your binary needs to be compiled by separate and different compilers.

But it doesn't fully convince me, there is a lot of projects which are just using one compiler. Compared to C++, exceptions are heavily used in C# and Java and the reason can only be that exception are not bringing enough benefit.

One point is debugbility in practice. Exception can not get the call stack in C++ code, but in C# and Java you can get the call stack from exception, it is significant and makes debugging easier. No-callstack is not the fault of the exception, it is the language difference, but it impacts the exception usage.

So what's the reason that exceptions are frowned upon in c++ programs?


问:
Should my colleagues review each others code from source control system?
So that's my story: one of my colleagues uses to review all the code, hosted to revision system. I'm not speaking about adequate review of changes in parts that he belongs to. He watches the code file to file, line to line. Every new file and every modified. I feel just like being spied on!

My guess is that if code was already hosted to control system, you should trust it as workable at least. My question is, maybe I'm just too paranoiac and practice of reviewing each others code is good?

P.S: We're team of only three developers, and I fear that if there will be more of us, colleague just won't have time to review all the the code we'll write.

答:
This happened in my team once. Unfortunately it resulted in a blame game. People continously waited for others to check-In the code and would always try to find something wrong in it and played the blame game all the time.

I hope you have a more matured audience.

ContentLicense: CC BY-SA 2.5


答:
A few points suggest that either "hash function" isn't the right term for what you want, or that what you want does not exist.

First, a function cannot be strictly increasing unless it is 1-1, and typically by "hash" we mean getting a result that is smaller than the input (usually by many orders of magnitude).  In this context the most you could ask for is a weakly increasing function, e.g retain a high-order byte or something similar.

Second, the linked Question suggests to me that what is wanted is to make the results uniformly distributed (at least roughly) while preserving order (monotonicity).  It's possible to achieve this for a specific data set (sort and enumerate), but defining a master function to do this for arbitrary input data ("generic integers" as Nicos puts it) is impossible.

Third, a monotone function would be useless as a cryptographic hash function because monotone functions are easy to invert (defeating the purpose of making it difficult to generate a message having a given hash).


答:
The reasoning I followed to choose not to use exceptions in C++ code unless no viable alternative was immediately visible was the overhead of managed resources and breaking the normal flow of things.

http://ptgmedia.pearsoncmg.com/images/020163371x/supplements/exception_handling_article.html

The article in the above link is a reproduction of something that was first published in 1994 C++ Report by Tom Cargill.

I would say it is just harder to do Exception handling properly in C++ code and bugs can be very subtle hard to find. As Tom Cargill writes

The really hard part of using exceptions is to write all the intervening code in such a way that an arbitrary exception can propagate from its throw site to its handler, arriving safely and without damaging other parts of the program along the way.


问:
How should I start with Lisp?
I've been programming for years now, working my way through various iterations of Blub (BASIC, Assembler, C, C++, Visual Basic, Java, Ruby in no particular order of "Blub-ness") and I'd like to learn Lisp. However, I have a lot of intertia what with limited time (family, full time job etc) and a comfortable happiness with my current Blub (Java).

So my question is this, given that I'm someone who would really like to learn Lisp, what would be the initial steps to get a good result that demonstrates the superiority of Lisp in web development? Maybe I'm missing the point, but that's how I would initially see the application of my Lisp knowledge.

I'm thinking "use dialect A, use IDE B, follow instructions on page C, question your sanity after monads using counsellor D". I'd just like to know what people here consider to be an optimal set of values for A, B, C and perhaps D. Also some discussion on the relative merit of learning such a powerful language as opposed to, say, becoming a Rails expert.

Just to add some more detail, I'll be developing on MacOS (or a Linux VM) - no Windows based approaches will be necessary, thanks.

Notes for those just browsing by

I'm going to keep this question open for a while so that I can offer feedback on the suggestions after I've been able to explore them. If you happen to be browsing by and feel you have something to add, please do. I would really welcome your feedback.

Interesting links

Assuming you're coming at Lisp from a Java background, this set of links will get you started quickly.

Using Intellij's La Clojure plugin to integrate Lisp (videocast)

Lisp for the Web

Online version of Practical Common Lisp (c/o Frank Shearar)

Land of Lisp a (+ (+ very quirky) game based) way in but makes it all so straightforward

Install Clojure and Sublime 2 on MacOS an excellent getting started guide

Look at the Clojure in Action book. Worked for me.

答: Well,You can hit two birds by reading SICP ;One is learning LISP(i mean scheme is a dialect of Lisp) and other is great insight into how to program.Period!!</p>

Title: How should I start with Lisp?


问:
Is spreading code with refactoring comments a good idea?
I am working on a "spaghetti-code" project, and while I am fixing bugs and implementing new features, I also do some refactoring in order to make the code unit-testable.

The code is often so tightly coupled or complicated that fixing a small bug would result in a lot of classes being rewritten. So I decided to draw a line somewhere in the code where I stop refactoring. To make this clear, I drop some comments in the code explaining the situation, like:

```<code>class RefactoredClass {
    private SingletonClass xyz;

    // I know SingletonClass is a Singleton, so I would not need to pass it here.
    // However, I would like to get rid of it in the future, so it is passed as a
    // parameter here to make this change easier later.
    public RefactoredClass(SingletonClass xyz) {
        this.xyz = xyz;
    }
}
</code>```

Or, another piece of cake:

```<code>// This might be a good candidate to be refactored. The structure is like:
// Version String
//    |
//    +--> ...
//    |
//    +--> ...
//          |
//    ... and so on ...
//
Map map = new HashMap<String, Map<String, Map<String, List<String>>>>();
</code>```

Is this a good idea? What should I keep in mind when doing so?

答:
I would make such considerations /// @todo comments for doxygen or an easy-to-install custom tag for javadoc, so it gets automatically extracted to the todo section of the API docs. Plain comments will be overlooked too easily and eventually get lost in the depths of code.

[Edit] BTW: is this a good idea:

while I am fixing bugs and implementing new features, I also do some refactoring in order to make the code unit-testable

I think (know by experience!), refactoring can be very dangerous, especially when there are still no unit tests. So you'd better restrict your extra work (when fixing bugs etc.) on adding todo comments ...We all know: when ever possible ;)


问:
Is Java's ElementCollection Considered a Bad Practice?
From my understanding, an ElementCollection has no primary key, is embedded with the class, and cannot be queried. This sounds pretty hefty, but it allows me the comfort of writing an enum class which also helps with internationalization (using the enum key for lookup in my ResourceBundle).

Example: We have a user on a media site and he can choose in which format he downloads the files

```<code>@Entity
@Table(name = "user")
public class User {
    /* snip other fields */

    @Enumerated
    @ElementCollection(
            targetClass = DownloadFilePreference.class,
            fetch = FetchType.EAGER
    )
    @CollectionTable(name = "download_file_preference",
        joinColumns = @JoinColumn(name = "user_id")
    )
    @Column(name = "name")
    private Set<DownloadFilePreference> downloadFilePreferences = new HashSet<>();
}

public enum DownloadFilePreference {
    MP3, WAV, AIF, OGG;
}
</code>```

This seems pretty great to me, and I suppose it comes down to "it depends on your use case", but I also know that I'm quite frankly only an initiate when it comes to Database design. Some best practices suggest to never have a class without a primary key- even in this case? It also doesn't seem very extensible- should I use this and gamble on the chance I will never need to query on the FilePreferences?

答:
I had a simple case with I first solved with an enum that was stored inside a ElementCollection. It would suffice in your situation because you would most likely just query the whole collection and then you can do your operations on the Set. If you want on the other hand to have your choices to be more extensible you should create another Entity for DownloadFilePreferences.

ome best practices suggest to never have a class without a primary key

Just because you didn't declare a primary it doesn't mean that your JPA implementation doesn't create an id on the mapping table.


答: If Integer is bounded, trivially impossible because of the pigeonhole principle. If it isn't, trivially impossible because Hash(2,0) - Hash(1,0) is finite, and yet an infinite number of hash values Hash(1,y) must lie in between.


问:
Is it reasonable to null guard every single dereferenced pointer?
At a new job, I've been getting flagged in code reviews for code like this:

```<code>PowerManager::PowerManager(IMsgSender* msgSender)
  : msgSender_(msgSender) { }

void PowerManager::SignalShutdown()
{
    msgSender_->sendMsg("shutdown()");
}
</code>```

I'm told that last method should read:

```<code>void PowerManager::SignalShutdown()
{
    if (msgSender_) {
        msgSender_->sendMsg("shutdown()");
    }
}
</code>```

i.e., I must put a NULL guard around the msgSender_ variable, even though it is a private data member. It's difficult for me to restrain myself from using expletives to describe how I feel about this piece of 'wisdom'. When I ask for an explanation, I get a litany of horror stories about how some junior programmer, some-year, got confused about how a class was supposed to work and accidentally deleted a member he shouldn't have (and set it to NULL afterwards, apparently), and things blew up in the field right after a product release, and we've "learned the hard way, trust us" that it's better to just NULL check everything.

To me, this feels like cargo cult programming, plain and simple. A few well-meaning colleagues are earnestly trying to help me 'get it' and see how this will help me write more robust code, but... I can't help feeling like they're the ones who don't get it.

Is it reasonable for a coding standard to require that every single pointer dereferenced in a function be checked for NULL first—even private data members? (Note: To give some context, we make a consumer electronics device, not an air traffic control system or some other 'failure-equals-people-die' product.)

EDIT: In the above example, the msgSender_ collaborator isn't optional. If it's ever NULL, it indicates a bug. The only reason it is passed into the constructor is so PowerManager can be tested with a mock IMsgSender subclass.

SUMMARY: There were some really great answers to this question, thanks everyone. I accepted the one from @aaronps chiefly due to its brevity. There seems to be fairly broad general agreement that:

Mandating NULL guards for every single dereferenced pointer is overkill, but

You can side-step the whole debate by using a reference instead (if possible) or a const pointer, and

assert statements are a more enlightened alternative to NULL guards for verifying that a function's preconditions are met.

答:
I agree with trapping the null in the constructor.  Further, if the member is declared in the header as:

```<code>IMsgSender* const msgSender_;
</code>```

Then the pointer cannot be changed after initialisation, so if it was fine at construction it will be fine for the lifetime of the object containing it.  (The object pointed to will not be const.)

Body:
I agree with trapping the null in the constructor.  Further, if the member is declared in the header as:

IMsgSender* const msgSender_;

Then the pointer cannot be changed after initialisation, so if it was fine at construction it will be fine for the lifetime of the object containing it.  (The object pointed to will not be const.)


答:
This is quite standard practice in industry. The companies I have worked at have very tight code review guidelines. One even wouldn't let you commit unless code had been reviewed.

Don't take offence, or feel watched. Think of it as a safety net and a learning experience.

Title: Should my colleagues review each others code from source control system?

Body:
So that's my story: one of my colleagues uses to review all the code, hosted to revision system. I'm not speaking about adequate review of changes in parts that he belongs to. He watches the code file to file, line to line. Every new file and every modified. I feel just like being spied on!

My guess is that if code was already hosted to control system, you should trust it as workable at least. My question is, maybe I'm just too paranoiac and practice of reviewing each others code is good?

P.S: We're team of only three developers, and I fear that if there will be more of us, colleague just won't have time to review all the the code we'll write.


问:
What is an effective git process for managing our central code library?
Quick background: we're a small web agency (3-6 developers at any one time) developing small to medium sized Symfony 1.4 sites. We've used git for a year now, but have previously used Subversion.

For the past 6 months we've put a lot of development time into a central Symfony plugin that powers our custom CMS.  This plugin includes a number of features, helpers, base classes etc. that we use to build custom functionality. This plugin is stored in git, but branches wildly as the plugin is used in various products and is pulled from/pushed to constantly.  The repository is usually used as a submodule within a major project.

The problems we're starting to see now are a large number of Merge conflicts and backwards incompatible changes brought into the repository by developers adding custom functionality in the context of their own project.

I've read Vincent Driessen's excellent git branching model and successfully used it for projects in the past, but it doesn't seem to quite apply well to our particular situation; we have a number of projects concurrently using the same core plugin while developing new features for it.

What we need is a strategy that provides the following:

A methodology for developing major features within the code repository.

A way of migrating those features into other projects.

A way of versioning the core repository, and of tracking which version each major project uses.

A plan for migrating bug fixes back to older versions.

A cleaner history that's easier to see where changes have come from.

Any suggestions or discussion would be greatly appreciated.

答:
I use branches for short-lived code changes - usually days to a week or two.

Branches should be used purely for managing new features/fixes/upgrades/chores, they should not be a mechanism for managing product features themselves.

One option is multiple master for each product but that doesn't address the need to keep common code in sync.

Seem like you need to develop software / API's / configurations that let you configure these things through switches without changing code.

The main 'long-lived' branches that I expect to see are versions for non-application related differences such as ruby and/or rails versions.


问:
Managing Personal Projects As Solo Developer - Getting out of depth and failing projects
I need some advice on project management.

I start a project, and often times it will a large project for a solo developer. Usually its a web project. I handle everything from the UI, to the JS, PHP, server management etc. Half way in I feel out of my depth. I lose where I am, so I spend a couple of days away from the project to avoid the stress and before you know it, it becomes another unfinished project.

I try to use frameworks and code libraries to make my developments easier on myself. Sometimes I will complete a project so it "works" and then go back and handle errors, design the UI properly and stuff. But without fail I will always end up out of my depth.

I've though about outsourcing tasks such as the UI, and the behaviour, and focusing just on the PHP - which I feel is my strong point. But then pride kicks in, and I don't feel at one with a project I haven't completed myself. Does this make sense?

I am sure there are many others who have felt like this either at home, or at work, and I would love some advice on managing my projects better.

答:
Just from pure product perspective, I would ask what is the rationale of creating such projects? What is the end goal you try to reach?

If you are doing it for hobby, then I would argue that probably you don't have too good motivation and reason for doing those projects (getting bored etc.). And if there's no clear understanding why one does some things, it's very hard to expect this to be engaging for a long period of time.

If you are doing it for money, then I personally would seriously recommend taking a look into your business model and would argue that there are inefficiencies that have to be overcome. Every business or product needs a set of different people to be successful, there are good one man armies, but very seldom.

I feel that the problem here is more general and can not be solved with just adapting to new tools or development methodologies.


问:
Could someone help me understand SQL TDE Database encryption?
I don't quite follow how it works. According to the MSDN Article there is a big hierarchy of keys protecting other keys and passwords. At some point the database is encrypted. You query the database which is encrypted, and it works seamlessly.

If you're able to simply connect to the database as normal and not have to worry about any of the encryption from a developer point of view, how exactly is it secure? Surely anyone can simply connect and do select * from x and the data is revealed.

Sorry my question is a bit scattered, I am just very confused by the article.

答:
You are right, the point of database encryption is not to protect data from the users of the data base - that is the task of role-based access and privilege levels.

Encryption protects you against someone physically stealing the server from its rack, ripping out the hard disk and then reading the confidential data from the file system. It's a bit more complicated than that - obviously you can't just keep the decryption key lying around, in particular not on the same disk where you store the encrypted DB, otherwise the thief could just decrypt the data - but done right, it can add a functional layer of data security, and security is all about defense in depth.


答:
Somewhat on a tangent, but I find that having personal projects of the same type that I do at work (eg. Rails developer by day, personal Rails projects at night) can lead to much increased risk of burnout, fatigue, and just total unwillingness to work on it.

Focusing on the parts that are different (eg. the UI, or the server stuff) can help with the motivation I think, rather than doing 'just more of the same'.

Title: Managing Personal Projects As Solo Developer - Getting out of depth and failing projects

Body:
I need some advice on project management.

I start a project, and often times it will a large project for a solo developer. Usually its a web project. I handle everything from the UI, to the JS, PHP, server management etc. Half way in I feel out of my depth. I lose where I am, so I spend a couple of days away from the project to avoid the stress and before you know it, it becomes another unfinished project.

I try to use frameworks and code libraries to make my developments easier on myself. Sometimes I will complete a project so it "works" and then go back and handle errors, design the UI properly and stuff. But without fail I will always end up out of my depth.

I've though about outsourcing tasks such as the UI, and the behaviour, and focusing just on the PHP - which I feel is my strong point. But then pride kicks in, and I don't feel at one with a project I haven't completed myself. Does this make sense?

I am sure there are many others who have felt like this either at home, or at work, and I would love some advice on managing my projects better.


问:
Improving the performance of a db import process
I have a program in Microsoft Access that processes text and also inserts data in MySQL database. This operation takes 30 mins or less to finished.

I translated it into VB.NET and it takes 2 hours to finish.

The program goes like this:

A text file contains individual swipe from a corresponding person,
it contains their id, time and date of swipe in the machine, and an
indicator if it is a time-in or a time-out.

I process this text, segregate the information and insert the time-in and time-out per row.

I also check if there are double occurrences in the database.

After checking, I simply merge the time-in and time-out of the corresponding person into one row only.

This process takes 2 hours to finished in VB.NET considering I have a table to compare which contains 600,000+ rows.

Now, I read in the internet that python is best in text processing, i already have a test but i doubt in database operation.

What do you think is the best programming language for this kind of problem?

How can I speed up the process? My first idea was using python instead of VB.NET, but since people here telling me here on PSE that this most probably won't help I am searching for different solutions.

Here is a link to the test i've done.

答:
I agree with Doc Brown that you should first try to find the bottleneck in your application, otherwise we are all only hypothesing and there's no real use to it. Anything could be suspect.

The first thing you could is to measure the duration of:

the entire process, consisting in reading a line in the file, checking for duplicates, inserting a row

the entire process again, but when you found a duplicate and as such, when you are not inserting a line

Also, please take measures of individual operations. How long does it take to read from the file on average ? How long does it take to check for duplicates ? How long does it take to insert a record ? Maybe the duration keeps growing along the size of the table ?

The only way to answer these questions is get the numbers.

Finally, I think there are several possible optimizations depending on the location of the bottleneck. For example, you could:

read more than 1 line at once, say 10 for example, and dispatch the processing of these lines to 10 separate threads.

more generally, separate the process of reading the file from writing to the database by using the publish/subscribe pattern.

optimize indexes in your database table.

instead of checking every time if there is no duplicate, maybe store the entire table in a map in memory before you start processing the file. This could take a few seconds (and some memory size) but you avoid one IO round trip per line.

To conclude, I think you are right about not changing your language, the langue is not the problem.


问:
How do you cope with ugly code that you wrote?
So your client asks you to write some code, so you do. He then changes the specs on you, as expected, and you diligently implement his new features like a good little lad. Except... the new features kind of conflict with the old features, so now your code is a mess. You really want to go back and fix it, but he keeps requesting new things and every time you finish cleaning something, it winds up a mess again.

What do you do? Stop being an OCD maniac and just accept that your code is going to wind up a mess no matter what you do, and just keep tacking on features to this monstrosity? Save the cleaning for version 2?

答:
With all the comments about over-estimating I think that there's a modest amount of point (well opportunity) being missed.

Its not about estimating the time taken make the change (just) and then adding some, its about estimating the time required to modify the code (refactor!) to bring it to a point where the change may be safely made and to then make the change (probably somewhat munged together). Ok, this amounts to the same thing... but there's no question of fudging or stretching or over-estimating, its simply a matter of saying that in order to do this I first need to do that and this is how long it will take in total. Key here is that you work on those parts of the system that the change is dependent on and no more - if there's horrid code elsewhere... tough, catch it when you're there.

To come back to the original question a bit - after a lot of years its come down to this for me, when you implement something unless you know (not believe, not expect (suspect?), not think but know) that additional stuff is also required then you should do what you need to implement that requirement and no more in as tidy and elegant a fashion as you can.

When you come to implement the next thing - some while later - you take the steps required to bring the codebase (and the database and whatever) to the state needed to implement that functionality as in as tidy and elegant a fashion as you can. This refactoring is where you deal with the mess that arises naturally as a project evolves - and hopefully avoid creating more mess (or at least keep the level consistent).

One of the discussion areas here is "Technical Debt" - its like an overdraft, you have to pay it back and the longer you leave it the more interest (in this case time required to correct) you will accrue - which gives you a good argument for spending some of your time minimising the technical debt.

This is also where unit testing and other automated testing starts to come in (if I could do as well as I say I'm fairly certain I would be a happier person!) combined with a proper build server (that can run at least some of your tests). Combined with those - but of value in an of themselves - are patterns like dependency injection and inversion of control (never quite sure how much the "same" those two are) because they make it easier to change the plumbing and hence deal with changes in isolation.

Lastly - remember, if it ain't broke don't fix it. Tidying your code purely for the sake of tidying it may be satisfying but its also an opportunity to introduce errors so whilst it can be painful if you don't need to change it and are not building upon it it may be better to leave some lumps alone - the opportunity to fix or replace will roll around eventually!


问:
Licensing a collaborative research project
I am involved with an international research project which involves many different universities, national labs, and companies. The project is developed by national grants and in-kind support.

One task in the project is to develop code to streamline workflow in our domain (energy simulation) by scripting common pre- and post-processing tasks for different tools. We want this code to be freely distributable to the simulation community.

How can we ensure that this effort is digestible by the legal departments of these different parties such that the people involved can freely code?

答:
First, you need to check and see if there are any restrictions placed upon you by your funding sources.  In the United States, research projects backed by federal funds are generally required to be licensed with a permissive license.  Other agencies in other countries may have different requirements.

Second, take a look at the licenses listed by the Open Source Initiative and you'll find a number of popular licenses that should meet your needs.  The OSI FAQ on licenses should address some of your questions as well.

Third, consider having an entity created that all contributors can assign the copyright to.  Remember that copyright is not the same as a license.  Copyright is ownership of the code, licensing is how others can use the code.  By having contributors assign their copyright to the project you'll be in a better position to handle any future licensing changes that may be required.

Finally, if you are concerned about the legal department review, stick with the more popular and benign licenses.  Copyleft within the GPL attracts a lot of negative attention (some warranted, some not) from legal departments.  BSD, MIT, eclipse, and Apache are generally benign but with varying degrees of restrictiveness.  That's not an exhaustive list, but the well-known licenses are more likely to have already been reviewed by legal departments and will be more likely to receive approval.


问:
Why is there a "new" in Go?
I'm still puzzled as why we have new in Go.

When you want to instantiate a struct, you do

```<code>t := Thing{}
</code>```

and you can get a pointer to a new instance by doing

```<code>t := &Thing{}
</code>```

But there's also this possibility :

```<code>t := new(Thing)
</code>```

This last one seems a little alien to the rest of the language. &Thing{} is as clear and concise as new(Thing) and it uses only constructs you often use elsewhere. It's also more extensible as you might change it to &Thing{3} or  &Thing{Feets:7}.

In my opinion, having a supplementary keyword1 is costly, it makes the language more complex and adds to what you must know. And it might mask to newcomers what's behind instantiating a struct.

It also makes one more reserved word.

So what's the reasoning behind new ? Is it sometimes useful ? Should we use it ?

1 : Yes, I know it's not a keyword at the grammar level, you can shadow it, but that doesn't change the fact it's, for the reasonable developer, a reserved word.

答:
The best way to ask is probably to the people working on it; exactly what I did!

Tl;dr: it was originally there before make and &{}, and it's still the function to use in some situations.

Basically, here are the most important parts quoted:

So what's the reasoning behind new ? Is it something useful ? Should we
    use it ?

You cannot do this without new

```<code>v := new(int)
*v++
fmt.Println(*v)
</code>```

new isn't a headline feature of Go, you won't find it used often, but
  when you need it, it is there.

Cheers

Dave

After another answer showing this kind of solution:

```<code>vv := 0
v := &vv
*v++
fmt.Println(*v)
</code>```

I asked for further clarification:

So basically, Dave's point doesn't really stand?

There are places where it's inconvenient to sneak in a new
  variable just to take its address.

new(T) has an immediately straightforward meaning, rather than
  being a multi-step idiom.

Dave's point only falls if mere technical possibility (of doing without
  new) is compelling on its own.

Wasn't this discussed because it was just obvious that Go should have it because almost every language has it?

The "shall we keep new?" discussion pops up from time to time.
  Since we can't take it out until Go 2, if I understand the Promise correctly,
  there doesn't seem to be much to be had from going round the
  loop again; by the time Go 2 is thinkaboutable, we might have some
  different and better ideas ...

Chris

It's also there mostly for historical reasons:

you need to consider the history of the project. i think new is introduced
    first before there is make.

That is true.  In fact we struggled for a while before coming up with
  the idea of make.  If you look at the repository logs you can see that
  make only shows up in January 2009, revision 9a924177598f.

The new builtin function also preceded the idea of &{} for taking the
  address of a composite literal (and that syntax is in some sense
  wrong; it probably ought to be (*T){fields of T} but there wasn't
  enough reason to change it).

The new function is not strictly necessary but code does seem to use
  it in practice.  It's hard to get rid of it at this point.

Ian

ContentLicense: CC BY-SA 4.0

Title: Why is there a "new" in Go?

Body:
The best way to ask is probably to the people working on it; exactly what I did!

Tl;dr: it was originally there before make and &{}, and it's still the function to use in some situations.

Basically, here are the most important parts quoted:

So what's the reasoning behind new ? Is it something useful ? Should we
    use it ?

You cannot do this without new

v := new(int)
*v++
fmt.Println(*v)

new isn't a headline feature of Go, you won't find it used often, but
  when you need it, it is there.

Cheers

Dave

After another answer showing this kind of solution:

vv := 0
v := &vv
*v++
fmt.Println(*v)

I asked for further clarification:

So basically, Dave's point doesn't really stand?

There are places where it's inconvenient to sneak in a new
  variable just to take its address.

new(T) has an immediately straightforward meaning, rather than
  being a multi-step idiom.

Dave's point only falls if mere technical possibility (of doing without
  new) is compelling on its own.

Wasn't this discussed because it was just obvious that Go should have it because almost every language has it?

The "shall we keep new?" discussion pops up from time to time.
  Since we can't take it out until Go 2, if I understand the Promise correctly,
  there doesn't seem to be much to be had from going round the
  loop again; by the time Go 2 is thinkaboutable, we might have some
  different and better ideas ...

Chris

It's also there mostly for historical reasons:

you need to consider the history of the project. i think new is introduced
    first before there is make.

That is true.  In fact we struggled for a while before coming up with
  the idea of make.  If you look at the repository logs you can see that
  make only shows up in January 2009, revision 9a924177598f.

The new builtin function also preceded the idea of &{} for taking the
  address of a composite literal (and that syntax is in some sense
  wrong; it probably ought to be (*T){fields of T} but there wasn't
  enough reason to change it).

The new function is not strictly necessary but code does seem to use
  it in practice.  It's hard to get rid of it at this point.

Ian


问:
Code structure for multiple applications with a common core
I want to create two applications that will have a lot of common functionality. Basically, one system is a more advanced version of the other system. Let's call them Simple and Advanced. The Advanced system will add to, extend, alter and sometimes replace the functionality of the Simple system. For instance, the Advanced system will add new classes, add properties and methods to existing Simple classes, change the behavior of classes, etc.

Initially I was thinking that the Advanced classes simply inherited from the Simple classes but I can see the functionality diverging quite significantly as development progresses, even while maintaining a core base functionality. For instance, the Simple system might have a Project class with a Sponsor property whereas the Advanced system has a list of Project.Sponsors. It seems poor practice to inherit from a class and then hide, alter or throw away significant parts of its features.

An alternative is just to run two separate code bases and copy the common code between them but that seems inefficient, archaic and fraught with peril. Surely we have moved beyond the days of "copy-and-paste inheritance".

Another way to structure it would be to use partial classes and have three projects: Core which has the common functionality, Simple which extends the Core partial classes for the simple system, and Advanced which also extends the Core partial classes for the advanced system. Plus having three test projects as well for each system. This seems like a cleaner approach.

What would be the best way to structure the solution/projects/code to create two versions of a similar system? Let's say I later want to create a third system called Extreme, largely based on the Advanced system. Do I then create an AdvancedCore project which both Advanced and Extreme extend using partial classes? Is there a better way to do this?

If it matters, this is likely to be a C#/MVC system but I'd be happy to do this in any language/framework that is suitable.

答:
The best approach here is to have the Model of the MVC triad in a separate, common project. This Model should provide all the model features that are needed by the various versions. But the model should not implement the restrictions that differentiate the Simple version from the Advanced version.
To take your example, if the Advanced version supports multiple Sponsors for a project, and the Simple version only one Sponsor, then the model should cater for a list of Sponsors, even though this is less efficient for the Simple version.

For each version, you create a separate project containing the View and Controller parts of the MVC triad. This is also where you put the version-specific restrictions in, like having only a single Sponsor for a Project in the Simple version.


问:
How do I manage the technical debate over WCF vs. Web API?
I'm managing a team of like 15 developers now, and we are stuck at a point on choosing the technology, where the team is broken into two completely opposite teams, debating over usage of WCF vs. Web API.

Team A which supports usage of Web API, brings forward these reasons:

Web API is just the modern way of writing services (Wikipedia)

WCF is an overhead for HTTP. It's a solution for TCP, and Net Pipes, and other protocols

WCF models are not POCO, because of [DataContract] & [DataMember] and those attributes

SOAP is not as readable and handy as JSON

SOAP is an overhead for network compared to JSON (transport over HTTP)

No method overloading

Team B which supports the usage of WCF, says:

WCF supports multiple protocols (via configuration)

WCF supports distributed transactions

Many good examples and success stories exist for WCF (while Web API is still young)

Duplex is excellent for two-way communication

This debate is continuing, and I don't know what to do now. Personally, I think that we should use a tool only for its right place of usage. In other words, we'd better use Web API, if we want to expose a service over HTTP, but use WCF when it comes to TCP and Duplex.

By searching the Internet, we can't get to a solid result. Many posts exist for supporting WCF, but on the contrary we also find people complaint about it. I know that the nature of this question might sound arguable, but we need some good hints to decide. We're stuck at a point where choosing a technology by chance might make us regret it later. We want to choose with open eyes.

Our usage would be mostly for web, and we would expose our services over HTTP. In some cases (say 5 to 10 percent) we might need distributed transactions though.

What should I do now? How do I manage this debate in a constructive way?

答:
Hence I'm facing the same choice now, I asked myself what is the subset of features from WCF our team is using at the moment.
Do we use different protocols? No. Do we use transaction support? No (although we use custom eventual consistency mechanisms). Do we use duplex? No.

Why we would like to use Web API? Easier frontend integration (removes additional service layer existing now), SignalR for pushing reply to clients, caching for GETs.

May be, one could find other reasons :) As well, reasons to stay with WCF.


问:
How can I use JIRA for project management with Green Hopper
I am thinking of using JIRA + GreenHopper for my project management.

I have seen that Green Hopper is for making User stories, sprints.

I am not able to find how do i need to add tasks, or how to break user stories in to sub stories.

Do I first need to  create project in JIRA and then use Green Hopper or I can use use Green Hopper standalone for project management?

I am thinking of JIRA as issue tracker not project management.

答:
Greenhopper is a pure agile process tool, with the main focus on planning for the sprints, ordering the backlog and tracking the work using the computerized agile boards. We've been trying to use it as a project tracking tool, but it's not really suitable for that.

Also we've been using Jira plugin called Structure to do the work which is more or less similar to the project management - it allows to create overarching items, subitems, tracks process and helps to do a lot of things that are relevant for the status communication part. On splitting up the user stories and creating subtasks etc. I would assume that your problem is creating a view how they are linked together, this is where Structure does help, as it allows the n depth tree structure to be created.

And after spending almost 4 years trying to figure out ways how to make things work, so that the needs of a project manager are satisfied and also the teams and it doesn't deviate too far away from the scrum - conclusion is that neither of them are perfect. Structure messes up the planning process as you have items in a tree and they don't easily correspond to sprints. And Greenhopper does not really let you do meaningful project management, as it does not show the relation between issues.

Atlassian, if I'm not mistaken, has recently launched a really neat tool, called Rapid Boards, which actually combines aspects of these quite nicely, so this is one of the things I recommend to take a look into.

P.S. I am more and more also inclining towards the opinion that post-it (tm) based agile board is really the most suitable thing for planning and paper and pencil are one of the most important tools for a project manager :).


答:
The use of a pointer instead of a reference would tell me that msgSender is only an option and here the null check would be right. The code snippet is too slow as to decide that. Maybe there are other elements in PowerManager that are valuable (or testable)...

When choosing between pointer and reference, I thoroughly weigh both options. If I have to use a pointer for a member (even for private members), I have to accept the if (x) procedure each time I dereferencing it.

Title: Is it reasonable to null guard every single dereferenced pointer?

Body:
At a new job, I've been getting flagged in code reviews for code like this:

```<code>PowerManager::PowerManager(IMsgSender* msgSender)
  : msgSender_(msgSender) { }

void PowerManager::SignalShutdown()
{
    msgSender_->sendMsg("shutdown()");
}
</code>```

I'm told that last method should read:

```<code>void PowerManager::SignalShutdown()
{
    if (msgSender_) {
        msgSender_->sendMsg("shutdown()");
    }
}
</code>```

i.e., I must put a NULL guard around the msgSender_ variable, even though it is a private data member. It's difficult for me to restrain myself from using expletives to describe how I feel about this piece of 'wisdom'. When I ask for an explanation, I get a litany of horror stories about how some junior programmer, some-year, got confused about how a class was supposed to work and accidentally deleted a member he shouldn't have (and set it to NULL afterwards, apparently), and things blew up in the field right after a product release, and we've "learned the hard way, trust us" that it's better to just NULL check everything.

To me, this feels like cargo cult programming, plain and simple. A few well-meaning colleagues are earnestly trying to help me 'get it' and see how this will help me write more robust code, but... I can't help feeling like they're the ones who don't get it.

Is it reasonable for a coding standard to require that every single pointer dereferenced in a function be checked for NULL first—even private data members? (Note: To give some context, we make a consumer electronics device, not an air traffic control system or some other 'failure-equals-people-die' product.)

EDIT: In the above example, the msgSender_ collaborator isn't optional. If it's ever NULL, it indicates a bug. The only reason it is passed into the constructor is so PowerManager can be tested with a mock IMsgSender subclass.

SUMMARY: There were some really great answers to this question, thanks everyone. I accepted the one from @aaronps chiefly due to its brevity. There seems to be fairly broad general agreement that:

Mandating NULL guards for every single dereferenced pointer is overkill, but

You can side-step the whole debate by using a reference instead (if possible) or a const pointer, and

assert statements are a more enlightened alternative to NULL guards for verifying that a function's preconditions are met.


问:
Effective and simple matching for 2 unequal small-scale point sets
I need to match two sets of 3D points, however the number of points in each set can be different. It seems that most algorithms are designed to align images and trimmed to work with hundreds of thousands of points. My case are 50 to 150 points in each of the two sets.

So far I have acquainted myself with Iterative Closest Point and Procrustes Matching algorithms. Implementing Procrustes algorithms seems like a total overkill for this small quantity. ICP has many implementations, but I haven't found any readily implemented version accounting for the so-called "outliers" - points without a matching pair.

Besides the implementation expense, algorithms like Fractional and Sparse ICP use some statistics information to cancel points that are considered outliers. For series with 50 to 150 points statistic measures are often biased or statistic significance criteria are not met.

I know of Assignment Problem in linear optimization, but it is not suitable for cases with unequal sets of points.

Are there other, small-scale algorithms that solve the problem of matching 2 point sets? I am looking for algorithm names, scientific papers or C++ implementations. I need some hints to know where to start my search.

答:
In the end I had to resort to a naive algorithm.

Choose a random point. Find the closest point from the second set inside a given threshold range. Find the closest point from the first set which corresponds to the point in the second set. If the two are the same, store them as pair and eliminate them from list. Continue until no more candidate points can be found.

This is a simple and a straight-forward solution approach, since unfortunately I could not persuade my company to spend more time on implementing a scientific algorithm from scratch.


问:
How to handle editing a large file for a non-technical user
I have a client who is given a tab delimited .txt file containing hundreds of thousands of rows.

I have a user story as follows:

As a user I want to take the text file and add a new value at the end of each line which contains the concatenated value of two of the columns.

for example if the file read

```<code>text_one    text_two
</code>```

I need to output the following (preferably to a .txt file)

```<code>text_one    text_two    text_onetext_two
</code>```

My first approach was to ask the vendor supplying the file to do the concatenation before providing the file, the easiest way to solve a problem is to eliminate it right? however they are very uncooperative and have point blank refused.

I've looked at building a simple javascript application that does this client side so a non-technical user could select the file using a file selector. This approach has a few problems

The file could be over a GB in size and so can't be loaded straight into memory, I've tried and the browser crashes

There is no means to write a file in javascript so I'd need to output the content to the screen and have the user save it (somehow)

I was thinking if I could get around the filesize limitations I could just output the edited content to the page and have the user save the page as a .txt file, however I think there is a better way than using javascript that will still accommodate the users lack of technical know-how.

Please consider this question to be stack agnostic, but bear in mind that a nice little shell script or python script would be deemed unsuitable for a non technical user unless there is a way of "packaging" it nicely for a non-technical user.

Updates

The file is too large to open in excel.
The process needs to be run weekly, but it doesn't require scheduling or automation...(yet)

答:
The code bit is trivial. The best way to do the process is to put something between the vendor and the user.

My preference with these things is to get the vendor to transfer the file via FTP (which I hope they are already doing, given it's size). Write your code to grab the file, process it, and put it where the user expects to find it, then set it to run as appropriate (every 5 minutes, every day or whatever).

This is a very common problem.


答:
Had the exact same problem:

If I'd put my project away for a few days, I'd come back to it and have a real hard time picking it back up. I'd had to re-familiarize myself with it, try to re-understand where my thought train was going and sometimes wonder why did I ever decide to code it the way I did so far. More often than not this would simply make me abandon the project or restart it from scratch. All this became a real problem once I realized that the situation forced me to:

only start small, simple projects. Starting a big project would just bring me past the confusion threshold where I could no longer resume it after leaving it alone for a few days

if I really wanted to do something more involved and complex I'd have to do it all in "one sitting", namely work on it every day until it was done with no breaks

This was just stressful. I decided that before I'd go on making solo personal projects, I'd have to try to address this issue.

I started complex projects on purpose, just to see exactly why I was having a hard time resuming them after a few days' break. I'd start implementing an algorithm I wasn't familiar with until then (like A* or Mini-max, etc), stop coding in the middle of it all, and try to finish it a few days later.

Here's some of my conclusions on this:

monolithic anything is bad. Make small concise functions/methods, classes, packages, modules. Don't ever, not once, cut corners on this, thinking "oh come on, I'll just add the draw.sprite code to the updateGameState() function this time only, how bad can it be" or that "I'll just add this calculateArea function right in the SelectButtonWidget class, just this once, it'll be so much faster than making a separate class and instance for it". This makes code hard to follow, and thus hard to resume after a few days break, when you no longer remember all the "mental aliases" you've just done like "all drawing code is in the drawables package except the one for drawing a sprite which is in the updateState() function of the X class of the Y package".

unit tests make the code a whole lot more readable than comments Did you, out of desperation, add "pages and pages" of comments to your code in the hope that it will be more readable next week when you resume your project after returning from your vacation? I know I did. Never helped. Not only did it not make the code more readable (some times it made it less, as it didn't help me re-understand what my code was doing while at the same time clutter my code), but it was taking A LOT of time to add comments everywhere. Also this lead to such conundrums as "do I comment at the class level, method/function level, for each line of code, only for the important ones, etc etc". Investing all that comment writing time into unit tests was a lot better. Unit tests check for something, and you can readily tell what they're checking for and hence what the code is supposed to do. Also, the "where to place them comments" conundrum was gone. Unit test everything. Start with whatever you think is most important but make a goal of unit testing anything.

unit tests just rock Not only is your code more readable and easier to (re)understand, you'll not even need to to (re)understand it all when you come back to it, but only those areas you want to work on presently. Just peek at the unit tests of the parts of the code you wanna modify right now to see what they're exactly supposed to do. Then modify them, add more unit tests if necessary, and just re-run the whole unit test battery to make sure you haven't broken anything. If they all pass, you instantly know all is good with your code, without having to go through all of it by hand.

never optimize until the very end In the grand scheme of it all, this point is debatable. Will my code run faster if I delay optimizations as much as possible, or of I do them incrementally while I implement things? I honestly don't know. But strictly from the point of view of code readability and how easy it is to resume a project after putting it aside for a while, I'm pretty sure that the practice of delaying optimization until the very end is the best way to go. What I mean is: as long as your implementation is not close to being done, focus on having nice, readable, modularized code. Once the functionality is all in, debugged, unit and functional tested, you can get to the task of optimizing the code. You'll have a nice version of your code which you are confident that it works ok, and is clear and easy to read, and to which you can always return if the refactoring for the purpose of optimization gives you trouble.


答:
Yes you can still query select * from x

I have little experience with Transparent Data Encryption, but as the article itself states:

TDE works at the file level, which is similar to two Windows®
  features: the Encrypting File System (EFS) and BitLocker™ Drive
  Encryption, the new volume-level encryption introduced in Windows
  Vista®

It notes that:

TDE does not replace cell-level encryption, EFS, or BitLocker.

It further states:

TDE operates at the I/O level through the buffer pool. Thus, any data
  that is written into the database file (*.mdf) is encrypted.

To give you an idea of the extensiveness:

Encrypting at the I/O level also allows the snapshots and backups to
  be encrypted, thus all snapshots and backups created by the database
  will be encrypted by TDE.

As for cell-encryption:

Cell-level encryption is implemented as a series of built-ins and a
  key management hierarchy.

It basically invokes an additional en/decryption function for each cell-access, and requires schema modification of the cell's to 'varbinary' and re-casting to the appropriate data type upon reading.

See also:

4 Ways to Encrypt Data in Microsoft SQL Server

What are impacts of TDE enabling on database performance?

Title: Could someone help me understand SQL TDE Database encryption?

Body:
I don't quite follow how it works. According to the MSDN Article there is a big hierarchy of keys protecting other keys and passwords. At some point the database is encrypted. You query the database which is encrypted, and it works seamlessly.

If you're able to simply connect to the database as normal and not have to worry about any of the encryption from a developer point of view, how exactly is it secure? Surely anyone can simply connect and do select * from x and the data is revealed.

Sorry my question is a bit scattered, I am just very confused by the article.


答:
For me the solution is not to start big projects. Instead I make a small project and evolve it.

It may not be publishable after first step, but on every step it has some known state, list of future improvements, may be some plans.

Motivation may affect how much time I spend, but the code and other things (plans, bugs, features) should be maintained in a state that allows to easily continue after at least a month delay (year delay will likely require some refreshing).


问:
Design pattern: static function call with input/output containers?
I work for a company in software research department. We use algorithms from our real software and wrap them so that we can use them for prototyping. Every time an algorithm interface changes, we need to adapt our wrappers respectively.

Recently all algorithms have been refactored in such a manner that instead of accepting many different inputs and returning outputs via referenced parameters, they now accept one input data container and one output data container (the latter is passed by reference). Algorithm interface is limited to a static function call like that:

```<code>class MyAlgorithm{
static bool calculate(MyAlgorithmInput input, MyAlgorithmOutput &output);
}
</code>```

This is actually a very powerful design, though I have never seen it in a C++ programming environment before. Changes in the number of parameters and their data types are now encapsulated and they don't change the algorithm callback.

In the latest algorithm which I have developed I used the same scheme. Now I want to know if this is a popular design pattern and what it is called.

答:
This is actually a very powerful design, though I have never seen it
  in a C++ programming environment

I think the word powerful is open to interpretation, and really depends on perspective,   agreeing with Kilian Foth comment, in what he calls an anti-pattern.

Most C++ projects you will see are likely open sourced, or open in one way or another, thus licencing modules and code obfuscation techniques will be hard to come by.

After all, with the technique you described you will sacrifice to some degree meaningful IDE code assistance, which is used by a gross section of programmers.

This kind of wrapping makes things more obfuscated, and I am guessing part of the motivation to do so may actually be protection of Intellectual property, notwithstanding the slight performance impact due to the extra stack operations.

On the other hand it depends on the programming language and its idiosyncrasies:
In JavaScript for instance I use such a 'pattern' for instance to pass strings by reference.

In C++ it would make sense for additional parameter, type, sanity, pointer and memory bounds testing. As such it takes a page from managed code minus the application virtual machine. Please, correct me if I am wrong.

I could see this type of wrapping being employed frequently in game engine development. Is your company by any means involved in that field?


问:
What is the (craziest, stupidest, silliest) thing a client/boss asked you to do?
See title, but I am asking from a technical perspective, not

Take my 40 year old virgin niece on a date or you're fired.

答: I was working on a digital marketing website where we were selling DRM-protected Windows Media audio files; audio books for the most part. Microsoft pushed out some sort of required compliance update for all vendors to implement, assuming the vendors are deploying desktop client applications on end-user machines. The update required the application to check DLL versions on the client's machine to ensure they're up-to-date. My boss was literally drilling me for every conceivable way in which our website was allowed to check a web client's DLL file versions in his/her system folder, even to the point of suggesting we write our own Windows Media Player skin to do so.</p>


答:
Initial proper design cannot help avoiding the problem. And it is almost impossible (or very-very hard) to consider all future "maybe" requirements. So after some time the Big Re-factoring will arrive. And the best solution is to re-write everything.

In a few words: instead of fitting a turret onto the red Ferrari, re-consider the requirements and build a tank.

Title: How do you cope with ugly code that you wrote?

Body:
So your client asks you to write some code, so you do. He then changes the specs on you, as expected, and you diligently implement his new features like a good little lad. Except... the new features kind of conflict with the old features, so now your code is a mess. You really want to go back and fix it, but he keeps requesting new things and every time you finish cleaning something, it winds up a mess again.

What do you do? Stop being an OCD maniac and just accept that your code is going to wind up a mess no matter what you do, and just keep tacking on features to this monstrosity? Save the cleaning for version 2?


问:
What is a byte stream actually?
Can anyone explain me what byte stream actually contains? Does it contain bytes (hex data) or binary data or english letters only? I am also confused about the term "raw data". If someone asked me to "reverse the 4 byte data", then what should I assume the data is hex code or binary code?</p>

答:
Byte streams contain, well, bytes.  Broken down into what it is actually, it is 8 bits composed of 1s and 0s.  If it were representing a number, it would be any number from 0 to 255 (which, I may add, is no coincidence why the 4 numbers in an IP address always range from 0 to 255).  Byte streams are usually sophisticated interfaces meant to hide the underlying basic byte array used to hold a circular buffer (you fill up the buffer and wait for someone to empty it, at which time it simply fills up the buffer again).

What the heck does that represent?  Well, it could represent a text file, or an image, or a live video stream.  What it is is entirely dependent upon the context of who is reading it.  Hex representation is another way of saying the same thing, though it is sometimes more convenient to manage bytes in terms of their hex representation rather than numbers however it is the same thing.

When you're referring to raw data, you are usually referring to byte data.  The data comes without a tag saying "I am an image file!"  Usually you only deal with raw data when you don't really care what the data represents overall.  For example, if I wanted to convert an image to its black and white version, I might say to read an image's raw data and for every 3 bytes read (which would actually be representation of red color, representation of green color, and representation of blue color), add its number value and divide by 3, then write that value 3 times.  Essentially what I'd be doing is averaging a pixel's red, green, and blue values and making its gray equivalent pixel from that.  However, when you talk about performing operations to data at the level of "byte by byte", you don't really care about the big picture, so to speak.

Or, perhaps you wish to save a file in a database, but it asks you to insert its "raw data" in a blob data type.  This simply means to convert the data of a file into a large byte array that the database can understand and manage.  You'll find that when you retrieve that value from the database, it will be simply one large byte array as you initially provided to the database to begin with.  If that data was a file, then you, the programmer, must reinterpret that byte data as if you were reading a file one byte at a time.

If someone asked you to "reverse the 4 byte data", I would assume it refers to big-endian vs little-endian interpretation of numbers, which writes numbers starting with the most or least significant byte.  It does not matter if a number is represented as big-endian or little-endian, just that all systems reading the number interpret it consistently.

This isn't to say that the actual number representation (or hex representation for that matter) is changed, simply that the order in which these 4 bytes make a number should be reversed.  So say you have 0x01, 0x02, 0x03, and 0x04.  To reverse these, you'd have 0x04, 0x03, 0x02, 0x01 instead.  The system would presumably read these 4 bytes in the reverse order and since you've already reversed it, the value is interpreted to be the very same as what was intended in the raw data.

I hope that explains it!

Title: What is a byte stream actually?


答:
A byte is simply a unit of information - it can be anything.  A byte by itself doesn't mean anything, you have to attach some sort of meaning to it.

So, to expand on that -

Does it contain bytes (hex data) or binary data or english letters only?

Hex data is the same as binary data.  It's just a different way of displaying the data.  For example, 0x41 = 0b01000001 = 'A' = 65 (decimal).  English letters would be just a subset of that.

If someone asked me to "reverse the 4 byte data", then what should I assume the data is hex code or binary code?

Since hex is just a representation of the data, it doesn't matter how you think about it.  If you have data of 0x65 0x66 0x67 0x68, to reverse it you would get 0x68 0x67 0x66 0x65.  If you were looking at this data in terms of characters, you would originally have A B C D, but now you have D C B A.

Back to a byte stream - it's just a sequence of data.  You need to know what the data represents in order to use it.  If we're reading a text file, the byte stream that you would get when you're reading the file would just be characters of some kind.  An executable file would have a bunch of unprintable characters in it, which is why it would be called a binary file.  Clearly, it's possible to open up an executable in a text editor, but it doesn't do anything useful.

Title: What is a byte stream actually?

Body: Can anyone explain me what byte stream actually contains? Does it contain bytes (hex data) or binary data or english letters only? I am also confused about the term "raw data". If someone asked me to "reverse the 4 byte data", then what should I assume the data is hex code or binary code?</p>


问:
Python Multiprocessing with Queue vs ZeroMQ IPC
I am busy writing a Python application using ZeroMQ and implementing a variation of the Majordomo pattern as described in the ZGuide.

I have a broker as an intermediary between a set of workers and clients. I want to do some extensive logging for every request that comes in, but I do not want the broker to waste time doing that. The broker should pass that logging request to something else.

I have thought of two ways :-

Create workers that are only for logging and use the ZeroMQ IPC transport

Use Multiprocessing with a Queue

I am not sure which one is better or faster for that matter. The first option does allow me to use the current worker base classes that I already use for normal workers, but the second option seems quicker to implement.

I would like some advice or comments on the above or possibly a different solution.

答:
These are radically different approaches, each with its own sets of pros and cons, which you will most likely see panning out at a later development stage:

I have thought of two ways :-

Create workers that are only for logging and use the ZeroMQ IPC transport

Use Multiprocessing with a Queue

One way you could try is to have an additional logging-worker, as in approach 1. You could let your workers log to a memcache logging cluster, and the logging worker monitors the current resource load and upon deceeding a given resource load parameter, the worker logs to an IOPs limited device (e.g. harddisk) .

I also like Jonathan's approach with the caveat that I too mostly use Python 2.x, and that you would likely have to setup your own logging backend to really push the performance-envelope.

Correct me if I am wrong, but my take is that you are doing some really data-intensive task, with storage IOPs being your bottleneck.

A convenient way still would be to let the broker do the brokerage logging - in the form as described- with all the disadvantages of a central broker instance.
For instance if the broker is in such high demand that it never gets some breathing room to write the memcached logs back to storage, you would need to take another approach.

You may ultimately end up with a brokerless model. That is with the workers managing their work among themselves. In a simple example, through a Distributed round-robin algorithm.


答:
If I understand correctly, the problem is that you or the user in question is mostly proficient in JavaScript and would like to employ it. If you just drag an drop a file into the Browser window, the DOM and subsequent rendering step will incur great overhead, in your case even leading to premature application termination due to an out of memory exception.

You can use JavaScript standalone.
So you could use either use

node.js, Rhino or

the Browser's FileReader API and a simple REST Endpoint

You should be able to effortlessly open GB sized files in a FileReader object, with the caveat that the following approach is not the best performing. Then in a loop you would read blocks of say 5MB into memory, process these and push these to a webserver (ideally at localhost/127.0.0.1) You could use a simple php script for that (name it index.php). Something like:

```<code><?php file_put_contents("outfile".date().".tsv", implode('', $_POST); , FILE_APPEND); ?>
</code>```

Notes:

You should probably do some filtering on the $_POST variable first, and put a PHP_Session ID into the filename.

Through another short php/some-language script you could offer the user to compress said output-file and let the user download it. The script checks if the archive already exists, and if not inform the user to try again later.

You should do the POSTing of the processed content through AJAX calls. If you do not want to use a comprehensive library such as jQuery for that purpose consider using is-lib's setFile function.

Reference:

For an hands-on example of the File-Reader see this project, and look up the MDN Reference for more information.

Good luck.

Title: How to handle editing a large file for a non-technical user

Body:
I have a client who is given a tab delimited .txt file containing hundreds of thousands of rows.

I have a user story as follows:

As a user I want to take the text file and add a new value at the end of each line which contains the concatenated value of two of the columns.

for example if the file read

```<code>text_one    text_two
</code>```

I need to output the following (preferably to a .txt file)

```<code>text_one    text_two    text_onetext_two
</code>```

My first approach was to ask the vendor supplying the file to do the concatenation before providing the file, the easiest way to solve a problem is to eliminate it right? however they are very uncooperative and have point blank refused.

I've looked at building a simple javascript application that does this client side so a non-technical user could select the file using a file selector. This approach has a few problems

The file could be over a GB in size and so can't be loaded straight into memory, I've tried and the browser crashes

There is no means to write a file in javascript so I'd need to output the content to the screen and have the user save it (somehow)

I was thinking if I could get around the filesize limitations I could just output the edited content to the page and have the user save the page as a .txt file, however I think there is a better way than using javascript that will still accommodate the users lack of technical know-how.

Please consider this question to be stack agnostic, but bear in mind that a nice little shell script or python script would be deemed unsuitable for a non technical user unless there is a way of "packaging" it nicely for a non-technical user.

Updates

The file is too large to open in excel.
The process needs to be run weekly, but it doesn't require scheduling or automation...(yet)

Body:
If I understand correctly, the problem is that you or the user in question is mostly proficient in JavaScript and would like to employ it. If you just drag an drop a file into the Browser window, the DOM and subsequent rendering step will incur great overhead, in your case even leading to premature application termination due to an out of memory exception.

You can use JavaScript standalone.
So you could use either use

node.js, Rhino or

the Browser's FileReader API and a simple REST Endpoint

You should be able to effortlessly open GB sized files in a FileReader object, with the caveat that the following approach is not the best performing. Then in a loop you would read blocks of say 5MB into memory, process these and push these to a webserver (ideally at localhost/127.0.0.1) You could use a simple php script for that (name it index.php). Something like:

<?php file_put_contents("outfile".date().".tsv", implode('', $_POST); , FILE_APPEND); ?>

Notes:

You should probably do some filtering on the $_POST variable first, and put a PHP_Session ID into the filename.

Through another short php/some-language script you could offer the user to compress said output-file and let the user download it. The script checks if the archive already exists, and if not inform the user to try again later.

You should do the POSTing of the processed content through AJAX calls. If you do not want to use a comprehensive library such as jQuery for that purpose consider using is-lib's setFile function.

Reference:

For an hands-on example of the File-Reader see this project, and look up the MDN Reference for more information.

Good luck.


问:
How is reactive logic programming different from functional programming?
As Reactive Programming model is perfectly suited to address the change propagation required for transaction processing, that is, when the server receives change (Insert, Update, or Delete) requests. The key approach borrows heavily from Domain Driven Design to provide declarative Domain Logic. But how does it differ from functional programming approach?</p>

答:
From the referenced wikipedia page cited by your link, It can be said that functional approach is orthogonal to being reactive. From the wikipedia article follows that reactive programming can be integrated in several programming paradigms (like procedural, logic, OO or functional).

My understanding is that reactive programming is based on the concept of dataflow, meaning that variable can be built from other variables and their value is constantly updated from its "dependencies", much like what happens in a electronic spreadsheet.

So reactive programming is mostly about data and how data changes.

The functional programming model is essentially based upon the definition of functions and the composition of those to build higher level operations.

The logic programming model is based on logical statements (usually in the form of assertions or equations) and the composition of those. The program executes by finding data that satisfies the proposed logic. This is similar to using a rule engine.

Both approaches can be used to create reactive programs, which will lead to programs built upon different reasoning approach to solving a problem. This reflects on the building blocks offered by the language as can be seen reading the OP link.


问:
How do web servers enforce the same-origin policy?
I'm diving deeper into developing RESTful APIs and have so far worked with a few different frameworks to achieve this. Of course I've run into the same-origin policy, and now I'm wondering how web servers (rather than web browsers) enforce it. From what I understand, some enforcing seems to happen in the browser's end (e.g., honoring an Access-Control-Allow-Origin header received from a server). But what about the server?

For example, let's say a web server is hosting a Javascript web app that accesses an API, also hosted on that server. I assume that the server would enforce the same-origin policy --- so that only the javascript that is hosted on that server would be allowed to access the API. This would prevent someone else from writing a javascript client for that API and hosting it on another site, right? So how would a web server be able to stop a malicious client that would try to make AJAX requests to its API endpoints while claiming to be running javascript that originated from that same web server? What's the way most popular servers (Apache, nginx) protect against this kind of attack? Or is my understanding of this somehow off the mark?

Or is the cross-origin policy only enforced on the client end?

答:
Web servers generally prevent attacks of this sort by checking the (infamously misspelled) Referer line in the HTTP header, to ensure that a request is coming from a page on their own site.  There's no good way to guard against a malicious client, but that's not how XSRF attacks work.

The client isn't malicious; it's generally an ordinary user who's been tricked by a malicious third-party into opening a document that silently makes a HTTP request using the client's stored cookies.  So if the server can verify via the Referer that the HTTP request is coming from gmail.com, and not MyAwesomeWebsite.com, it can shut the attack down.


答:
The same origin policy is a wholly client-based restriction, and is primarily engineered to protect users, not services. All or most browsers include a command-line switch or configuration option to to turn it off. The SOP is like seat belts in a car: they protect the rider in the car, but anyone can freely choose not to use them. Certainly don't expect a person's seat belt to stop them from getting out of their car and attacking you (or accessing your Web service).

Suppose I write a program that accesses your Web service. It's just a program that sends TCP messages that include HTTP requests. You're asking for a server-side mechanism to distinguish between requests made by my program (which can send anything) and requests made by a browser that has a page loaded from a permitted origin. It simply can't be done; my program can always send a request identical to one formed by a Web page.

The same-origin policy was invented because it prevents code from one website from accessing credential-restricted content on another site. Ajax requests are by default sent with any auth cookies granted by the target site. For example, suppose I accidentally load http://evil.com/, which sends a request for http://mail.google.com/. If the SOP were not in place, and I was signed into Gmail, the script at evil.com could see my inbox. If the site at evil.com wants to load mail.google.com without my cookies, it can just use a proxy server; the public contents of mail.google.com are not a secret (but the contents of mail.google.com when accessed with my cookies are a secret).

Title: How do web servers enforce the same-origin policy?

Body:
I'm diving deeper into developing RESTful APIs and have so far worked with a few different frameworks to achieve this. Of course I've run into the same-origin policy, and now I'm wondering how web servers (rather than web browsers) enforce it. From what I understand, some enforcing seems to happen in the browser's end (e.g., honoring an Access-Control-Allow-Origin header received from a server). But what about the server?

For example, let's say a web server is hosting a Javascript web app that accesses an API, also hosted on that server. I assume that the server would enforce the same-origin policy --- so that only the javascript that is hosted on that server would be allowed to access the API. This would prevent someone else from writing a javascript client for that API and hosting it on another site, right? So how would a web server be able to stop a malicious client that would try to make AJAX requests to its API endpoints while claiming to be running javascript that originated from that same web server? What's the way most popular servers (Apache, nginx) protect against this kind of attack? Or is my understanding of this somehow off the mark?

Or is the cross-origin policy only enforced on the client end?


答:
The same-origin policy is enforced on the client-side. If the browser supports CORS, the server can send back headers that tell the browser to make exceptions to the same-origin policy. For example, sending the header

```<code> Access-Control-Allow-Origin: www.example.com
</code>```

would tell the browser to allow cross-origin requests from www.example.com.

```<code> Access-Control-Allow-Origin: *
</code>```

tells the browser to allow all cross-origin requests to that resource.

Body:
The same-origin policy is enforced on the client-side. If the browser supports CORS, the server can send back headers that tell the browser to make exceptions to the same-origin policy. For example, sending the header

 Access-Control-Allow-Origin: www.example.com

would tell the browser to allow cross-origin requests from www.example.com.

 Access-Control-Allow-Origin: *

tells the browser to allow all cross-origin requests to that resource.


答:
I like the approach of using standard tools like what Jonathan proposed. You didn't mention which OS you are doing the work on, but another alternative that follows that same spirit could be to use Python's standard logging module together with logging.handlers.SysLogHandler and send the logging messages to rsyslog service (available on any linux/unix, but I think there's also a windows option, but I've never used that one).

Essentially that whole system implements same thing that you are thinking off.  Your local process queues up log message to be handled/processed/written by someone else. In this case, the someone else (rsyslog) is a well-known, proven service that has a lot of built-in functionality and flexibility.

Another advantage of this approach is that your product will integrate that much better with other sysadmin tools that are built on top of syslog. And it wouldn't even require you to write any code to get that option.

Title: Python Multiprocessing with Queue vs ZeroMQ IPC

Body:
I am busy writing a Python application using ZeroMQ and implementing a variation of the Majordomo pattern as described in the ZGuide.

I have a broker as an intermediary between a set of workers and clients. I want to do some extensive logging for every request that comes in, but I do not want the broker to waste time doing that. The broker should pass that logging request to something else.

I have thought of two ways :-

Create workers that are only for logging and use the ZeroMQ IPC transport

Use Multiprocessing with a Queue

I am not sure which one is better or faster for that matter. The first option does allow me to use the current worker base classes that I already use for normal workers, but the second option seems quicker to implement.

I would like some advice or comments on the above or possibly a different solution.


问:
When does it become overkill?
First off, I apologize cause I don't know how to make a community thread; so someone help me out please.

As a developer, across many platforms, technologies and even on an infrastructure level; I always find myself asking, when am I doing TOO much?!?

It's been a never ending learning process, since I started. One (1) thing I learned is that requirements are barely valid for an extended period of time, and as such a little foresight may go a long way.

But where is the balance, and how do you know when you're losing time, not gaining it?!

答:
a never ending learning process indeed! ...and i think it stays that way! The balance is when things are efficient enough and you have the time to do something else apart from programming as well. I 'll agree with gablin " a matter of discipline and prioritizing" and jim hopkins on that it becomes instinct after a while. I know that perfecting the small parts is what makes us happy but in the end it's all about what makes the end user happy. so i 'd say the balance (or maybe the compromise) is make the end user / customer / client happy first (which should be plan A) and then if there's time work on perfecting - making more efficient your "small parts" and/or everything else you please. At some point you have to say "enough" though :) otherwise yes, it will become an overkill!

the worst case scenario of course is when the end user / customer / client is you! :)

Title: When does it become overkill?

OwnerDisplayName: phdem

LastEditorDisplayName: phdem


答:
Functional programming is about well.. functions :) You're entire program corresponds to the application and reduction of functions. If you want to get back to its roots, functional programming is lambda calculus supplemented with whatever you like (types, records, concurrency, primitive data).

Notably, FP has no notion of "time". Functions are referentially transparent and it shouldn't matter when you evaluate them.

Reactive programming talks about different stuff, it focuses on "behaviors" and "events". behaviors are like variables that change all on their own, for example: In a GUI the value of a slider changes over time, we could model a slider with a behavior. We also talk about events, which are fired at discrete moments, this corresponds to a button, which a user might click.

Notice that these aren't mutually exclusive, but reactive programming is pretty much orthogonal to functional programming. You can model behaviors and events with functions, objects, logical relations, you name it! We've simply run out of buzzwords and are beginning to reuse them :)

Title: How is reactive logic programming different from functional programming?

Body: As Reactive Programming model is perfectly suited to address the change propagation required for transaction processing, that is, when the server receives change (Insert, Update, or Delete) requests. The key approach borrows heavily from Domain Driven Design to provide declarative Domain Logic. But how does it differ from functional programming approach?</p>


答:
How do web servers enforce the same-origin policy?

In short, they do not, as apsillers and Dirk pointed out.

One important reason is that the ACAO header protects the servers themselves from rampant DDOS, - Distributed Denial of Service- attacks.

Who:

The ACAO as a HTTP response-header is meant for the web client to be interpreted, operating under the assumption that the majority of human internet users are browsing the web through major browser vendors who adhere and implement the W3C recommended draft. They should, after all most of them benefit from a fast, accessible internet.

How:

Otherwise anyone, could just copy and paste a few lines of javascript code into a malicious website that runs a simple loop, which makes an Ajax GET's or POST's request to a foreign domain. Without user interaction, and the ability to multithread.

That is why you have to opt-in to access a cross-origin site, through the ACAO HTTP header. You, the user, can access said site any time through a user-aware interaction, i.e. an internet link. Just like you can user-aware copy or paste content from or to your clipboard, but not any other way - plugins aside.

Future:

At that point, heed the web-browser maker's direction of:

default plugin removal, as a step towards a fully sandboxed environment and

removing the user-option to turn off javascript

Security restrictions can be decently established using a combination of TSL 2/3, strong session-IDs, TANs, two factor authentication etc..

'Google' has this to show and say about DDOS

Lastly, anyone is free to proxy any web content and add a desired ACAO header to access proxied cross-site content. Likewise, this proxy is then as open to a DDOS attack, as the ACAO setting allows it to be. I actually don't know of a single, free public service offering. Please correct me if I am wrong.