问:
What is an Anti-Corruption layer, and how is it used?
I'm trying to figure out what the Anti-Corruption layer really means. I know that it's a way to transition/work around legacy code or bad APIs. What I don't understand is how it works and what makes it a clean separation from the undesirable layer.

I've done some searching, but I can't find any simple examples or explanations, so I'm looking for someone who understands it and can explain it with simple examples. An answer that would satisfy my question should be simple (not necessarily short), and provide understandable examples of implementation and use.

See this question, for my use case.

答:
Imagine you have to use someone else's code that is designed as shown below:

    class Messy {
        String concat(String param, String str) { /* ... */ }
        boolean contains(String param, String s) { /* ... */ }
        boolean isEmpty(String param) { /* ... */ }
        boolean matches(String param, String regex) { /* ... */ }
        boolean startsWith(String param, String prefix) { /* ... */ }
    }
```

<p>Now imagine you find out that your code that depends on it looks like the following:</p>

<pre class="lang-java prettyprint-override"><code>String process(String param) {
    Messy messy = new Messy();
    if (messy.contains(param, "whatever")) {
        return messy.concat(param, "-contains");
    }
    if (messy.isEmpty(param)) {
        return messy.concat(param, "-empty");
    }
    if (messy.matches(param, "[whatever]")) {
        return messy.concat(param, "-matches");
    }
    if (messy.startsWith(param, "whatever")) {
        return messy.concat(param, "-startsWith");
    }
    return messy.concat(param, "-whatever");
    // WTF do I really need to repeat bloody "param" 9 times above?
}
</code>```

...and that you want to make it easier to use, in particular, to get rid of repetitive usage of parameters that just aren't needed for your application.

Okay, so you start building an anti-corruption layer.

First thing is to make sure that your "main code" doesn't refer to Messy directly. For example, you arrange dependency management in such a way that trying to access Messy fails to compile.

Second, you create a dedicated "layer" module that is the only one accessing Messy and expose it to your "main code" in a way that makes better sense to you.

Layer code would look like the following:

    class Reasonable { // anti-corruption layer
        String param;
        Messy messy = new Messy();
        Reasonable(String param) {
            this.param = param;
        }
        String concat(String str) { return messy.concat(param, str); }
        boolean contains(String s) { return messy.contains(param, s); }
        boolean isEmpty() { return messy.isEmpty(param); }
        boolean matches(String regex) { return messy.matches(param, regex); }
        boolean startsWith(String prefix) { return messy.startsWith(param, prefix); }
    }
```

<p>As a result, your "main code" does not mess with <code>Messy</code>, using <code>Reasonable</code> instead, about as follows:</p>

<pre class="lang-java prettyprint-override"><code>String process(String param) {
    Reasonable reasonable = new Reasonable(param);
    // single use of "param" above and voila, you're free
    if (reasonable.contains("whatever")) {
        return reasonable.concat("-contains");
    }
    if (reasonable.isEmpty()) {
        return reasonable.concat("-empty");
    }
    if (reasonable.matches("[whatever]")) {
        return reasonable.concat("-matches");
    }
    if (reasonable.startsWith("whatever")) {
        return reasonable.concat("-startsWith");
    }
    return reasonable.concat("-whatever");
}
</code>```

Note there is still a bit of a mess messing with Messy but this is now hidden reasonably deep inside Reasonable, making your "main code" reasonably clean and free of corruption that would be brought there by direct usage of Messy stuff.

Above example is based on how Anticorruption Layer is explained at c2 wiki:

If your application needs to deal with a database or another application whose model is undesirable or inapplicable to the model you want within your own application, use an AnticorruptionLayer to translate to/from that model and yours.

Note example is intentionally made simple and condensed to keep explanation brief.

If you have a larger mess-of-API to cover behind the anti-corruption layer, same approach applies: first, make sure your "main code" doesn't access corrupted stuff directly and second, expose it in a way that is more convenient in your usage context.

When "scaling" your layer beyond a simplified example above, take into account that making your API convenient is not necessarily a trivial task. Invest an effort to design your layer the right way, verify its intended use with unit tests etc.

In other words, make sure that your API is indeed an improvement over one it hides, make sure that you don't just introduce another layer of corruption.

For the sake of completeness, notice subtle but important difference between this and related patterns Adapter and Facade. As indicated by its name, anticorruption layer assumes that underlying API has quality issues (is "corrupted") and intends to offer a protection of mentioned issues.

You can think of it this way: if you can justify that library designer would be better off exposing its functionality with Reasonable instead of Messy, this would mean you're working on anticorruption layer, doing their job, fixing their design mistakes.

As opposed to that, Adapter and Facade do not make assumptions on the quality of underlying design. These could be applied to API that is well designed to start with, just adapting it for your specific needs.

Actually, it could even be more productive to assume that patterns like Adapter and Facade expect underlying code to be well designed. You can think of it this way: well designed code shouldn't be too difficult to tweak for particular use case. If it turns out that design of your adapter takes more effort than expected, this could indicate that underlying code is, well, somehow "corrupted". In that case, you can consider splitting the job to separate phases: first, establish an anticorruption layer to present underlying API in a properly structured way and next, design your adapter / facade over that protection layer.

Body:
Imagine you have to use someone else's code that is designed as shown below:

    class Messy {
        String concat(String param, String str) { /* ... */ }
        boolean contains(String param, String s) { /* ... */ }
        boolean isEmpty(String param) { /* ... */ }
        boolean matches(String param, String regex) { /* ... */ }
        boolean startsWith(String param, String prefix) { /* ... */ }
    }

Now imagine you find out that your code that depends on it looks like the following:

String process(String param) {
    Messy messy = new Messy();
    if (messy.contains(param, "whatever")) {
        return messy.concat(param, "-contains");
    }
    if (messy.isEmpty(param)) {
        return messy.concat(param, "-empty");
    }
    if (messy.matches(param, "[whatever]")) {
        return messy.concat(param, "-matches");
    }
    if (messy.startsWith(param, "whatever")) {
        return messy.concat(param, "-startsWith");
    }
    return messy.concat(param, "-whatever");
    // WTF do I really need to repeat bloody "param" 9 times above?
}

...and that you want to make it easier to use, in particular, to get rid of repetitive usage of parameters that just aren't needed for your application.

Okay, so you start building an anti-corruption layer.

First thing is to make sure that your "main code" doesn't refer to Messy directly. For example, you arrange dependency management in such a way that trying to access Messy fails to compile.

Second, you create a dedicated "layer" module that is the only one accessing Messy and expose it to your "main code" in a way that makes better sense to you.

Layer code would look like the following:

    class Reasonable { // anti-corruption layer
        String param;
        Messy messy = new Messy();
        Reasonable(String param) {
            this.param = param;
        }
        String concat(String str) { return messy.concat(param, str); }
        boolean contains(String s) { return messy.contains(param, s); }
        boolean isEmpty() { return messy.isEmpty(param); }
        boolean matches(String regex) { return messy.matches(param, regex); }
        boolean startsWith(String prefix) { return messy.startsWith(param, prefix); }
    }

As a result, your "main code" does not mess with Messy, using Reasonable instead, about as follows:

String process(String param) {
    Reasonable reasonable = new Reasonable(param);
    // single use of "param" above and voila, you're free
    if (reasonable.contains("whatever")) {
        return reasonable.concat("-contains");
    }
    if (reasonable.isEmpty()) {
        return reasonable.concat("-empty");
    }
    if (reasonable.matches("[whatever]")) {
        return reasonable.concat("-matches");
    }
    if (reasonable.startsWith("whatever")) {
        return reasonable.concat("-startsWith");
    }
    return reasonable.concat("-whatever");
}

Note there is still a bit of a mess messing with Messy but this is now hidden reasonably deep inside Reasonable, making your "main code" reasonably clean and free of corruption that would be brought there by direct usage of Messy stuff.

Above example is based on how Anticorruption Layer is explained at c2 wiki:

If your application needs to deal with a database or another application whose model is undesirable or inapplicable to the model you want within your own application, use an AnticorruptionLayer to translate to/from that model and yours.

Note example is intentionally made simple and condensed to keep explanation brief.

If you have a larger mess-of-API to cover behind the anti-corruption layer, same approach applies: first, make sure your "main code" doesn't access corrupted stuff directly and second, expose it in a way that is more convenient in your usage context.

When "scaling" your layer beyond a simplified example above, take into account that making your API convenient is not necessarily a trivial task. Invest an effort to design your layer the right way, verify its intended use with unit tests etc.

In other words, make sure that your API is indeed an improvement over one it hides, make sure that you don't just introduce another layer of corruption.

For the sake of completeness, notice subtle but important difference between this and related patterns Adapter and Facade. As indicated by its name, anticorruption layer assumes that underlying API has quality issues (is "corrupted") and intends to offer a protection of mentioned issues.

You can think of it this way: if you can justify that library designer would be better off exposing its functionality with Reasonable instead of Messy, this would mean you're working on anticorruption layer, doing their job, fixing their design mistakes.

As opposed to that, Adapter and Facade do not make assumptions on the quality of underlying design. These could be applied to API that is well designed to start with, just adapting it for your specific needs.

Actually, it could even be more productive to assume that patterns like Adapter and Facade expect underlying code to be well designed. You can think of it this way: well designed code shouldn't be too difficult to tweak for particular use case. If it turns out that design of your adapter takes more effort than expected, this could indicate that underlying code is, well, somehow "corrupted". In that case, you can consider splitting the job to separate phases: first, establish an anticorruption layer to present underlying API in a properly structured way and next, design your adapter / facade over that protection layer.


问:
Using Subversion as an artifact repository vs a specific artifact management tool
TL;DR: Why use something like Apache Archiva or Sonatype Nexus as an artifact repository instead of Subversion?

The build system I use currently has a lot of binary blobs (images, sound files, compiled binaries, etc), both as input and output to our builds. Our system for managing these is very ad hoc; some are checked into our Subversion repository alongside our code, some are stored elsewhere outside any formal version control.

I'm looking at consolidating this, so we have something that's more self-consistent and easy to use, and which separates binary artifacts from code.

Google tells me there are a selection of artifact repositories available (Archiva, Nexus, Artifactory, …), but from reading around, I can't see any advantage to using these over Subversion. That will look after the binaries for us – it already does that for some of our binaries, we'd just want to rearrange the repository layout to separate them from code – and has the notable advantage that we already have Subversion servers and expertise.

So. What's the advantage of using a dedicated artifact management system over using a general version control tool like Subversion?

答:
Short answer: Generally, you don't need a history of binary artifacts and changes to those artifacts, you just need specific versions.

Longer answer: Every time you commit a small change to a binary file, version control systems don't have any way to create a delta -- a diff between the two files -- so it creates a whole new copy.

In a CVCS, like SVN, that's not such a big pain, because you only have one central copy of your repository -- your local copy is only one version. (Although, even then, your repository can become very large, making checkins slower.) But what happens if you later switch to a DVCS, where every copy of a repository has the full history of every file? The size of changes becomes very relevant there.

And what does it give you in return for the pain? The only thing it offers is being able to go back to a previous version of your repository and know that you have the correct binaries for that version.

But do you need the whole binary in your repository to do that? Or can you get away with simply having a text file, telling the build process which versions to pull from another repository elsewhere?

The latter is what is offered by artifact repositories generally.

In addition, some of the more professional ones, such as Nexus, will also give you information about licensing for third-party artifacts, so that you don't risk falling afoul of some subtle clause in what you believe to be a FOSS library.

ContentLicense: CC BY-SA 3.0


问:
What should you bring to the table as a Software Architect?
There have been many questions with good answers about the role of a Software Architect (SA) on StackOverflow and Programmers SE. I am trying to ask a slightly more focused question than those. The very definition of a SA is broad so for the sake of this question let's define a SA as follows:

A Software Architect guides the
  overall design of a project, gets
  involved with coding efforts, conducts
  code reviews, and selects the
  technologies to be used.

In other words, I am not talking about managerial rest and vest at the crest (further rhyming words elided) types of SAs. If I were to pursue any type of SA position I don't want to be away from coding. I might sacrifice some time to interface with clients and Business Analysts etc., but I am still technically involved and I'm not just aware of what's going on through meetings.

With these points in mind, what should a SA bring to the table? Should they come in with a mentality of "laying down the law" (so to speak) and enforcing the usage of certain tools to fit "their way," i.e., coding guidelines, source control, patterns, UML documentation, etc.? Or should they specify initial direction and strategy then be laid back and jump in as needed to correct the ship's direction?

Depending on the organization this might not work. An SA who relies on TFS to enforce everything may struggle to implement their plan at an employer that only uses StarTeam. Similarly, an SA needs to be flexible depending on the stage of the project. If it's a fresh project they have more choices, whereas they might have less for existing projects.

Here are some SA stories I have experienced as a way of sharing some background in hopes that answers to my questions might also shed some light on these issues:

I've worked with an SA who code reviewed literally every single line of code of the team. The SA would do this for not just our project but other projects in the organization (imagine the time spent on this). At first it was useful to enforce certain standards, but later it became crippling. FxCop was how the SA would find issues. Don't get me wrong, it was a good way to teach junior developers and force them to think of the consequences of their chosen approach, but for senior developers it was seen as somewhat draconian.

One particular SA was against the use of a certain library, claiming it was slow. This forced us to write tons of code to achieve things differently while the other library would've saved us a lot of time. Fast forward to the last month of the project and the clients were complaining about performance. The only solution was to change certain functionality to use the originally ignored approach despite early warnings from the devs. By that point a lot of code was thrown out and not reusable, leading to overtime and stress. Sadly the estimates used for the project were based on the old approach which my project was forbidden from using so it wasn't an appropriate indicator for estimation. I would hear the PM say "we've done this before," when in reality they had not since we were using a new library and the devs working on it were not the same devs used on the old project.

The SA who would enforce the usage of DTOs, DOs, BOs, Service layers and so on for all projects. New devs had to learn this architecture and the SA adamantly enforced usage guidelines. Exceptions to usage guidelines were made when it was absolutely difficult to follow the guidelines. The SA was grounded in their approach. Classes for DTOs and all CRUD operations were generated via CodeSmith and database schemas were another similar ball of wax. However, having used this setup everywhere, the SA was not open to new technologies such as LINQ to SQL or Entity Framework.

I am not using this post as a platform for venting. There were positive and negative aspects to my experiences with the SA stories mentioned above. My questions boil down to:

What should an SA bring to the table?

How can they strike a balance in their decision making?

Should one approach an SA job (as defined earlier) with the mentality that they must enforce certain ground rules?

Anything else to consider?

Thanks! I'm sure these job tasks are easily extended to people who are senior devs or technical leads, so feel free to answer at that capacity as well.

答:
1 What should an SA bring to the table?

A thick skin

Good negotiation skills

A good understanding of the various software tiers (from whizzy AJAX goodness to low level networking I/O).  You don't necessarily be a hands on expert, but you're going to have make important decisions about what software should be executed at which layer(s).

Willingness to get their hands dirty in the code, white paper designs just don't cut it.

Encouragement of software craftsmanship - be the cheerleader for doing things the right way whenever possible and the best way given the limitations in other cases.  So things like source control, TDD, build and CI, coding DOJOs, code reviews, a good issue tracking system etc

2 How can they strike a balance in their decision making?

Much of it depends on your team(s) and how capable they are.

Your environment (you might be forced to use a particular vendor's product for example)

Overall it's best not to be an ivory tower architect, be hands on, be part of the team - people will understand your decisions that way.

3 Should one approach an SA job (as defined earlier) with the mentality that they must enforce certain ground rules?

Yes, things like version control and a build system are pretty darn important and developers need to use these.  However it's aways best to make them part of the solution.

There's lots more, I think there are going to be some really good answers coming through for this one.

ContentLicense: CC BY-SA 2.5


问:
Aggregate Root and Lots of Data Efficiency
It's more of a scenario, but it isn't far fetched at all. Let's say I have an Aggregate Root (AR) Warehouse which it's used to manage product stock. The Product itself is an AR in a different bounded context (BC) but in this BC is represented only by an id. In the Warehouse I can add a new product (must be unique), Ican remove it and i can update stock. Of course, I can communicate the stock for a product and maybe even keep the in/out flow for a product.

THe problem is you can easily reach hundreds or thousands of products. So, for any wareohuse action you'll have to load everything even if that action won't use all that info. It's highly inefficient.

A solution I could think of is to pretty much 'break' the warehouse AR in specialiazed objects for different actions. But this means we no longer have an AR and we're back to a CRUD like solution. Even more so, the AR was split not becasue the Domain need it, but because the technical details need it.

It looks like you can do DDD until a certain point, after that you go CRUD or buy a MUCH BIGGER and expensive server.

What do you think? Can we have both DDD and efficiency when lots of data is involved?

答:
Can we have both DDD and efficiency when lots of data is involved?

Yes, however DDD is not entirely free of technical constraints, especially in more modern architectures. An aggregate can be thought of as a transactional consistency boundary and not simply a pure representation of the business domain. When thinking of an aggregate as a consistency boundary it can be designed in a way that optimizes corresponding use cases.

In your scenario, having a large Warehouse aggregate will result in overlapping boundaries - the stock information of one product has no integrity constraints associated with other products. The problem may be that the Warehouse aggregate contains behavior for managing stock in addition to serving as a sort of anchor for your domain model. One solution would be to shift behavior to a ProductInventoryAggregate. Another could be to introduce eventual consistency.

Take a look at Effective Aggregate Design by Vaughn Vernon for more on this. Overall however, I do see the tension your observe between focusing on the pure domain on the one hand, but considering technical constraints on the other. This isn't too much of a conflict, for me at least, since I view the central value of DDD as being the strategic patterns not tactical ones.


问:
When creating an api should I stick with small functions and many calls, or a few calls and large functions?
I have a rails platform that I maintain. It has a lot of different web applications built on top of it. However now a client is asking for an API so that they can keep users on their site, but take advantage of some of the automated tasks we have.

The platform is used to build insurance applications and allows for their purchase online, as well as providing ways to download documentation related to your policy.

So my question when building the API is this:

When I have to do a lot of things, like validate, create a user, user profile, and policy, pretty much at the same time. Should I make 4 separate API calls, and make the client build 4 calls on their side. OR should I have one call that excepts a lot of parameters, that validates the client and creates all 3 of those things at the same time, simplifying things for the client?

The client, in this case, gets all of the required information at the same time, so its not like there is a natural flow in their application where it pauses and they can make an API call to my platform.

Having been on the client side using many API's before, my gut is to make it as simple for the client as possible and have them make just one call. However this is leading to rather large functions in the API, which I'm not a fan of either.

How do you suggest I tackle this?

As a note, I am not very confident in the clients ability to implement a complicated API on their side.

答:
How about doing both?  Have a "low level" (so to speak) API that exposes functions of the system and have another "layer" that exposes services that a client might want to do.  This layer would use the necessary low level API's required but those are still exposed if the client wants them.

UPDATE:
To also include some of the great points and comments made by others.

Consider if the client is ever going to need to call one of the smaller API methods e.g. Is it feasible to call createUserProfile without also calling createUser?  If not then don't expose that method.  Thanks NoobsArePeople2

A simple but excellent point. Key point with exposing something in an API - you can't ever unexpose it. Expose the minimum necessary to function and then expand rather than exposing everything and... well, then its naked and making changes can be embarrassing and awkward.  Thanks MichaelT


答:
I think your gut feeling is right - make the API simple for consumers. To some extent, this is the philosophy behind Consumer Driven Contracts.

More specifically, the API should expose suitable business use cases. Consider the business domain at hand - is there really a need for those low-level functions? What is the drawback of encapsulating them? Large functions in the API are not a problem in and of themselves. What would be a problem is if a large function sequences operations that may need to be partitioned to allow consumer intervention.

Title: When creating an api should I stick with small functions and many calls, or a few calls and large functions?

Body:
I have a rails platform that I maintain. It has a lot of different web applications built on top of it. However now a client is asking for an API so that they can keep users on their site, but take advantage of some of the automated tasks we have.

The platform is used to build insurance applications and allows for their purchase online, as well as providing ways to download documentation related to your policy.

So my question when building the API is this:

When I have to do a lot of things, like validate, create a user, user profile, and policy, pretty much at the same time. Should I make 4 separate API calls, and make the client build 4 calls on their side. OR should I have one call that excepts a lot of parameters, that validates the client and creates all 3 of those things at the same time, simplifying things for the client?

The client, in this case, gets all of the required information at the same time, so its not like there is a natural flow in their application where it pauses and they can make an API call to my platform.

Having been on the client side using many API's before, my gut is to make it as simple for the client as possible and have them make just one call. However this is leading to rather large functions in the API, which I'm not a fan of either.

How do you suggest I tackle this?

As a note, I am not very confident in the clients ability to implement a complicated API on their side.


答:
I think you're looking at it in the wrong way.  Don't worry about large | small calls or lots of | few calls.

Think about the business objects and the actions that can be performed with | for | against those objects.

You've got:

Users

Policies

Rates

...

So you should create API calls around those objects.

User.Create

User.UpdatePassword

Policy.Validate

...

The idea is to create atomic or near-atomic operations based upon the business objects and their actions.  This will simplify design and coding - calls that do what the business object should do which matches the mental model or expectations of the programmers.  When the programmers or architects are discussing requirements with the business analysts then they can all use the same terminology and general flow of operations.

The problem with larger, all-in-one type calls is the risk of side effects.  If Policy.Create also spawns a User and triggers some other action, then that would break the expectation of the programmers.  Likewise, lots of small calls force the programmer to remember to call A and then B and then C in order to perform a "single" business operation.

And how you name the calls will be based upon what Rails and your supporting web services will support.

To be more prescriptive, this will create some calls that take a number of parameters and may have multiple calls on the back-end that are obscured to the client.  You will also end up with some fairly quick / simple calls where the API is little more than a wrapper to the underlying routine.


问:
Retrieving hierarchy of tree-nodes stored in relational database
I have a hierarchical ADT that I want to store in a database.

Nodes have exactly one parent element that is a node as well. Multiple nodes can have the same parent node.

My current approach uses an adjacency list with a table structure similar to the following:
```<code>+----------------+------------------+------+-----+---------+----------------+
| Field          | Type             | Null | Key | Default | Extra          |
+----------------+------------------+------+-----+---------+----------------+
| node_id        | int(10) unsigned | NO   | PRI | NULL    | auto_increment |
| node_name      | varchar(255)     | NO   |     | NULL    |                |
| parent_node_id | int(10) unsigned | YES  | MUL | NULL    |                |
+----------------+------------------+------+-----+---------+----------------+</code>``` where parent_node_id stores other node_id values and cascades on update and delete.

To display a node's hierarchy, I implemented a loop in my script that queries for the respective parent_node_id, which can result in 5 to 10 database queries for a node. This struck me as incredibly inefficient, but I am at a loss as to how to avoid this. From my limited understanding of the inner workings of relational databases, I have deduced that stored procedures is not necessarily the way to go, since the issue seems to be with my approach to how I am storing my ADT.
I looked at discussions on how to store trees in databases, but couldn't find anything that addressed my specific needs, which are:

The integrity of the node hierarchy is important, which to me rules out storing the complete hierarchy in a column (Path Enumeration.)

Whenever I retrieve a node, I will also need to query for it's complete hierarchy.

答: This blog post goes into some detail on the issues you're having. You can do hierarchal queries in MySQL and also do complete chains of hierarchies. You should also be able  to maintain the hierarchy from what I can tell.</p>


问:
Client-side oriented website approach name (less server-side code, more client-side code)
I have a terminology question.

I was searching about tips to design a website that is using less server-side code and more client-side code, and realized I don't really know which exact term to search if there is one. Example: Maximizing CDNs usage for library, using widgets when possible, integration of client-side social-media SDKs.

So my question is not about tips themselves, but about the existence of such a term.

Edit:
I already know about SPAs and Client-Side Javascript Frameworks. This is what brings me to wonder if there's a name to an approach where you focus on client-side code at a point that you maximize external resources.

答:
I can't guarantee less server-side code, but you might want to search for:

Single Page Applications (a.k.a. SPAs)


问:
How do scalable bloom filters work?
I was reading up on scalable bloom filters and could not understand how each time a constituent bloom filters fills up, a new bloom filter with larger size is added.

The elements that contributed to the set bits in the initially created filters cannot be looked up for presence. Perhaps I am wrong in my understanding of this?

I do understand basic bloom filters. However, I cannot wrap my head around dynamic bloom filters.

答:
Let me try to give this a shot to see how much I can butcher it. :-)

So, to start off, you need to be able to create a regular bloom filter that allows a finite number of elements with a maximum probability of a false positive. The addition of these features to your basic filter are required before attempting to build a scalable implementation.

Before we try to control and optimize what the probability is, lets figure out what the probability is for a given bloom filter size.

First we split up the bitfield by how many hash functions we have (total number of bits / number of hash functions = slices) to get k slices of bits which represent each hash function so every element is always described by k bits.

If you increase the number of slices or the number of bits per slice, the probability of false positives will decrease.

It also follows that as elements are added, more bits are set to 1, so false positives increase. We refer to this as the "fill ratio" of each slice.

When the filter holds a large amount of data, we can assume that the probability of false positives for this filter is the fill ratio raised to the number of slices (If we were to actually count the bits instead of using a ratio, this simplifies into a permutation with repetition problem).

So, how do we figure out how to pick a probability of false positives in a bloom filter? We can modify the number of slices (which will affect the fill ratio).

To figure out how many slices we should have, we start off with figuring out the optimal fill ratio for a slice. Since the fill ratio is determined by the number of bits in a slice which are 1 versus the number of bits which are 0, we can determine that each bit will remain unset with probability of (100% - (1 / bits in a slice)). Since we're going to have multiple items inserted, we have another permutation with reputation problem and we expand things out to the expected fill ratio, which is (100% - ((100% - (1 / bits in a slice)) ^ "elements inserted")). Well, it turns out that this is very similar to another equation. In the paper, they relate the fill ratio to another equation so it fits nicely into a taylor series (1-e^(-n/m)). After a bit of futzing with this, it turns out that the optimal fill ratio is always about 50%, regardless of any of the variables that you change.

So, since the probability of a filter is the fill ratio raised to the number of slices, we can fill in 50% and get P=(50%)^k or k=log_2(1/P). We can then use this function to compute the number of slices we should generate for a given filter in the list of filters for a scalable bloom filter.

```<code>    def slices_count(false_positive_probability):
        return math.ceil(math.log(1 / false_positive_probability, 2))
</code>```

Edit: After writing this, I came across a mention of the "fifty-percent rule" when reading up on buddy system based dynamic memory allocation in TAoCP Vol 1, pp 442-445 with a much cleaner reasoning versus fitting the curve to (1-e^(-n/m)). Knuth also references a paper "The fifty percent rule revisited" with a bit of background on the concept (pdf available here).

Title: How do scalable bloom filters work?

Body:
Let me try to give this a shot to see how much I can butcher it. :-)

So, to start off, you need to be able to create a regular bloom filter that allows a finite number of elements with a maximum probability of a false positive. The addition of these features to your basic filter are required before attempting to build a scalable implementation.

Before we try to control and optimize what the probability is, lets figure out what the probability is for a given bloom filter size.

First we split up the bitfield by how many hash functions we have (total number of bits / number of hash functions = slices) to get k slices of bits which represent each hash function so every element is always described by k bits.

If you increase the number of slices or the number of bits per slice, the probability of false positives will decrease.

It also follows that as elements are added, more bits are set to 1, so false positives increase. We refer to this as the "fill ratio" of each slice.

When the filter holds a large amount of data, we can assume that the probability of false positives for this filter is the fill ratio raised to the number of slices (If we were to actually count the bits instead of using a ratio, this simplifies into a permutation with repetition problem).

So, how do we figure out how to pick a probability of false positives in a bloom filter? We can modify the number of slices (which will affect the fill ratio).

To figure out how many slices we should have, we start off with figuring out the optimal fill ratio for a slice. Since the fill ratio is determined by the number of bits in a slice which are 1 versus the number of bits which are 0, we can determine that each bit will remain unset with probability of (100% - (1 / bits in a slice)). Since we're going to have multiple items inserted, we have another permutation with reputation problem and we expand things out to the expected fill ratio, which is (100% - ((100% - (1 / bits in a slice)) ^ "elements inserted")). Well, it turns out that this is very similar to another equation. In the paper, they relate the fill ratio to another equation so it fits nicely into a taylor series (1-e^(-n/m)). After a bit of futzing with this, it turns out that the optimal fill ratio is always about 50%, regardless of any of the variables that you change.

So, since the probability of a filter is the fill ratio raised to the number of slices, we can fill in 50% and get P=(50%)^k or k=log_2(1/P). We can then use this function to compute the number of slices we should generate for a given filter in the list of filters for a scalable bloom filter.

    def slices_count(false_positive_probability):
        return math.ceil(math.log(1 / false_positive_probability, 2))

Edit: After writing this, I came across a mention of the "fifty-percent rule" when reading up on buddy system based dynamic memory allocation in TAoCP Vol 1, pp 442-445 with a much cleaner reasoning versus fitting the curve to (1-e^(-n/m)). Knuth also references a paper "The fifty percent rule revisited" with a bit of background on the concept (pdf available here).


问:
Core data syncing (deletion) using a custom REST web service
I need to sync my local core data stack against a server running a persistent data storage underneath the covers. (I've gone through other posts here and on SO on similar topics)

My requirements:

One way syncing. Meaning my server is always updated and my clients just merely need to be fed incremental deltas which they feed into their core data stack.

The server could arbitrarily insert new records, modify existing ones or even delete them.

I would rather not use RestKit because I can do the work of making queries and updating local state myself. I don't need an extra library to do this.

My solution:

I keep a last updated date (UTC) locally. I send this date to my server periodically and my server responds with the data I need to update or insert, based on a server timestamp for each record.

My problem:

I don't know how to handle deletes.

How would I handle deletes in such a scenario? If I delete the record from the server, how will clients know to delete that record as well.

Thoughts? I can think of a number of complicated ways to do it (e.g. don't really delete the record, or keep track separately of deleted records, etc.), but I'm looking for the simplest.

答:
Your solution is good but as you found, you need to keep the deleted record somewhere to inform the clients.

Another solution is to require that the clients "registers" to your server. Then keep a queue of all changes per client on the server. It consumes more space on the server side but can keep track of all changes and not only the last state of a record.


问:
My boss has a bad case of "Not Invented Here"
My department specializes in converting customer data into our database schema so that they can use our software.

Right now, we have C# applications that take an IDataReader (99% of the time it is a SqlDataReader), perform some cleaning and mapping, insert it into a DataRow object, and then use a SqlBulkCopy to insert it in to our database.

Sometimes (especially when the source database contains images as varbinary objects), this process can really bog down with a SQL transfer from the server to the app to just then turn right around and go back to the server.

I feel that if we re-wrote some of the conversions as SSIS packages it could speed things up a lot. However the biggest stonewall I keep running into is when my boss, in Not Invented Here fashion, pushes back and says "What if Microsoft drops support for SSIS? We will have all this obsolete code and we will be screwed."

This is not the first time that I have hit a "What if they remove that feature...?" answer from my boss. I don't have the time to write the conversion the old way, self-teach myself SSIS, and also write it the new way to demonstrate/test the benefits (None of us have used SSIS so there would be a period where we would have to learn how to use it).

What should I do in this situation? Stop pushing the new technology? Wait till he leaves the department (I am the 2nd most Sr. person in the department after him, but it could be years before he quits/retires)? Find a new way of getting him to stop being afraid of 3rd party tools?

答:
While there is merit to “don’t fix what is not broken” I disagree with the accepted answer.

The accepted answer comes from the perspective that problem is not broken. From a mid-level management's perspective this is true. If a cost analysis were to be done on the time spent by developers to create and maintain an ETL solution in C# compared to buying an out-of-the-box solution, it would show the C# solution takes 3 to 4 times longer to create and maintain and cost as much as 10 times more (I looked for the reference on the numbers, but I could not find it).

Unless it’s the company’s core competency there is very little reason to write and maintain data transformations in C#. The homegrown solution will be slow, there will be mistakes (i.e. corrupt data), there will be edge cases, there will be little reuse between ETL classes, and it will be fragile. Honestly, what developer wants to spend his/her days writing and maintaining ETL in C#? I’ve done it. It’s a load of crap. It’s about as far away from creative work as one can get.

Use a tool like Informatica, a company whose business is ETL. They’ve been working this problem for over 15 years. They have it solved. Their tools are drag and drop, reusability is built-in, as is performance. Most anyone (i.e. developers don’t have too) can create ETL with Informatica tooling. I’m not trying to sell Informatica, use any tool. Just don’t re-invent the wheel.

In the long run buying a tool, such as Informatica or using SSIS will save the company potentially millions over the long-haul.  And best of all you will not have to maintain the ETL or be responsible for it when it breaks.


问:
What non-programming books should a programmer read to help develop programming/thinking skills?
There are a lot of questions about what programming books should be on the programmer's bookshelf. How about non-programming related books that can help you become a better programmer or developer? It would also be interesting to know why they would help.

My first choice would be Sun Tzu's "Art of War" (however cliché), because it made it obvious that the success of any project depends on the strength of its weakest link (and warfare is a big project).

答: The Inmates are Running the Asylum has been helpful to me during the planning stage of any software, or interaction project.</p>


问:
Do I need to understand algorithms and data structures to be called a programmer?
It has been six years since I have been coding. Coding into all kinds of things like ActionScript, JavaScript, Java, PHP, Ajax, XML HTML, ASP, etc. I have used arrays, maps, linked lists, sets, etc and wherever I worked people like me. But whenever I am interviewed it is very likely that people ask me question on hashes, trees, stacks, and queues. Some questions are on juggling some sorting algorithms.  I don't know if I should really know them or should I stop calling myself a programmer. There is something in me which also tells me even if people who are asking all these questions select me, they will never be making me work of these things. Am I really required to know all this?</p>

答: If all you know how to do is write glue code, you may call yourself a code monkey.  Lots of glue code needs to be written and you can make a decent living as a code monkey.  To call yourself a Real Programmer TM and be trusted when code needs to be written from scratch, you have to know algorithms, data structures, memory management, pointers, assembly language, etc. and understand how to use this knowledge to evaluate tradeoffs.


问:
LGPLv3 in closed embedded environment
I'm thinking of using a LGPLv3 library in a closed source commercial program running in a closed embedded environment.

According to what I know from LGPLv3 licensing, I can use it as long as the closed source software dynamically links with the LGPLv3 library and leaves the freedom to the user to change or update the LGPLv3 library by changing the library binaries.

Since my software runs in a closed, controlled, validated environment, it would be really odd to respect the part of allowing the user to change the LGPLv3 library.
I could add a functionality to my environment allowing to update the library binaries, but it would be odd to present this functionality to our customers since they would have no interest into changing that library and would see that as a security hole, so they'll want us to remove it (or restrict the access to no one).

Is adding an update binary functionality the only way to respect LGPLv3 ? If by using it, I require to call my company to give access to it (by a password), Will I still respect the license ?

答:
I am, of course, not a lawyer, so this is just my reading of the relevant licenses.

Now, in the LGPL, we're talking section 4.d.1, "Use a suitable shared library mechanism", and 4.e, "Provide Installation Information".

Now, 4.e says you have to provide installation info per section 6 of the GPL v3.

First off, can your device be upgraded? If it can't, then the 'Installation instructions' are moot and you don't have to do anything - GPL, section 6, third paragraph from the end

"But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM)."

Assuming your device is upgradable (and so few devices aren't anymore) what happens if your device doesn't fit the definition of a 'User Product'? And is your device a User Product?

Because if I'm reading section 6 of the GPL v3 correctly, you only have to provide Installation Instructions for consumer goods - stuff sold to consumers, or for use in houses. My reading of this is that industrial stuff (for instance, the medical monitors that my company makes) wouldn't be covered by the 'Installation Instructions' requirement.

If my reading is correct, and if you aren't making consumer goods, then you're in the clear. Obviously you'd want to check that with a lawyer, but I suspect you are good without it.

Title: LGPLv3 in closed embedded environment


问:
Given that I voluntarily choose not to implement virtual memory in my kernel, how would the memory-management-unit work?
For my basic kernel I refuse to implement the dreaded confusion of virtual memory scheming, so I want only real memory addresses for everything. Some people have argued with me that since virtual memory is hardware supported, and sometimes the MMU hardware is integrated with the CPU, is that a rule that virtual memory is mandatory by hardware-design?

I have at least one good reason to not care about virtual memory at this point, being that the real memory I have is plenty (32 GB), and that I don't care to go through trouble and workarounds with virtual addressing and such(and that I honestly don't like the idea, regardless of the fact that it helps in many areas).

So if my question wasn't clear, is it possible to write a dedicated kernel and not at all implement virtual memory and use only real addresses for everything, with a MMU?

(Some of these low-level programming and hardware-concepts are mind-boggling, even after hours and hours of studying and implementing).

PS: I'm guessing the answer to some point is "yes", but hearing someone with more knowledge on the subject would definitely be a good pointer for me and others who read this.

答:
It's fairly common for embedded kernels not to use virtual memory, because protecting processes' memory spaces from each other isn't as important, and you generally have a relatively small amount of memory.  In that case, the MMU is primarily used for protecting appropriate parts of the address space as read only, so you get segfaults when you want them.

With 32GB of RAM, though, you'd require a 64-bit machine to address it all natively, or some other paging scheme.  Also, you still have to understand the MMU in order to initialize it properly, so it might not save you that much effort.  Unless you're building a dedicated machine that only runs a handful of processes, I would highly recommend biting the bullet and implementing virtual memory.


答:
Although 20c and Ant have both proposed reasonable solutions, I really didn't want to maintain any extra state such as a deleted flag or per-client data. It just becomes too complicated for my taste.

The solution that I'm going to implement for deletion will simply be the following:

Client issues a delete request. This request can only be made once update/inserts are up to date.

Server responds with the primary key (a 4 byte integer) of every single record. This response is also compressed to save space. For reference, 40k records worth of primary keys compressed is about 20k bytes.

Client then does a cross reference and deletes records that have primary keys which the server did not include.

I just wanted to throw this out there is a viable solution as well.

Title: Core data syncing (deletion) using a custom REST web service

Body:
I need to sync my local core data stack against a server running a persistent data storage underneath the covers. (I've gone through other posts here and on SO on similar topics)

My requirements:

One way syncing. Meaning my server is always updated and my clients just merely need to be fed incremental deltas which they feed into their core data stack.

The server could arbitrarily insert new records, modify existing ones or even delete them.

I would rather not use RestKit because I can do the work of making queries and updating local state myself. I don't need an extra library to do this.

My solution:

I keep a last updated date (UTC) locally. I send this date to my server periodically and my server responds with the data I need to update or insert, based on a server timestamp for each record.

My problem:

I don't know how to handle deletes.

How would I handle deletes in such a scenario? If I delete the record from the server, how will clients know to delete that record as well.

Thoughts? I can think of a number of complicated ways to do it (e.g. don't really delete the record, or keep track separately of deleted records, etc.), but I'm looking for the simplest.


答:
Since you don't take Karl's answer, I will try to answer your question: with no virtual memory, a useful MMU will be based on memory segmentation, pretty much like what the DOS did in the old (good) days - you allocate memory in segments and addresses are always local to the corresponding segments. Another approach is to go with no MMU at all (for embedded systems) - all processes uses a pre-allocated address schema and you will quickly find out 32G is not big enough.

Again, as Karl suggested, you should bite the bullet.

Added more detail to Robert's comment

For a system that needs to dynamically load and unload modules, the free physical memory is always dynamic. To a module that assumes direct physical memory access, there is a hard way to locate it's data and code: when the module is loaded, scan the machine code and rewrite all pointers to the allocated addresses. I have not read anything in detail about this technique but it just sounds pain to me (while windows does this for dlls). An easier way to address this is to use local addresses to segments - the location of a segment may change but the address relative to its beginning never changes.

Whether segmentation is supported in modern computer is not relavent, it's just an memory management solution that is easy to implement.

Title: Given that I voluntarily choose not to implement virtual memory in my kernel, how would the memory-management-unit work?

Body:
For my basic kernel I refuse to implement the dreaded confusion of virtual memory scheming, so I want only real memory addresses for everything. Some people have argued with me that since virtual memory is hardware supported, and sometimes the MMU hardware is integrated with the CPU, is that a rule that virtual memory is mandatory by hardware-design?

I have at least one good reason to not care about virtual memory at this point, being that the real memory I have is plenty (32 GB), and that I don't care to go through trouble and workarounds with virtual addressing and such(and that I honestly don't like the idea, regardless of the fact that it helps in many areas).

So if my question wasn't clear, is it possible to write a dedicated kernel and not at all implement virtual memory and use only real addresses for everything, with a MMU?

(Some of these low-level programming and hardware-concepts are mind-boggling, even after hours and hours of studying and implementing).

PS: I'm guessing the answer to some point is "yes", but hearing someone with more knowledge on the subject would definitely be a good pointer for me and others who read this.


答:
You can certainly do that, but it may not save you as much complexity as you think.

One of the main benefits of virtual memory is that it keeps different processes from needing to know which parts of memory other processes are using.

On a system with virtual memory, one of the main jobs of the MMU is to give each process the illusion that the whole machine is theirs. As an example, if you have three programs you want to run on your OS, each can be compiled to start at memory address 0 -- and none of them has to be changed if you want to run all three at the same time. You are right, however, that this adds complexity to the kernel.

If you decide to save this complexity by making it the program's job to not use the same addresses used by other programs, there are two main ways you can go:

One option is to make each program use a different range of memory -- if program1 only uses addresses 0 .. 8191, and program2 only uses addresses 8192 .. 16383, then (barring a bug), they won't interfere with each other, but -- and this is a big deal -- you need to plan in advance what range of memory each program will use, so you need to know beforehand all the programs which will run on your OS. This may be workable for an embedded system (and may be the only choice there), but is obviously a show stopper for an OS which you hope other people will build software for.  Historically, a method like this was also used for some early shared library implementations, and proved very hard to get right.

Another option is to compile each program using what's called position independent code -- code where no absolute addresses are compiled into the program, and all access to variables and code within the program is done by first checking a register (possibly the program counter) for a value pointing to where the program actually is in memory. This allows multiple programs to coexist, but requires the compiler and linker to do more work to make the program work this way, requires the operating system to do more work when loading each program, and has a cost in performance every time access to a variable or function has to be computed using this method.  Historically, some operating systems, including classic MacOS before version 7.1, have taken this approach, and something like this is still used in modern shared library implementations.

So, you can move the complexity from the kernel to each running program, or from the kernel to your record-keeping of which program gets loaded where -- but you can't get rid of it altogether.


答:
An MMU plays two major roles: it translates virtual addresses into physical addresses, and it allows the possibility of trapping (also known as faulting and various other names), i.e. executing code (in the kernel context) when a process dereferences certain virtual addresses.

There is a third related capability, that of having different mappings and different traps for different processes. This is a consequence of executing code to change MMU settings during a context switch. This doesn't require anything special about the MMU (but most processor designs have features in the MMU to make this more efficient).

Some processors (typically high-end microcontrollers, such as the ARM Cortex-M3) do not support translations but nonetheless support trapping: they have a memory protection unit (MPU). An MPU allows the kernel to prevent certain addresses from being accessed. Thus you can enforce memory separation between processes at runtime with only an MPU, you don't need an MMU.

If you have an MMU, you can use it as an MPU. It's up to you to design your kernel such that every virtual address is either always mapped to the corresponding physical address or unmapped. (An unmapped address triggers a trap.)

There is a version of the Linux kernel that supports processors with an MPU but no MMU: µClinux. The code is integrated in the Linux source tree under nommu architectures. The lack of an MMU brings some limitations, in particular the impossibility of implementing fork.

An important thing that you give up if you have no virtual memory is the ability to load every program in the same address space. This makes it considerably easier to compile programs. For example, with virtual memory, you can choose to always load the code of a program at a certain predefined address, store global variables at a predefined address, etc. If every process is stored at a different address, you need to generate position-independent code all the time.

Another limitation of having no virtual memory is that memory management becomes more difficult, because of fragmentation. Suppose you have four consecutive physical memory pages of 4kB each, numbered 0,1,2,3, and you first allocate four page-sized objects, one in each page; then you free the objects in pages 1 and 3. With virtual memory, you can allocate an 8kB object made of pages 1 and 3 by mapping them at consecutive addresses. With a direct physical mapping, page 1 is lost until you need a single-page object or the surrounding objects are freed.

If you want, you can use your processor without activating its MMU. All processors boot with the MMU turned off, because you can't do anything useful until the MMU tables have been initialized to something sensible. With no MMU, you can't enforce any isolation between processes. You also can't swap, or have memory-mapped files.

With address 0 mapped, take care if you program in C or most other languages. There are a lot of runtime environments and non-fully-portable C code that assume that an address of all-bits-zero is the NULL pointer, which does not point to any object.

Depending on the architecture, there may be no way to access some hardware without mapping it into memory. This requires having the MMU up and going. You can still pretty much ignore the MMU by mapping the physical RAM from address 0, mapping devices above the last RAM address, and never changing the MMU mapping.

All in all, by eschewing virtual memory, you're simplifying some parts of the kernel, but making it a lot more difficult to write programs that do anything useful. There's a reason why all high-end processors have an MMU: it's difficult to do anything sophisticated without it. MMU management is not the most difficult part of writing a kernel if you keep it simple; if you find it “mind-boggling”, you have a lot to learn about low-level programming. Writing a kernel with MMU management would be a very good learning exercise.


答:
You mention hashes, trees, stacks, queues and sorting algorithms.  Well, the technologies you've mentioned are mostly webpage and web-scripting related.  You should definitely understand trees at the very least, so you can work with the DOM well.  But if scripting is all you need to do, then you're probably OK.  You won't need most of the tools of the trade for a real programmer.  But that's because there's a big difference between the string-juggling that makes up most of Web scripting and what most of us consider "writing programs."

I work with hashes and trees practically on a daily basis, and stacks and queues less often but frequently enough.  Sorting is basically a solved problem; just about any language has a quicksort built into the standard library, a Sort method on basic collection types, etc, but you should know under what circumstances quicksort's performance can degrade severely and the right strategies to delay sorting.

If I didn't know these principles and how they work, I could probably hack up coding solutions that work, but they wouldn't be very good quality solutions.  They'd run slowly, be hard to read, and hard to modify, reuse or extend.  So if you want to learn to be a good programmer, you should definitely read up on your algorithms and data structures.  They'll really improve the quality of your code.

Title: Do I need to understand algorithms and data structures to be called a programmer?

Body: It has been six years since I have been coding. Coding into all kinds of things like ActionScript, JavaScript, Java, PHP, Ajax, XML HTML, ASP, etc. I have used arrays, maps, linked lists, sets, etc and wherever I worked people like me. But whenever I am interviewed it is very likely that people ask me question on hashes, trees, stacks, and queues. Some questions are on juggling some sorting algorithms.  I don't know if I should really know them or should I stop calling myself a programmer. There is something in me which also tells me even if people who are asking all these questions select me, they will never be making me work of these things. Am I really required to know all this?</p>


答:
What if we flip this around and talk about how the client and server interacts and not necessarily which has more code. I've seen this called (and this is a phrase I like):

API-centric

Title: Client-side oriented website approach name (less server-side code, more client-side code)

Body:
I have a terminology question.

I was searching about tips to design a website that is using less server-side code and more client-side code, and realized I don't really know which exact term to search if there is one. Example: Maximizing CDNs usage for library, using widgets when possible, integration of client-side social-media SDKs.

So my question is not about tips themselves, but about the existence of such a term.

Edit:
I already know about SPAs and Client-Side Javascript Frameworks. This is what brings me to wonder if there's a name to an approach where you focus on client-side code at a point that you maximize external resources.


答:
Personally, I like APIs that:

are optimized in a way that common use cases can be easily performed

calls related to CRUD operations are batch oriented or have batch versions

allows reflection (so people writing plugins or bindings for other computer languages can automate the process)


问:
R vs Python for data analysis
I have been programming for about a year and I am really interested in data analysis and machine learning. I am taking part in a couple of online courses and am reading a couple of books.

Everything I am doing uses either R or Python and I am looking for suggestions on whether or not I should concentrate on one language (and if so which) or carry on with both; do they complement each other?

-- I should mention that I use C# in school but am familiar with Python through self-study.

答:
Background: I'm a data scientist at a startup in Austin, and I come from grad school (Physics). I use Python day-to-day for data analysis, but use R a bit. I also use C#/.NET and Java (just about daily), I used C++ heavily in grad school.

I think the main problem with using Python for numerics (over R) is the size of the user community. Since the language has been around for ever, lots of people have done things that you're likely to want to do. This means that, when faced with a hard problem, you can just download the package and get to work. And R "just works": you give it a dataset, and it knows what summary statistics are useful. You give it some results, and it knows what plots you want. All the common plots you'd want to make are there, even some pretty esoteric ones that you'll have to look up on Wikipedia. As nice as scipy/numpy/pandas/statsmodels/etc. are for Python, they're not at the level of the R standard library.

The main advantage of Python over R is that it's a real programming language in the C family. It scales easily, so it's conceivable that anything you have in your sandbox can be used in production. Python has Object Orientation baked in, as opposed to R where it feels like kind of an afterthought (because it is). There's other stuff that Python does nicely too: threading and parallel processing are pretty easy, and I'm not sure if that's the case in R. And learning Python gives you a powerful scripting tool, too. There are also really good (free) IDEs for Python, much better ones if you're willing to pay (less than $100), and I'm not sure this is the case for R--the only R IDE I know of is R Studio, which is pretty good, but isn't as good as PyDev + Eclipse, in my experience.

I'll add this as a bit of a kicker: since you're still in school, you should think about jobs. You'll find more job postings for highly skilled Python devs than you will for highly skilled R devs. In Austin, jobs for Django devs are kind of falling out of the sky. If you know R really well, there are a few places where you'll be able to capitalize that skill (Revolution Analytics, for example), but lots of shops seem to use Python. Even in the field of data analysis/data science, more people seem to be turning to Python.

And don't underestimate that you may work with/for people who only know (say) Java. Those people will be able to read your Python code pretty easily. This won't necessarily be the case if you do all of your work in R. (This comes from experience.)

Finally, this may sound superficial, but I think the Python documentation and naming conventions (which are religiously adhered to, it turns out) is a lot nicer than the utilitarian R doc. This will be hotly debated, I'm sure, but the emphasis in Python is readability. That means that arguments to Python functions have names that you can read, and that mean something. In R, argument names are often truncated---I've found this less true in Python. This may sound pedantic, but it drives me nuts to write things like 'xlab' when you could just as easily name an argument 'x_label' (just one example)---this has a huge effect when you're trying to learn a new module/package API. Reading R doc is like reading Linux man pages---if that's what floats your boat, then more power to you. When I have a question about how something works in R, I avoid the R documentation, whereas I START with the Python doc when I'm confused about Python.

All of that being said, I'd suggest the following (which is also my typical workflow): since you know Python, use that as your first tool. When you find Python lacking, learn enough R to do what you want, and then either:

Write scripts in R and run them from Python using the subprocess module, or

Install the RPy module.

Use Python for what Python is good at and fill in the gaps with one of the above. This is my normal workflow---I usually use R for plotting things, and Python for the heavy lifting.

So to sum up: because of Python's emphasis on readability (search gooogle for "Pythonic"), the availability of good, free IDEs, the fact that it's in the C family of languages, the greater possibility that you'll be able to capitalize the skillset, and the all-around better documentation-style of the language, I'd suggest making Python your go-to, and relying on R only when necessary.

Ok, this is (by far) my most popular answer ever on a stack site, and it's not even #1 :) I hope this has helped a few people along the path.

At any rate, I've come to the following conclusion after several years in the field:

This is probably the wrong question to ask.

Asking "should I learn this particular technology" is a bad question. Why?

Technology changes. You'll always have to learn another technology. If you go work at Twitter, they run Scala. Some places are Python shops. Some places don't care. You're not going to be hired because you know or don't know some particular piece of tech--if you can't learn a new tech, you can (and should be) fired. It's like, if a new pipe wrench comes out, and you're a plumber, and you can't figure out how the new pipe wrench works, you're probably a pretty lousy plumber.

Given the choice of "Do I learn this technology" or "Do I spend more time solving real problems", you should always choose the latter, without exception.

As a data scientist, your job is to solve problems. That single bit of wisdom is pretty much always lost at every conference or meetup you go to--every "big data" talk I've ever seen has focused on tech, not on solving problems. The actual problem solving is usually relegated to a few slides at the end:

[Talk title = "Deep learning at Cool New Startup"]...[45 minutes of diagrams and techno-babel during which I zone out and check my phone]...And, after implementing our Hadoop cluster and [Ben zones out again] we can run our deep learning routine, [wake up: this is why I came!] the details of which are proprietary. Questions?

This gives a bad impression that the field is about tech, and it's just not true. If you're really good at Scala, or Python, or R, but you're really bad at solving problems you will make a lousy data scientist.

Paco Nathan was in Austin a few months ago at a day long "big data" conference, and said something like "Chemistry isn't about test tubes". That pretty much sums it up--data science isn't about Scala, or Hadoop, or Spark, or whatever-other-tech-du-jour pops up. At the end of the day, I want to hire people who think, not people who are adept at using Stack Overflow to learn toolkits.

Likewise, if you go to a job interview, and they don't hire you just because you don't know some programming language, then that company sucks. They don't understand what "data scientist" means, and it's probably better for you if it didn't work out.

Finally, if your problem solving abilities are marginal (be honest with yourself), or you really just enjoy the tech side of things, or learning tech is what you really love (again, be honest) then learn a lot of tech. You'll always be able to find "data engineer" type roles that fit your skill set. This isn't a bad thing, data engineers grease the wheels and make it possible for you to do your job as a data scientist. (The difference is akin to software architect vs. the development team.)

Title: R vs Python for data analysis


问:
Why exactly can't PHP have full unicode support?
Everybody knows, that PHP has problems with Unicode. Version 6 is effectively abandoned, because of Unicode implementation difficulties. But I wonder if anyone knows what are the exact reasons? Architecture/design problems, performance concerns, community problems (I bet not), something other?

答:
One of the primary reasons the old PHP 6 work was stopped was due to the internal complexity it brought and the amount of work to do, which barely anybody fully unerstood.

A bit of history: PHP 6's Unicode imlementation was designed by the need of a larger PHP user and tried to do Unicode "right". After some evaluation the primary designer of PHP's to-be-Unicode-support has choosen to add a new string type which internally is Utf-16 and to allow different encdings to be used in different places. So the code might be written in one encoding, the output might use a different encoding and "runtme operations" some other encoding. The reason for choosing UTF-16 was that the work should be based on the ICU livrary which uses UTF-16 and it was found that this encoding makes common string operations in a fast way while conversi between utf- and utf-16 is relatively cheap. So far so good.

Now the consequence of doing this is at foremost the introduction of a new string type. PHP's internal type system till then had a few types (NULL, bool, int/long, float/double, string, array, resource, object) and lots of code had some assumptions on this being the case. Besides such assumptions all functions operating on strings, and there are a lot of those, have to be evaluated individually and it has to be decided how to handle encodings. Should they work on binary strings or unicode strings? If a conversion is required which encoding should be used etc. and this is a lot of work and in some cases quite complicated to do right. Additionally the internal APIs became quite complicated, as most key APIs in PHP got versions for binary strings (the old one) and then often a version for "runtime encoded" strings, as well as utf-16 strings, creating quite a mess there ...

Over the process of doing that many developers stumbled over the coplexity, became annoyed by utf-16 and didn't like the fact that this would more than double memory usage and spend lots of time converting strings while breaking most existing applications. So, PHP being driven by volunteers, fewer and fewer developers were working on it and other things piled up and contributors became unhappy and in the end it had to be abandoned.

Now what might the future bring? - There is a slow evolution happening that more and more things in PHP ae built around utf-8. Not in a strong way with a custom type and forcing everything and currently the developrs aren't motivated to touch this hot iron. One can hope that somebody has a good proposal to make it work nicely, but currently "everybody" will run away if they only hear the word. :)


问:
Common Lisp Implementations - threading and multiplatform?
I'm learning Common Lisp, mostly as a "mind gym" hobby thing, but I want to end up with a set of skills that would also be usable "in real life", because when you learn a language you also accumulate knowledge about module and package management, deployment and stuff, whether you like it or not, so I want to learn with a CL implementation that is also usable in production. So my question is:

What CL implementation is most usable "in production", having attributes like these? :

it's multi-platform - one should be able to use the same language implementation and languages on a Linux server, a Windows dev machine and a MacOS one (the coolest thing would be being able to also compile for Android, Windows Phone and RT and maybe even iOS)

it has good concurrency, parallelism, multi-threading features - and these features work cross-platform, otherwise it doesn't count

it's easy to interface with other languages - working with C libraries is a must, but anything else, like an easy CL-Python interface would be nice

I insist on the multi-platform part because I know that one of the serious arguments for Reddit ditching CL (CMUCL - btw, I know SBCL is fork of CMUCL, but how do they compare?) was that developers couldn't just use their Macs to develop without being ssh'd into a dev server environment (quote: "On my Mac, my choices of threaded Lisp implementations was limited to OpenMCL, and in FreeBSD it's CMUCL."). Yeah, one can use a dev VM, but still...

Note #1: I was close to settling on CLISP but I don't know about its concurrency features, and I also came across this: "it all-but-forces your code to be released as GPL" (I can't understand what this means or how can a languages's license become forced on your code, but I'm not fluent in legalese...).

Note #2: I'm not really fond of the JVM, and if I were to use a Lisp on it it would be Clojure and nothing else, so no JVM Lisps suggestions, please :)

答:
Disclaimer:  I'm a (minor) contributor to SBCL, but I was a user first. As a happy user of CLISP, CMUCL, and SBCL, here's why SBCL is my "go to" CL implementation:

Steel Bank Common Lisp meets the criteria you mention:

Runs on MacOS, Windows, and Linux (among other platforms)

Provides an identical API for threading and concurrency control on all of these platforms

Has a comprehensive foreign function interface to allow interfacing with C

In addition, SBCL is a good Lisp environment in general:

it's a native Lisp -- no JVM, no bytecode

it's well supported by SLIME and Quicklisp

it's a mature and complete implementation

it's actively developed, and I've seen good turnaround when I've hit bugs


答:
You may well be a good programmer
   right now, but knowledge of Data
   Structure, Algorithms and knowledge
   of other topics in computer science
   would certainly help a lot in
   improving yourself in many ways:

You may be able
to be more effective and faster at
doing things. Even folks who already
have a Computer Science degree and
know a lot of these topics would tend
to keep themselves abreast with
latest advances in order to improve
themselves.

This knowledge will also be good for,
if upto a lesser extent, say if you
move from programmer to management
track later, because you would still
be able to understand the technical
aspects of projects better with this
knowledge.

Of course data structure and
algorithms are asked a lot in
interviews, so that is one more
reason why it might be useful to know
them.


问:
How should a developer reject impossible requirements?
Here's the problem I'm facing:

Quote From Project Manager:

Hey Spark, I'm assigning you the task of developing a framework that could be used for many different iOS applications. Here are the requirements:

It should be able to detect the thickness of the thumb or fingers being used to manipulate the UI.

With this information, all elements of the UI should be arranged & sized automatically.

For a larger thumb, elements should be arranged nearer the center of the screen.

For a smaller thumb, elements should be arranged nearer the corners of the screen.

For a larger thumb, all fonts should be smaller. (We're assuming an adult in this case.)

For a smaller thumb, all fonts should be larger. (We're assuming a younger person in this case.)

Summary:

This framework is required for creating user-friendly user interfaces programmatically. The framework should be developed in such a way that we can use for as many projects as needed, so it must also be very developer-friendly.

I am the developer given this task, so my questions are as follows:

How can I explain that these requirements are a little ridiculous?

How can I explain that it would be better to concentrate on developing actual projects?

How can I explain that even if this were possible, I wouldn't recommended developing such a thing?

How do I say NO to this project politely, gently, and respectfully?

How can I explain that even for a developer with 3 years of experience, this might not be possible?

答:
What I like to do, is to approach the problem from a team perspective. In this case the project manager and I need to come up with a solution. I also like to show that I come with an open mind. If I hear what I think are hearing impossibilities, there is a chance there is a misunderstanding between me and the project manager. Or perhaps we have not reached that aha moment.

Once we decide to take a leap and have an open conversation, I find it easiest treating the problem as an economic one. In technology almost anything is possible given an infinite amount of resources. Discussing what it would take to do, many times is the easiest way to respectfully "say no". And that would be the way to clear any possible misunderstandings. Although, to be honest, we are not quite saying no. And, what is the purpose of saying no anyway. If the work is not worth doing, it does not matter whether we want to do it or not.


问:
Should Version Control Softwares be used from Command Line or GUI
I was going through some tutorials on TortoiseHg. Despite having a rich GUI, the first examples are given using command line options.
Does the ideal usage involve command line or it was started that way so one has idea of what is happening under the hood and GUI internally uses this command anyway.</p>

答:
If you have a lot of repeated operations, such as
adding some but not all files

```<code>git add file1
git add file2
git add file3
...
git add fileN
</code>```

then GUI with checkboxes is definitely better.

If you need to enter hashes as command arguments, then GUI is definitely better (allow to select and no need to remember the hashes)

Body:
If you have a lot of repeated operations, such as
adding some but not all files

git add file1
git add file2
git add file3
...
git add fileN

then GUI with checkboxes is definitely better.

If you need to enter hashes as command arguments, then GUI is definitely better (allow to select and no need to remember the hashes)


答:
In addition to the excellent answer by Michael Kohne.

If by using it, I require to call my company to give access to it (by a password), Will I still respect the license ?

Yes, because the (L)GPL allows you to give a " written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model" for obtaining the (L)GPL-covered source code.

You can offer to send them a CD-ROM with the source code and installation instructions, along with the warning that if they do an unofficial update of the software, then the warranty becomes void.

Title: LGPLv3 in closed embedded environment

Body:
I'm thinking of using a LGPLv3 library in a closed source commercial program running in a closed embedded environment.

According to what I know from LGPLv3 licensing, I can use it as long as the closed source software dynamically links with the LGPLv3 library and leaves the freedom to the user to change or update the LGPLv3 library by changing the library binaries.

Since my software runs in a closed, controlled, validated environment, it would be really odd to respect the part of allowing the user to change the LGPLv3 library.
I could add a functionality to my environment allowing to update the library binaries, but it would be odd to present this functionality to our customers since they would have no interest into changing that library and would see that as a security hole, so they'll want us to remove it (or restrict the access to no one).

Is adding an update binary functionality the only way to respect LGPLv3 ? If by using it, I require to call my company to give access to it (by a password), Will I still respect the license ?


问:
Implementing the command pattern in a RESTful API
I'm in the process of designing an HTTP API, hopefully making it as RESTful as possible.

There are some actions which functionality spreads over a few resources, and sometime needs to be undone.

I thought to myself, this sounds like a command pattern, but how can I model it into a resource?

I will introduce a new resource named XXAction, like DepositAction, which will be created through something like this

```<code>POST /card/{card-id}/account/{account-id}/Deposit
AmountToDeposit=100, different parameters...
</code>```

this will actually create a new DepositAction and activate it's Do/Execute method.
In this case, returning a 201 Created HTTP status means the action has been executed successfully.

Later if a client wishes to look at the action details he can

```<code>GET /action/{action-id}
</code>```

Update/PUT should be blocked I guess, because it is not relevant here.

And in order to Undo the action, I thought of using

```<code>DELETE /action/{action-id}
</code>```

which will actually call the Undo method of the relevant object, and change it's status.

Let's say I'm happy with only one Do-Undo, I don't need to Redo.

Is this approach ok?

Are there any pitfalls, reasons not to use it?

Is this understood from the POV of the clients?

答:
The main reason for REST existence is resilience against network errors. To which end all operations should be idempotent.

The basic approach seems reasonable, but the way you describe the DepositAction creation does not sound to be idempotent, which should be fixed. By having client provide unique ID that will be used to detect duplicate requests. So the creation would change to

```<code>PUT /card/{card-id}/account/{account-id}/Deposit/{action-id}
AmountToDeposit=100, different parameters...
</code>```

If another PUT to the same URL is made with the same content as previously, the response should still be 201 created if the content is the same and error if the content is different. This allows the client to simply retransmit the request when it fails, since the client can't tell whether the request or response got lost.

It makes more sense to use PUT, because it just writes the resource and is idempotent, but using POST wouldn't really cause any problem either.

To look at the transaction details the client will GET the same URL, i.e.

```<code>GET /card/{card-id}/account/{account-id}/Deposit/{action-id}
</code>```

and to undo it, it can DELETE it. But if it actually has anything to do with money as the sample suggests, I would suggest PUTting it with added "cancelled" flags instead though for accountability (that there remains trace of created and cancelled transaction).

Now you need to choose a method of creating the unique id. You have several options:

Issue client-specific prefix earlier in the exchange that must be included.

Add a special POST request to get blank unique ID from the server. This request does not have to be idempotent (and can't, really), because unused IDs don't really cause any trouble.

Simply use UUID. Everybody uses them and nobody seems to have any problem with neither the MAC-based ones nor the random ones.

Body:
The main reason for REST existence is resilience against network errors. To which end all operations should be idempotent.

The basic approach seems reasonable, but the way you describe the DepositAction creation does not sound to be idempotent, which should be fixed. By having client provide unique ID that will be used to detect duplicate requests. So the creation would change to

PUT /card/{card-id}/account/{account-id}/Deposit/{action-id}
AmountToDeposit=100, different parameters...

If another PUT to the same URL is made with the same content as previously, the response should still be 201 created if the content is the same and error if the content is different. This allows the client to simply retransmit the request when it fails, since the client can't tell whether the request or response got lost.

It makes more sense to use PUT, because it just writes the resource and is idempotent, but using POST wouldn't really cause any problem either.

To look at the transaction details the client will GET the same URL, i.e.

GET /card/{card-id}/account/{account-id}/Deposit/{action-id}

and to undo it, it can DELETE it. But if it actually has anything to do with money as the sample suggests, I would suggest PUTting it with added "cancelled" flags instead though for accountability (that there remains trace of created and cancelled transaction).

Now you need to choose a method of creating the unique id. You have several options:

Issue client-specific prefix earlier in the exchange that must be included.

Add a special POST request to get blank unique ID from the server. This request does not have to be idempotent (and can't, really), because unused IDs don't really cause any trouble.

Simply use UUID. Everybody uses them and nobody seems to have any problem with neither the MAC-based ones nor the random ones.


问:
How can I clone an svn repository that changed layouts with git maintaining full history?

Problem

I want to clone an svn repo with git-svn.
The problem is the subversion repository changed layouts at r1235.

Specifics

From revision 1 to 1234, foo used the standard svn folder layout with:

/trunk/src

/tags/version/src

At revision 1235, the author added bar to the repo. To do so they restructured the master repository making the folder layout look like this:

/trunk/foo/src

/trunk/bar/src

/tags/version/foo/src

/tags/version/bar/src

Goal

I want a new git repository that has the history of foo as if it was never rearranged to include bar.

Relevant Documentation

Working with non standard layouts is discussed in git-svn's documentation.

If the entire history in unnecessary (or if I just want bar), I can get what I want from r1235 on:

git svn init --stdlayout http://path.to/repo

edit repo/.git/config adding /foo to relevant paths (or /bar)

`

```<code>[svn-remote "svn"]
   url = http://path.to/repo
   fetch = trunk/foo:refs/remotes/trunk
   branches = branches/*/foo:refs/remotes/*
   tags = tags/*/foo:refs/remotes/tags/*
</code>```

`

git svn fetch -r 1235:HEAD

My attempts

git svn clone -r 1:1234 --stdlayout http://path.to/repo

edit repo/.git/config as above

git svn fetch -r 1235:HEAD

Result

The last command seems to succeed because r1235 - HEAD get downloaded. However, git log only shows commits up to 1234. The working directory also seems to be the old revision.

答:
You did just git svn fetch, so you now have to check out the latest revision.

Any of git svn rebase (hm, I am least certain with this one after the path change, but normally it's what you use; it includes fetch), git reset --hard trunk (forcibly discards local changes), git rebase trunk (re-applies committed local changes) or git merge --ff-only trunk (check there are no local changes) should do and since you don't have local changes yet are equivalent.


问:
Why has C prevailed over Pascal?
My understanding is that in the 1980s, and perhaps in the 1990s too, Pascal and C were pretty much head-to-head as production languages.

Is the ultimate demise of Pascal only due to Borland's neglect of Delphi? Or was there more, such as C being a more robust language? If the latter, what were the perceived advantages of C over Pascal?

I'm interested in historical facts and observations one can back up, rather than likes and dislikes.

答:
Pascal only ever became popular in a single rather limited environment PC/DOS.

Even then there were as many MicroFocus COBOL applications running on PCs as there were pascal applications.

C was the basic of the UNIX operating system and all the MS/Windows operating systems.

The combination of efficient execution on limited hardware, and, native access to the underlying OS and GUI libraries were probably the main reasons for C's success. Pascal never really hacked it on windows, and, Delphi arrived too late to make a difference.

ContentLicense: CC BY-SA 4.0

Title: Why has C prevailed over Pascal?


问:
Is using unit tests to tell a story a good idea?
So, I have an authentication module I wrote some time ago. Now I'm seeing the errors of my way and writing unit tests for it. While writing unit tests, I have a hard time coming up with good names and good areas to test. For instance, I have things like

RequiresLogin_should_redirect_when_not_logged_in

RequiresLogin_should_pass_through_when_logged_in

Login_should_work_when_given_proper_credentials

Personally, I think it's a bit ugly, even though it seems "proper". I also have trouble differentiating between tests by just scanning over them (I have to read the method name at least twice to know what just failed)

So, I thought that maybe instead of writing tests that purely test functionality, maybe write a set of tests that cover scenarios.

For instance, this is a test stub I came up with:

```<code>public class Authentication_Bill
{
    public void Bill_has_no_account()
    { //assert username "bill" not in UserStore
    }
    public void Bill_attempts_to_post_comment_but_is_redirected_to_login()
    { //Calls RequiredLogin and should redirect to login page
    }
    public void Bill_creates_account()
    { //pretend the login page doubled as registration and he made an account. Add the account here
    }
    public void Bill_logs_in_with_new_account()
    { //Login("bill", "password"). Assert not redirected to login page
    }
    public void Bill_can_now_post_comment()
    { //Calls RequiredLogin, but should not kill request or redirect to login page
    }
}
</code>```

Is this a heard of pattern? I've seen acceptance stories and such, but this is fundamentally different. The big difference is that I'm coming up with scenarios to "force" the tests out. Rather than manually trying to come up with possible interactions that I'll need to test. Also, I know that this encourages unit tests that don't test exactly one method and class. I think this is OK though. Also, I'm aware that this will cause problems for at least some testing frameworks, as they usually assume that tests are independent from each other and order doesn't matter(where it would in this case).

Anyway, is this an advisable pattern at all? Or, would this be a perfect fit for integration tests of my API rather than as "unit" tests? This is just in a personal project, so I'm open to experiments that may or may not go well.

答:
Yes, it is a good idea to give your tests names of the example scenarios you are testing. And using your unit testing tool for more than just unit tests maybe ok, too, lots of people do this with success (me too).

But no, it is definitely not a good idea to write your tests in a fashion where the order of execution of the tests matters. For example, NUnit allows the user to select interactively which test he/she wants to be executed, so this will not work the way intended any more.

You can avoid this easily here by separating the main testing part of each test (including the "assert") from the parts which set your system in the correct initial state. Using your example above: write methods for creating an account, logging on and post a comment - without any assert. Then reuse those methods in different tests. You will also have to add some code to the [Setup] method of your test fixtures to make sure the system is in a properly defined initial state (for example, no accounts so far in the database, noone connected so far etc.).

EDIT: Of course, this seems to be against the "story" nature of your tests, but if you give your helper methods meaningful names, you find your stories within each test.

So, it should look like this:

```<code>[TestFixture]
public class Authentication_Bill
{
    [Setup]
    public void Init()
    {  // bring the system in a predefined state, with noone logged in so far
    }

    [Test]
    public void Test_if_Bill_can_create_account()
    {
         CreateAccountForBill();
         // assert that the account was created properly
    }

    [Test]
    public void Test_if_Bill_can_post_comment_after_login()
    {
         // here is the "story" now
         CreateAccountForBill();
         LoginWithBillsAccount();
         AddCommentForBill();
        //  assert that the right things happened
    }

    private void CreateAccountForBill()
    {
        // ...
    }
    // ...
}
</code>```

Body:
Yes, it is a good idea to give your tests names of the example scenarios you are testing. And using your unit testing tool for more than just unit tests maybe ok, too, lots of people do this with success (me too).

But no, it is definitely not a good idea to write your tests in a fashion where the order of execution of the tests matters. For example, NUnit allows the user to select interactively which test he/she wants to be executed, so this will not work the way intended any more.

You can avoid this easily here by separating the main testing part of each test (including the "assert") from the parts which set your system in the correct initial state. Using your example above: write methods for creating an account, logging on and post a comment - without any assert. Then reuse those methods in different tests. You will also have to add some code to the [Setup] method of your test fixtures to make sure the system is in a properly defined initial state (for example, no accounts so far in the database, noone connected so far etc.).

EDIT: Of course, this seems to be against the "story" nature of your tests, but if you give your helper methods meaningful names, you find your stories within each test.

So, it should look like this:

[TestFixture]
public class Authentication_Bill
{
    [Setup]
    public void Init()
    {  // bring the system in a predefined state, with noone logged in so far
    }

    [Test]
    public void Test_if_Bill_can_create_account()
    {
         CreateAccountForBill();
         // assert that the account was created properly
    }

    [Test]
    public void Test_if_Bill_can_post_comment_after_login()
    {
         // here is the "story" now
         CreateAccountForBill();
         LoginWithBillsAccount();
         AddCommentForBill();
        //  assert that the right things happened
    }

    private void CreateAccountForBill()
    {
        // ...
    }
    // ...
}