问:
How to test code and functionality effectively while working as a freelancer?
I work as an independent developer. I find it difficult to test functionality of a feature and test code. As project goes complex, I loose focus resulting in each feature partially developed. I want to know how to write test cases, manage smooth workflow and handle features effectively and while remain focused?</p>

答:
Staying focused while working on your own, be it personal projects or freelancing, is certainly a challenge. Here are some tips for what I'd try to consider (and use myself whenever I'm dealing with these kind of projects):

Try to mix boring tasks with more interesting tasks. If you find yourself doing a lot of CRUD tasks and alike, consider distributing them more over time.

Keep a backlog of work items, and one backlog of bugs, preferrably prioritized.

Don't aim for perfection. Build a basic version of the application first, and then improve it.

Try to avoid distractions. Go working to a room which is seldom visited. Use the time when you're alone in the house, perhaps early mornings when everybody but you has already left or even better evenings when everybody but you is already asleep.


答:
In addition to what Andiaz said:

Increase modularity in your project.

Keep doin the unit-testing in parallel to your development work

Use a versioning software like they do in big companies. (Dont know if these softwares have free versions)

Decide and follow your own procedure for development process. Keep doin reviews.

One suggestion: Read some Software Eng. book, like Pressman.

ContentLicense: CC BY-SA 3.0

Title: How to test code and functionality effectively while working as a freelancer?

Body: I work as an independent developer. I find it difficult to test functionality of a feature and test code. As project goes complex, I loose focus resulting in each feature partially developed. I want to know how to write test cases, manage smooth workflow and handle features effectively and while remain focused?</p>


问:
Do I have to write a lot of boilerplate code if I keep working using Java?
I'm working for a company writing ERP applications. My problem is that I have to write tons of boilerplate code. I came up with ideas to automatize/prevent the drudgery but only some of them were accepted. I have been told by the lead developer that my ideas tend to be go far afield and I should write code everyone can understand.

I had a discussion about this lately and it seems to me that this kind of code ramp is within java's philosophy. I have to write lots of code to achiveve simple things not because it is necessary but because this is the way most of the people at the company think.

Is this universally applicable to most of the companies out there using java or this is just my company's view? Do I have to get used to the drudgery if I keep working for java-based firms?

答:
It's one thing to invent a new language that gets translated into the target language (think coffeescript->javascript) and a completely different thing to have a generator that only creates the first version of a source file.

The former is expensive and difficult to make, and should be left to people who know how to design and implement languages.

The later is helpful in environments where lots of redundant boilerplate code is required, and usually it's not very difficult to design and make. Using such a tool generates boilerplate code that looks like it was written by hand, just a bit more consistent and typo-free. Once generated, the source file is further worked on, edited, adapted, extended etc. The input-file for the generator is one-off; no attempt is made to allow round-trips etc.

To answer your question: Go for the second option, don't bother trying the first.


问:
Dealing with the customer / developer culture mismatch on an agile project
One of the tenets of agile is ...

Customer collaboration over contract negotiation

... another one is ...

Individuals and interactions over processes and tools

But the way I see it, at least when it comes to interaction with the customer, there is a fundamental problem:

How the customer thinks is different to how a software engineer thinks

That may be a bit of a generalisation, yes.  Arguably, there are business domains where this is not necessarily true---these are few and far between though.  In many domains though, the typical customer is:

Interested in daily operational concerns--short-range tactics ... not necessarily strategy;

Understandably, only concerned with the immediate solution;

Practical thinkers, not abstract thinkers;

Much more interested in "getting the job done" than considering how the solution will support future concerns.

On the other hand, in the ideal, software engineers who practice agile are:

People who think alot about quality;

Individuals who appreciate how a little bit of upfront work can save a ton in effort down the line;

Experienced, analytical thinkers.

So there seems to be a culture discrepancy that tends to inhibit "customer collaboration".

What's the best way to address this?

答:
Remember that agile doesn't mean that the customer is involved in daily standups or some of the other day-to-day aspects of agile. Agile, from the customer perspective, is about communication. That doesn't mean they are communicating with engineers about implementation details.

Customers collaborate with the product owner to get and give constant feedback. The topic is features, but not how they are implemented. Are you delivering the proper features? Are you on schedule? Do they have changing requirements you need to adapt to?

Agile helps you and your customer answer those questions.


问:
Why is verbosity bad for a programming language?
I have seen many people around complaining about verbosity in programming languages. I find that, within some bounds, the more verbose a programming language is, the better it is to understand. I think that verbosity also reinforces writing clearer APIs for that particular language.

The only disadvantage I can think of is that it makes you type more, but I mean, most people use IDEs which do all the work for you.

So, What are the possible downsides to a verbose programming language?

答:
If the language verbosity detracts from code readability, then it is bad.

Some languages have such verbose syntax that grasping the meaning of the code takes longer (I am thinking about VB.NET versus C#):

```<code>If something Then
End If

if(something)
{}
</code>```

It really boils down to what a coder is familiar and comfortable with.

Body:
If the language verbosity detracts from code readability, then it is bad.

Some languages have such verbose syntax that grasping the meaning of the code takes longer (I am thinking about VB.NET versus C#):

If something Then
End If

if(something)
{}

It really boils down to what a coder is familiar and comfortable with.


答:
At the end of the day, anything which is 'different' will cause people to howl. I am comfortable with C style syntax and hence ok with other languages which share similar syntax (C++, Java, C#). So i tend to favour this particular style.

Languages tend to find a balance between descriptive enough, and not verbose.

Perl is an example of where you can write very concise code. For the uninitiated, code written by a Perl ninja is nothing short of magic.

COBOL is an example of too verbose.

Title: Why is verbosity bad for a programming language?

Body:
I have seen many people around complaining about verbosity in programming languages. I find that, within some bounds, the more verbose a programming language is, the better it is to understand. I think that verbosity also reinforces writing clearer APIs for that particular language.

The only disadvantage I can think of is that it makes you type more, but I mean, most people use IDEs which do all the work for you.

So, What are the possible downsides to a verbose programming language?


答:
First off, you can have boilerplate code in software written in any language. It's not java specific. I've written and worked with big java applications which had almost zero boilerplate code.

Like in every other software, there may be some simple boilerplate code here and there. But usually, it's small and it's not really a problem to type the couple of extra lines, nor does it harm readability. On the other hand, it may help debugging, maintenance, etc...

That being said, the aim of all source code should be to strive for simplicity and ease of understanding. If the boilerplate is excessive and you have a much simpler solution, leading to more readable, maintainable, understandable code, then, by all means, do it. Show it to your superior, that it's simpler, safer, less code, etc. and if it's indeed that obvious, his decision should be obvious as well.

Now, on the other hand, if you want to make an super complicated inflexible unsafe construct just to save a couple of boilerplate lines, then, I agree with your boss. It may be a source of more complexity then it tries to solve. Remember that you don't code for yourself, you code for others.

Title: Do I have to write a lot of boilerplate code if I keep working using Java?

Body:
I'm working for a company writing ERP applications. My problem is that I have to write tons of boilerplate code. I came up with ideas to automatize/prevent the drudgery but only some of them were accepted. I have been told by the lead developer that my ideas tend to be go far afield and I should write code everyone can understand.

I had a discussion about this lately and it seems to me that this kind of code ramp is within java's philosophy. I have to write lots of code to achiveve simple things not because it is necessary but because this is the way most of the people at the company think.

Is this universally applicable to most of the companies out there using java or this is just my company's view? Do I have to get used to the drudgery if I keep working for java-based firms?


答:
It affects how much code you can see and parse in a single glance

x++  rather than set(x,Integeradd(get(x),1))

ps. It's not just a matter of reading code. In the days of 40x25 screens then APL type languages, or Perl, were useful over Cobol or Fortran for the amount of code you could read/page. But now it's more about your own internal cache - if a statement has a single operator it's easier for my ageing brain to parse than one with 3 symbols and 4 function calls.


答:
Go look at AppleScript, one of the most verbose languages I can think off that could be considered current, try to write something trival in it and come back and try and argue that verbosity is a good trait in a language.

Trying to remember all the semantics of where the keywords go and what they are is non-trivial if you don't do it every day.

Just look at all the keyword noise in this trivial example:

tell application "Finder"
    if folder "Applications" of startup disk exists then
        return count files in folder "Applications" of startup disk
    else
        return 0
    end if
end tell
```

granted the same thing in bash with traditional Unix tools would be terse but cryptic to most OSX users, so there is a balance that must be struck.

Body:
Go look at AppleScript, one of the most verbose languages I can think off that could be considered current, try to write something trival in it and come back and try and argue that verbosity is a good trait in a language.

Trying to remember all the semantics of where the keywords go and what they are is non-trivial if you don't do it every day.

Just look at all the keyword noise in this trivial example:

tell application "Finder"
    if folder "Applications" of startup disk exists then
        return count files in folder "Applications" of startup disk
    else
        return 0
    end if
end tell

granted the same thing in bash with traditional Unix tools would be terse but cryptic to most OSX users, so there is a balance that must be struck.


答:
Usually, people favor readable/verbose languages over cryptic/terse ones. However, I think most people assimilated "verbosity" with "clutter", which are not the same thing.

For instance: while(<>){print if($.==2 || $& && !$x++); $.=0 if (/^--+$/)}

is a very terse statement. It contains everything you want to know. However, because of it's terseness, it's hard to understand. On the other hand, a slightly more verbose version of the same program would be fairly straightforward to understand because it is more readable. This is of course not a property of the language per-se, but some languages tend to more verbosity while others tend to more conciseness in their way of programming.

To the other extreme of verbosity, is http://en.wikipedia.org/wiki/Literate_programming
, which I consider a pretty interesting concept.

One other aspect is "unwanted verbosity", also called "clutter". Things you have to add for technical reasons, lack of syntactic sugar, bloated APIs, and so on.

I think a clear and intuitive syntax is very important. It also has to be verbose enough to facilitate an easy reading. Too short statements packed with too much logic can often lead to more time to decypher than their slightly longer counterparts. On the other hand, useless bloated code full of boilerplate lines to do something utterly simple is equally a nuisance.


问:
What is meant by elaboration of a declaration statement?
Intuitively I understand "elaboration" and have a very mild idea. But I want to know exactly what is elaboration of a declaration?

I came across this word in this context:

Stack dynamic storage bindings are created for variables when their declaration statements are elaborated.

The quote is from "Concepts of Programming languages" Robert Sebesta .

答:
From Robert Sebesta's Concepts of Programming Languages 9th edition:

Elaboration of a declaration refers to the storage allocation and
  binding process that takes place when the code containing the
  declaration is executed.

Okay, what does that mean?

Stack Dynamic variables are allocated to the stack. Hardware in the case of native code. Software in the case of a a virtual machine. This means that they are pushed on to the stack when the code containing them is executed. Static variables on the other hand are bound and given values at load time when the program begins running.

How then is this useful? Well, functions generally get sent to the stack when they execute along with the passed parameters. In addition to being faster than the heap, this allows for recursive calls.

As a historical note, Elaboration as a term originated with Algol 68.


问:
How are Implicit-Heap dynamic Storage Binding and Dynamic type binding similar?
"Concepts of Programming languages" by Robert Sebesta says -

Implicit Heap-Dynamic Storage Binding:

Implicit Heap-Dynamic variables are bound to heap storage only when they are assigned values.

It is similar to dynamic type binding.

Can anyone explain the similarity with suitable examples. I understand the meaning of both the phrases, but I am an amateur when it comes to in-depth details.

答:
The example once again from Robert Sebesta's Concepts of Programming Languages 9th edition:

```<code>highs = [74, 84, 86, 90, 71];
</code>```

This Javascript array is only brought into being when its assigned values. It then gets placed in the heap as a collection of those values.

It would bind those values to that variable. It would not necessarily tell you what type they were. That part will be dependent on other language semantics.

Body:
The example once again from Robert Sebesta's Concepts of Programming Languages 9th edition:

highs = [74, 84, 86, 90, 71];

This Javascript array is only brought into being when its assigned values. It then gets placed in the heap as a collection of those values.

It would bind those values to that variable. It would not necessarily tell you what type they were. That part will be dependent on other language semantics.


答:
I think you need to turn the question on it's head and ask: Why do some people think terse code is good?

My answer would be that there are two basic reasons:

Reasons Why Terse Can Be Better

These include being more readable, in the same way that a short sentence can be more understandable than a flowery, verbose sentence. Often programming languages that try and emulate the grammar of the English language end up being horribly long-winded, requiring huge expanses of code to perform the simplest actions. Often you'll find that the more a language emulates a written language then the harder it is to coax it into performing logically complex operations.

Reasons Why Terse Can Be Worse

Some languages (and I'm thinking of Perl) use an array of strange symbols, which seem to have been almost arbitrarily selected, as part of their syntax. To anyone not familiar with these hieroglyphs then the language becomes impenetrable. It also becomes easy to make typo mistakes that aren't easily seen. Regular expressions perhaps epitomise this kind of terseness.

Also, some people like to show-off by writing terse code because, frankly, they think it makes them look clever. You see this sometimes on StackOverflow, where people often will submit answers that are extremely compact at the expense of readability. This is a kind of "impress your peers" with how much you know philosophy, but good programmers realise that "code golf" isn't the way to create maintainable software.


问:
Teacher demands excessive/unjustified use of Design Patterns
I study computer science and I have a class called "Programming Techniques". Its purpose is to teach (us) good object oriented design principles. During the semester we have homeworks, programs that we must write to demonstrate what we've learned.

The lab assistant demands for each of these homeworks that specific design patterns should be used. For example, the current homework is an application used for processing customer orders. We are demanded to use either "Factory Method" or "Abstract Factory" design patterns for this. It gets even worse: at the end of the semester we must write a program (something more complex) that must use at least one creational pattern, at least one structural pattern and at least one behavioural pattern.

Is it normal to demand this ? I mean, forcing us to design our programs in such a way that a specific design pattern makes sense is just beyond what I consider ok. If I'm a car mechanic and have a huge tool box, then I will use a certain tool from that box if and when the situation demands it. Not more, not less. If my design of the application doesn't demand at all the use of "Abstract Factory" (for example), then why should I implement it ?

I'm not sure yet if the senior lecturer agrees with what the lab assistant is demanding, but I want to talk to him about it and I need solid arguments to do so. How should I approach this problem with him ?

PS: I'm sure there must be a better way to teach us these things. Maybe making us each week read about 3 design patterns and the next week giving us a test with small but specific programming or architectural situations/problems. The goal in that test would be to identify what design patterns would make sense and how they could be implemented. This way, he can see if we understand them.

EDIT: These homeworks are not just 100-line programs, they have quite a lot of requirements and are fairly complicated. This is the reason we have about 2 - 3 weeks of deadline for each of them.

I agree that practicing this is the best way to learn. But shouldn't smaller programs/applications be used for this ? Something just for demonstrating purposes. Not big programs with lots of requirements/classes/etc.

答:
If you feel your teacher is doing something wrong, discuss it with him or her.

Imposing specific practices in an homework is very common. It helps you learn the thing much better than just reading about it. In short, practice is an essential learning step.

I hear and I forget. I see and I remember. I do and I understand. [Confucius]


答: you used a mechanic example so lets try to stay with that. When the mechanic was in training and they were teaching tire replacement they told him to use the lug wrench, they didn't hand him a tool box and say go. Its the same with you, you are in classes to learn to implement these techniques so when a programming assignment is intended to use a certain tool they tell you. This problem is not really a problem, unless they are telling you to use a hammer on those screws.</p>

Title: Teacher demands excessive/unjustified use of Design Patterns

Body:
I study computer science and I have a class called "Programming Techniques". Its purpose is to teach (us) good object oriented design principles. During the semester we have homeworks, programs that we must write to demonstrate what we've learned.

The lab assistant demands for each of these homeworks that specific design patterns should be used. For example, the current homework is an application used for processing customer orders. We are demanded to use either "Factory Method" or "Abstract Factory" design patterns for this. It gets even worse: at the end of the semester we must write a program (something more complex) that must use at least one creational pattern, at least one structural pattern and at least one behavioural pattern.

Is it normal to demand this ? I mean, forcing us to design our programs in such a way that a specific design pattern makes sense is just beyond what I consider ok. If I'm a car mechanic and have a huge tool box, then I will use a certain tool from that box if and when the situation demands it. Not more, not less. If my design of the application doesn't demand at all the use of "Abstract Factory" (for example), then why should I implement it ?

I'm not sure yet if the senior lecturer agrees with what the lab assistant is demanding, but I want to talk to him about it and I need solid arguments to do so. How should I approach this problem with him ?

PS: I'm sure there must be a better way to teach us these things. Maybe making us each week read about 3 design patterns and the next week giving us a test with small but specific programming or architectural situations/problems. The goal in that test would be to identify what design patterns would make sense and how they could be implemented. This way, he can see if we understand them.

EDIT: These homeworks are not just 100-line programs, they have quite a lot of requirements and are fairly complicated. This is the reason we have about 2 - 3 weeks of deadline for each of them.

I agree that practicing this is the best way to learn. But shouldn't smaller programs/applications be used for this ? Something just for demonstrating purposes. Not big programs with lots of requirements/classes/etc.


答:
Is the point of the class/exercises to learn design patterns? If yes, then do what he says and work on learning the design patterns. Don't focus on the fact that the pattern is ill-suited to the sample project.

If the class/exercise is about choosing the right design pattern for the right situation, then you might have an acceptable complaint and can talk with your teacher about it. It would be even better if you can suggest some good sample projects that would benefit from the design pattern you're trying to learn.

Just don't forget to be respectful and accept the fact that your teacher might not be able to change the program.


问:
What are some good tips for a developer trying to design a scalable MySQL database?
As the question states, I am a developer, not a DBA.  I have experience with designing good ER schemas and am fairly knowledgeable about normalization and good schema design.  I have also worked with data warehouses that use dimensional modeling with fact tables and dim tables.

However, all of the database-driven applications I've developed at previous jobs have been internal applications on the company's intranet, never receiving "real-world traffic".  Furthermore, at previous jobs, I have always had a DBA or someone who knew much more than me about these things.

At this new job I just started, I've been asked to develop a public-facing application with a MySQL backend and the data stored by this application is expected to grow very rapidly.  Oh, and we don't have a DBA.  Well, I guess I am the DBA. ;)

As far as designing a database to be scalable, I don't even know where to start.  Does anyone have any good tips or know of any good educational materials for a developer who has been sort of shoved into a DBA/database designer role and has been tasked with designing a scalable database to support an application like this?  Have any other developers been through this sort of thing?  What did you do to quickly become good at this role?

I've found some good slides on the subject here but it's hard to glean details from slides.  Wish I could've attended that guy's talk.

I also found a good blog entry called 5 Ways to Boost MySQL Scalability which had some good information, though some of it was over my head.

tl;dr

I just want to make sure the database doesn't have to be completely redesigned when it scales up, and I'm looking for tips to get it right the first time.  The answer I'm looking for is a "list of things every developer should know about making a scalable MySQL database so your application doesn't perform like crap when the data gets huge".

答:
I found that the defaults for MySQL are very, very conservative. This means you can run it out of the box on really old hardware, but that it's slow until you configure it. I went through these steps (just hacking on it really):

Determine what engine you will use (Isam, the default, does not support transactions)

Go through all the available options and read the docs

Adjust options upwards as much as possible

Not every option matters, but this covers most of the ground. Secondly, realize that alot of the back end is IO bound by your hard drive (tmp files, reading/writing large chunks of data) so pointing your mysql-data and /tmp folder to a SSD drive has a big boost in performance. That is easily done through the magic of symlinks.

Oh, and make sure that the DB schema supports and encourages good queries. Recognize that SQL is set based and you should use UUIDs.


问:
Can't work on real projects because I keep thinking about my side projects
I have a "real" project I'm working on to make money from, and a side project that is vastly more interesting.

I'm having a problem where every time I go to work on my "real" project, all I can think about is ideas for my side project. How do you guys deal with that? Just power through the real work?

答:
I learned this "technique" a while back and it's so simple, it seems stupid.  However, it works wonders for some of us.

Open a session of Notepad (or other simple text editor) and in HUGE point font type in your immediate responsibility.

Keep the document open, and if you are prone to not "full screen" the applications you work with, make sure that a small portion of the document is always visible.  If/when you get distracted, click over to the document and read the short statement reminding you of what your current task is.

Yes, this sounds silly, but for some of us, seeing something helps jar our minds to focus on what we need to focus on.  Eventually, your mind will be trained to just see the corner of Notepad and think "Oh, yeah let me get back to that."

It's just a way to re-condition your mind to stay focused on your work.

Now I have to go.  Notepad is telling me to create a few, new user accounts...

ContentLicense: CC BY-SA 2.5


答:
The goal is quick comprehension

"Verbose" means "uses too many words". The question is what "too many" is.

Good code should be easy to comprehend at a glance. This is easier if most of the characters directly serve the purpose of the code.

Signal vs Noise

If a language is verbose, more of your code is noise. Compare Java's "Hello World":

```<code>class HelloWorldApp {
    public static void main(String[] args) {
        System.out.println("Hello World!");
    }
}
</code>```

... with Ruby's:

```<code>print "Hello World!"
</code>```

Noise wastes mental energy.

Cryptic vs Clear

On the other hand, excessive terseness in a language also costs mental energy. Compare these two examples from Common Lisp:

```<code>(car '(1 2 3))   # 3 characters whose meaning must be memorized
# vs
(first '(1 2 3)) # 5 characters whose meaning is obvious
</code>```

Body:
The goal is quick comprehension

"Verbose" means "uses too many words". The question is what "too many" is.

Good code should be easy to comprehend at a glance. This is easier if most of the characters directly serve the purpose of the code.

Signal vs Noise

If a language is verbose, more of your code is noise. Compare Java's "Hello World":

class HelloWorldApp {
    public static void main(String[] args) {
        System.out.println("Hello World!");
    }
}

... with Ruby's:

print "Hello World!"

Noise wastes mental energy.

Cryptic vs Clear

On the other hand, excessive terseness in a language also costs mental energy. Compare these two examples from Common Lisp:

(car '(1 2 3))   # 3 characters whose meaning must be memorized
# vs
(first '(1 2 3)) # 5 characters whose meaning is obvious


答:
Sorry, but I think your teachers are right.

If you were developing software for a customer, and the customer or your boss requires you to use specific design patterns, I would definitely say that that was a big mistake. But there is a difference between class assignments and software development for a customer: both serve completely different purposes:

If you are developing for a customer, the purpose is the software you create. It should work, it should be maintainable, it should satisfy your customer's requirements.

The purpose of your assignments is not the software you create, it is the experience you gather during the whole creation process. During the assignment you will learn things about implementing  patterns, which will help you understand how the patterns work, when to use a pattern, and (hopefully) when not to use it, as well.

To use your car mechanic analogy: You are not a mechanic, you are learning how to become one. And maybe your teacher will tell you which wrench to use, in order to change the  tires. That is completely ok. And if the owner of the garage tells you to change the tires with a screwdriver, because he wants you to learn something from the failure that will certainly occur, that is ok, too.


问:
What's your favourite quote about programming?
What's your favourite quote about programming?

One quote per answer, and please check for duplicates before posting!

答: Keep it Simple!</p>

LastEditorDisplayName: Roger Pate

Body: Keep it Simple!


答:
Firstly, is this an excellent way of teaching design patterns?  No, probably not.  Is there a better way to teach them?  I'm sure there is.  Should you be organising a revolt because of this?  Absolutely not.  Let me elaborate:

You are in a class which seems to be dealing heavily with design patterns.  Design patterns are your "software engineering toolbox", and you should definitely get very familiar with them.  In general, they have many positive aspects, and one of those is that they allow you to think on a higher level of abstraction.  Instead of trying to explain things on the level of individual methods or variables, you now think in terms of high level patterns.

It gets even worse: at the end of the semester we must write a program (something more complex) that must use at least one creational pattern, at least one structural pattern and at least one behavioural pattern.

You make this sound like it's a bad thing.  Yes, there are some people who go pattern-crazy, and use them inappropriately and over-complicate everything with them.  One creational pattern, one structural pattern, and one behavioural pattern is not it.  Any non-trivial system will have at least this much, so it's a good idea to get comfortable with them.  I suspect this is the general idea in your class - you should give it another shot.


答:
I think the usual rules apply:

Keep tables small (don't waste space unnecessarily).

Don't query for more information than you need.

If you use ORMs, beware of common pitfalls such as the N+1 problem.

Stay away from troublesome operators (e.g. Like '%Smith%').

Design your indexes intelligently and make sure they cover the majority of the uses (here is a decent, if light, treatment of indexes).

Remember that set-based querying is usually far superior in terms of performance than iterating through the data.

Know when to denormalise data for performance reasons.

Cache whatever can be cached (economically) to relieve the DB.

Of course, vertical scaling can only get you so far, and then you might have to look into horizontal scaling.  Having said that, a good single-DB design can still take you very far - as far as I know, StackOverflow is still running a single DB instance.  If you think you will need to handle far more data than that, consider sharding (or alternative DBs) early on.

Title: What are some good tips for a developer trying to design a scalable MySQL database?

Body:
As the question states, I am a developer, not a DBA.  I have experience with designing good ER schemas and am fairly knowledgeable about normalization and good schema design.  I have also worked with data warehouses that use dimensional modeling with fact tables and dim tables.

However, all of the database-driven applications I've developed at previous jobs have been internal applications on the company's intranet, never receiving "real-world traffic".  Furthermore, at previous jobs, I have always had a DBA or someone who knew much more than me about these things.

At this new job I just started, I've been asked to develop a public-facing application with a MySQL backend and the data stored by this application is expected to grow very rapidly.  Oh, and we don't have a DBA.  Well, I guess I am the DBA. ;)

As far as designing a database to be scalable, I don't even know where to start.  Does anyone have any good tips or know of any good educational materials for a developer who has been sort of shoved into a DBA/database designer role and has been tasked with designing a scalable database to support an application like this?  Have any other developers been through this sort of thing?  What did you do to quickly become good at this role?

I've found some good slides on the subject here but it's hard to glean details from slides.  Wish I could've attended that guy's talk.

I also found a good blog entry called 5 Ways to Boost MySQL Scalability which had some good information, though some of it was over my head.

tl;dr

I just want to make sure the database doesn't have to be completely redesigned when it scales up, and I'm looking for tips to get it right the first time.  The answer I'm looking for is a "list of things every developer should know about making a scalable MySQL database so your application doesn't perform like crap when the data gets huge".


答:
The scalability highly depends on database schema design, and much more less depending on database performance.

I'd suggest following scenario of creating good database schema.

Normalize schema as much as possible.

Then, if you have any link tables - duplicate all keys through target tables.

Now - slightly denormalize all tables in order to get faster reports.


问:
Data base structure of a subscriber list
I am building a application that allow different user to store the subscriber information

To store the subscriber information , the user first create a list
For each list, there is a ListID.

Subscriber may have different attribute : email phone fax ....

For each list,  their setting is different , so a require_attribute table is introduced.
It is a bridge between subscriber and List

That store Listid, subid, attribute, datatype

That means the system have a lot of list, each user have their own list, and the list have different attribute, some list have email , phone , some may have phone, address, name mail.. And the datatype is different, some may use 'name' as integer, some may use 'name' as varchar

attribute means email phone, it is to define for which list have
which subscriber attribute

datatype means for each attribute, what is its datatype

```<code>Table :subscriber :
Field :subid , name,email

Table :Require Attribute:

Field : Listid ,subid , attribute, datatype
</code>```

The attribute here is {name, email}

So a simple data is

```<code>Subscriber: 1 , MYname, Myemail

Require Attribute :

 Listid , 1 , 'email', 'intger'

 Listid , 1 , 'name', 'varchar'
</code>```

I found that this kind of storage is too complex to handle with,  Since the subscriber is share to everybody, so if a person want to change the datatype of name, it will also affect the data of the other user.

Simple error situation:

Subscriber:

```<code>list1,  Subscriber 1 , name1, email1

list2, Subscriber 2  , name2 , email2

Require Attribute :

     List1 , Subscriber 1 , 'email', 'varchar',

     List1 , Subscriber 1  , 'name', 'varchar',

     Listid , Subscriber 2 , 'email', 'varchar',

     Listid , Subscriber 2, 'name', 'integer',
</code>```

if user B change the data type of name in require attribute from varchar to integer, it cause a problem. because list 1 is own by user A , he want the datatype is varchar, but list 2 is won by user B, he want the datatype to be integer

So how can I redesign the structure?

答:
You can solve this problem using something I have called a vertical table solution in the past.  There are both pros and cons to this solution.  I can provide a sample database upon request.

Cons

•   Loss of horizontal structure makes queries difficult to write and slower to perform.  The difficulty in writing queries can be helped with views.  Views introduce their own set of issues as well.

•   Data integrity becomes more difficult.  Since you are defining the datatype abstractly in a table and storing as as the most common type, usually some form of text field, it puts a burden on verifying the data on input.  Previously I have investigated using foreign keys to insure valid data but have not implemented a solution.

•   Does not support 'many to one' relationships well, but it can be done.

Pros

•   Customer is able to define their own data definition.

•   Change in definition does not adversely affect other clients. However if a client changes their own datatype or meaning there can be issues.

•   Can become 'brittle' once a list is defined and data is collected.

Sample Query

```<code>Select
    rst.responseSetID,
    Max(case when lvdt.CustomerFieldName = 'Email' then vst.InputValue end) as Email,
    Max(case when lvdt.CustomerFieldName = 'Name' then vst.InputValue end) as Name
From SubscriberTable st
    join ListDefinitionTable ldt on
        st.SubID  = ldt.SubID
    join VerticalStorageTable vst on
        ldt.ListID = vst.ListID
    join ListValueDefinitionTable lvdt on
        vst.ListDefintionID  = lvdt.ListDefinitionID
    join AttributeDefinition ad on
        lvdt.AttributeTypeID = ad.AttributeTypeID
    Join ResponseSetTable rst on
        vst.ResponseSetID = rst.ResponseSetID
where
    st.SubID = 1
    and vst.ListID = 1
Group by
    rst.responseSetID
</code>```

Subscriber Table

•   SubID (ID) (PK)

•   CustomerName (Text)

Attribute Definition

•   AttributeTypeID (ID) (PK)

•   AttributeType (Text)

List Definition Table

•   ListID (ID) (PK)

•   SubID (ID)

•   CustomerListName (Text)

List Value Definition Table

•   ListDefinitonID (ID) (PK)

•   ListID (ID)

•   AttributeTypeID (ID)

•   CustomerFieldName (Text)

Response Set Table

•   ResponseSetID (ID) (PK)

•   SubID (ID)

•   ListID (ID)

•   TimeInput (DateTime) (Default – GetDate())

Vertical Storage Table

•   InputID (ID) (PK)

•   ResponseSetID (ID)

•   ListID (ID)

•   ListDefinitonID (ID)

•   InputValue (Text)

OwnerDisplayName: foodil

Body:
You can solve this problem using something I have called a vertical table solution in the past.  There are both pros and cons to this solution.  I can provide a sample database upon request.

Cons

•   Loss of horizontal structure makes queries difficult to write and slower to perform.  The difficulty in writing queries can be helped with views.  Views introduce their own set of issues as well.

•   Data integrity becomes more difficult.  Since you are defining the datatype abstractly in a table and storing as as the most common type, usually some form of text field, it puts a burden on verifying the data on input.  Previously I have investigated using foreign keys to insure valid data but have not implemented a solution.

•   Does not support 'many to one' relationships well, but it can be done.

Pros

•   Customer is able to define their own data definition.

•   Change in definition does not adversely affect other clients. However if a client changes their own datatype or meaning there can be issues.

•   Can become 'brittle' once a list is defined and data is collected.

Sample Query

Select
    rst.responseSetID,
    Max(case when lvdt.CustomerFieldName = 'Email' then vst.InputValue end) as Email,
    Max(case when lvdt.CustomerFieldName = 'Name' then vst.InputValue end) as Name
From SubscriberTable st
    join ListDefinitionTable ldt on
        st.SubID  = ldt.SubID
    join VerticalStorageTable vst on
        ldt.ListID = vst.ListID
    join ListValueDefinitionTable lvdt on
        vst.ListDefintionID  = lvdt.ListDefinitionID
    join AttributeDefinition ad on
        lvdt.AttributeTypeID = ad.AttributeTypeID
    Join ResponseSetTable rst on
        vst.ResponseSetID = rst.ResponseSetID
where
    st.SubID = 1
    and vst.ListID = 1
Group by
    rst.responseSetID

Subscriber Table

•   SubID (ID) (PK)

•   CustomerName (Text)

Attribute Definition

•   AttributeTypeID (ID) (PK)

•   AttributeType (Text)

List Definition Table

•   ListID (ID) (PK)

•   SubID (ID)

•   CustomerListName (Text)

List Value Definition Table

•   ListDefinitonID (ID) (PK)

•   ListID (ID)

•   AttributeTypeID (ID)

•   CustomerFieldName (Text)

Response Set Table

•   ResponseSetID (ID) (PK)

•   SubID (ID)

•   ListID (ID)

•   TimeInput (DateTime) (Default – GetDate())

Vertical Storage Table

•   InputID (ID) (PK)

•   ResponseSetID (ID)

•   ListID (ID)

•   ListDefinitonID (ID)

•   InputValue (Text)


问:
How to make hardware sizing sheet?
We build some software platform, and now we need to make hardware sizing sheet for the server-side piece of this software. So, I should prepare some mathematical model that, for example, given the number of concurrent users, can predict the number of physical servers or storage capacity or network bandwidth. How to make such model?</p>

答:
You may want to start with a mathematical model, but I'd advise instead to do a simulation - make a "user": a program that acts like a user and generate randomized user-like activity. Then, load the system with a real-life like dataset (sized accordingly) and deploy your users against the structure you created. Rinse and repeat until you get as many measures for every data point so that you have a reasonable confidence the number represents the performance at a given load and that noise has been reduced to a level that doesn't interfere with your measurements. Keep an eye on maximums, because they may appear in real-world usage and derail the application. Also, remember to test it with absurd numbers of users and absurd volumes of data, as it often offers you some insight on problems that may not be evident on more normal scenarios.

If possible, publish alongside your chart the methodology you used (how your users behave, what hardware did you use - if real metal, if EC2 instances - and how many measurements you made).

This is a highly worthy exercise, BTW. You may discover some bottlenecks you never imagined you'd have, as well as new ways to tune your application on different hardware configurations that never occurred to you.

I come from a web background and I try to make every system I build capable of generating one or more JMeter configuration files for load testing purposes. Nobody knows how to better simulate load than the system itself - it has access to previous usage patterns, as well as to its own working database.

And then when you have a ton of experimental data, you can refine your mathematical models in order to predict conditions you may not easily simulate.

Title: How to make hardware sizing sheet?


答:
Something no one else has hit is in the amount of code you can fit in one screen. It helps for several reasons:

Less scrolling back and forth when you're working on (or reading) multiple functions in one file.

Less horizontal scrolling (stop reading, click and drag scroll bar, stop reading, click and drag scrollbar...).

Faster scanning, since there's less useless syntax getting in the way of finding what you want.


答:
I read somewhere once that a great deal of the verbose debate comes from new programmers vs old hands. The analogy used was that when we learn something new, we tell ourselves a "story" to get through it (think of when you learned algebra in HS). As we gain experience, we move beyond the need for a story explaining individual components and we comprehend a greater amount. Our code becomes denser and more terse, with comments only explaining the necessary items, instead of reiterating what the code says.

I've found this to be true between me and a coworker. She writes code with lots of comments explaining what the lines of code are, and lots of white space. If I'm reading it, I find that I can only fit a few lines of code on one screen because of this. That makes understanding each individual line far easier, but grokking the whole function becomes much harder.

I tend to write code without the extra whitespace or comments. This makes seeing the whole far easier, but you have to be able to simply "get" the individual code "words". If you tried to read this answer with each word being on it's own line, with lots of whitespace/comments/extra verbiage then it would be far easier to understand each individual word, but far harder to understand the whole.


答:
Select by PK whenever possible;

Keep tables small enough for all indices to fit in the memory. If that is not possible shard vertically. Should be easy enough if some of the tables are rarely or never used together;

Avoid JOINs whenever possible, but on the other hand don't fall into 1+N trap;

Avoid SELECT *, especially if any of the columns is a LOB (like TEXT). LOBs are never cached in memory, they are always fetched from HDD;

Have dedicated slaves used for interactive queries, and separate ones for slow, complicated, aggregate report queries;

Use InnoDB (default in current MySQL);

Use slaves for SELECTs, master for INSERT/UPDATE/DELETEs;

Batch similar queries, eg. if you have an insert on each page view, don't do  database immediately, put it in cache and later on insert with multiple values instead;


答: Like with so many things, the answer depends on the concrete definition of "verbosity". If the definition is such that verbosity implies redundancy then I would argue it is always bad. Note that with such a definition the often cited Java hello world program would not qualify as "verbose", because there is nothing in it that is redundant.


问:
Call constructor using an arguments object in javascript?
Is it possible to call the constructor using an arguments object?</p>

```<code>var MyClass = function(a, b){
  this.a = a;
  this.b = b;
};
var myClassInstance = function(){
  //This line would not work, but is what I'm asking. Is there a way besides eval?
  return new MyClass.apply(?, arguments);
}('an A value', 'a B value');
</code>```

答:
Yes it is.

However, I had to rewrite your code a bit, as the method you're currently using appears to put the function calls into the global scope.

```<code>function MyClass(a, b){
    this.a = a;
    this.b = b;
};

function myClassInstance(){

    //The apply function will apply MyClass attributes to this object.
    //The apply function itself returns nothing.
    MyClass.apply(this, arguments);
    console.log(this); //Should show the a and b variables

    return this;
}

new myClassInstance('an A value', 'a B value');
</code>```

Body:
Yes it is.

However, I had to rewrite your code a bit, as the method you're currently using appears to put the function calls into the global scope.

function MyClass(a, b){
    this.a = a;
    this.b = b;
};

function myClassInstance(){

    //The apply function will apply MyClass attributes to this object.
    //The apply function itself returns nothing.
    MyClass.apply(this, arguments);
    console.log(this); //Should show the a and b variables

    return this;
}

new myClassInstance('an A value', 'a B value');


问:
Should Business Interfaces be part of the Model layer?
In an oriented-services enterprise application, isn't it an antipattern to mix Service APIs (containing interface that external users depends on) with Model objects (entities, custom exceptions objects etc...) ?

According to me, Services should only depends on Model layer but never mixed with it.

In fact, my colleague told me that it doesn't make sense to separate it since client need both. (model and service interfaces)

But I notice that everytime a client asks for some changes, like adding a new method in some interface (means a new service), Model layer has to be also delivered...

Thus, client who has not interested by this "addition" is constrained to be concerned by this update of Model... and in a large enterprise application, this kind of delivery is known to be very risked...

What is the best practice ? Separate services(only interfaces so) and model objects or mix it ?

答: It depends on whether client is using exactly the same domain model as the server/service. Typically it doesn't, so they should be kept separate IMO.</p>


答: It does sounds strange that there making you come up with your own assignments to create specific patterns, however, if you can pull it off I would recommend that you embrace the class and do it. It'll be a great learning experience and an extremely great thing to put on your resume. Being a college grad in the real world I can tell you that that last point I made makes all the difference in the world if you ever want to get a job in this field. A degree alone means nothing without the work to back it up.</p>


答:
C++: an octopus made by nailing extra
  legs onto a dog.

Body:
What's your favourite quote about programming?

One quote per answer, and please check for duplicates before posting!


答:
Yes, you could do this:</p>

```<code>var myClassInstance = function(){
  return MyClass.apply(Object.create(MyClass.prototype), arguments);
}('an A value', 'a B value');
</code>```

Title: Call constructor using an arguments object in javascript?

Body:
Is it possible to call the constructor using an arguments object?</p>

```<code>var MyClass = function(a, b){
  this.a = a;
  this.b = b;
};
var myClassInstance = function(){
  //This line would not work, but is what I'm asking. Is there a way besides eval?
  return new MyClass.apply(?, arguments);
}('an A value', 'a B value');
</code>```

Body:
Yes, you could do this:

var myClassInstance = function(){
  return MyClass.apply(Object.create(MyClass.prototype), arguments);
}('an A value', 'a B value');


问:
Moving from Winforms to WPF
I am a long time experienced Windows Forms developer, but now it's time to move to WPF because a new WPF project is comming soon to me and I have only a short lead time to prepare myself to learn WPF.

What is the best way for a experienced Winforms devleoper?

Can you give me some hints and recommendations to learn WPF in a very short time!

Are there simple sample WPF solutions and short (video) tutorials? Which books do you recommend? Is www.windowsclient.net a good starting point? Are there alternatives to the official Microsoft site?

答: I've found WPF Programmer's Reference by Rod Stephens really good.  It's not a reference manual, but it is written by someone who can explain things.  It won't last you your entire WPF career, but it will shorten the initial learning curve.</p>

Title: Moving from Winforms to WPF


答:
I agree that practicing this is the best way to learn. But shouldn't
  smaller programs/applications be used for this ? Something just for
  demonstrating purposes. Not big programs with lots of
  requirements/classes/etc.

Design patterns really aren't that interesting in a very small program. Anybody can implement a factory or an observer for a class that doesn't do anything useful. The power of patterns comes from their ability to help you organize larger programs -- once you recognize that a pattern is being used, you don't have to read through the entire implementation to have a pretty good idea of what the code does. Patterns are the tools that you use to translate "lots of requirements" into sensible classes. So asking you to use the patterns that you're learning in the context of a program that does something interesting seems like a pretty good idea.

Besides, if you're writing these programs as homework over 2-3 weeks, you're definitely still writing small programs.


答:
Scrum, agile, and Test driven development techniques aren't just for big teams or big companies, they can help on freelance projects too.

Creating a backlog of requirements/user stories can help focus on what needs to be done and when it needs doing. For each iteration you can decide what you want to achieve and how much time you want to dedicate to it. Try to strike a balance between keeping yourself busy and avoiding biting off more than you can chew though - both extremes can be demoralising.

If you feel yourself getting bored or losing focus, try switching to another user story. Remember that it is often the case that a change is as good as a rest. With a good DVCS like git or mercurial you can easily switch contexts by switching branches. When sets of features are ready at the end of a sprint, merge them together and you have a sound basis for the next iteration.

Finally, TDD can really help focus the mind on what you need right now (YAGNI) and it strongly encourages you to write your code with testability in mind. It takes me about the same time to write code using TDD as it takes me to write without it, but with TDD subsequent debugging, re-factoring and regression testing are all much cheaper and can be done with much more confidence.


答:
How your customer views themselves:

I have a project that I need to get done ASAP

I know my business needs

I need to solve the problem in a way that is not going to be disruptive to current business practices

I do have a  limtied budget to get this done.

On the other hand, they see your group as:

Guys who dont get that they are sucking money out of my budget.

Do not understand the needs of our business

Want to redesign everything even though it is going to make implementing it harder on the business.

Want to have clever slick solution when all i have the budget for is functional and effective.

Your primary problem seems to be neither of you is understanding what they need from the other party.

Title: Dealing with the customer / developer culture mismatch on an agile project

Body:
One of the tenets of agile is ...

Customer collaboration over contract negotiation

... another one is ...

Individuals and interactions over processes and tools

But the way I see it, at least when it comes to interaction with the customer, there is a fundamental problem:

How the customer thinks is different to how a software engineer thinks

That may be a bit of a generalisation, yes.  Arguably, there are business domains where this is not necessarily true---these are few and far between though.  In many domains though, the typical customer is:

Interested in daily operational concerns--short-range tactics ... not necessarily strategy;

Understandably, only concerned with the immediate solution;

Practical thinkers, not abstract thinkers;

Much more interested in "getting the job done" than considering how the solution will support future concerns.

On the other hand, in the ideal, software engineers who practice agile are:

People who think alot about quality;

Individuals who appreciate how a little bit of upfront work can save a ton in effort down the line;

Experienced, analytical thinkers.

So there seems to be a culture discrepancy that tends to inhibit "customer collaboration".

What's the best way to address this?


答:
It sounds like you're referring to data models, not business models, and to me data models are just dumb data objects. Their sole purpose in life is to hold data, and they don't perform any business logic or functionality. There shouldn't be any problem with keeping them completely separate from the rest of the application code.

So typically my data models reside in their own library, which the business layers can reference, while business models will usually be part of the business layer.

Title: Should Business Interfaces be part of the Model layer?

Body:
In an oriented-services enterprise application, isn't it an antipattern to mix Service APIs (containing interface that external users depends on) with Model objects (entities, custom exceptions objects etc...) ?

According to me, Services should only depends on Model layer but never mixed with it.

In fact, my colleague told me that it doesn't make sense to separate it since client need both. (model and service interfaces)

But I notice that everytime a client asks for some changes, like adding a new method in some interface (means a new service), Model layer has to be also delivered...

Thus, client who has not interested by this "addition" is constrained to be concerned by this update of Model... and in a large enterprise application, this kind of delivery is known to be very risked...

What is the best practice ? Separate services(only interfaces so) and model objects or mix it ?


问:
Why does PHP have interfaces?
I noticed that as of PHP5, interfaces have been added to the language. However, since PHP is so loosely typed, it seems that most of the benefits of using interfaces is lost. Why is this included in the language?</p>

答:
My guess.

PHP is used by many entry level programmers, entry level programmers are taught java in college.

After their Programming 101 course they start nagging Zend they want java features because that's the way they have been taught to think, an thinking on your own terms (or understanding duck typing) it's hard, when you're only 20.

Zend is pragmatic, it's easier to add the feature other than to pretend they where right all along.

This also buys in more users instead of making them leave, so it must be good.

Another instance of this process? People fresh out of .NET and Java courses also want Frameworks of Foundation Classes, they nag about it until Zend sprouts out the Zend Framework. This buys in even more users. And on and on...

(the only language feature the PHP team is known to have struggled against, over the years, is goto)

Title: Why does PHP have interfaces?


问:
Why aren't there automated translators from one programming language to another?
Most programming languages are Turing complete, which means that any task that can be solved in one language can be solved in another one, or even on Turing machine. Then why aren't there automatic translators that can convert programs from any given language to any other language? I've seen couple attempts for two languages, but they always work only on a limited subset of a language and can hardly be used for converting real projects.

Is it possible, at least in theory, to write 100% correct translator between all languages? What are the challenges in practice? Are there any existing translators that work?

答:
There are some automatic translators. If your goal is to produce compilable code, rather than readable code, it is quite possible and occasionally useful, just not very often. Famously, the first C++ compiler was not actually a compiler, but translated C++ into (really complicated) C source that was then compiled by the C compiler. Many compilers can generate assembly code on request-- but instead of spitting out assembly text and then translating it to machine code, they can normally generate machine code directly.

Given a complete specification of language A, it is not that hard in principle to write a program that expresses its directives in some language B. But usually anyone who goes to the trouble will choose something really low level for "language B": Machine code, or these days bytecode: Jython is an implementation of python that generates java byte code, which is interpreted by the Java VM. No need to bother writing out and compiling java class hierarchies!


问:
What's shell script's advantage over interpreted programming languages?
(I'm not sure if it's a appropriate question here)

Shell scripts, like those written in bash, can do many things. They can call Unix programs, pipe their output, redirect I/O from/to files, control flow, check whether a file exists, etc.

But a modern programming language, e.g, python and ruby, can also do these things. And, they are (I think) more readable and maintainable.

bash enjoys wide spread adoption. But many distributions have installed python interpreter, too.

So what's the advantage of shell a script? If I could write python, ruby or perl, is it worth to learn bash?

答:
I would say a shell script has the advantage in situations where you just want to whip up a super-simple automated task.  For example, if you wanted to write a script that backs up a directory from one server to another every day at 5:00, it would be overkill to use a heavy duty programming language.

On the other hand, if your programming task has more of a need for control structures (if/then/else), iterative structures(looping), database & file I/O, etc, then it is probably better suited for a more capable programming language than a shell script.

I think a good rule of thumb is:

```<code>automatingSimpleTask ? shellScript() : programmingLanguage();
</code>```

Body:
I would say a shell script has the advantage in situations where you just want to whip up a super-simple automated task.  For example, if you wanted to write a script that backs up a directory from one server to another every day at 5:00, it would be overkill to use a heavy duty programming language.

On the other hand, if your programming task has more of a need for control structures (if/then/else), iterative structures(looping), database & file I/O, etc, then it is probably better suited for a more capable programming language than a shell script.

I think a good rule of thumb is:

automatingSimpleTask ? shellScript() : programmingLanguage();


答: While you can launch programs from a python/ruby/whatever program, it's different in the shell. You may need to use an API to launch something - fork() for example. In a shell script, programs behave more like functions in a conventional programming language and can be executed, piped, etc with minimal effort. The flow of data is very clear using pipes too.</p>

Title: What's shell script's advantage over interpreted programming languages?

Body:
(I'm not sure if it's a appropriate question here)

Shell scripts, like those written in bash, can do many things. They can call Unix programs, pipe their output, redirect I/O from/to files, control flow, check whether a file exists, etc.

But a modern programming language, e.g, python and ruby, can also do these things. And, they are (I think) more readable and maintainable.

bash enjoys wide spread adoption. But many distributions have installed python interpreter, too.

So what's the advantage of shell a script? If I could write python, ruby or perl, is it worth to learn bash?


答:
Shells have specialized features for working with files and getting data from one program into another (assuming that data is text). For those tasks, shell scripts can be less cumbersome than a scripting language like Python.

Shell scripting also has the advantage that the commands you use are basically the same commands you'd use from the command line -- so if you can do something in the shell, you're more than halfway to scripting the same operation.

Here for example is a bash script that moves all the PNG files from the current directory to a specified directory.

```<code>#!/usr/bin/sh
mv *.png $1
</code>```

Here's a Python version.

```<code>#!/usr/bin/python
import sys, shutil, glob
for filename in glob.iglob("./*.png"):
    shutil.move(filename, sys.argv[1])
</code>```

You'll notice:

The bash script is a third as long as the Python if you count lines (excluding the shebang line) -- even less by character count. In fact, you probably wouldn't even write a script for it, you'd just create an alias.

The Python script requires three libraries to be imported, while everything you need for this task is natively available in bash.

The Python script requires an explicit loop to move the files, whereas that is part of the semantics of the mv command in bash.

The bash script can run faster -- you'll probably invoke it from bash, and you can use source to run it in the same instance of the shell.

glob.iglob("./*.png") is quite a mouthful just to say *.png

If you wanted to write a basic pipe operation in Python, you would be astounded at the verbosity. (Of course, some things, like piping through grep, can be replaced by Python code rather than using an external program, so you often don't need to pipe quite as much.)

As a counterexample, I once had to write a routine that checked to see how long each of the filenames were in a particular directory.  If they were longer than supported by a particular OS, they had to be shortened. This could result in duplicate filenames, which I needed to rectify, and since they would be linked from a Web page, the shortened names needed to be stable, i.e., they should be generated in such way that the same long filename would always result in the same shortened filename. I did this by generating a hex md5 of the long filename and appending the first four characters of that to the shortened name (names could still collide, but it was very unilkely, so I just checked for that condition and bailed if it should happen). It also had to record the rename operation so a batch search-and-replace could later be done on the files to fix the links between them (I wrote out a sed command file and passed that to sed for each file).

I did this in bash because it was part of our build system which was already written in bash. It was exactly as hard to get right as you are probably thinking. It would have taken a lot less time to write in Python and probably would have been clearer, too.

In short: different languages are designed for different kinds of tasks; choose the language available to you that is best suited to the task at hand.

Body:
Shells have specialized features for working with files and getting data from one program into another (assuming that data is text). For those tasks, shell scripts can be less cumbersome than a scripting language like Python.

Shell scripting also has the advantage that the commands you use are basically the same commands you'd use from the command line -- so if you can do something in the shell, you're more than halfway to scripting the same operation.

Here for example is a bash script that moves all the PNG files from the current directory to a specified directory.

#!/usr/bin/sh
mv *.png $1

Here's a Python version.

#!/usr/bin/python
import sys, shutil, glob
for filename in glob.iglob("./*.png"):
    shutil.move(filename, sys.argv[1])

You'll notice:

The bash script is a third as long as the Python if you count lines (excluding the shebang line) -- even less by character count. In fact, you probably wouldn't even write a script for it, you'd just create an alias.

The Python script requires three libraries to be imported, while everything you need for this task is natively available in bash.

The Python script requires an explicit loop to move the files, whereas that is part of the semantics of the mv command in bash.

The bash script can run faster -- you'll probably invoke it from bash, and you can use source to run it in the same instance of the shell.

glob.iglob("./*.png") is quite a mouthful just to say *.png

If you wanted to write a basic pipe operation in Python, you would be astounded at the verbosity. (Of course, some things, like piping through grep, can be replaced by Python code rather than using an external program, so you often don't need to pipe quite as much.)

As a counterexample, I once had to write a routine that checked to see how long each of the filenames were in a particular directory.  If they were longer than supported by a particular OS, they had to be shortened. This could result in duplicate filenames, which I needed to rectify, and since they would be linked from a Web page, the shortened names needed to be stable, i.e., they should be generated in such way that the same long filename would always result in the same shortened filename. I did this by generating a hex md5 of the long filename and appending the first four characters of that to the shortened name (names could still collide, but it was very unilkely, so I just checked for that condition and bailed if it should happen). It also had to record the rename operation so a batch search-and-replace could later be done on the files to fix the links between them (I wrote out a sed command file and passed that to sed for each file).

I did this in bash because it was part of our build system which was already written in bash. It was exactly as hard to get right as you are probably thinking. It would have taken a lot less time to write in Python and probably would have been clearer, too.

In short: different languages are designed for different kinds of tasks; choose the language available to you that is best suited to the task at hand.

ContentLicense: CC BY-SA 4.0


答:
If you are totally free to choose the programming language you use for certain tasks, you could probably almost completely avoid learning bash and do every script programming task with Perl, Python or Ruby. This may sometimes lead to a more verbose solution, especially in the case when you just need a script calling a sequence of other Unix tools, but in more complex cases you are right, those modern script languages give you more possiblites to write better maintainable code (they give you also more possibilites to write obfuscated code, but thats your choice).

On the other hand, often we are not free to choose the language we like most, because we have to work with legacy code or code written by one who knew bash but did not know one of the scripting languages you named.


答:
The existing answers are also valid, but there's one reason that no one has mentioned yet:  because it WILL be there.

Any given *nix install is done with some set of optional packages which may or may not be loaded, and not all systems will have Python or Perl or Ruby.  But if the system is expected to have any interactive capability at all, it will have a shell.  This means that shell scripts will work on systems from servers to programmer's desktops to secretarial thin-client desktops to embedded devices, on any system that supports a writable file system and a bash command line.

For someone who only ever works in a consistent server environment, this is something that can be taken for granted, and isn't so important.  For someone working in a more varied environment, this should not be overlooked.


问:
javascript - would you consider 'prototype' to be the same as 'parent'
I'm learning javascript.

I see that with an object, I can use .prototype, is it fair to say that, in a class/tree hierarchy I am effectively using the 'parent' (or 'ancestor' perhaps) by doing that?  Can a class only have 1 prototype at a time as in Ruby?

Call it Personal Homework that I set myself.

答: Yes, the prototype property on a given object refers to the parent object, which allows the given object to inherit all its properties and methods. However, the definition of parent probably should be defined here. In a prototype-based language, that means that a single object can be the parent of many other objects. A side effect of this is that a property that changes in the parent is reflected in all child objects, assuming the child objects don't override it. This essentially forms a tree structure of objects that you can traverse. This is different then in a class-based language where the parent refers to the super class, and does not represent a tree of objects. The object also inherits the properties and methods from the parent, but it is all self contained within that one object.


问:
Do you think we will ever settle on a "standard" platform?
The recent explosion of phone platforms has depressed me (slightly), and made me wonder if we will ever reach any kind of standard for presentation?

I don't mean language or IDE.  Different languages have different strengths and I can see that there may always be a need for disparity, although I do note that languages are merging somewhat in functionality, with traditional imperitive languages like C++ now supporting things like lambdas.

What I'm really talking about is a common presentation mechanism. Before smart phones and tablets came along, the web seemed to be finally becoming a reasonable platform for presenting an application that was globally accessible, not just geographically, but by platform too. Sure there are still (sometimes infuriating) implementation differences and quirks, but if you wrote a decent site you knew it could be accessed on anything from a PC to a phone to a C64 running the right software. "Write Once Run Anywhere" seemed to finally be becoming a reality.

However, in the last few years we've seen an explosion of mobile operating systems, and the ubiquitous "app". A good site is no longer enough, you need a native "app", and of course we have a sudden massive disparity in OS, language, and APIs needed to write them as each battles for supremecy.

It's kind of weird how the cycle of popularity goes.
Mainframes with terminals - thin client.
PC - thick client.
Web browser - thin client.
Phone app - thick(ish) client.

I just wonder if you think there will ever be a global standard for clients, or whether the "shiny and different" cycle will always continue along with the battle of the tech du jour.

答:
No, I don't think there will ever be a "standard" platform.  "Ain't it cool".  Why would you want that anyway?  It's how you get innovation in the market.  You don't like what one platform does, then use another or you make another.  You improve.

It's also fun (at least for me).  I like learning new hardware, frameworks and approaches.  It's part of the tinkering that a lot of us like to do.  At least for me, it doesn't get old (at least it hasn't yet).

It's kind of weird how the cycle of popularity goes. Mainframes with
  terminals - thin client. PC - thick client. Web browser - thin client.
  Phone app - thick(ish) client.

I don't see this as a "cycle of popularity", but as evolution and improvement.


答:
Not as long as people have ideas they think are better than the existing [your thing here]. [your thing here] could be hardware, software, user interaction methods (think wii vs PlayStation/XBox), or whatever.

It will always be this way, hopefully.  When it stops, new ideas are gone.

Title: Do you think we will ever settle on a "standard" platform?

Body:
The recent explosion of phone platforms has depressed me (slightly), and made me wonder if we will ever reach any kind of standard for presentation?

I don't mean language or IDE.  Different languages have different strengths and I can see that there may always be a need for disparity, although I do note that languages are merging somewhat in functionality, with traditional imperitive languages like C++ now supporting things like lambdas.

What I'm really talking about is a common presentation mechanism. Before smart phones and tablets came along, the web seemed to be finally becoming a reasonable platform for presenting an application that was globally accessible, not just geographically, but by platform too. Sure there are still (sometimes infuriating) implementation differences and quirks, but if you wrote a decent site you knew it could be accessed on anything from a PC to a phone to a C64 running the right software. "Write Once Run Anywhere" seemed to finally be becoming a reality.

However, in the last few years we've seen an explosion of mobile operating systems, and the ubiquitous "app". A good site is no longer enough, you need a native "app", and of course we have a sudden massive disparity in OS, language, and APIs needed to write them as each battles for supremecy.

It's kind of weird how the cycle of popularity goes.
Mainframes with terminals - thin client.
PC - thick client.
Web browser - thin client.
Phone app - thick(ish) client.

I just wonder if you think there will ever be a global standard for clients, or whether the "shiny and different" cycle will always continue along with the battle of the tech du jour.


问:
Drawbacks of code kata
Code kata is a concept that proposes to hone programmer's skill by doing small problems many times trying to improve the code at each iteration.  The name comes from an analogy to martial art kata where forms (aka kata) are practices done over and over leading to improvements.  </p>

<p>From the reaction I got to my last question on the topic, I wish to know what are the draw backs of this approach? </p>

答:
Doing only katas, in a martial art context, is useless. As said before, katas are choregraphy of movements that teaches you the form. You know the movements but you don't know how to apply them. You don't know in which situations they are useful. If you really want to do something useful with them, you need the application(s), which is the Bunkai.

In a software context, the code kata would be the algorithm, tool, design pattern or any other technology. Knowing it is good, but you need to apply it to understand it. You need to use it in different contexts to really master it. The Bunkai would be a concrete situation where that kata or part of kata is useful.

I don't see anything wrong with that methodology, this is how about anything is mastered: learn something, practice it, add a detail, practice, add another detail, practice, etc.

Title: Drawbacks of code kata


问:
Why is verbosity bad for a programming language?
I have seen many people around complaining about verbosity in programming languages. I find that, within some bounds, the more verbose a programming language is, the better it is to understand. I think that verbosity also reinforces writing clearer APIs for that particular language.

The only disadvantage I can think of is that it makes you type more, but I mean, most people use IDEs which do all the work for you.

So, What are the possible downsides to a verbose programming language?

答:
A number of people have hinted at something that I think should probably be stated explicitly.

Verbosity tends to favor understanding at the microscopic level -- it generally leads to an individual statement being easy to read and understand. Verbosity also tends to reduce the degree of familiarity with the language necessary to understand it (to at least some degree). The most verbose languages (e.g., COBOL) can be at least somewhat readable even by complete non-programmers.

Concision tends toward the opposite: it helps understanding at the macroscopic level, especially by programmers with the most intimate familiarity with that particular language. At the same time, lack of familiarity with that specific language can prevent even rudimentary understanding, even from programmers who are otherwise quite experienced.

As such, readability varies widely depending on the target audience. On one hand, consider a large, complex project being written and maintained by people who work almost exclusively on that project, most of whom have substantial programming background and education (e.g., lots of PhDs). This will tend to favor a more concise language.

At more or less the opposite extreme, consider a project that's considerably simpler (though possibly still quite large) that's maintained primarily by people who really specialize in the business associated with the software, rather than in the software itself. This will almost certainly favor a much more verbose language.


答:
The point of compiling is to get something useful for the computer. ie something that can run. Why compile to something that may even be higher level than what you wrote it in?

I like the strategy of .NET better. Compile everything to a common language. This gives the benefit of the languages being able to communicate without needing to create (N^2)-N cross language compilers.

For example if you had 10 programming languages you would only need to write 10 compilers under the .NET model and they could all communicate with each other. If you made all possible cross language compilers you would need to write 90 compilers.  That's a lot of extra work for little benefit.

Title: Why aren't there automated translators from one programming language to another?

Body:
Most programming languages are Turing complete, which means that any task that can be solved in one language can be solved in another one, or even on Turing machine. Then why aren't there automatic translators that can convert programs from any given language to any other language? I've seen couple attempts for two languages, but they always work only on a limited subset of a language and can hardly be used for converting real projects.

Is it possible, at least in theory, to write 100% correct translator between all languages? What are the challenges in practice? Are there any existing translators that work?


问:
Functions that only call other functions. Is this a good practice?
I'm currently working on a set of reports that have many different sections (all requiring different formatting), and I'm trying to figure out the best way to structure my code. Similar reports we've done in the past end up having very large (200+ line) functions that do all of the data manipulation and formatting for the report, such that the workflow looks something like this:

```<code>DataTable reportTable = new DataTable();
void RunReport()
{
    reportTable = DataClass.getReportData();
    largeReportProcessingFunction();
    outputReportToUser();
}
</code>```

I would like to be able to break these large functions up into smaller chunks, but I'm afraid that I'll just end up having dozens of non-reusable functions, and a similar "do everything here" function whose only job is to call all these smaller functions, like so:

```<code>void largeReportProcessingFunction()
{
    processSection1HeaderData();
    calculateSection1HeaderAverages();
    formatSection1HeaderDisplay();
    processSection1SummaryTableData();
    calculateSection1SummaryTableTotalRow();
    formatSection1SummaryTableDisplay();
    processSection1FooterData();
    getSection1FooterSummaryTotals();
    formatSection1FooterDisplay();

    processSection2HeaderData();
    calculateSection1HeaderAverages();
    formatSection1HeaderDisplay();
    calculateSection1HeaderAverages();
    ...
}
</code>```

Or, if we go one step further:

```<code>void largeReportProcessingFunction()
{
    callAllSection1Functions();

    callAllSection2Functions();

    callAllSection3Functions();
    ...
}
</code>```

Is this really a better solution? From an organizational point of view I suppose it is (i.e. everything is much more organized than it might otherwise be), but as far as code readability I'm not sure (potentially large chains of functions that only call other functions).

Thoughts?

答:
I think this is mostly personal preference. If your functions were going to be reused (and are you sure you couldn't make them more generic and reusable?) I would definitely say split them up. I've also heard that it's a good principle that no function or method should have more than 2-3 screens of code in them. So if your big function just goes on and on, it might make your code more readable to split it up.

You don't mention what technology you're using to develop these reports. It looks like you might be using C#. Is that correct? If so, you might also consider the #region directive as an alternative if you don't want to split your function into smaller ones.


答:
Yes.

If you have a code segments that needs to be used in multiple places own function.  Otherwise if you have to make a change to the functionality you will need to make the change in multiple places.  This not only increase work but increases the risk of bugs showing up as the code gets changed in one place but not another or where a but is introduced in one place but not the other.

It also helps make the method more readable if descriptive method names are used.

Title: Functions that only call other functions. Is this a good practice?

Body:
I'm currently working on a set of reports that have many different sections (all requiring different formatting), and I'm trying to figure out the best way to structure my code. Similar reports we've done in the past end up having very large (200+ line) functions that do all of the data manipulation and formatting for the report, such that the workflow looks something like this:

```<code>DataTable reportTable = new DataTable();
void RunReport()
{
    reportTable = DataClass.getReportData();
    largeReportProcessingFunction();
    outputReportToUser();
}
</code>```

I would like to be able to break these large functions up into smaller chunks, but I'm afraid that I'll just end up having dozens of non-reusable functions, and a similar "do everything here" function whose only job is to call all these smaller functions, like so:

```<code>void largeReportProcessingFunction()
{
    processSection1HeaderData();
    calculateSection1HeaderAverages();
    formatSection1HeaderDisplay();
    processSection1SummaryTableData();
    calculateSection1SummaryTableTotalRow();
    formatSection1SummaryTableDisplay();
    processSection1FooterData();
    getSection1FooterSummaryTotals();
    formatSection1FooterDisplay();

    processSection2HeaderData();
    calculateSection1HeaderAverages();
    formatSection1HeaderDisplay();
    calculateSection1HeaderAverages();
    ...
}
</code>```

Or, if we go one step further:

```<code>void largeReportProcessingFunction()
{
    callAllSection1Functions();

    callAllSection2Functions();

    callAllSection3Functions();
    ...
}
</code>```

Is this really a better solution? From an organizational point of view I suppose it is (i.e. everything is much more organized than it might otherwise be), but as far as code readability I'm not sure (potentially large chains of functions that only call other functions).

Thoughts?


答:
It depends. If all the functions you are creating are truly a single unit of work then it's fine, if they are just lines 1-15, 16-30, 31-45 of their calling function that is bad. Long functions are not evil, if they do one thing,  if you have 20 filters for your report having a validate function that checks all twenty is going to be long. That's just fine, breaking it up by filter or groups of filters is just adding extra complexity for no real benefit.

Having a lot of private functions also makes automated unit testing more difficult, so some will try to avoid it for that reason, but this is rather poor reasoning in my opinion as it tends to create a larger more complex design to accommodate testing.


答:
You said you're using C#, so I wonder if you could build a structured class hierarchy taking advantage of the fact that you're using an object-oriented language? For example, if it makes sense to break up the report into sections, have a Section class and extend it for client specific requirements.

It may make sense to think about it as if you were creating a non-interactive GUI. Think about the objects you might create as if they were different GUI components.  Ask yourself what a basic report would look like? How about a complex one? Where should the extension points be?


答:
Since you are using C#, try and think more in objects. It looks like you are still coding procedurally by having "global" class variables which subsequent functions depend on.

Breaking out your logic in separate functions is definitely better than having all the logic in a single function.  I would bet you could go further by also separating the data the functions do work on by breaking this out in several objects.

Consider having a ReportBuilder class where you can pass in the report data. If you can break ReportBuilder into separate classes do this as well.


答:
This is commonly referred to as functional decomposition and is generally a good thing to do, if done correctly.

Also, the implementation of a function should be within a single level of abstraction. If you take largeReportProcessingFunction, then its role is to define which different processing steps are to be taken in which order. The implementation of each of those steps is on an abstraction layer below and largeReportProcessingFunction shouldn't depend on it directly.

Please note that this however is a bad choice of naming:

```<code>void largeReportProcessingFunction() {
    callAllSection1Functions();
    callAllSection2Functions();
    callAllSection3Functions();
    ...
}
</code>```

You see callAllSection1Functions is a name that doesn't actually provide an abstraction, because it doesn't actually say what it does, but rather how it does it. It should be called processSection1 instead or whatever is actually meaningful in the context.

Body:
This is commonly referred to as functional decomposition and is generally a good thing to do, if done correctly.

Also, the implementation of a function should be within a single level of abstraction. If you take largeReportProcessingFunction, then its role is to define which different processing steps are to be taken in which order. The implementation of each of those steps is on an abstraction layer below and largeReportProcessingFunction shouldn't depend on it directly.

Please note that this however is a bad choice of naming:

void largeReportProcessingFunction() {
    callAllSection1Functions();
    callAllSection2Functions();
    callAllSection3Functions();
    ...
}

You see callAllSection1Functions is a name that doesn't actually provide an abstraction, because it doesn't actually say what it does, but rather how it does it. It should be called processSection1 instead or whatever is actually meaningful in the context.


问:
Learning embedded systems development without supervision
I have graduated this year and got my first job involved with C programming, Linux administration and embedded systems development. I feel quite content with my job, but I'm afraid I won't become a successful programmer in this field. I'm a lone developer on my job now, with my teammates being hardware experts, there is no one to guide me or teach me in the ways of embedded programming, I have to study all on my own.

So here are my questions. Is it possible to become a good embedded systems developer starting from an entry level position without any supervision by senior programmers? How could I become one (perhaps with the help of forums, IRC channels, good textbooks)? How long would it take?

UPDATE: So far, I have received a handful of helpful answers, but I realized that I need some specific pointers on the subject.

So, here are my particular questions:

What are some good textbooks one can use to learn embedded development?

What specialized online communities can be helpful for an aspiring embedded developer (not counting general stuff like Stack Overflow, Reddit and so on)?

What are the most interesting blogs dedicated to embedded development?

答:
I'm a lone developer on my job now,
  with my teammates being hardware
  experts

Don't hesitate to use them as a resource.  One of the hardest things for embedded programmers who are not EE's is getting comfortable with the hardware.  It is very useful to be able to read a schematic (even if you can't design one), and understand how to use an scope to look at signals coming out of the microcontroller.


答:
The fact that you use numbering to distinguish functions suggests that there are underlying problems with duplication or a class doing too much.  Breaking up a large function into many smaller functions can be a useful step in refactoring out duplication, but it should not be the last one.  For instance, you create the two functions:

```<code>processSection1HeaderData();
processSection2HeaderData();
</code>```

Aren't there any similarities in the code used to process section headers?  Can you extract a common function for both of them by parameterizing the differences?

Body:
The fact that you use numbering to distinguish functions suggests that there are underlying problems with duplication or a class doing too much.  Breaking up a large function into many smaller functions can be a useful step in refactoring out duplication, but it should not be the last one.  For instance, you create the two functions:

processSection1HeaderData();
processSection2HeaderData();

Aren't there any similarities in the code used to process section headers?  Can you extract a common function for both of them by parameterizing the differences?


问:
Multi-tenancy - single database vs multiple database
We have a number of clients, whose systems share some functionality, but also have quite a degree of diversity. The number of clients is growing - always a healthy thing! - and the diversity between their businesses is also increasing.

At present there is a single ASP.Net (Web Forms) Web Site (as opposed to web project), which has sub-folders for each tenant, with that tenant's non-standard pages. There is a separate model project, which deals with database access and business logic.

Which is preferable - and most importantly, why - between having (a) 1 database per client, with only the features associated with that client; or (b) a single database shared by all clients, where only a subset of tables are used by any one client.

The main concerns within the business are over:

maintenance of multiple assets - backups, version control and the like

promoting re-use as much as possible

How would you ensure these concerns are addressed, which solution is preferable, and why? (I have been also compiling responses to similar questions)

答:
You're missing some concerns. Problems will come with growth. If you can assume that someday you'll grow bigger than one DB server - one complex database will definitely cause you a headache. Unless you'll invest in architecture in advance. But it is also expensive step )

So, just do not forget, that it is both many times cheaper and many times easier to scale out few different databases, than huge ones )


答: If you are using SQL Server, use one database but use schemas. Use dbo for stuff that is general to all clients and create a schema for each client and make that the default schema for users from that client. Now you can have a general object (say a getBudget proc) in the dbo schema and a customized one for the client in their schema with the same name. </p>

Title: Multi-tenancy - single database vs multiple database

Body:
We have a number of clients, whose systems share some functionality, but also have quite a degree of diversity. The number of clients is growing - always a healthy thing! - and the diversity between their businesses is also increasing.

At present there is a single ASP.Net (Web Forms) Web Site (as opposed to web project), which has sub-folders for each tenant, with that tenant's non-standard pages. There is a separate model project, which deals with database access and business logic.

Which is preferable - and most importantly, why - between having (a) 1 database per client, with only the features associated with that client; or (b) a single database shared by all clients, where only a subset of tables are used by any one client.

The main concerns within the business are over:

maintenance of multiple assets - backups, version control and the like

promoting re-use as much as possible

How would you ensure these concerns are addressed, which solution is preferable, and why? (I have been also compiling responses to similar questions)


答:
A significant portion of your question is answered here:

Why are scripting languages (e.g. Perl, Python, Ruby) not suitable as shell languages?

Here's an excerpt from my answer to that question:

There are a couple of differences that I can think of; just thoughtstreaming here, in no particular order:

Python & Co. are designed to be good at scripting. Bash & Co. are designed to be only good at scripting, with absolutely no compromise. IOW: Python is designed to be good both at scripting and non-scripting, Bash cares only about scripting.

Bash & Co. are untyped, Python & Co. are strongly typed, which means that the number 123, the string 123 and the file 123 are quite different. They are, however, not statically typed, which means they need to have different literals for those, in order to keep them apart. Example:

Ruby: 123 (number), Bash: 123

Ruby: '123' (string), Bash: 123

Ruby: /123/ (regexp), Bash: 123

Ruby: File.open('123') (file), Bash: 123

Ruby: IO.open('123') (file descriptor), Bash: 123

Ruby: URI.parse('123') (URI), Bash: 123

Ruby: `123` (command), Bash: 123

Python & Co. are designed to scale up to 10000, 100000, maybe even 1000000 line programs, Bash & Co. are designed to scale down to 10 character programs.

In Bash & Co., files, directories, file descriptors, processes are all first-class objects, in Python, only Python objects are first-class, if you want to manipulate files, directories etc., you have to wrap them in a Python object first.

Shell programming is basically dataflow programming. Nobody realizes that, not even the people who write shells, but it turns out that shells are quite good at that, and general-purpose languages not so much. In the general-purpose programming world, dataflow seems to be mostly viewed as a concurrency model, not so much as a programming paradigm.

I have the feeling that trying to address these points by bolting features or DSLs onto a general-purpose programming language doesn't work. At least, I have yet to see a convincing implementation of it. There is RuSH (Ruby shell), which tries to implement a shell in Ruby, there is rush, which is an internal DSL for shell programming in Ruby, there is Hotwire, which is a Python shell, but IMO none of those come even close to competing with Bash, Zsh, fish and friends.