text:
How can neural networks be used to generate rather than classify?
In my experience with Neural Nets, I have only used them to take input vectors and return binary output.
But, here in a video, https://youtu.be/ajGgd9Ld-Wc?t=214, Kai Fu Lee, renowned AI Expert shows a deep net which takes thousands of samples of Trump's speeches and generates output in the Chinese Language.
In short, how can deep nets/neural nets be used to generate output rather than giving answer yes or no? Additionally, how are these nets being trained? Can anyone here provide me a simple design to nets that are capable of doing that?

Think of a neural network as a universal function approximator (With infinite width under a set of constraints this is actually provable). Now when discussing generation in the context you have provided, you essentially want to draw from some distribution $p(y|c)$ where $y$ is your output and $c$ is your context or input.
Theorem: For any distribution $\Omega$, if we take $z \sim \mathcal{N}(0,I)$, there exists a function $f$ where $f(z) \sim \Omega$.
Given the above theorem (for the purposes of this post I don't need to prove it, but its very similar to the universal approximation theorem proof) and if we take neural networks as a pseudo-universal function approximator, if we have a valid objective or training procedure that can learn the parameters of $f$, sampling is as easy as sampling $\mathcal{N}(0,I)$ and then applying $f$.
So the trick really is finding a good training procedure, and this is where you see GANs, VAEs and other models/schemes come into play.
Everything I've said above works really well when there isn't autocorrelation like in text, but when there is, the above methodology would result in a combinatorially large output space which isn't realistic with a vocabulary size usually spanning somewhere between a couple thousand and a couple hundred thousand. So to handle this they model the joint by taking advantage of that autocorrelation by modeling the joint probability as its bayesian decomposition.
$$p(\vec w) = p(w_0)\prod_{i=1}^{N-1}p(w_i|w_{<i})$$
Now that there is a framework to efficiently model this type of output, were back into the position as before where we are looking for clever training schemes. In this case you'll see commonly RNN's or other sequential model training with teacher forcing (@nbro described this in his answer too), or using GAN like compositions using either reinforcement learning to handle the lack of differentiability in sampling or using approximations like Gumbel-Softmax or Intermediate Loss Sampling (method I actually developed)
I hope this answered your question.

I think I really need to pass high school doing mathematics before reaching the discussion to the level of mathematics you are talking...

Look into basic probability and I think your golden for the understanding. If you want something to get your hands dirty, just google "GAN GitHub" and you'll find tons of repos for starting point of generative models. Add more specifics to your search if your looking for something more specific. Being in high school shouldn't stop you-- Its actually great you're getting into this so early!

hehe... I think I'm too much addicted to doing AI using C++ , Python... I've recently made an AI that generates Music using Long Term/Short Term Memory manipulation using AI, wanted to add some lyrics but I needed a better voice, so, I'm now learning to generate voice using AI.

What is the meaning of $f(z) $~ $\omega$?

@AbhasKumarSinha: It means that if you have some imagined target distribution over your data $\Omega$ that you want to generate but don't have a random function for it, then you can create that random function by mapping from the normal distribution in $z$. The function $f()$, if well trained, and fed with samples from the normal distribution as inputs, will create samples over your output space - which is what you want, a random generator for your target data.

If the output can either be yes or no, then you have a discrete and binary output, so this problem is called binary classification, that is, it is the task of classifying (or categorizing) the input into one of two categories (or classes). You can also have a neural network with an output that can take more than two possible discrete values, which is can be used to solve a multi-class classification problem. For example, a neural network that outputs a sentence, which is composed of $n$ words, where $n>1$. In general, the output does not necessarily need to take a value from a set of discrete values (or classes), but it can also take a numeric value (e.g. a floating-point number). In that case, the problem is called regression. For example, the task of predicting the height of a person (a numeric value) given a picture of the same.
There are different types of neural networks. The simplest neural network is either a perceptron (if you consider it a neural network) or a multi-layer feed-forward neural network, that is, a neural network with only forward connections, with possible multiple layers. There are also convolutional neural networks (CNNs) and recurrent neural networks (RNNs), which are more sophisticated neural networks that are more suited for processing respectively imagery or sequences. There are also generative neural networks (e.g. variational auto-encoders), which are trained to learn a distribution, from which you can then sample.
In your specific example, the sentence could have been generated with a recurrent neural network or a generative model (or a combination of both). More precisely, a recurrent generative model could have been trained to learn the rules of either the English or Chinese language. Then you sample from this distribution to generate sentences. In principle, a sentence could also be generated with simpler neural networks (such as a multi-layer perceptron), but, in practice, this may be more inefficient.


text:
What are methods human actors use to imitate robots?
Robot technology is usually thought from an engineering perspective. A human programmer writes a software this executed in a robot who is doing a task.
But what would happen, if the project is started with the opposite goal? The idea is, that the human becomes the robot by himself. That means, the human is using makeup to make his face more mechanically, buys special futuristic clothing which mirrors the light and imitates in a roleplay the working of a kitchen robot.
What are methods human actors use to imitate robots?

"What is the preferred method to imitate a robot for a human actor?", If you're talking about real actors (like Daniel Day-Lewis), I think this question is probably off-topic here. Only because a robot is related to AI, you're specifically asking for acting techniques, but people here probably have never had an experience in acting (maybe apart from me). Furthermore, the question in the title is different than the question in the body. Yes, of course, a human can try to imitate a robot. How well can it do it? It depends on the robot. It may be difficult for a human to imitate a quadrotor.

This is an interesting question--need to think about whether it's on-topic.  Humans imitating robots has a history of many decades, from the original Westworld to Lieutenant Data, Terminator  & Stepford Wives, Blade Runner to the Alien franchise.  (Fassbender's recent performance  in Covenant was especially interesting b/c he played two androids who look identical, but have different neural structures.)  Also the Westworld reboot and the recent Ex Machina.

Pretty much all robots in all sci-fi films to the present are "humans imitating robots", even if it is via CGI, someone has animated that CGI to be an imitation of how they think a robot could act, and someone wrote dialog. I don't see this as on-topic, as it is too focused on specific acting skills - which will lead to conjecture and opinionated answers. However related questions about representation of AI in media or in public imagination could be - e.g. What led to specific choices of "AI" representation and behaviour by the director and actors in the movie Ex Machina?

I've added a couple of tags as the way human actors portray androids informs the perception of the general public in regard to robots/AI.  In the context of mythology of AI and social impacts, this question would seem to be on-topic.

The great acting teacher Stella Adler wrote about mannerisms being a powerful tool for actors. Method acting in general focuses on natural performances based roughly on understanding the mindset of the character portrayed.
It's possible actors who have portrayed androids have observed industrial robots to inform their physicality, and many performances convey the idea, via movement, of a mechanical inner structure.  (It is often said that an "actor's body is their instrument".)
What is more interesting is actors trying to convey the cognitive structure of the androids.
With Arnold, and Terminator robots in general, the baseline performance is decidedly robotic, to convey their inhumanity. But the more advanced Terminators are able to mimic naturalistic human mannerisms, and even established human characters, to trick humans.
Lieutenant Data often used head motions, such as cocking his head slightly, to convey computation.  Here the character arc involved working to become more human, as this character draws heavily on Pinocchio, the wooden puppet that became a boy.  Overall Data's performance conveyed a lack of emotion, a definite reference to the logic-oriented Mr. Spock, although I recall episodes where Data experimented with "emotional circuits" and "humor circuits", where the output was intentionally inconsistent with natural human behavior.
Blade Runner, where the Tyrell Corporation's motto was "More Human than Human", presented the cutting edge Nexus-6 androids as having emotions, but, due to their artificially short life-spans, were portrayed as childlike in trying to reconcile extremely powerful feelings.  The Voight-Kampff Test, a form of Turing Test, used in the film to identify androids, relied on the emotional response to questions.
The key plot point of Do Androids Dream of Electric Sheep, the novel the film was based on, utilized what would be formalized as evolutionary game theory to hypothesize that empathy is a natural function of intelligence sufficiently advanced.  Deckard, who may or may not have been an android, and Rachel, who definitely was, are both capable of love.  This capacity informed their performances, to the extent that the androids came off as more human than the actual humans, due to the depth of their emotion.  This is also reflected in Blade Runner 2049 via the girlfriend-bot Joi, who us the most limited android, but the most human character in the film per her capacity to love (or at least simulate it.)
In the recent HBO Westworld reboot, the Androids replicate natural human mannerisms when playing their designated roles, but reset to more mechanical mannerisms when acting under their own agency.  This is reflected in Ex Machina, where the android mimics human emotions to pass a Turing Test and trick the human subject, only to revert to purely alien mannerisms after the android is free.  ("Alien" here used in the sense of non-human--it's possible the android is sentient as it seems to convey some degree of emotion in regarding the simulated human skin it will wear.)
The most interesting recent android performance may come from the recent Alien: Covenant where Michael Fassbender plays two identical androids, David and Walter, which have two distinct neural structures.  (David has the capacity to be creative, where Walter cannot.  In the film it is mentioned that David made people uncomfortable, so the creative functions were removed from subsequent models.)  The key difference in the performance seems to be that David demonstrates passion, and even emotions, where Walter is more clearly "robotic".

In general, the underlying approach of actors seems to have been to show the androids being distinct from humans, drawing a clear, though sometimes subtle, contrast.
Actors portraying androids have typically utilized robotic mannerisms to convey an artificial entity.

Actually this answer has convinced me that the question is on-topic, but perhaps poorly worded/framed.

@nbro Method acting originates with Stanislavski and is linked to modernism in theater (Ibsen, Stringberg, Checkov, initially) where naturalistic performances were called for.  I particularly like Adler, as opposed to Strasberg, b/c he focused on channeling real emotions in performance, where Adler felt this would drain the actors. She focused on accessing the emotions to find the mannerisms, so emotions could be convincingly simulated without the emotional drain per ad nauseam repetition.

@DukeZhou I deleted my previous comment. By the way, there are many professional actors that have not gone to an acting school. Marlon Brando used to think that everyone is an actor. So, in principle, you do not need to be taught to act. Anyway, I think this question is off-topic given we are now talking about acting and actors and not really about AI, but you have given an interesting answer and you put it into the AI context.

@nbro It is interesting though that Adler's technique focuses on simulating human emotions convincingly, which is the goal of human-like AI, sentience notwithstanding.

Disclaimer: The intent of this answer is to suggest a a parallel between methods of acting and machine learning, both in intent and application, and theory.  A large number of links are included for the convenience of readers new to the field, and there is not an exact correspondence of AI concepts to acting preparation techniques.
In my prior answer, I mentioned the method acting technique, and Stella Adler's interpretation of Stanislavski's method.  Bear in mind that the method is a post-empiricism approach, an attempt, in some sense, to create a science of acting in the sense of analysis, and an approach that is fundamentally algorithmic in the sense of process.  (The original manual is titled An Actor Prepares.)  Note that areas covered include action, imagination (creativity), units and objectives, emotional memory (accessing memory), and adaptation.  See Also: Classical Acting.)
Note also that plays are aptly named.  Drama and comedy arise out of interplay of individuals, and the process of refining performance is the process of play—searching within a rule-space for the most optimal outcome.

Strong actors will rigorously research the character to create a mental model of the character's experience of the world, similar to a model-based agent.
Modern actors seek objectives, sometimes referred to as motivations, similar to goal-based agents.

The model has many dimensions, and there may be multiple layers of objectives in the sense of the subconscious.  (What does the character want?  What does it really want?  What does it really really want?) This also applies to the contexts for any choice, which are multiple (personal, societal, economic, etc.)

Actors observe human behavior for the purpose of imitating it, commonly referred to as "people watching".

As you note, actors preparing for role of robot may observe machinery, with the purpose of indicating for an audience that quality.  Actors may also observe other actors, although novelty in performance is typically understood to be optimal.

Actors will access emotional memory, alternately referred to as "sense memory" & "emotional recall" (affective memory), either to produce a physical effect or or analysis. Output are signifiers. They create a state space which they can return to access on command.

Essentially it's a form of "memory palace" (method of loci) where events take the place of locations.

Actors will improvise in preparation, to identify and test choices (actions and mannerisms), which involves decision theory. The choices of the other actors (rational agents) are factor, and influence each other.
Choices are selected in a genetic process, for fitness in environment, here defined as audience response.  The improvisation that leads to the performance is evolutionary, in that it optimizes via the rehearsal process, with director as audience, and later, in the case of live theater, in response to live audiences. (See also the Actor-critic model.)

It's not quite a monte-carlo, being more of an informed search, but does not exclude randomness.
Essentially, it's a process of analysis, trial-and-error, more analysis, repeat, similar to machine learning with heuristics.  It wouldn't be far off to say that there is a convergence, leading to what is perceived to be the optimal set of choices, (although it is more typical to say a performance "gels" or "comes together".)

It can be said that modern acting methods are themselves algorithmic processes, where the intent is maximizing utility, here audience response, which can carry significant economic consequences.

Modern actors are using methods similar to modern AI methods to imitate intelligent androids!
In the sense of Adler specifically, the technique involves simulating natural emotions to "trick" the observer, a form of Affective computing.   In other words, via training, the actor is doing what AI's are being trained to do in the context of interacting with humans.
The underlying method can be understood as a form of applied psychology and neuroscience, where the actor is accessing emotion for the purpose of analysis, and accessing specific parts of the brain on command to create observable signs.

I would remove some of the analogies you're giving (even though I understand you just want to compare actors to AI/CS concepts), because some of them may not be very accurate. For example, I would not compare humans to a ROM or an actor to a model- or goal-based agent. The analogies seem to be forced. I think your answer will be fine even without most of the analogies.

@nbro ROM in the sense that the memories (traumas & triumphs) are fixed, not rewritable--memory to be accessed only. That said, it may be shaky in that ROM is a general computing element, as opposed to something AI specific. In some sense I was trying to draw a parallel with intelligence, learning, and process in general.  (Actor/critic in AI was surely named in reference to the performing arts!)  Essentially, this is me explaining acting technique in using the formal language of AI--should be taken as a "fun" answer to supplement my prior one.

Human memory doesn't exactly work like that, even though I am not an expert or neuroscientist. Memories are fuzzy and tend to vanish. I just think that this answer can be a little bit misleading if people interpret it too literally (because of certain analogies)

@nbro That's the thing though, actors use "sense memory"/"emotional recall" to access these deeply ingrained emotions triggered by memories of specific experiences. Removed ROM and replaced with affective memory.
Also a form of memory palace.  I also added a disclaimer explaining why I include the links, but emphasizing your point that there is not an exact correspondence. Let me know if you have more thoughts on improvements.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How can I use deep reinforcement learning for vehicle rerouting in SUMO?
I want to use deep reinforcement learning for vehicle rerouting in SUMO, but I don't know how to start training the model.
I've already created road network and vehicle routing in SUMO-XML files (mymap.net.xml and mymap.rou.xml). Currently, I'm trying to train the model on Jupyter Notebook, importing TraCI library to control the SUMO simulator and allow for a reinforcement learning approach. However, I'm still confused in training step.

Do I need any traffic data to train my agent to take actions in the environment?
How can I train based on these SUMO-XML files I created?
Is it possible to run the simulation on Windows? or I need to change to Ubuntu instead?

I would appreciate if someone could guide me. Thank you in advance.

There is a platform called Flow, sounds like its what you're looking for
https://github.com/flow-project/flow

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How can an AI train itself if no one is telling it if its answer is correct or wrong?
I am a programmer but not in the field of AI. A question constantly confuses me is that how can an AI be trained if we human beings are not telling it its calculation is correct?
For example, news usually said something like "company A has a large human face database so that it can train its facial recognition program more efficiently". What the piece of news doesn't mention is whether a human engineer needs to tell the AI program each of the program's recognition result is accurate or not.
Are there any engineers who are constantly telling an AI what it produced it correct or wrong? If no, how can an AI determine if the result it produces is correct or wrong?

You know those "prove you are a human" tests you get on web sites all the time these days?  You know how they are often "click on all the pictures that have a stop sign in them"? Guess what: you are the human engineer constantly telling an AI whether it has correctly classified a stop sign.

@EricLippert obligatory xkcd

@StianYttervik: Even more obligatory: https://xkcd.com/1897/

It depends on what task the AI is trained to perform. If an AI were trained to generate mathematical proofs, an automated proof checker could ensure that its proofs were correct.

By "company A has a large human face database so that it can train its facial recognition program more efficiently" the article probably means that there is a training dataset $S$ of the form
$$
S = \{ (\mathbf{x}_1, y_1), \dots,(\mathbf{x}_N, y_N) \}
$$
where $\mathbf{x}_i$ is an image of the face of the $i$th human and $y_i$ (which is often called a label, class or target) is e.g. the name of the $i$th human. So, the programmer provides a supervisory signal (the label) for the AI to learn. The programmer also specifies the function that determines the error the AI program is making, based on the answer of the AI model and $y_i$.
This way of learning is called supervised learning (SL). However, there are other ways of training an AI. For example, there is unsupervised learning (UL), where the AI needs to find patterns in the data by aggregating objects based on some similarity measure, which is specified by the programmer. There's also reinforcement learning (RL), where the programmer specifies only certain reinforcement signals, that is, the programmer tells the AI which moves or results are "good" and which ones are "bad" to achieve its goal, by giving to the AI, respectively, a positive or negative reward. You can also combine these three approaches and there are other variations.

Are there any engineers who are constantly telling an AI what it produced it correct or wrong?

Yes, in the case of SL. In the case of RL, the programmer also needs to provide the reinforcement signal, but it doesn't need to explicitly tell the AI which action it needs to take. In UL, the programmer needs to specify the way the AI needs to aggregate the objects, so, in this case, the programmer is also involved in the learning process.

Comments are not for extended discussion; this conversation has been moved to chat.

Taking your example of the faces data, keep in mind that when the model is run on a new unseen image the model can only return the already seen identity which emerges as the closest match. The result may be incorrect. The chances of mis-identification are much lower as the number of features incorporated increases.
The input of the engineers lies at the level of the training data. Say we have a new photo of an individual that needs to be included in the model. The engineering task is now to morph that image to simulate different environments, angles of view, atmospheric conditions, lighting and so on to provide a large number of data input cases all of which will be "true" since the underlying features are all unchanged since the images are based on the same individual. Then the model is recalculated using the additional data.
Keep in mind too that adding a new set of data to an existing training set has the advantage that the parameters of the model are largely in the right ballpark already, and adding the new faces will make only small changes. Cross validation will show whether the addition has improved or spoiled the model.

"The chances of mis-identification are much lower as the number of features incorporated increases." This is not necessarily true.

how can an AI be trained if we human beings are not telling it its calculation is correct?

What you are looking for is called self-supervised learning. Yann LeCun, one of the originators behind modern neural network systems, has suggested that machines can reason usefully even in the absence of human-provided labels simply by learning auxiliary tasks, the answers for which are already encoded in the data samples. Self-supervision has already been successfully applied to a variety of tasks, showing improvement in multitask performance due to self-supervision. Unsupervised learning would in general be a subset of self-supervision.
Self-supervision can be performed in a variety of ways. One of the most common is to use parts of the data as input and other parts as labels, and using the "input" subset of the data to predict the labels.
Supervised learning looks like this:
model.fit(various_data, human_labels)

The human_labels correspond to entries in various_data, which we expect the model to predict.
Meanwhile, self-supervised learning can look something like this:
model.fit(various_data[:,:500], various_data[:,500:])

(Using Python array slice notation, some of the input data are used as training labels.)
For example, a machine could use half of the pixels in an image of a handwritten digit to try to predict the missing pixels. This is a form of self-supervision: Since the machine knows which pixels belong together in the same sample, it can "automatically" produce its own labeled data from the input itself, simply by using some inputs as outputs.
However, predicting pixels from other pixels is often not the desired task.
So instead, a neural network is often pretrained using self-supervised or unsupervised learning techniques, and then subsequently trained on some amount of human-labeled data as a form of transfer learning.
What the summary of the hypothetical news article promises is that self-supervision made the learning more efficient, not that it outgrew the need for any kind of human intervention. This is exactly what we get from successful self-supervision in pretraining.
In the best possible case, the machine learns to "recognize" each class of digit 0-9 but it still does not know how to ground its own internal labels to the human's labels. Then a human supplying the mapping between the machine's labels and the human-specified IDs would be the only step necessary to upgrade the self-supervised machine to one that is directly useful for digit recognition.
There will always be a need for humans to train a machine via direct supervision in order for the machine to learn the intended task. In order to solve a specific problem, a sufficient degree of supervision is always required, and sufficient labels to reflect the intention must be provided.

I highly doubt that the article was specifically talking about self-supervised learning, but I think it is useful to mention this approach.

@nbro the OP seems to be asking about the principle generally. Many forms of unsupervised learning (autoencoders and their ilk) are included in self-supervised learning, so this is a fairly general response. I am not aware of any form of unsupervised pretraining or auxiliary training that doesn't fit into the paradigm of self-supervision.

I think you're probably looking at this the wrong way around. A conventional, old-fashioned AI doesn't make a guess, then require confirmation as to whether that guess was right or wrong. Instead, (in the simplest case) it undergoes a one-off computationally intensive "training"/"learning" phase, during which you feed it an enormous number of correct answers (which are labelled as correct) and an even more enormous number of incorrect answers (which are labelled as incorrect). Using whatever learning mechanism it has at its disposal, it then identifies some underlying structure in the "corrects" that doesn't exist in the "incorrects". When, in the future, it encounters something new that seems to also exhibit this structure, then it will classify this as a "correct". It might do rather well, or it might do terribly. Once the one-off training phase is done, it's stuck with whatever capability it has.
Let's say the company you mention is called Facebook and they have a feature that allows you to "tag" your friends in photos. No need to pay engineers to create the largest labelled image database in human history in order to train your AI.

Facebook's face-identification receives constant feedback from users confirming (or denying) the identity of a potential match

@Valorum Correct.

What you are missing is what the news story does't mention and gloss over. When a news article says:

company A has a large human face database so that it can train its facial recognition program more efficiently

What it really means is:

company A has a large database of human faces along with additional information such as the identity of the person the face belong to that was created by other humans so that they can use this data set to train its facial recognition program

How training works is basically as follows:

You have a large database of correct (or almost all correct, ideally it should be correct) information that you want to relate one to the other. For example images of faces along with who that face belongs to.
You split this large database into several sets.
You use one set to train the AI.
After looping through the training set you use one or more of the other sets to test the AI and check if the training works.
If you've done this before compare the performance of the current AI to previous AI. Else go to 6.
Tweak some parameters of the AI to try to improve performance.
Go to 2 until you are satisfied with the performance of the AI.

All the steps above are normally automated by scripts. The key here is that the original database has both the question you want to ask the AI (face) and the answer you want the AI to learn (person).
Yes, humans are involved in training the AI but the involvement happens earlier at the database gathering stage.

The trick with unsupervised learning is that the AI doesn't learn that something is a face or not, it just sees unnamed patterns that the researchers need to then name.
Let's say you feed it a dataset with one million pictures in order to train a facial recognition algorithm. After training, the AI will have found a few patterns in the pictures based on the parameters of each picture such as color, lighting, topography, etc. However, without labels (supervised learning) the AI doesn't know what exactly it found, so a researcher then needs to label those patterns. You don't need a label to tell that a picture of a face is mostly different than the picture of a building. You need a label to tell you that one is a "face"and the other is a "building".

But what if the program is designed to verify the identity of people? Let's say the AI program is used to determine "if the newly uploaded images from a bank branch of Alice matches our database record of Alice photos/3D face models". In this case, the program would always produce an answer (be it a percentage point or a boolean value) and if no human is checking its result, I suppose the AI would only reinforce its original pattern it found by "training" itself (no matter the pattern is accurate or not)?

@Mamsds You would need some kind of supervised label to verify identity. Given a new image, an unsupervised method could return a set of similar faces and a set of dissimilar faces, but you'd need at least some faces that are labeled "Alice" to see if they land in the set of similar faces or not. Without labels, the algorithm could say the new image belongs to Face Cluster 157, for example, but there's no way to link that cluster to Alice's identity unless you tell it.

I can't remember the researcher's name, but he specializes in psychology in Great Britain and has done a lot of work with machine learning and artificial intelligence.
The project he was working on that I read about earlier this year was one where they tried to deduce how humans learn.  They came up with the theory that we learn by making guesses about plausible and possible outcomes and that creates our expectations about reality.  When we are wrong, depending on the degree, we are possibly surprised or shocked or not affected at all.  They are working on creating AI that does not need human intervention, but to make guesses about outcomes before it performs tasks, and then update those expectations as it experiences more varying outcomes.
Extremely interesting stuff, and definitely closer to how sentient beings gain experience and grow as individuals.


text:
How to perform regression with multiple numeric (positive and negative) inputs and one numeric output?
I have a dataset with different types of numerical values (both negative and positive numerical values) for the inputs (for example, -40, -35, 1, 25, 39, etc., that is, multiple inputs) and single output numerical value (either negative or positive).
I have tried to use linear regression, but I haven't been so successful and I think one of the reasons is negative values.
What is the best way to deal with this scenario? What model should I use?
I am using Keras for my AI model.

Can you provide more details? e.g. how many input features do you have? Are they all integer? Output value is limited to a range or could be any integer?

negative values are not the problem, you need to scale your data. For example if your data range is $[-200, 200]$ then scale the data to range $[-1, 1]$ or you could scale them to have mean of $0$ and standard deviation of $1$. You can use scikit-learn library for data preprocessing for example.


text:
What are the properties of a model that is well suited for for high performance real-time inference
What are general best practices or considerations in designing a model that is optimized for real-time inference?

The only difference between any normal model and "real time" is the fact that the the real time model needs to be as fast as possible. Rephrasing your question to "What are the quick/fast models/techniques for real time [USAGE]". This question is lacking the usage. Is this image recognition in real time? Is it audio detection? Voice generation? What do you want to do in real time, then maybe people can help

I wouldn't agree that it depends on usecase and more on how inference is genrally implemented on the GPU. So for example keeping convolution kernels small but having more layers may present very different bottlenecks in gpu implementations than having large convolution kernels and few layers.

That's true, but we still don't know what you are even doing. What do you want to do in real time

Many things. I want to start with some pixel classification - but I want to do this only to learn about inference performance and implementation specifics. My whole goal is to acquie general best practices to follow during modeling when inference performance (on mobile gpus in particular) is a concern


text:
Curiosity Driven Learning affect optimal policy
I am trying to understand some of the different approaches used to overcome  sparse rewards in a reinforcement learning setting for a research project. Particularly, I have looked at curiosity driven learning, where an agent learns an intrinsic reward function based on the uncertainty of the next state that the agent will end up in as he takes action a in state s. The greater the uncertainty of the next state, the higher the rewards. This will incentive agent's to be more exploratory and it is used particularly in some games where a huge number of steps is needed before the agent reaches the terminal state where is he only then rewarded.
The curiosity driven approach as demonstrated in this paper:
https://pathak22.github.io/noreward-rl/ is able to learn faster than if a 0 rewards were used for each state, action.
To my knowledge, using different reward functions will affect the optimal policy obtained. Would curiosity driven learning therefore lead to a different policy as compared to whether a 0 reward was used ? Assume that for a 0 immediate reward system, it is able to derive a policy that reaches the goal state. Which of these 2 policies will be more optimal ?

The goal of curiosity driven learning is obviously to get better average original reward for trained policy. Global maximum on complete policy space for original reward will be bigger or same, but (if curiosity driven learning really works) trained approximation from curiosity could be better then trained approximation from original reward.

@mirror2image That is practically an answer, consider making it one


text:
Semi-supervised: Can I predict the label of purposely unlabelled observations?
Let's say I have a data set with of length N. A small proportion N2 is labeled. Can I remove some labels and then 'reverse' this action with a trained neural network? I could then use the same process to fill the other (N - N2) rows with labels.

What is the purpose of your trained neural network? Is it to label the data like N2 is labelled? Or is it to provide new labels?

purpose is to label to have N labels.

N different labels? So each data point has a different label? Or do you mean each data point is labeled with a label already existing in N2?

not N labels, let's say 2 labels. N2 rows have 2 labels, the other N - N2 have no labels.

How is this different from supervised learning, trained on your labeled data, and used on your unlabeled data?


text:
How can the V and Q functions take the expectation over a sum where the number of summands is random?
Assume the existence of a Markov Decision Process consisting of:

State space $S$
Action space $A$
Transition model $T: S \times A \times S \to [0,1]$
Reward function $R: S \times A \times S \to \mathbb{R}$
Distribution of initial state $p_0: S \to [0,1]$

and a policy $\pi: S \to A$.
The $V$ and $Q$-functions take expectations of the sum of future rewards.
Let's start off by $r_0:= R(x_0,\pi(x_0),x_1)$, where $\pi$ is the current policy while $x_0 \sim p_0$ and $x_1 \sim T(x_0,\pi(x_0),-)$ are random variables. With setting $\mu_i:= T(x_i,\pi(x_i),-),\rho_i:=R(x_i,\pi(x_i),-)$, I obtain
$$E[r_0]= \int_{\mathbb{R}} r d\mu_0^{\rho_0} = \int_S R(x_0,\pi(x_0),-)d\mu_0,$$
where $\mu_i^{\rho_i}:= \mu_i\circ \rho_i^{-1}$ is the pushforward of $\mu_i$ under random variable $\rho_i$. But the above quantity still depends on $x_0$ as both $\mu_o$ and $\rho_0$ depend on $x_0$. Intuitively, I would guess that one has to calculate the integral over every occuring random variable to obtain the overall expectation, i.e. $$E[r_0]= \int_S\int_S R(x_0,\pi(x_0),x_1)d\mu_0(x_1)dp_0(x_0)$$ is that correct ?
Now, the $V$ and $Q$-functions take the expectation over the sum $R_{\tau} = \sum^T_{t=\tau}\gamma^{t-\tau}r_t$, where the instant of termination $T$ itself is a random variable, and, besides that, the agent does not know its distribution, as it is not even included in the MDP model.
How can I take the expectation over a sum where the number of summands is random?
We cannot just calculate $\sum^{E[T]}(\dots)$, because $E[T]$ might not even be an integer.

Hi. Can you please simplify your question? Ask one question per post, if you want to attract the attention of more people. It seems to me you're asking multiple questions here.

For the question related to $V$ and $Q$ I feel like you are overthinking it (either that, or I don't understand your question). The sum of $R_\tau$ is a random variable so it is fine for it to have an expectation. You can circumvent the problem of $E[T]$ (the length of the trajectory) by just assuming, that the trajectory is infinite and that after $T$ the reward is zero.

@nbro Yes, I will split it into serveral questions.


text:
How to understand and visualize a trained RL agent's policy when the state space is high dimensional?
What are typical ways to understand and visualize a trained RL agent's policy when the state space is of high dimension (but not images)?
For example, suppose  state and action are denoted by $s=(s_1,s_2,\cdots,s_n)$ and $a=(a_1,a_2,\cdots,a_k)$. How do I determine which attribute of the state (e.g. an image pixel of video game) is most responsible for a particular action $a_j$? I would like to have, for each action $a_j, j=1,2,...,k$, a table that ranks the attributes of the observation.
My question may be a little bit vague, but if you have any thoughts on how to improve it please let me know!

What kind of policy is the agent following? Greedy? If it is random there is no relationship

I'm not really sure the answer to your question but maybe Figure 4 in this paper might guide you along the right path. Human level control through deep reinforcement learning


text:
Understanding Bayesian Optimisation graph
I came across the concept of Bayesian Occam Razor in the book Machine Learning: a Probabilistic Perspective. According to the book:

Another way to understand
  the Bayesian Occam’s razor effect is to note that probabilities must
  sum to one. Hence $\sum_D'  p(D' |m) = 1$, where the sum is over all possible data sets. Complex
  models, which can predict many things, must spread their probability mass thinly, and hence
  will not obtain as large a probability for any given data set as simpler models. This is sometimes called the conservation of probability mass principle.

The figure below is used to explain the concept:

Image Explanation: On the vertical axis we plot the predictions of 3 possible models: a simple one, $M_1$ ; a medium one, $M_2$ ; and a complex one, $M_3$ . We also indicate the actually observed
    data $D_0$ by a vertical line. Model 1 is too simple and assigns low probability to $D_0$ . Model 3
    also assigns $D_0$ relatively low probability, because it can predict many data sets, and hence it
    spreads its probability quite widely and thinly. Model 2 is “just right”: it predicts the observed data with a reasonable degree of confidence, but does not predict too many other things. Hence model 2 is the most probable model.

What I do not understand is when a complex model is used, it will likely overfit data and hence the plot for a complex model will look like a bell shaped with its peak at $D_0$ while simpler models will more likely have a broader bell shape. But the graph here shows something else entirely. What am I missing here?

Which chapter (page) of the book are these excerpts taken from?

@nbro from section 5.3.1 page 156.

The original graph for the aforementioned Bayesian Optimisation is similar to the graph in these slides (slide 18) along with the calculations.
So, according to the tutorial the graph shown should actually have the term $p(D|m)$ on the y-axis, thus making it a generative model.Now the graph starts to make sense, since a model with low complexity cannot produce very complex datasets and will be centred around 0, while very complex models can produce richer datasets which makes them assign probability thinly over all the datatsets (to keep $\sum_{D'}p(D'|m) = 1$).


text:
How do I deal with an erratic validation set loss when the loss on my training set is monotonically decreasing?

Model used
mobilenet_model = MobileNet(input_shape=in_dim, include_top=False, pooling='avg', weights='imagenet')
mob_x = Dropout(0.75)(mobilenet_model.output)
mob_x = Dense(2, activation='sigmoid')(mob_x)

model = Model(mobilenet_model.input, mob_x)

for layer in model.layers[:50]:
    layer.trainable=False

for layer in model.layers[50:]:
    layer.trainable=True

model.summary()

The rest of the code
in_dim = (224,224,3)
batch_size = 64
samples_per_epoch = 1000
validation_steps = 300
nb_filters1 = 32
nb_filters2 = 64
conv1_size = 3
conv2_size = 2
pool_size = 2
epochs = 20
classes_num = 2
lr = 0.000004
train_datagen = ImageDataGenerator(
    rescale=1. / 255,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True)
test_datagen = ImageDataGenerator(rescale=1./255)
train_generator = train_datagen.flow_from_directory(
        'output/train',  # this is the target directory
        target_size= in_dim[0:2],  # all images will be resized to 224*224
        batch_size=batch_size,
        class_mode='categorical')
#Found 6062 images belonging to 2 classes.
validation_generator = test_datagen.flow_from_directory(
        'output/val',
        target_size=in_dim[0:2],
        batch_size=batch_size,
        class_mode='categorical')
#Found 769 images belonging to 2 classes.
from keras.callbacks import EarlyStopping
#set early stopping monitor so the model stops training when it won't improve anymore
early_stopping_monitor = EarlyStopping(patience=3)
steps_per_epoch = 10
from keras import backend as K

def recall_m(y_true, y_pred):
        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
        possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
        recall = true_positives / (possible_positives + K.epsilon())
        return recall

def precision_m(y_true, y_pred):
        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
        predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
        precision = true_positives / (predicted_positives + K.epsilon())
        return precision

def f1_m(y_true, y_pred):
    precision = precision_m(y_true, y_pred)
    recall = recall_m(y_true, y_pred)
    return 2*((precision*recall)/(precision+recall+K.epsilon()))
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc',f1_m,precision_m, recall_m])

history = model.fit_generator(
        train_generator,
        steps_per_epoch=2000// batch_size ,
        epochs=50,
        validation_data=validation_generator,
        validation_steps=800// batch_size,
        callbacks = [early_stopping_monitor],
       )
test_generator = train_datagen.flow_from_directory(
        'output/test',
        target_size=in_dim[0:2],
        batch_size=batch_size,
        class_mode='categorical')
loss, accuracy, f1_score, precision, recall = model.evaluate(test_generator)
print("The test set accuracy is ", accuracy)
#The test set accuracy is  0.9001349538122272

From what I have gathered from this post and this article, I understand that the validation set is much smaller with respect to the training set. I have applied augmentation to the test set due to this and that boosted test set accuracy by 1%.
Please note that the test train split is "stratified" as here is a breakdown of each individual class
in test/train/validation folders
Test: Class 0: 7426
      Class 1: 631
Train: Class 0: 928
       Class 1: 80
Val: Class 0: 928
     Class 1: 79

I have used an 80/10/10 split for train/test/val respectively.
Can someone guide me on what to do so that I can ensure the accuracy is  95%+  and the validation loss graph is less erratic?

I am thinking of tuning the learning rate though it doesn't seem to be working by much.
Another suggestion is to use test time augmentation.
Also, the link on fast.ai has a comment like so

That is also part of the reasons why a weighted ensemble of different
performing epoch models will usually perform better than the best
performing model on your validation dataset. Sometimes choosing the
best model or the best ensemble to generalize well isn’t as easy as
selecting the lower loss/higher accuracy model.
4. Should I use L2 regularization in addition to the current dropout?

Applying augmentation of any kind to the validation set is a strict no-no and the dataset is generated by my company which I cannot get more of.

I've quickly looked at your source code. You're using a dropout rate of 0.75, which means you will be training only 0.25 percent of the network at every iteration. Maybe try reducing it to 0.5 and see if you get any improvement.

Thanks. I will try it out.


text:
Is there any resource that describes in detail a naive example-based machine translation algorithm?
I'm looking to develop a machine translation tool for a constructed language. I think that the example-based approach is the most suitable because the said language is very regular and I can have a sufficient amount of parallel translations.
I already know the overall idea behind the example-based machine translation (EBMT) approach, but I can't find any resource that describes a naive EBMT algorithm (or model) that would allow me to easily implement it.
So, I'm looking for either:

a detailed description,
pseudocode or
a sufficiently clear open-source project (maybe a GitHub one)

of a naive EBMT algorithm. So, I'm not looking for a software library that implements this, but I'm looking for a resource that explains/describes in detail a naive/simple EBMT algorithm, so that I am able to implement it.
Note that there are probably dozens of variations of EBMT algorithms. I'm only looking for the most naive/simple one.
I have already looked at the project Phrase-based Memory-based Machine Translator, but, unfortunately, it is not purely based on examples but also statistical, i.e. it needs an alignment file generated by, for example, Giza++ or Moses.

See https://github.com/topics/statistical-machine-translation. Maybe you will find something useful there!

I have not found any simple implementation of a naive EBMT system, but I found some articles, papers and books that may be helpful (although I haven't read them, apart from the first and last one), so I will list them below.
The web article Example-based machine translation provides a decent high-level explanation of example-based machine translation.
The paper Example-Based Machine Translation: A New Paradigm (2002) by Chunyu Kit et al. also seems to provide a detailed description of the EBMT approach, so this paper should provide you with details you need to implement an EBMT system.
The paper A framework of a mechanical translation between Japanese and English by analogy principle (1984) by Makoto Nagao introduced the example-based machine translation approach, so it will be at least historically relevant.
Additionally, the paper Example-Based Machine Translation of the Basque Language and the book Recent Advances in Example-Based Machine Translation (2003), which is not apparently freely available online, could also be useful.
Finally, the article Machine Translation. From the Cold War to Deep Learning gives a nice high-level overview of the main machine translation approaches, so that you can understand the differences between EBMT and other approaches (especially, in case you are not able to distinguish between EBMT and other MT, e.g. those that use a parallel corpus, such as the supervised statistical machine translation approaches).


text:
Derivation of regularized cost function w.r.t activation and bias
In regularzied cost function a L2 regularization cost has been added.

Here we have already calculated cross entropy cost w.r.t $A, W$.
As mentioned in the regularization notebook (see below) in order to do derivation of regularized $J$ (cost function), the changes only concern $dW^{[1]}$, $dW^{[2]}$ and $dW^{[3]}$. For each, you have to add the regularization term's gradient.(No impact on $dA^{[2]}$, $db^{[2]}$, $dA^{[1]}$ and $db^{[1]}$ ?)

But I am doing it using the chain rule then I am getting change in values for $dA^{[2]}$ , $dZ^{[2]}$, $dA^{[1]}$, $dW^{[1]}$ and $db^{[1]}$.
Please refer below how I calculated this ?

Can someone explain why I am getting different results?
What is the derivative of L2 reguarlization w.r.t $dA^{[2]}$ ? (in equation 1)
So my questions are
1) Derivative of L2 regularization cost w.r.t $dA^{[2]}$
2) How adding regularization term not affecting $dA^{[2]}$, $db^{[2]}$, $dA^{[1]}$ and $db^{[1]}$ (i.e. $dA$ and $db$) but changes $dW$'s ?

Its a good thing you are trying to derive the equations yourself, but it is quite difficult for someone to debug your equations for you. So I would suggest you to try to find the mistake yourself, it'll prepare you for future mistakes you are going to make. As for your question the $L2$ is w.r.t weights and hence impact is directly on weights, you don't have to go through chain rule of activations to reach a weight. All get the same weightage in the L2 term. A better way to derive this would be just to forget the cross entropy term and make your cost just L2.

This will result in a cost dependent linearly only on square of weights, whose derivative w.r.t weights you can easily take....there is no interfering terms of $dA$ and other stuff. The cost function $J_{reg}$ is modularized/independent. So try to find the derivatives of cross entropy and L2 individually like I suggested above and just add them. This is the most cited tutorial (as far as I have seen) for backprop: https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/

@DuttaA One of my question is why we need to avoid chain rule in case of derivative of regularised cost function especially for regularization part ? I know if I separate cost function into two part where part_1 is cross_entropy and part_two is l2_regularization cost and then calculate derivative where for part_1  we can use chain rule and for part_2 we don't need to use chain rule as this makes reglarization impact on dW zero then we can easily arrive at the derivative.


text:
How should I define the loss function when using DQN to estimate the probability density?
I'm doing a Deep Q-learning project. All of my rewards are positive and there are two terminal states. One of them has a zero reward and the other has a high positive reward.
The rewards are stochastic and my Q-network must generate non-zero Q-values for all states and actions. Based on my project, I must use these numbers to create a probability density. In other words, the normalized Q-values of each state generated by network define a probability density for choosing an action.
How should I define my loss function? Is there a project or paper which I could look at and decide how to define the loss function? I am searching for similar projects and their proposed Deep Q-learning algorithms.

Hi. Please, ask your second question in a different post (I removed it from this post). Also, can you clarify what you mean by "probability density"? Do you mean that you need to estimate the transition probability matrix of the underlying MDP using deep Q learning? If yes, how does this relate to the loss function?

@nbro: I would guess that "the normalized Q-values of each state generated by network define a probability density for choosing an action" is referring to generating a policy. But it is not clear which policy - e.g. is this a desire to make a behaviour policy based on Boltzmann sampling? That would be relatively normal Q learning. Whilst if this was the target policy i.e. end result from training, then it would be less standard, and the OP would be wanting to learn a non-optimal policy for some unexplained reason.

Also neither of those things should affect loss function choice for the Q values approximation. But that could be part of any answer. It may help if @bitWise explains why they think that the Q network should have a different loss function due to these choices.

@Neil Slater: Q-values generated by the network define the policy of choosing actions.  Instead of using an epsilon-greedy policy, I'm using the normalized q-values as the policy of my algorithm.

@bitWise: Which policy are you wanting to adjust? 1) The behaviour policy to guide exploration and focus more on "reasonable" options  . . . . or 2) The target policy which is the final output of the learning process? Both are suitable goals, although the second one is uncommon in Q learning, you could use Expected SARSA off-policy which includes Q learning.


text:
What is the role of convex optimisation in AI systems?
Convex optimisation is defined as:

I have seen a lot of talk about convex loss functions in Neural Networks and how we are optimising rewards or penalty in AI/ML systems. But I have never seen any loss function formulated in the aforementioned way. So my question is:
Is there any role of convex optimization in AI? If so, in what algorithms or problem settings or systems?

Is there any role of convex optimization in AI?

Yes, of course!

If so, in what algorithms or problem settings or systems?

The problem of finding the parameters of a support vector machine can be formulated as a convex optimization problem. Another example is linear regression.
See also the paper Convex Optimization: Algorithms and Complexity (2014) by Sébastien Bubeck, which also mentions SVM as a typical example.

I'll accept the answer after I do a bit of my own research.

@DuttaA Yes, don't worry. The first thing that came to mind when I read your question was SVM (because I am familiar enough with its details and I used it in the past). Anyway, I only skimmed through the first paragraphs of the paper I am linking to, but it looks like a paper that goes into the direction of your question.


text:
Non-linear regression with a neural network
I have to perform a regression on three curves as shown in the following plot. Here, accA (y-axis) is the dependent variable, and w (x-axis) is the independent variable. The sum of the three curves adds up to 1.
To perform regression for the three curves, I would like to use a neural network. What architecture/set-up should I use for this task?

Hi and welcome to AI SE! Just to clarify, what exactly do you want to predict here? Do you want to predict the value for each of the 3 curves individually?

Yes, exactly. I would like to fit three curves individually, basically, but in a way that the sums of their ouput always adds up to 1, for each choice of the variable w. I would like to use a neural network, instead of performing a regression manually.

@Fq00 It looks like you want to do time series prediction maybe? So the real goal is to predict the next value of of the 3 accA functions, given the past values of accA (and maybe w)?


text:
Are No Free Lunch theorem and Universal Approximation theorem contradictory in the context of neural networks?
To my understanding NFL states that, we cannot have an hypothesis (let's assume it is an approximator like NN in this case) class that can't achieve certain accuracy parameters $\leq \epsilon$ with probability greater than a certain $p$ given the number of points from which we can sample is upper bounded to $m$.
Whereas, the UAC states that an approximator like a NN given enough hidden units can approximate any function (to my knowledge the function must be bounded).
The point where these 2 clashes (as per my knowledge) is that if we increase the paramters in a NN, the UAC will start to hold good, but the VC dimension will increase (or hypothesis class becomes richer) and for the same $m$ our $\epsilon$ increases or $p$ decreases (not sure which one is affected).
So what are the gaps in my knowledge here? How do we make these 2 consistent with each other?


text:
How do I prove that $\mathcal{H}$, with $\mathcal{VC}$ dimension $d$, shatters all subsets with size less than $d-1$?
If a certain hypothesis class $\mathcal{H}$ has a $\mathcal{VC}$ dimension $d$ over a domain $X$, how can I prove that $H$ will shatter all subsets of $X$ with size less than $d$, i.e. $\mathcal{H}$ will shatter $A \subset X$ where $|A| \leq d-1$?

We can show that it is not true by a counterexample. For example, $X = \{1,2,3\}$ and $\mathcal H = \{\{\},\{1\},\{2\},\{1,2\}\}$ is the finite set hypothesis class. By definition, in this case, the $\mathcal{VC}$ dimension of $\mathcal H$ over the domain $X$ is $d=2$. Although $A = \{3\} \subset X$, whose size is smaller than the $\mathcal{VC}$ dimension, i.e $|A|<d=2$, it is not shattered by $\mathcal H$.


text:
Interpreting I/O Transformation Matrix in Convolution
I've been reading this article on convolutional neural networks (I'm a beginner) - and I'm stuck at a point.

What I understand: We have a 4x4 input, and want to transform it to a 2x2 grid. I'm visualising this as a kernel sliding over the 4x4 grid, with just the right number of strides, so as to get 4 outputs which constitute the 2x2 grid (there are animations right above this part of the page in the link attached). The model chooses to represent the 4x4 grid as a vector of length 16. Also, the 4x16 transformation matrix when pre-multiplied to the input vector, produces the output vector, which is mapped back to a 2D grid. Is this right?
Moving on, another screenshot from the same page-

Is it that both the matrices are really the same, and a lot of weights just happen to be zero in the weight matrix, which is what the second matrix is depicting? In that case even, why so many zeros?
P.S. Thanks a lot for being so patient and reading to the end of this post, I really appreciate it. I'm a beginner, and really interested in these topics, and I'm self studying from various online resources - hence, any help is appreciated. I hope this is the right platform for this post.

Hi and welcome to AI SE! Although I acknowledge you are confused, please, ask one question per post! See https://ai.stackexchange.com/help/on-topic (specifically, the section notes) for more details. Please, edit your post to ask only 1 question. Ask other questions in their separate post. I will be happy to answer your question, if I am able to. This is one of the reasons why you should ask one question per post: maybe someone is able to answer one question, but not the others. Btw, your questions are very interesting!

Maybe later I will provide a formal answer. Meanwhile, to provide you with some info, have a look at this answer. This should help you with the second part of your post. Btw, once you have an answer to your questions, you can also answer your own questions!


text:
What is the theoretical basis for the use of a validation set?
Let's say we use an MLE estimator (implementation doesn't matter) and we have a training set. We assume that we have sampled the training set from a Gaussian distribution $\mathcal N(\mu, \sigma^2)$.
Now, we split the dataset into training, validation and test sets. The result will be that each will have maximum likelihoods for the following Gaussian distributions $\mathcal N(\mu_{training}, \sigma^2_{training}), \mathcal N(\mu_{validation}, \sigma^2_{validation}), \mathcal N(\mu_{test}, \sigma^2_{test})$.
Now, let's assume the  case where $\mu_{validation}<\mu_{training}<\mu_{test}$ and $\mu_{training}<\mu<\mu_{test}$.
Clearly, if we perform validation using this split, then the model that gets selected will be closer to  $\mu_{validation}$, which will worsen the performance on actual data, whereas if we only used the training set, the performance could actually be better (this is the simplest case without taking into account the effect of variance).
So, we will have a $4!$ combinations between the means, and each one might improve or worsen the performance (probably in $50 \%$ cases performance will be worsened, assuming symmetry).
So, what am I missing here? Were my aforementioned assumptions wrong? Or does the validation set has a completely different purpose?

I think Cross-Validation serves a completely different purpose.
From your post, it looks like you think we would use CV to get a better estimate of the parameters of our model (i.e. the model parameters after cross validation are closer to the parameters of the test data).
In fact, we use CV to get an estimate of generalization error while keeping our test set outside the training process. That is, we use it to answer the question "What is the size of the difference between my training and testing performance likely to be?". If you have an estimate of this that you are confident in, you can be confident that when you deploy a model to your customers, the model will actually work as you expect.
If you're only going to build a single model, then you don't need cross validation. You just train the model on the training data, and test it on the test data. Then you have an unbiased estimate of generalization error.
However, we might want to try out many different kinds of models, and many different parameters (broadly, we might want to do hyperparameter tuning). To do this, we need to understand how generalization error changes as we change our hyperparameters, and then use this information to pick hyperparameter values that we think will minimize the actual error when we deploy the model.
You could do this by training different models on the training set, and then testing them on the test set, recording the difference in model performance on the two sets. If you use this as a basis to pick a model though, you have effectively pulled the test set inside your training process (model parameters were implicitly selected using the test set, since you picked the parameters with the lowest test error). This bias will make your true generalization error much larger than what you observed.
As a stop gap, you could split your training set into a 'real training' set and a validation set. You could train models on the 'real training' set, and then measure their performance on the validation set. The difference would be a biased (but hopefully still useful) estimate of generalization error. You could then test against the test set just once (at the end) to get an unbiased estimate that you can use to decide whether or not to deploy the model.
A better workflow is to use CV on the training set to get an estimate of generalization error during hyperparameter optimization. You get K samples for k-fold cross-validation, so you can do statistical testing to see whether one model truly has better generalization error than another, or whether it's just a fluke. This decreases the degree of bias in your estimates of generalization error. Then, once you've completed hyperparameter optimization, you can run your final model against the test set once to obtain a truly unbiased estimate of your final generalization error.

So the thing I am skeptical about is the part 'hopefully'. If the CV set is farther from the test set or true distribution in terms of mean and variance, then it'll potray the wrong model/hyperparameter choices as correct.

@DuttaA Ah, I think I see what you mean. This is actually a matter of some debate. I think the current consensus is that boorstrapping has better theoretical support than CV, but CV works fine in practice. I think I can write up a better version at a later date.


text:
Can feature engineering change the selection of the model according to the minimum description length?
The definition of MDL according to these slides is:

The minimum description length (MDL) criteria in machine learning says that the best description of the data is given by the model which compresses it the best. Put another way, learning a model for the data or predicting it is about capturing the regularities in the data and any regularity in the data can be used to compress it. Thus, the more we can compress a data, the more we have learnt about it and the better we
  can predict it.
MDL is also connected to Occam’s Razor used in machine learning which states that "other things being equal, a simpler explanation is better than a more complex one." In MDL, the simplicity (or rather complexity) of a model is interpreted as the length of the code obtained when that model is used to compress
  the data.

To put it in short according to MDL principle, we prefer predictors with relatively smaller description length (i.e can be described within a certain length) for a given description language (this definition is without delving into exact technical details as it is not necessary to the question).
Since MDL is dependent on the description language we use, can we say feature engineering can cause a change in the selection of the predictor?
For example as this picture shows:

To me, it seems that, in the first picture, we will require a longer description length predictor in Cartesian coordinates, as compared to a predictor in polar co-ordinates (just a single discerning radius needs to be specified). So, to me, it seems feature engineering changed the selection of the predictor to a relatively simple one (in the sense that it will have a shorter description length). Thus feature engineering has changed the description length required for out predictor. Did I make any wrong assumptions? If so why?

You're completely right. For points in cartesian coordinates, your final model would require at least 3 parameters a,b,c since the equation of a circle is: $(x-a)^2+(y-b)^2=c^2$ while in polar coordinates you final model would require only one parameter since a vertical line $p=a$ is enough to separate the two classes.

I think the wrong assumption here is that you've forgotten the cost of encoding the new features!
MDL should be considered relative to the original or raw dataset. The idea is that you want to find an expression you could send to someone else that encodes the structure of the dataset in terms of the original variables. If you define new features, you need to send a description of those features along with your model.
To make this clearer, imagine you call me up on the phone, and we're both looking at your left hand image. You say to me 'Yeah, you just draw a line through it at $p=a$'. The natural question for me to ask is 'What's p?'. If you have to tell me what $p$ is, then it's part of the description length.
A circular model for your lefthand image does have a MDL for its decision boundary of something like $(x−a)^2+(y−b)^2=c$. However, the feature transformation you've selected has a description length of $p=(x-a)^2 + (y-b)^2$. It should be clear that the description lengths for the linear model through $p$ and the circular model through $x$ and $y$ are identical.

The main thing I was trying to look into was what if I change the basis to cylindrical coordinates, or polar coordinates. I used to do this in electromagnetics to simplify the sum without incurring  great cost of calculation. Does the problem here result in a tradeoff not so favourable?

So here, we're interested in the size of the information you need to communicate to someone who's just looking at the original features.
So if you have to say: "Transform to polar coordinates with center at (a,b), and then draw a line at c", that's really not any savings over saying "Draw a circle centered at (a,b) with radius c".
See the issue?

Hmmm...I get it, but is this an idea used commonly? Say I already know the decision boundary will be a certain algebraic basis, so I collect points in that algebraic basis i.e my original dataset is already feature engineered (compared to the Cartesian system). Does such universal basis exist? (the idea is somewhat similar to cross entropy loss, even though its not perfect, we know it works for a wide range of problems)

If you know in advance that certain features are useful, then indeed, you could collect them. I would say MDL is useful when comparing two models that were developed for the same data. It is not useful if we compare two models developed on different representations of the same data, except insofar as we can say that one model learned the feature transforms that the other model got for free.


text:
What is the degree of linearity in the error propagated by Gradient Descent?
Neural Network is trained to learn a non-linear function, the more layers it has, the more is the quality of the prediction and the ability to match the real-world function correctly (lets leave aside overfitting for now).
Now, given that the concept of a derivative of a function (if it is not a line) is only defined at a single point , and worse than that, it is defined as a tangent line (so, its linear in essence) , and backpropagation uses this tangent line to project weight updates, does this mean that Gradient Descent is linear at its fundamental level ?
Then if Gradient Descent is using linearity to propagate back weight updates, imagine how big the error would be for non linear function? For example , you  have calculated that the weight number 129 of your network has to be decreased by 0.003412, but at that point with this new weight, the function might have already reversed its direction, and the real update for this weight must be a negative number!!! Isn't this the reason that our deep fully connected networks have such a difficulty to learn , because with more layers stacked up, the more non-linear the model becomes, and thus, the weight updates we propagate back to lower layers could be treated as "best guess" values instead of something that can be fully trusted?
I am correct in assuming that Gradient Descent is not calculating correct weight updates on each backwards step and that the only reason that the network eventually converges to required model is because these imprecisions are fixed in a loop (called epoch). So, if we use an analogy, Gradient Descent would be like navigating the World on a boat with maps developed with the assumption that the Earth is flat. With such maps, you can sail in near-by areas, but if you would travel around the world without knowing that the Earth is round you will never arrive to your destination, and that's exactly what we are experiencing when we train deep fully connected networks without making them converge.
This means, if Gradient Descent is broken in such a way, a correct Gradient Descent algorithm would only have to do SINGLE backwards step and update all the weights in one pass, giving the minimum error that is theoretically possible in 1 epoch .... I am right ?
So, my question basically is: is Gradient Descent a really broken algorithm or I am missing something?

What? Very confusing question to me. Are you saying that by the time you reach weight 129 after updating all weights till 128 the function value has already changed, or are you saying that the derivative changes value due to non linearity as it propagates backwards? Or you are saying none of the above.

@DuttaA, well there might be a condition that  after a change in this single weight (w129) by 0.003412, the Error will be totally different, and the function at that interval might be concave up now (after the change), while before this weight update gradient descent was assuming that the function is concave down. So, the weight update was incorrect

I mean we adjust weights to change the error, so the error will change and also the error surface will change (due to effect of other weights). Can't really understand your question though.

@DuttaA the question is easy: is it correct to change weights using a linear function when dealing with non-linear function? the real world function is non-linear , and it may not even be continous, and not differentiable at all

I mean second order derivatives exist which probably takes care of this, the learning rate is also responsible.

@DuttaA, well backpropagation uses first order derivatives of composite function. This is all based on the perpendicular line (tangent) to the function at some point. Weight updates assume this tangent is continuing in a linear way, and since the real world funciton is not linear, the error changes, and all this stuff doesn't work when networks are deep enough, becuase the depth of the network increases the non-linearity. Imagine that neural network (6 layers) is learning something similar to a cryptographic hash, you change single bit, and it all goes the other way. Butterfly Effect

That's what the learning rate is for.

@DuttaA, the learning rate is a softener on the amount of error. If Gradient Descent wouldn't be broken, no learning rate would be necessary.

I would not call it broken because it fails to do something. It was not designed for it, it is computationally efficient than other methods, which is what's important.

@DuttaA, It is computationally efficient for 2-3 layered networks. Then it becomes terribly inefficient and so far, Fully Connected Multi-layer Perceptron Networks haven't been built with more than 5 layers. The only progress that was made in last 2 decades is due to human-engineered convolutional layers, which can be stacked deeply because they don't have so many parameters in each kernel. But MLP networks can't be build for more than 5 layers. 20 years ago we had 50 times less computing power but we still can't train FC-MLP. Its not the hardware, its the algorithm.


text:
Do Multi-resolution CNN exist?
I am currently working on a problem for which the topographic data is in very different resolution. Let say I have data of 20x20 with 1km2 tiles and also high resolution data of 50m2 tiles. I would like to combine both for input in a CNN.
To make things more spicy I don't care about the 50m2 when it is far away from the center, that is why I would like to use an 'image' multi resolution, aka resolution low in the edges but higher in the center. That would be like human vision, only high detailed in the center...
Then I would combine that multi-resolution image with my 1km2 data
Do you know any research done on such a CNN ?
Only found that one for now Multi-Resolution Feature Fusion for Image
Classification of Building Damages with
Convolutional Neural Networks
Thank you for you help

So your input is an image divided into 20x20 tiles where some of them represent $1$ km$^2$ area and some of them represent $50$ m$^2$ area, and they are all mixed up ? What are you trying to classify ? Did you try "regular" architectures ?

I have many layers of 1 km2 in my image of 20x20. I want to add layers from data given at 50 m2 resolution, but I want to represent the same 20km2. Hence the idea of multi-resolution layer.... Actually I do avg pooling to produce a lower (1km2) resolution layer.
That being said, topographic at 1km2 resolution is useless for my application

I think Multi-resolution so far I have not seen a multi resolution CNN, and in my opinion it hasn't been there yet.


text:
If a neural network is a universal function approximator, can it have any prior beliefs?
Let us confine ourselves to the case where we have a $n$ dimensional input and a $+1$ or $-1$ output. It can be shown that:

For every $n$, there exists a dense NN of depth 2, such that it contains all functions from ${±1}^n$ to ${±1}$. (given sign activation functions and some other very simple assumptions).

Check section 20.3 for the proof.
So, if a neural net can approximate any function, then it has a $\mathcal V \mathcal C $ dimension of infinite (considering the $n$ dimensional set of points as our universe).
Thus, it can realize all types of functions (or its hypothesis set contains all set of functions), and hence cannot have prior knowledge (prior knowledge in the sense used in the No Free Lunch theorem).
Are my deductions correct? Or did I make wrong assumptions? Are there actually any prior beliefs in a neural network that I am missing?
A detailed explanation would be nice.

I think your deduction is mostly correct.
Neural networks of depth are universal function approximators. This means that in principal, for any function of the form you describe, there's a NN that approximates it.
However, a particular NN architecture of fixed width and depth, with fixed connections is not a universal approximator for all functions. Only an infinitely wide NN is, and that's a theoretical construct, not something you can make in practice.
Typically, a practitioner using a NN injects their prior beliefs about the problem  by selecting an architecture. For example, it is common to use the ImageNet or ResNet architectures for image processing tasks. Those architectures are less effective on other types of tasks. For instance, it should be clear that they are ineffective on say, a time-series analysis task.

So in theory it still should have infinite VC dimension, just like in theory of UAT a super wide NN will have arbitrarily low error?

Yes. In fact, the two ideas are really sides of the same coin: If you can make the error arbitrarily low for any function by making an NN infinitely wide, then it follows that, for a binary function (where we just clip, say, >0.5 outputs of the NN to 1), you can always learn it. However, the class of models you're learning is "NNs of arbitrary width", which is not a class we know how to optimize nicely.

@JohnDoucette Thanks for the clarification, just another question, why does arbitrarily wide not optimize nicely? (without taking into consideration computation cost)


text:
Cannot fine-tune L2-regularization parameter
I have a data set of 1600 examples. I am using 1280 (80%) for training, 160 (10%) for testing, and 160 (10%) for validation. The training goes one of two ways no matter how I fine-tune the L2 parameter:
1) The validation and training error converge, albeit around 75% error
2) The training error settles to around 0%, but the validation error stays around 75%
I don't think my network is too large either. I have trained networks with two hidden layers, both with the same number of nodes as the input. I also tried dropout layers and that did not seem to help.
Does this just mean that I need to add more training examples? Or how do I know that I have reached the limitations of what I am having the network learn?

How was the validation data selected, randomly or hand selected. I don;t think L2 has anything to do with the problem. Try running the network without L2 and see what the results are.

It was selected randomly. I have also tried without a validation set and the training set error is very low while the testing set error is very high

Seems the issue is more complex. Can't give further help without seeing the code.

Acutally I am able to get 100% accuracy on the training set and 33% on the testing set with a very small network. Hope you are familiar with Mathematica.
func = "ReLU";
net = NetChain[{LinearLayer[in, "Input" -> 50], ElementwiseLayer[func],
LinearLayer[l4, "Input" -> 50], ElementwiseLayer[func],
SoftmaxLayer["Input" -> 4]}]


text:
Can neural networks handle redundant inputs?
I have a fully connected neural network with the following number of neurons in each layer [4, 20, 20, 20, ..., 1]. I am using TensorFlow and the 4 real-valued inputs correspond to a particular point in space and time, i.e. (x, y, z, t), and the 1 real-valued output corresponds to the temperature at that point. The loss function is just the mean square error between my predicted temperature and the actual temperature at that point in (x, y, z, t). I have a set of training data points with the following structure for their inputs:

(x,y,z,t):
(0.11,0.12,1.00,0.41)
(0.34,0.43,1.00,0.92)
(0.01,0.25,1.00,0.65)
...
(0.71,0.32,1.00,0.49)
(0.31,0.22,1.00,0.01)
(0.21,0.13,1.00,0.71)

Namely, what you will notice is that the training data all have the same redundant value in z, but x, y, and t are generally not redundant. Yet what I find is my neural network cannot train on this data due to the redundancy. In particular, every time I start training the neural network, it appears to fail and the loss function becomes nan. But, if I change the structure of the neural network such that the number of neurons in each layer is [3, 20, 20, 20, ..., 1], i.e. now data points only correspond to an input of (x, y, t), everything works perfectly and training is all right. But is there any way to overcome this problem? (Note: it occurs whether any of the variables are identical, e.g. either x, y, or t could be redundant and cause this error.)
My question: is there any way to still train the neural network while keeping the redundant z as an input? It just so happens the particular training data set I am considering at the moment has all z redundant, but in general, I will have data coming from different z in the future. Therefore, a way to ensure the neural network can robustly handle inputs at the present moment is sought.

Hi. Try to reduce the number of layers. Btw, how many layers and parameters do you have in your neural network, and which activation functions are you using? Maybe you're facing an "exploding" or "vanishing" gradient problem".

i would suspect that there is some error in the configuration of your nn, normally if an input is "redundant", it would learn a very low weight

@nbro I have attempted to vary the size and number of layers, but it appears to not improve the situation as it still outputs nan. For example, I have tried the following combinations of [# layers, #neurons per layer]: [10,50], [8,20], [4,20], [1,10]. I am currently using tanh activation functions with Xavier initialization. As you mention, I believe it is related to the gradients being unable to update weights for z. For example, if I add a few training data points with z != 1.0, everything works again, which indicates this is the origin. But uncertain of a good general fix.

@Mathews24 Hyperbolic tangent (tanh) is a so-called "saturating" activation function. Try to use ReLU, which is non-saturating. Also, if the problem persists, try to use even fewer layers.

@nbro I just attempted using ReLU and even fewer layers (e.g. ranging from 1 to 8 layers with 20 neurons each) but it still does not resolve the problem.


text:
Doubt regarding the proof of convergence of $\epsilon$ soft policies without exploring starts
In page 125 of Sutton and Barto (second last paragraph) the proof for equality of $v_{\pi}$ and $v_*$ for $\epsilon$ soft policies is given. But I could not understand the statement explaining the proof:

Consider a new environment that is just like the original environment, except with the requirement that policies be $\epsilon$-soft “moved inside” the environment. The new environment has the same action and state set as the original
and behaves as follows. If in state $s$ and taking action $a$, then with probability
$1 - \epsilon$ the new environment behaves exactly like the old environment. With
probability $\epsilon$ it repicks the action at random, with equal probabilities, and
then behaves like the old environment with the new, random action. The best
one can do in this new environment with general policies is the same as the
best one could do in the original environment with $\epsilon$-soft policies.

What is the meaning of environment here? And what is this new thing/argument (provided above) the authors are describing to arrive at the proof?

Let's first clarify a couple of details:

The policy $\pi$ we're talking about is an $\epsilon$-soft policy (defined to mean that $\pi(a \vert s) \geq \frac{\epsilon}{\vert \mathcal{A}(s) \vert}$ for all states and all actions).
We're not trying to prove equality of $v_{\pi}$ and $v_*$, but of $v_{\pi}$ and $\tilde{v}_*$, where $\tilde{v}_*$ denotes the optimal value function in this "new environment" that we're constructing.

So, "environment" is basically the "world" that our agent "lives" and acts in. You can think of it as the "rules" that we "play" by. So, you could think of the definitions of our complete state and action spaces as part of the environment, and the function that tells us which successor state $s'$ we end up in whenever we pick an action $a$ in a state $s$ (i.e. the state transition dynamics) are a part of the environment. And the function that tells us what Rewards we'll obtain in what situations is also a part of the environment. The policy $\pi$ is not a part of the environment; this is the "brain" of the agent itself.
Now, recall that here we're not interested in proving that $v_{\pi}$ moves towards the true optimal value function $v_*$ of the "real" environment. We know for a fact that we won't ever become completely equal to that, because we're forcing our policies to have exploratory behaviour by requiring them to be $\epsilon$-soft, so it would be hopeless to prove such a thing. Instead, we're interested in proving that $v_{\pi}$ will move towards whatever value function is the best one that we could possibly achieve under the restriction that we must have an $\epsilon$-soft policy.
What we do in the book here is that we slightly "transform" our environment into a new environment (i.e. we change the rules that we play by just a little bit). This is done in a clever way, such that the thing I just described that we want to prove becomes mathematically equivalent to just proving that $v_{\pi}$ moves towards (or becomes equal to) $\tilde{v}_*$. Now, if we can prove this for the new environment, we'll automatically have proven the thing that we actually wanted to prove for the "real" environment.

@ Dennis Soemers  1- Consider a new environment that is just like the original environment, except with the requirement that policies be ε-soft “moved inside” the environment. 2-The best one can do in this new environment with general policies is the same as the best one could do in the original environment with ε-soft policies. Are 1 and 2 not conflicting each other because in (1)  only epsilon soft policies are in the new environment and in (2) the author is talking about general policies?

@DSPinfinity (1) does not say that only epsilon soft policies are in the new environment. We had that requirement for the old environment. (1) states that we no longer explicitly have that same constraint on the policy, but we modify the environment in such a way that the environment actually enforces something like "epsilon-softness" on general / deterministic policies whenever they are used to play the new environment. The new environment simply has a bunch of randomness baked into it.

What does this mean exactly: "Consider a new environment that is just like the original environment, except with the
requirement that policies be $\varepsilon$-soft  "moved inside" the environment? Does it mean that in new environment we can have all general policies in contrast to the old environment where we have only  $\varepsilon$-soft policies? The English language use of the authors is very weird sometimes for me.

@DSPinfinity with "moved inside the environment" they mean exactly what they write afterwards in more explicit detail: "The new environment has the same action and state set as the original and behaves as follows. If in state $s$ and taking action $a$, then with probability $1 - \epsilon$ the new environment behaves exactly like the old environment. With probability $\epsilon$ it repicks the action at random, with equal probabilities, and then behaves like the old environment with the new, random action." So the "$\epsilon$-randomness" is made an inherent part of the environment.


text:
What is the difference between the prediction and control problems in the context of Reinforcement Learning?
What is the difference between the prediction (value estimation) and control problems in reinforcement learning?
Are there scenarios in RL where the problem cannot be distinctly categorised into the aforementioned problems and is a mixture of the problems?
Examples where the problem cannot be easily categorised into one of the aforementioned problems would be nice.

Prediction is the problem of predicting any feature of the environment. In reinforcement learning, the typical feature is the reward or return, but this doesn't have to be always the case. See Multi-timescale nexting in a reinforcement learning robot (2011) by Joseph Modayil et al.
Control is the problem of estimating a policy. Clearly, the term control is related to control theory. In fact, the term control is often used as a synonym for action. See Is there any difference between a control and an action in reinforcement learning?. Similarly, the term controller is also used as a synonym for agent. For example, in the paper Metacontrol for Adaptive Imagination-Based Optimization (2017) by Jessica B. Hamrick et al. the term meta-controller is used to refer to an agent. A controlled system can also refer to the environment.
Section 14.1 of the book Reinforcement learning: an introduction (2nd edition) provides more details about the distinction between prediction and control and how this distinction is related to psychological concepts.

Nbro's answer already addresses the basic definitions, so I won't repeat that. Instead I'll try to elaborate a bit on the other parts of the question.

Are there scenarios in RL where the problem cannot be distinctly categorised into the aforementioned problems and is a mixture of the problems?

I'm not sure about cases where the "problem" can't be distinctly categories... but often, when we're actually interested in control as a problem, we still also actually deal with the prediction problem as a part of our training algorithm. Think of $Q$-learning, Sarsa, and all kinds of other algorithms related to the idea of "Generalized Policy Iteration". Many of them work (roughly) like this:

Initialise (somehow, possibly randomly) a value function
Express a policy in terms of that value function (greedy, $\epsilon$-greedy, etc.)
Generate experience using that policy
Train the value function to be more accurate for that policy (prediction problem here)
Go back to step 2 (control problem here)

You could view these techniques in this way, as handling both of the problems at the same time, but there's also something to be said for the argument that they're really mostly just tackling the prediction problem. That's where all the "interesting" learning happens. The solution to the control problem is directly derived from the solution to the prediction problem in a single, small step. There are different algorithms, such as Policy Gradient methods, that directly aim to address the control problem instead.

An interesting (in my opinion :)) tangent is that in some problems, one of these problems may be significantly easier than the other, and this can be important to inform your selection of algorithm. For example, suppose you have a very long "road" where you can only move to the left or the right, you start on the left, and the goal is all the way to the right. In this problem, a solution to the control problem is trivial to express; just always go right. For the prediction problem, you need something much more powerful to be able to express all the predictions of values in all possible states.
In other problems, it may be much easier to quickly get an estimate of the value, but much more complicated to actually express how to obtain that value. For example, in StarCraft, if you have a much larger army, it is easy to predict that you will win. But you will still need to execute some very specific, long sequences of actions to achieve that goal.


text:
Can Reinforcement Learning be used for UAV waypoint control?
I want to make a drone which can follow static and dynamic waypoints. I am a total beginner in the drone field so I can't figure out that should I use Reinforcement Learning or any other learning methods for the drone to make it follow both static and dynamic waypoints. If RL is the best choice for the task, then how would I go about training the model and upload it to the flight controller. And if RL is not required, then what should I use in order to achieve this task.
Please let me know how should I begin with this task

You shouldnt use RL, you should use some trajectory planning algorithm that gives you trajectory from the waypoints and then use  some controller to follow trajectory (simple PID will suffice). There is numerous literature (here is one example), look it up on Google scholar by typing keywords like 'UAV', 'trajectory', 'waypoints'.

Okay I will look at it, thanks!

https://ieeexplore.ieee.org/document/8795855
https://arxiv.org/pdf/2005.05057.pdf


text:
My Double DQN with Experience Replay produces a no-action decision most of the time. Why?
I've written a Double DQN-based stock trading bot using mainly time series stock data. The internal network of the Double DQN is a LSTM which handles the time series data. An Experience Replay buffer is also used. The objective function is cumulative stock return over the test period. My epsilon used for exploration is 0.1 (which I think is already very high).
My trading bot has a very simple action space, trade or no-trade.
-- When it decides to trade, it sends a signal to buy and own a stock for a day. I'd get a positive return if stock price has gone up from today to tomorrow; equally would get a negative return if stock price has gone down.
-- When it decides to not trade, I own no stock and the daily return is 0 because there is no trading.
Strangely, my algorithm gives a daily 'no trade' signal most of the time when I run the algo through a number of different test periods.
Very often, after giving a 'no trade' signal for many days, the algo would finally give a 'trade' signal but the next day reverse back to giving 'no trade' right away.
My questions:
Why am I getting this phenomenon? Most importantly, what can I do to make the algo not stuck in giving out 'no trade' signal most of the time?

It could be simply a bug. If it's not a bug it's property of your data. If input give zero (or negative) expectation of reward for any action  it's perfectly normal to do no action. By other word network unable to extract any useful information from input. Which is correct if your data are realistic - stock prices unpredictable from their history on medium frequency trading.

Hi mirro2image, I see your reasoning. Since I am training the network using sections of time series data over 20 years, I'd be very surprised that most of these data make the network decide there is zero expectation of reward. I am guessing it's something else, but I don't know what it might be.

PS: You can try simple time-series prediction without RL with the same (or close) architecture and see if it works

Interesting. I built a very similar agent. It would initially start trading very frequently but then would converge on no trading. My interpretation was that it was being rewarded for not losing money and refusing to take a position because it was unable to come up with a profitable trading strategy... just like me. :). Have you tried just rewarding it for some profit target?
Also, from the literature, it appears to me that targeting the log of the returns is the way to go.

@JoshuaDeFord Thanks for sharing your experience. Could you share the literature you've read that used log of the returns please? Also in your own experiments how long was your trading window? I am doing intraday trading, trading every minute (if an action is taken), lasting for 100 minutes.