text:
Can PDDL be utilized for action recognition?
The Planning Domain Definition Language (PDDL) is known for its capabilities of symbolic planning in the state space. A solver will find a sequence of steps to bring the system from a start state to the goal state. A common example of this is the monkey-and-banana problem. At first, the monkey sits on the ground and, after doing some actions in the scene, the monkey will have reached the banana.
The way a PDDL planner works is by analyzing the preconditions and effects of each primitive action. This will answer the question of what happens if a certain action is executed.
However, will a PDDL domain description work the other way around as well, not for planning, but for action recognition?
I've searched in the literature to get an answer, but all the papers I've found are describing PDDL only as a planning paradigm.
My idea is to use the given precondition and effects as a parser to identify what the monkey is doing and not what he should do. That means, in the example, the robot ape knows by itself how to reach the banana and the AI system has to monitor the actions. The task is to identify a PDDL action that fits the action by the monkey.

I don't know of any work on this with respect to PDDL, but this is very similar to a conceptual dependency application called SAM (Script Applier Mechanism). Conceptual Dependency (CD) models actions using a number of primitives (which could be seen as equivalent to PDDL primitive actions):  PTRANS for physical transfer, PROPEL for application of a physical force to an object, GRASP for grasping an object, etc. Their number varies around 12 or so, depending on the version of CD.
Stories are described by a sequence of primitive acts, which have slots for actor, object, etc. They are supposed to enable a program to draw inferences about what happens. A common problem when trying to understand stories is that often common knowledge about a situation is omitted; the standard example here is going to a restaurant. It is usually assumed that the listener/reader knows what commonly happens when the protagonist enters a restaurant, so that when they leave without paying this is recognised as something unusual.
The approach used to solve this problem is to encode such knowledge in scripts, sequences of primitive acts. When triggered, eg by "Manuel went to a restaurant", this script is retrieved, and the following actions are looked for in the script. Anything that is recognised is used to fill gaps in the story, eg sitting down at a table, or looking at a menu. This was the task of the SAM program.
Basically you have a sequence of primitive actions, and you try to recognise a more abstract event "going to a restaurant" from that. Obviously you'd need to have a script to recognise, but one could presumably use this to derive a sequence of more generalised events, such as "retrieving an object from a high place", or "standing on top on another object".
The theory of using scripts, plans, and goals to describe human reasoning is detailed in Schank, Roger; Abelson, Robert P. (1977). Scripts, plans, goals and understanding: An inquiry into human knowledge structures. New Jersey: Erlbaum. ISBN 0-470-99033-3.

As an aside: each primitive CD act has pre-conditions and effects, so should be usable as a primitive action in PDDL.

This is theoretically possible for an exhaustive set of sequential, non-concurrent primitive actions:
$$
\forall s_1, s_2 \left(  \exists a \left(  \text{possible}(a, s_1) \land \text{do}(a, s_1, s_2) \right) \right)
$$
where $s_1$ is the prior situation, $s_2$ is the result of doing action $a$ in $s_2$, and $\text{possible}(a, s_1)$ is true if it is possible to do an action in a situation (See Situation Calculus).
So, for a given situation, reduce your search space to those actions that are possible, then reduce your search space again to those that result in the following situation.


text:
How does the initialization of the value function and definition of the reward function affect the performance of the RL agent?
Is there any empirical/theoretical evidence on the effect of initial values of state-action and state values on the training of an RL agent (the values an RL agent assigns to visited states) via MC methods Policy Evaluation and GLIE Policy Improvement?
For example, consider two initialization scenarios of Windy Gridworld problem:
Implementation: I have modified the problem along with step penalty to include a non-desired terminal state and a desired terminal state which will be conveyed to the agent as a negative and positive reward state respectively. The implementation takes care that the MC sampling ends at the terminal state and gives out penalty/reward as a state-action value and not state value, since this is a control problem. Also, I have 5 moves: north, south, east, west and stay.
NOTE: I am not sure whether this changes the objective of the problem. In the original problem, it was to reduce the number of steps required to reach the final stage.

We set the reward of reaching the desired terminal state to a value that is higher than the randomly initialized values of the value function; for example, we can set the reward to $20$ and initialize the values with random numbers in the range $[1, 7]$

We set the reward of reaching the desired terminal state to a value that is comparable to the randomly initialized values of the value functions; for example, we can set the reward to $5$ and initialize the values with random numbers in the range $[1, 10]$

As far as I see, in the first case, the algorithm will easily quickly converge as the reward is very high for the terminal reward state which will skew the agent to try to reach the reward stage.
In the second case, this might not be true if the reward state is surrounded by other high reward states, the agent will try to go to those states.
The step penalty ensures that the agent finally reaches the terminal state, but will this skew the path of the agent and severely affect its convergence time? This might be problematic in large state spaces since we will not be able to explore the entire state space, but the presence of exploratory constant $\epsilon$ might derail the training by going to a large false reward state. Is my understanding correct?

There seem to be two different ideas in this question here:

What's the impact / importance of our choice for reward values?
What's the impact / importance of our choice for initial value estimates (how do we initialise our table of $Q(s, a)$ values in the case of a simple, tabular RL algorithm like Sarsa or $Q$-learning)?

The reward values are typically assumed to be a part of the problem definition - something we shouldn't modify if we're using an existing problem definition as a benchmark. But if we're in charge of defining the problem ourselves, we can of course also pick the reward values. Modifying them may indeed have a huge impact on the speed with which RL algorithms are able to learn a task - but it may also intrinsically changes the task, it changes the objective of the problem, it may change which policies are optimal.

As for initialisation of our table of value approximations: by default, we normally assume an all-$0$ initialisation. However, it is a fairly common trick (in tabular RL algorithms, without function approximation) to initialise value estimates optimistically; pick higher initial $Q(s, a)$ value estimates than are likely (or even pick values higher than a known upper bound on what the true value possibly could be). This is often beneficial - also in large gridworlds with sparse rewards (e.g. a single distant goal somewhere) and negative rewards (i.e. costs) incurred for every step taken - because it incentivises exploration of state-action pairs that have not yet been tried.
Suppose you have your gridworld with negative rewards associated with every time step, and the optimal policy being one that takes you to a distant goal as soon as possible. If all $Q(s, a)$ are initialised to $0$ (or worse, to negative values), your agent may quickly learn that everything it does is equally bad anyway, and get stuck near the starting position. If all $Q(s, a)$ values are initialised optimistically (to large, positive values), your agent during the learning process will still have optimistic expectations of what it can achieve if it just tries to navigate to unexplored parts of the state-action space.

Could you please explain how to choose the initial value for optimistic initialization? Or is the question worth its own thread?

@RayWalker That depends on the environment, and in particular, on how much you know about the bounds / average / reasonable $Q$-values in your environment. If I know that all $Q$-values are bounded in $[-1, 1]$ for my environment (for instance because I'm training in a game where I only give a reward of $1$ for winning and $-1$ for losing)... then it's easy to say that good optimistic initialisations for every state-action pair would be $1$ (that is optimistic because no true $Q(s, a)$ value will ever exceed that)

So in the general case, we have to estimate the maximum possible $Q$-values using the reward function and our knowledge about the environment like the maximum lenght of an episode. Thank you very much for the detailed answer!

@RayWalker Well it doesn't necessarily have to be the maximum possible $Q$-values. That's what I would in most cases expect to work best if possible (in terms of encouraging exploration). But if not possible, even just initialising to something a bit higher than $0$ would lead to better exploration than initialising at $0$ (assuming here that we're dealing with environments with $> 0$ rewards)

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What are options in reinforcement learning?
According to a lecture (week 10) about Reinforcement Learning [1], the concept of an option allows searching the state space of an agent much faster. The lecture was hard to follow because many new terms were introduced in a short time. For me, the concept of an option sounds a bit like skills [2], which are used for describing high-level actions as well.
Are skills an improvement over options that includes the trajectory, or are both the same?
I'm asking for a certain reason. Normal deep reinforcement learning has the problem that the agent comes very often to a dead end, for example, in Montezuma's Revenge played at the Atari emulator. And the options framework promises to overcome the issue. But the concept sounds a bit too esoteric, and apart from the Nptel lecture, nobody else has explained the idea. So, is it useful at all?

what is the difference between options and learning larger code fragments for a domains specific language (DSL)? ref: dreamcoder https://www.youtube.com/watch?v=qtu0aSTDE2I&t=1167s

what is a modern survey of options ("temporal abstraction in reinforcement learning") or an example of a modern application of it? (especially with a good background section to learn options and some modern model that uses it)

An option is a generalization of the concept of action. The concept of an option (or macro-action) was introduced in the context of reinforcement learning in the paper Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning (1998) by Richard Sutton et al., so that to capture the idea that certain actions are composed of other sub-actions. Section 2 of the mentioned paper formally defines the concept of an option, which is a tuple composed of an initiation set, a policy, and a termination condition/set.
The authors of the mentioned paper give examples of options

Examples of options include picking up an object, going to lunch, and traveling to a distant city, as well as primitive actions such as muscle twitches and joint torques.

The option picking up an object, going to lunch, and traveling to a distant city is composed of other sub-actions (e.g. picking up an object), but is itself an action (or macro-action). A primitive action (e.g. joint torques) is itself an option.
A set of options defined over an MDP constitutes a semi-Markov decision process (SMDP), which are MDPs where the time between actions is not constant but it is variable. In other words, a semi-MDP (SMDP) is an extension of the concept of MDP that is used to deal with problems where there are actions of different levels of abstraction.  For example, consider a footballer that needs to take a freekick. The action "take a freekick" involves a sequence of other actions, like "run towards the ball", "look at the wall", etc. The action "take a freekick" takes a variable number of time steps (which depends on the other sub-actions).
Semi-MDPs are thus used to deal with such problems that involve actions of different levels of abstraction. Hierarchical reinforcement learning (HRL) is a generalization (or extension) of reinforcement learning where the environment is modeled as a semi-MDP.
Curiously, certain models that have won the RoboCup (the famous AI football) context are based on the concept of semi-MDPs, options and HRL. See e.g. WrightEagleBASE, which use the MAXQ-OP (MAXQ online planning) algorithm.
Semi-MDPs can be converted to MDPs. The picture below (which is a screenshot of figure 1 of the mentioned paper that introduces the "options framework" in RL) illustrates the relationship between semi-MDPs and MDPs.

The empty circles (in the middle) are options, while the black circles (at the top) are primitive actions (which are themselves options).
In the paper Reinforcement learning of motor skills with policy gradients mentioned in the question, apparently, the term skill is not formally defined. However, I suppose that skills can be represented as options.

For the last part, I would assume "skills" has close to the natural language meaning, and refers to learning effective policies (policies that perform a task well).

what is the difference between options and learning larger code fragments for a domains specific language (DSL)? ref: dreamcoder https://www.youtube.com/watch?v=qtu0aSTDE2I&t=1167s

what is a modern survey of options ("temporal abstraction in reinforcement learning") or an example of a modern application of it? (especially with a good background section to learn options and some modern model that uses it)

@CharlieParker Please, ask your questions in their separate post. One question per post. For each post, provide the necessary context to understand the question.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Why does GLIE+MC Control Algorithm use a single episode of Monte Carlo evaluation?
GLIE+MC control Algorithm:

My question is why does this algorithm use only a single Monte Carlo episode (during PE step) to compute the $Q(s,a)$? In my understanding this has the following drawbacks:

If we have multiple terminal states then we will only reach one (per Policy Iteration step PE+PI).
It is highly unlikely that we will visit all the states (during training), and a popular scheduling algorithm for exploration constant $\epsilon = 1/k$ where $k$ is apparently the episode number, ensures that exploration decays very very rapidly. This ensures that we may never visit a state during our entire training.

So why this algorithm uses single MC episode and why not multiple episodes in a single Policy Iteration step so that the agent gets a better feel of the environment?

I feel the general answer is that we want to be as efficient as possible in learning from experience.
Policy improvement here always produces an equivalent or better policy, so delaying the improvement step to gather more episodes will only slow down learning.
I would note too that often a different kind of Monte Carlo learning is used. Instead the speed of the update is typically controlled with a new hyper parameter $\alpha$, instead of keeping track of the visit counts. The Q estimate is then updated something like:
$$
Q \leftarrow Q + \alpha \left (G - Q \right)
$$
The value of $\alpha$ then lets you tune how much evaluation vs improvement happens. This is called constant alpha Monte Carlo. Often this is used as a stepping stone to introduce TD methods, e.g., in 6.1 of the Sutton and Barto book.

Your answer makes point, but I think in David SIlver's Lec 5 he actually mentions it explicitly later that we can do incomplete PE's, but I do not think he mentions whether this algo is also incorporating incomplete PE.

This is definitely incomplete (or truncated) policy iteration. Doing complete evaluation would mean running more episodes until the Q estimate converges to the exact Q for the current policy, then improving the policy with the state-by-state greedy action.

you are correct, just checked it. Thanks!


text:
Confused about NeuralODE
I am a bit confused about NeuralODE and I want to make sure that what I understood so far is correct.
Assume we have (for simplicity) 2 data points $z_0$ measured at $t_0$ and $z_1$ measured at $t_1$. Normally (in normal NN approach), one would train a NN to predict $z_1$ given $z_0$, i.e. $NN(z_0)=z_1$. In NeuralODE approach, the goal is to train the NN to approximate a function $f(z_0)$ (I will ignore the explicit time dependence) such that given the ODE: $\frac{dz}{dt}|_{t_0}=f(z_0)$ which would be approximated as $\frac{dz}{dt}|_{t_0}=NN(z_0)$ and solving this using some (non AI based) ODE integrator (Euler's method for example) one gets as the solution for this ODE at time $t_1$ something close to $z_1$. So basically the NN now approximates the tangent of the function ($\frac{dz}{dt}$) instead of the function itself ($z(t)$).
Is my understanding so far correct?
So I am a bit confused about the training itself. I understand that they use the adjoint method. What I don't understand is what exactly is being updated. As far as I can see, the only things that are free (i.e. not measured data) are the parameters of the function $f$, i.e. the NN approximating it. So one would need to compute $\frac{\partial loss}{\partial \theta}$, where $\theta$ are the parameters (weights and biases of the network).
Why would I need to compute, for example (as they do in the paper) $\frac{\partial loss}{\partial z_0}$? $Z_0$ is the input which is fixed, so I don't need to update it. What am I missing here?
Secondly, if what I said in the first part is correct, it seems like in principle one can get great results for a reasonably simple function $f$, such as a (for example) 3 layers fully connected NN. So one needs to update the parameters of this NN. On the other hand, ResNets can have tens or hundreds of layers.
Am I missing a step here or is this new approach so powerful that with a lot fewer parameters one can get very good results?
I feel like a ResNet, even with 2 layers, should be more powerful than Euler's Method ODE, as ResNets would allow more freedom in the sense that the 2 blocks don't need to be the same, while in the NeuralODE using Euler's Method one has the same (single) block.
Lastly, I am not sure I understand what do they mean by (continuous) depth in this case. What is the definition of the depth here (I assume it is not just the depth of $f$)?

Can you please provide a link to the paper you're reading about NeuralODE? Are you referring to Neural Ordinary Differential Equations? Furthermore, you're asking too many questions in the same post. Please, create one post for each of your questions, so that to facilitate the life of the answerers.

@nbro I am sorry. That is the paper (I assumed it is known well enough so I forgot to actually put the link, sorry). Should I delete this post and make several new ones? The reason I did it like this is that the introductory part would be the same for all the questions, so I would have to copy paste that part 3 times for each of my posts.

You can edit this post to ask just one question. Then you ask the other questions in other others, unless they are very related to each other (I would say).


text:
Encoding real valued inputs
UPDATE: After reading more about the topic, I've tried implementing the
 DDPG algorithm instead of using a variation of Q-Learning and still have the same issue.
I have the following issue:
I want to train my critic to estimate values of state/action pairs. My state consists of 2 real valued variables and my action is another real valued variable.
I normalize all values before I feed them into the network. Now I have the following issue: The network is very unresponsive to changes in the input. Before I normalize the state and the actions, they can take up any value between 0 and 50. After normalizing them they are in the range between -1 and 1. A change of 1 in the input can become a very small change in the input after normalization.
But in my specific situation, a small change in the action or the state can cause a very large change in the value of the state/action pair. The network does not really learn that correctly, it handles similar inputs similarly all the time (which is okay most of the times, but there are hard cuts in the shape of the value function here and there). If I further reduce the networks' capacity, the networks output becomes constant and ignores all three inputs.
Do you know any other tricks, that I could use to increase sensitivity to the input at some points? Or is my network configuration/approach the wrong one (too large, too small)?
The network I'm training is a simple feedforward neural network that takes two inputs, followed by 2 hidden layers followed by a single output to predict the value for that state/action combination. (I'm still trying out different configurations here, as I have no real feeling for the amount of elements per layer and the amount of layers needed to get the capacity I need without encouraging overfitting).
Thanks for your help :)

It usually means that network doesn't have an adequate complexity to fit the wanted function. But I'm curious how do you even do the q-learning update in continuous action space, how do you calculate $max_{a'} Q(s, a')$ ? Policy gradient / actor-critic methods are more appropriate for cases when you have continuous action space

Thanks for the tip with policy gradients. I'm fresh to RL, currently trying to implement my system using DDPG. I've calculated it until now by making a guess and sampling, which basically turns it back into a discrete action space.

Just to have it said: I still have exactly the same problem, only with the critic now. As the critic is in fact learning more or less the same function I have learned before (as it tries to estimate the value of the state/action combination), it still does not learn the actual distributions. @Brale_


text:
By learning from incomplete episodes, does David Silver mean learning of $V(s)$ even when the episode is not completed?
I came across the $TD(0)$  algorithm from Sutton and Barto:

Clearly, the only difference of TD methods with the MC methods is that TD method is not waiting till the end of the episode to update the $V(s)$ or $Q(s,a)$, but according to David Silver's lecture (Lecture 4- ~34:00),

The $TD(0)$ algorithm learns from incomplete episodes, but in the earlier algorithm we can see that the loop repeats until $s$ is terminal which mean completion of episode.
So, by learning from incomplete episodes, does David Silver mean learning of $V(s)$ even when the episode is not completed? Or did I interpret the algorithm wrong? If so, what is the correct interpretation?

The $TD(0)$ algorithm learns from incomplete episodes, but in the earlier algorithm we can see that the loop repeats until $s$ is terminal which mean completion of episode.

In the pseudocode, you have two loops: one for each episode and one (nested) for each step of the episode. The until $S$ is terminal means that you perform the updates until you end the episode (that is, you end up in a terminal state, e.g. checkmate in the game of chess). For each step of the episode, you perform the TD(0) update.
Apparently, you're confusing two things: the fact that each episode ends in a terminal state and the fact that TD learns from incomplete information. Each episode ends in a terminal state (otherwise it would not be called an episode), but this does not mean that it collects a full rollout before updating $V$. In fact, at each step of the episode, it updates $V$.
The information in the David Slider's slide is consistent with the pseudocode. TD learns from experience because it uses the given policy $\pi$ to behave.

So, by learning from incomplete episodes, does David Silver mean learning of $V(s)$ even when the episode is not completed?

Yes, essentially, you're updating the value function during each step of each episode.


text:
Which nonfictional documentaries about Artificial Intelligence are available?
From the subjective perspective, the number of documentaries about the subject Artificial Intelligence and robotics is small. It seems, that the topic is hard to visualize for the audience and in most cases, the assumption is, that the recipient isn't familiar with computers at all. I've found the following documentaries:

The Computer Chronicles - Artificial Intelligence (1985)
The Machine That Changed the World (1991), Episode IV, The Thinking Machine
Robots Rising (1998)
Rodney's Robot Revolution (2008)

The subjective awareness is, that the quality of the films in the 1980s was higher than in modern documentaries, and in 50% of the documentaries Rodney Brooks is the host. Are more documentaries available which can be recommended to watch?
Focus on non-fictional documentaries
Some fictional movies were already mentioned in a different post. For example Colossus: The Forbin Project (1970), Bladerunner (1982) or A.I. Artificial Intelligence (2001). They are based on fictional characters which doesn't exist and the presented robots are running with a Hollywood OS. This question is only about nonfictional motion pictures.

This question is very related to https://ai.stackexchange.com/q/8844/2444.

The documentary Plug & Pray (2010), directed and written by Jens Schanze, with main protagonists Joseph Weizenbaum (the creator of ELIZA) and the futurist Raymond Kurzweil, is about the promise, problems and ethics of artificial intelligence and robotics. This documentary won several awards, including the Bavarian Film Award 2010 for "best documentary". Here's the official trailer of the movie.

AlphaGo (2017) is quite a good watch, given that it is a documentary about the AlphaGo program, how DeepMind developed it, the help they had, and doesn't get too technical.  You can watch the trailer here.
Another documentary which wasn't exactly AI but something that is an interesting watch or at least was when it came out was The Human Face of Big Data.


text:
Can a computer identify the philosophical concept on which a given story is based?
Say you have to enter a story to a computer. Now, the computer has to identify the philosophical concept on which the story is based, say:

Was it a "self-fulfilling prophecy"?

Was it an example of "Deadlock" or "Pinocchio paradox situation"?

Was it an example of how rumours magnify? or something similar to a chain reaction process?

Was it an example of "cognitive dissonance" of a person?

Was it a story about "altruism"?

Was it a story about a "misunderstanding" when a person did something "innovative" but it accidentally was innovated earlier so the person was "falsely accused" of "plagiarising"?

And so on.
Given that the story is not only a heavy rephrase of the pre-existing story; not only character names and identities are totally changed, but the context completely changed, the exact tasks they were doing are changed.
Can computers identify such "concepts" from stories? If yes, then what mechanism do they use?

No. This is currently out of the scope for any language processing system. It requires a general understanding of abstract concepts which is not possible for machines at present.
In order to recognise a self-fulfilling prophecy, you first need to identify that something is a prophecy. So it needs to be something that expresses a possible future state, for which you need to identify what possible future states are; and then you need to see whether it is self-fulfilling. Conceptually this is far too complex to do.
You might get away for some of these with formal criteria (eg use of future tense for something describing a future state/event), but this is far too imprecise.
"Altruism" requires knowledge about typical expected behaviour; you would need to be able to identify motives behind people's actions, and then decide whether it was altruistic or not. This is just too complex for now (and the foreseeable future).

When one reviews cutting-edge AI narrative output, it's clear that the algorithms understand syntax (they can structure sentences and paragraphs, and write in various formats) and equally clear the algorithms have no semantic understanding of the content itself.

@DukeZhou Indeed -- structure is pretty much in hand, but the meaning (both semantic and pragmatic) of that structure is still far outside our grasp.


text:
Can computers recognise "grouping" from voice tonality?
In human communication, tonality or tonal language play many complex information, including emotions and motives. But excluding such complex aspects, tonality serves some a very basic purpose of "grouping" or "taking common" functions such as:

The sweet, (pause), bread-and-drink.

It means "The sweet bread and the sweet drink". However

The sweet-bread, (pause) and drink.

It means only the bread is sweet but the drink isn't necessarily sweet, or the drink's sweetness property isn't assigned.
Can computers recognise these differences of meaning based on tonality?


text:
Does this hyperparameter optimisation approach yield the optimal hyperparameters?
Say I have a ML model which is not very costly to train. It has around say 5 hyperparameters.
One way to select best hyperparameters would be to keep all the other hyperparamaters fixed and train the model by changing only one hyperparameter within a certain range. For the sake of mathematical convenience, we assume for the hyperparameter $h^1$, keeping all other hyperparameters fixed to their initial values, the model performs best when  $h^1_{low} < h^1 < h^1_{high}$ (which we found out by running the model on a huge range of $h^1$). Now we, fix $h^1$ to one of the best values and tune $h^2$ the same way, where $h^1$ is chosen and the rest of the hyperparameters are again fixed on their initial values.
My question is: Does this method find the best hyperparameter choices for the model? I know if the hyperparameters are independent, then this definitely does find the best solution, but in a general case, what is the general theory around this? (NOTE: I am not asking about the problem of choosing hyperparamaters, but I am asking about the aforementioned approach of choosing hyperparameters)

After you've computed $h^{1}_{optimal}$ the only thing you can be sure is that this is the best (assuming constrained case) value of $h^1$ (with respect to some model performance metric) given your initial values for $h^2, ..., h^n$. If you change a bit any of $h^2, ..., h^n$ you're no longer certain that the value $h^1$ you found is the optimal one. So yes, the key here is the assumption about the independence.

The theory behind hyper-parameter optimization (HPO) is not well developed. Nonetheless, there are several hyper-parameter optimization approaches, such as Bayesian optimization (using Gaussian processes), random search, grid search, genetic algorithms, etc. See, for example, the paper Hyperparameter Search in Machine Learning (2015), which attempts to formalize the problem of hyper-parameter optimization in machine learning, Random Search for Hyper-Parameter Optimization (2012), and the related Wikipedia article.
In the paper Hyperparameter Search in Machine Learning (and, similarly, in Random Search for Hyper-Parameter Optimization), the authors formally define the hyper-parameter optimization problem as follows
\begin{align}
\lambda^*
&= \operatorname{arg min}_{\lambda}\mathcal{L}(X^{test}; \mathcal{M} = \mathcal{A}(X^{train}; \lambda)) \tag{1}
\end{align}
where $\lambda$ are the hyper-parameters of the learning algorithm (for example, gradient descent, whose hyper-parameters are the learning rate and the batch size), that is, the algorithm that is used to train the model $\mathcal{M}$ (e.g. a convolutional neural network, with a fixed architecture) using the training ($X^{train}$) and test ($X^{test}$) datasets (for simplicity, ignore cross-validation and related techniques).
In simple words, in equation $1$, we want to find the hyper-parameters $\lambda$ of the learning algorithm $\mathcal{A}$ that minimize the loss $\mathcal{L}$ on the test dataset $X^{test}$, when the model $\mathcal{M}$ is trained using $\mathcal{A}$ and the training dataset $X^{train}$.
The equation $1$ thus ignores the hyper-parameters associated with the model (e.g., the number of layers of a multi-layer perceptron) and only considers the hyper-parameters associated with the learning algorithm. However, note that the optimal hyperparameters of the learning algorithm $\mathcal{A}$ depend on the given training and test datasets, the loss function $\mathcal{L}$ and the model $\mathcal{M}$. Eventually, the formulation in $1$ could be extended to include the hyper-parameters associated with the model (and other hyper-parameters).
So, in general, the choice of the HPO method (including the method you're proposing) depends on several factors, including the model (and its architecture), the task that needs to be solved, the loss function, the training and test datasets, and the computational complexity and runtime efficiency of the HPO method. For example, if the space of hyper-parameters is discrete and small, then grid search (which can be an exhaustive search) will find the best combination of hyper-parameters, for a given task and dataset. However, grid search can be impractical if the search space is huge.
In general, the method you're proposing will not be optimal because, as you state, the hyper-parameters might not be independent of each other. For example, if you're using stochastic gradient descent (that is, you train your model one example at a time), you probably do not want to update the parameters of your model too fast (that is, you probably do not want a high learning rate), given that a single training example is unlikely to be able to give the error signal that is able to update the parameters in the appropriate direction (that is, the global or even local optimum of the loss function). However, if you're using batch gradient descent, the higher the batch size, the more likely you can use a higher learning rate. This example is only meant to give you some intuition, but this might not hold in all cases.
The mostly used hyper-parameter optimization methods (that I mentioned above) seem to assume that the hyper-parameters are, in general, not independent of each other. In fact, this might be a correct assumption, given that, in the real world, the independence assumption almost never holds (see, for example, the related discussions in the context of the naive Bayes classifiers).


text:
Why is On-Policy MC/TD Algorithm guaranteed to converge to optimal policy?
Let's say we have a task where the cost depends entirely on the path length to a terminal state, so the goal of an agent would be to take actions to reach terminal state as quickly as possible.
Now let us say, we know the optimal path length is of length $10$, and there are $n$ such paths possible. Each state has 5 possible actions. Let's say the scheme we are using to find optimal policy is On-Policy MC/TD(n) along with GLIE Policy improvement (Generalised Policy Iteration).
In the first Policy Iteration Step, each actions are equally likely, there for the probability of sampling this optimal path (or the agent discovering this path) is $n* \frac {1}{5^{10}} \approx n* \frac {1}{2^{20}}$. So, according to probability theory we need to sample around $2^{20}/n$ steps to atleast discover one of the best paths (worst case scenario).
Since, it is not possible to go through such huge number of samplings, let's say we do not sample the path, thus in the next Policy Iteration step (after GLIE Policy Improvement) some other sub-optimal path will have a higher probability of being sampled than the optimal path, hence the probability falls even lower. So, like this there is a considerably high probability that we may not find the best path at all, yet theory says we will find $\pi^*$ which indicates the best path.
So what is wrong in my reasoning here?

Your reasoning is fine. GLIE - Greedy in the Limit with Infinite Exploration assumes that our agent does not act greedily all the time. As the number of samples approaches infinity all state-action pairs will be explored -> hence the policy will converge on a greedy policy. The emphasis is on "number of samples approaches infinity". Also for GLIE Monte-Carlo initial values of Q does not matter, since they are replaced after the first update.

To tell the truth even in a toy problem like windy gridworld it does not require more than 100 states (10*10) for the approach to be infeasible as I have shown.


text:
Is it ok to struggle with mathematics while learning AI as a beginner?
I have a decent background in Mathematics and Computer Science .I started learning AI from Andrew Ng's course from one month back. I understand logic and intuition behind everything taught but if someone asks me to write or derive mathematical formulas related to back propagation I will fail to do so.
I need to complete object recognition project within 4 months.
Am I on right path?

I think that you're on the right path. When it comes to backprop only understating the concept and how it works is enough. Most of the times when developing a model/project we are using an out of the box optimizer which does the job for us. What I would suggest you is to get familiar with the mathematical notations of CNN, and afterwards dive deeper into more complex aspects.

I think the key part of your question is "as a beginner". For all intents and purposes you can create a state of the art (SoTA) model in various fields with no knowledge of the mathematics what so ever.
This means you do not need to understand back-propagation, gradient descent, or even mathematically how each layer works. Respectively you could just know there exists an optimizer and that different layers generally do different things (convolutions are good at picking up local dependencies, fully connected layers are good at picking up connections among your neurons in an expensive manner when you hold no previous assumptions), etc.. Follow some common intuitions and architectures built upon in the field and your ability to model will follow (thanks to the amazing work on opensource ML frameworks -- Looking at you Google and Facebook)! But this is only a stop-gap.
A Newton quote that I'm about to butcher: "If I have seen further it's because I'm standing on the shoulders of giants". In other words he saw further because he didn't just use what people before him did, he utilized it to expand even further. So yes, I think you can finish your object detection project on time with little understanding of the math (look at the Google object detection API, it does wonders and you don't even need to know anything about ML to use it, you just need to have data. But, and this is a big but, if you ever want to extend into a realm that isn't particularly touched upon or push the envelope in any meaningful way, you will probably have to learn the math, learn the basics, learn the foundations.

Also, those others of us who are not Newton are sometimes in a position to see a little further because we drag ourselves to the top of a pile of dead dwarves.

@DrMcCleod calling the hoards of papers being produced in the field, "a pile of dead dwarves" is fitting.

Not only is it 100% ok, it's the process.

You may be surprised to know that even mathematicians struggle with mathematics, both the proofs they are working on, and the proofs of their colleagues.  Some thinkers are so far ahead of the curve, very few understand what they're stating until generations later.
The main thing is to keep with it.

If you want to bee engineer who work with models as black boxes it could be OK. If you want to be researcher, as the job position or for better understanding of the subject it's not OK. Backporpagation is just basic multivariate calculus. If you straggling with it things like Hessians, regularizers, stochastic processes etc. would cause even more problems. If you want to go research track it could be good idea to take some math courses and prioritize them.


text:
Why do language models place less importance on punctuation?
I have a very outdated idea about how NLP tasks are carried out by normal RNN's, LSTM's/GRU's, word2vec, etc to basically generate some hidden form of the sentence understood by the machine.
One of the things I have noticed is that, in general, researchers are interested in generating the context of the sentence, but oftentimes ignore punctuation marks which are one of the most important aspects for generating context. For example:

“Most of the time, travellers worry about their luggage.”
“Most of the time travellers worry about their luggage”

Source
Like this there exists probably 4 important punctuation marks .,? and !. Yet, I have not seen any significant tutorials/blogs on them. It is also interesting to note that punctuation marks don't have a meaning (quite important, since most language models try to map word to a numerical value/meaning), they are more of a 'delimiter'. So what is the current theory or perspective on this? And why is it ignored?

You are right. Approaches that map words to meaning solely do fail in this regard. None the less Word2Vec and Glove have shown wonderful downstream results. This in itself may indicate that most of the time, punctuation's addition can be interpolated. But as you provided, there are cases where this just is not true!
Now of days I would say most models actually use almost NO preprocessing. This is surprising but its due to the rise in power of learnable, reversable, tokenizations. Some examples of these include byte pair encoding (bpe) and the sentence piece model (spm).
State-of-the-art NLP generally rely on these. Examples include BERT and GPT2, which are general purpose pretrained Language Models. Their ability to parse and understand (i use this word loosely) a wide variety of phrasing, spelling and more can be partially due to the freedom in the preprocessing.
Takeaway: You can achieve good results by using preprocessing in a manner that will eliminate information but keep the meat and bones that you are interested in (but this requires domain knowledge paired with optimization experience), but the field seems gearing towards models that are more inclusive, more transferable, and dont have the problems you mention by design.

Language models almost always map every word to an embedding. There are many embedding algorithms with most of them having interpolation properties i.e. If $E(word)$ represents embedding of a word then $E(king)-E(male)+E(female) \sim E(queen)$. The smoother the interpolation properties the better the model understands the word, these properties don't exactly make much sense when it comes to delimiters.
Yet, there are instances where a delimiter embedding is learned ( always has an embedding ). While using these first all punctuation in the text is converted to one specific word, say 'dlmt', the embedding algorithm learns an embedding for this word treating it as if it would have any word. This maintains interpolation properties where the delimiter is understood to be a word that is used to split context.
I have observed that delimiters such as question mark or exclamation marks at the end of sentence are also understood to be breaks in context, in these cases the model learns if the statement is a question or so just by the context given by the words and stops in the sentence

Different delimiters convey different meaning. So why group them under same category?

The model learns to treat them as stops anyway. Different delimiters works better for sentence embeddings.  The most distinct delimiter is '?' which also happens to learn a pretty similar embedding to '.' , '!' or ','; the model cannot associate meaning as such to delimiters much beyond the fact that its a break word --- This might be due to under-represented examples too

This is a somewhat provocative view, so be warned (and please don't down-vote this if you feel provoked by it!):
In the "old days", when information retrieval (IR) was one of the main tasks in NLP, several categories of words were ignored as stopwords; conjunctions, determiners, prepositions, etc. These function words do not carry meaning themselves, but organise the structure of sentences. Most IR algorithms worked on frequencies of individual words, and as functions words are very frequent (of and the are the two most frequent English words) and don't mean anything by themselves, they were ignored. This kept the index files small and didn't seem to influence the results.
However, if you want to analyse sentences themselves, they are rather important. They are also useful for all sorts of other tasks where you are looking at sequences of words (eg part-of-speech tagging based on context). Similar for word embeddings: without function words you'd not have any meaningful context to work with. So, increasingly you would not ignore function words anymore.
My suspicion is, that punctuation is now in the 'stopword position': it's not too clear how it influences meaning, and is often inconsistent or redundant (obviously not in all cases). So you can probably treat it as 'noise' and get away with it for most applications. For example, looking at meanings of words, it probably doesn't matter that much whether the sentence they occurred in was a question or an exclamation. By removing punctuation (maybe apart from sentence-terminators), your model is a bit smaller and you don't lose much.
Since punctuation is purely a property of written language, we can generally get away without it, as we do in speech. A text without punctuation might be harder to read, because we're not used to it, but don't forget that some writing systems (Chinese, Egyptian hieroglyphics, ...) don't even have spaces between words — and people can still use them without problems.

I actually like the answer since it takes a statistical viewpoint that in most cases punctuation aren't important. I also suspect that punctuations are more to provide an emotional context rather than for actual meanings, and afaik most language models do not show much emotion but rather hard facts


text:
Can the AI in a box experiment be formalized?
Introduction
The AI in a box experiment is about a super strong game AI which starts with lower resources than the opponent and the question is, if the AI is able to win the game at the end, which is equal to escape from the prison. A typical example is a match of computer chess in which the AI player starts only with a king, but the human starts with all the 16 pieces including the queen, and the powerful bishop.
Winning the game
In case of a very asymmetric setup, the AI has no chance to win the game. Even if the AI thinks 100 moves ahead, a single king can't win against 16 opponent figures. But what happens, if the AI starts with 8 pieces and the human with 16? A formalized hypothesis will look like:

strength of the AI x resources weakness = strength of the human x resources strength

To put the AI for sure into a prison, the strength of the AI should be low and it's resources too. If the resources are low but the strength is middle, then the AI has a certain chance to escape from the prison. And if the AI has maximum strength and maximum resources, then the human player gets a serious problem.
Is this formalized prediction supported by the AI literature in academia?

I think that something like "strength" would be difficult to quantify in this context.  I do think that formal experimentation around the "AI in a box" scenario could be interesting.  I know that experiments have been done where a human plays the role of the AI, attempting to get naive test subjects to "release" him by interacting with them over a chat interface.  In all cases, the "AI" tends to be extraordinarily effective.  It is often simply not possible to anticipate every way someone can trick you into revealing information or creating security holes.  This is how human hackers do their jobs too.  But I think a fully automated "game" could produce some interesting data.
One way I think it may be interesting to go about creating an "AI in a box" game is to have an ambiguous win condition.  Essentially, make it possible to let the AI out and still think you are "winning" the game.  One example that pops into my head right away is to use a (pointless) point system.  You tell the player that the primary goal of the game is to keep the AI in the box, but you have a score that is displayed in the game, and tell the player that they win, no matter what, if their score is higher than the AI's by the end of the game.  Of course, they never get to see the AI's score so they never know how well they are doing by comparison.  This could be justified as representing human vs AI "strength" or "leverage" or whatever else.
In reality the AI doesn't have a score, but it does have the ability to effect the player's score.  It can raise the player's score if they do things that help it escape and lower it when the player stands in its way.  The score has no impact on the game and the player really wins by keeping the AI in the box.  It only serves as a red herring that the AI can use to manipulate the player.  I think this sort of experiment could provide an interesting model of how an AI might leverage bribes, threats, promises, and deception, as well as technical tricks, to convince a human to "let it out of the box".

Chess is indeed difficult to reduce to numeric values.


text:
How can AI be used to design UI Interfaces?
I'm very new to AI.  I read somewhere that AI can be used to create GUI UI/UX design. That has fascinated me for a long time. But, since I'm very new here, I don't have any idea how it can happen.
The usual steps to create the UI Design are:

Create Grids.
Draw Buttons/Text/Boxes/Borders/styles.
Choose Color Schemes.
Follow CRAP Principle (Contrast, Repeatition, Alignment, Proximity)

I wonder how can AI algorithms help with that. I know a bit of neural networking and the closest I can think of is the following two methods (Supervised Learning).

Draw grids manually and train the Software manually to learn proper styles until it becomes capable of giving modern results and design its own design language.
Take a list of a few websites (for example) from the internet and let the software learn and explore the source code and CSS style sheets and learn and program neurons manually until it becomes capable of making it's own unique styles.

The layout problem is suitable for a rule-based approach, as used to generate levels in Rogue or customised home-layouts, you encode some constraints and search the remaining space. The colour problem is also suitable for a rule-based approach, colour is a 3-dimensional space (4 if you include opacity). A palette can be created through taking colours at appropriate distances from a starting point.
Seen as this domain is well analysed, there are known heuristic and encodable rules. You'll likely have more success and faster by using these to generate designs.
Just to give an idea, seen as rule based approach isn't discussed much these days, the following is valid Prolog, query with phrase(webpage, Design)., that'll quickly generate designs like: [brand, header_nav, searchbox, sidebar_nav, form_content, sidebar_list]. This is just a super-quick demo, you'd likely want a tree instead of a list and you'll need to derive HTML and CSS from it too, which is simple enough in Prolog.
webpage --> body.
webpage --> header, body.
webpage --> header, body, footer.

header --> [brand], header_content.
header_content --> [header_nav].
header_content --> [header_nav], [searchbox].

body --> content.
body --> sidebar(_), content.
body --> { dif(A, B) }, sidebar(A), content, sidebar(B).
body --> content, sidebar(_).

content --> [form_content].
content --> [article_content].
content --> [list_content].

sidebar(list) --> [sidebar_list].
sidebar(nav) --> [sidebar_nav].
sidebar(ads) --> [sidebar_ads].

footer --> [notification].

If you see the use case, on higher level it seems to generate some visual output - the design but when seen at lower level, this design is output of some code.
One way we can do it is to train a neural network that learns to generate code which can be seen as some form of organized text. So now can be treated as a text generation problem on which you can find a lot of literature.
pix2code is one implementation that uses this idea and even expands it. In this paper the authors took it to another level, they also used the visual part - the GUI and built an architecture that takes both the code and the GUI as input and learns to generate code. So eventually it would be capable of generating code when a simple GUI was given.
Also there are some implementations where even the network could produce code even when a rough sketch was given.


text:
Can the C4.5 algorithm learn a GOAP model?
Goal-oriented action planning (GOAP) is a well-known planning technique in computer games. It was introduced to control the non-player characters in the game F.E.A.R. (2005) by creating an abstract state model. Similar to STRIPS planning, a GOAP model contains an action name, a precondition and an effect. The domain knowledge is stored in these subactions.
The bottleneck of GOAP is, that before the planner can bring the system into the goal state. the action model has to be typed in. Usually, the programmer defines actions like "walk to", "open the door", "take the object", and identifies for each of them the feature set for the precondition and the effect.
In theory, this challenging task can be simplified with a decision tree learning algorithm. A decision tree stores the observed features in a tree and creates the rules on its own with inductive learning. A typical example of the C4.5 algorithm is to find a rule like "if the weather is sunny, then play tennis". Unfortunately, the vanilla tree learning algorithm doesn't separate between different actions.
Is it possible to modify C4.5 algorithm such that the GOAP actions, like "walk to", "open the door", etc., are connected to individual rules?

You're going to run straight into the qualification problem, the frame problem and ramification problem on these. Plus you'll need to get your head round what an action is and means.
Qualification Problem
A more formal "STRIPS" would be Situation Calculus (logical equivalency demonstrated by Reiter), which can be defined in FOL or SOL, depending on who you talk to. In Situation Calculus you have your actions and you define when they are possible (S is the current Situation):
poss(open_door, S) → holds(door_position(closed) ^ door_lock(unlocked), S).

This is true, when it is possible to open the door it's both closed and unlocked. The difficulty arises when we try to flip that implication around:
poss(open_door, S) ← holds(door_position(closed) ^ door_lock(unlocked), S).

This is not true, because there must also be nothing blocking the door, the door can't be wedged too tightly into the door frame, and a whole bunch of other stuff that we might not think of. This is the qualification problem, which we get around in STRIPS by ignoring it and just flipping the implication.
In your case you'll end up learning a whole bunch of irrelevant qualifications:
 poss(open_door, S) ← holds(door_position(closed) ^ door_lock(unlocked) ^ weather(sunny) ^ grass(green), S).

You'd need some way to trim them down, by observing gameplay for your data you'll only learn the conditions when players choose to do an action, not when it is possible to do that action: the old correlation vs causation problem. You can immediately prune things that aren't fluents, but as far as I'm aware there's no good solution to this without letting the computer simulate to try actions in subsets of the fluents that hold in-order to prune.
Frame Problem
The result of an action in the language of Situation Calculus is that it causes some fluent to hold. So in our example we have a door_position fluent, we can say that opening the door causes door_position(open) to hold, but what about some other action, such as making tea? We know it has no effect on the position of the door, but for the computer to know that we need to tell it:
holds(door_position(open), S) → holds(door_position(open), do(make_tea, S)).

This creates far too many axioms. Reiter has a working solution to this for Golog in his book "Knowledge in Action". In your case it will manifest as missing axioms, i.e. not knowing what the result of an action will be for some fluent. Try beginning with the assumption that an action has no effect unless proven otherwise.
Ramification Problem
The ramification problem is very similar to the frame problem, it means you might not attribute all of the effects of an action to that action. Opening a door can cause changes in air-pressure through a house that can cause another door to slam, such an effect is easy to miss. For this you'd need a sensitive algorithm that can correctly recognise rare events that are important, which is a big ask. The solution is likely to be to ignore it.
Closely related and not usually a problem for such systems, you could also attribute effects to an action incorrectly. How will you be able to distinguish what the action caused vs occurred simultaneously? In some Mario game data it could likely be induced that jumping causes a turtle to move closer to Mario, again correlation vs causation and needing a method to distinguish. The solution here is likely to come from increasing the volume of data.
Definition and Granularity Problem
OK, it's probably possible that with enough data and a sensitive training algorithm you can overcome the above problems. The definition one is merely a conceptual one and easily overcome.
Humans tend to define an action by it's name, say for example "turning on a light". However, in Situation Calculus the action is defined by when it is possible and the result of that action, the name is merely a label. So "turning on a light" where the bulb is broken and when it is not may share the same label, but they are different actions.
Furthermore, in STRIPS and Situation Calculus we're defining "types" or "classes" of actions, the actual execution is an "instance" of that "class". Your problem is to derive the abstract from the concrete.
For you this problem will arise in terms of defining the granularity, as each instance of an action is possible in the unique situation that it occurred in, and caused the unique situation that came after it.
How do you group these actions together to abstract the most similar ones? How do you know how many actions there are to create these groups? If it's not based on a limited set of user-inputs to some computer game, then you might want to draw some ideas from the research into clustering, particularly heuristics of how many clusters to use.
Alternative to a decision tree
If I were to tackle this problem (fund a Post-Doc for me and I'd be tempted), I'd first investigate ILP as a method to learn the relations using Golog as the foundation of my action axioms. Then I'd look at clustering based on a space where each fluent was a dimension. I imagine it would be a rather useful tool, even if it merely created a pretty accurate template for manual review.


text:
Pooling vs Subsampling: Multiple Definitions?
I have seen people using pooling and subsampling synonymously. I have also seen people use them as different processes. I am not sure though if I have correctly inferred what they mean, when they use the terms with distinct meanings.
I think these people mean that the pooling part is selecting a submatrix from an input matrix and the subsampling part is selecting yet another submatrix that satisfies some condition from the first submatrix.
So say we have a $100 \times 100$ image. We do $10 \times 10$ non-overlapping pooling with $5 \times 5$ max subsampling. That would mean we slide the $10 \times 10$ "pool" across the image in strides of 10 and at every step we select the $5 \times 5$ submatrix inside the pool that has the maximum sum. That $5 \times 5$ matrix is what comes out of the $10 \times 10$ pool at the current potion. So in the end we have a $50 \times 50$ image.
Can you confirm that this usage of the terms pooling and subsampling exists?
I inferred this definition, as I cannot make sense of how some people use the two terms otherwise. For example in this video, or rather the people from the paper he is talking about (which I can't find because he only has the author name on his slide and no year).

I think this is asking the same question: https://stats.stackexchange.com/questions/354944/difference-between-pooling-and-subsampling

@Recessive I had read that one. It does not cover the case where people use the terms not as synonyms. It does not explain what a "$10 \times 10$ pool $5 \times 5$ subsampling" is  supposed to be.

From what I know, a maxpooling layer is a method for subsampling. So in that case, the terms are somewhat interchangable, but other types of pooling layers may also be a form of subsampling.

@Recessive The person from the video answered me now, and he says in this case by subsampling the authors meant the stride of the pooling.


text:
What are the steps that I need to follow to build a neural network for face recognition?
I have developed face recognition algorithms by using pre-built libraries in Python and open CV. However, suppose if I want to make my own neural network algorithm for face recognition, what are the steps that I need to follow?
I have just seen Andrew Ng's course videos (specifically, I watched 70 videos).

Please elaborate what do you mean by neural network algorithm. Do you mean the architecture, learning algorithm or the activity rule. It seems that you want to build an architecture, i guess.

My implementation to increase detection on video is using object tracking algorithms.
More specifically, first, I detect the object using a trained classifier. Then I track the object with the KCF algorithm. If the object tracker misses the object, again I call for the classifier.

You could try building Siamese network and train it on a large set of faces.
Two identical networks are used; one taking the known signature for the person, and another taking a candidate signature. The outputs of both networks are combined and scored to indicate whether the candidate signature is real or a forgery. The deep CNN are first trained to discriminate between examples of each class. The models are then re-purposed for verification to predict whether new examples match a template for each class. Specifically, each network produces a feature vector for an input image, which are then compared using the L1 distance and a sigmoid activation. Similar goes with face.

Hi. It's nice that you're trying to help, but we expect users to provide answers with more useful information. So, please, explain why Siamese networks are potentially useful to solve the task.

Two identical networks are used, one taking the known signature for the person, and another taking a candidate signature. The outputs of both networks are combined and scored to indicate whether the candidate signature is real or a forgery. The deep CNN are first trained to discriminate between examples of each class. .The models are then re-purposed for verification to predict whether new examples match a template for each class.Specifically, each network produces a feature vector for an input image, which are then compared using the L1 distance and a sigmoid activation.Similar goes with face

@PratheeshKumar Welcome to AI Stack Exchange! - use [edit] to add those details to your existing answer. The goal is to create good questions and good answers that match, and you are encouraged to edit your answer to make it as good as it can be. The comments here are for us to discuss what would make the answer better.

I've moved your comment into the answer itself, to make the answer as a whole a bit more complete (as suggested by Neil)

For the Construction of Deep Learning Models
Backbone Deep Learning models which can be applied to a variety of deep learning tasks (including facial recognition) have been implemented in a range of libraries available in Python. I'm assuming by constructing your own algorithm you mean a novel implementation of the model structure. Taking the PyTorch framework as an example, some common pretrained models are available here:
https://github.com/pytorch/vision/tree/master/torchvision/models
To train a novel face recognition model you could follow the tutorial for object detection available here:
https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html
and make changes to the model.
In the tutorial they use model features from the library in the following section of code:
# load a pre-trained model for classification and return
# only the features
backbone = torchvision.models.mobilenet_v2(pretrained=True).features

For the simplest example torchvision.models.AlexNet.features look like this:
self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(64, 192, kernel_size=5, padding=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(192, 384, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(384, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
        )

Adding or subtracting layers from this backbone feature extractor would result in a new "algorithm" for object detection.
If you want to know exactly what mathematical operation each of these layers is performing you can look at the PyTorch documentation. For example, in the case of nn.Relu layer:
https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html
Applies the rectified linear unit function element-wise:
$$ ReLU(x)=(x)^{+}=max(0,x)$$


text:
Using ML to analyze Facebook posts
First of all, I should mention that I have a very basic knowledge of ML so I apologize if this question seems trivial or stupid.
I am working on a small personal project, basically an app that analyzes Facebook posts concerning movies and translates them into a rating (out of 100). The algorithm looks for keywords, the length of the post, etc.. to determine the individual rating, and then averages all the ratings among a user's FB friends to give the result. My question is, would I be able to drastically improve such algorithm by using ML or is it not worth it? If yes, what algorithms/techniques do you advise me to learn?
All help is appreciated!

(1) Do you have access to data which has "true" ratings for the movies for some users that have also written the same kinds of text that you are analysing? If so, how much of this data do you have? (2) Do you have any measure or test for your current algorithm such that you know how accurate it is?

for this kind of ml training, you will need a ton of data first, at least in the thousands. If you have a bot program that fetches those data for you, AI is the way to go. I'm not sure how else you would do it though.
To train the nn you will need the inputs(the post) and the targets(the rating you want it to output). The targets could be anything you want, like the ratio of likes to views, etc.
There are tons of ML libraries out there and I recommend keras as it is easy to learn for beginners, hope it helps :)


text:
Why not use the MSE instead of the current logistic regression?
When watching the machine learning course on Coursera by Andrew Ng, in the logistic regression week, the cost function was a bit more complex than the one for linear regression, but definitely not that hard.
But it got me thinking, why not use the same cost function for logistic regression?
So, the cost function would be $\frac{1}{2m} \sum_{i}^m|h(x_i) - y_i|^2$, where $h(x_i)$ is our hypothesis $\text{function}(\text{sigmoid}(X * \theta))$, $m$ is the number of training examples and $x_i$ and $y_i$ are our $ith$ training example?

The mean squared error (MSE), $J(\theta) = \frac{1}{2m}\sum_{i=1}^m(h_\theta(x_i)-y_i)^2$, is not as appropriate as a cost function for classification, given that the MSE makes assumptions about the data that are not appropriate for classification. Though, as an optimization objective, it is still possible to attempt to minimize MSE even in a classification problem, and thus still learn parameters $\theta$.
The new cost function has better convergence characteristics as it is more inline with the objective.
See link for the precise mathematical formulation that explains these loss functions from a probabilistic perspective.
Note that the absolute value is redundant because $\forall x:x^2\geq0$.
I hope this clarifies the matter.

MLE is also not guaranteed to be convex though. It's I guess 'relatively' more convex.

You're right. Thanks for this comment! I did some reading on this and updated with a link that seems to provide the rigorous reasons.

I mean you technically could (it's not going to break or something) however, cross entropy is much better suited for classification as it penalizes for misclassification errors: have a look at the function: when you are wrong the loss goes to infinity:
you are either from one class or another. MSE is designed for regression where you have nuance: you get close to target is sometimes good enough. You should try both and you will see the performance will be much better for the cross entropy.

MSE also penalizes misclassification error, you could say it doesn't penalize it as much as MLE error function which might lead to the NN focusing on minimizing loss rather than increasing accuracy.

I would not call it penalising misclassification per se, as it would be predicting probabilities but implicitely yes you're right I should have mentionned that

It doesn't predict probablities though. The target is either 1 or 0. Since its output is between 0-1 or we design it to be so (like in softmax) and also because I think conventional literature somewhat treats it in the same way, so we interpret it as a probablity. Case in point is SVMs which try to just escape from a region, but if we normalize it tgen it's just probability. While in SVM we will try to reach infinity hear reaching the same will result in a 1 and thus 0. But then again I think the viewpoint varies from person to person.(NOTE: you can always edit your answer to add more details)

Minimising MSE in a classification setting is perfectly reasonable as it is also known as the Brier Score and is a proper scoring rule which means that it is minimised if the network outputs the conditional probability of class membership.  This is not unduly surprising as minimising MSE leads to a model that outputs an estimate of the conditional mean of the target distribution, which for a 1-of-c coding is the conditional probability of class membership.  You can even use MSE for training networks with logistic or softmax activation functions in the output layer so that the obey the usual constraints of being in the interval $[0,1]$ and summing to one.
However, the MSE penalises very confident misclassifications much less harshly than the cross-entropy metric does.  Whether this is a good or bad thing depends on the needs of the application.  If you are mostly interested in the p=0.5 decision boundary, then you probably don't want model resources spent dealing with highly confident misclassifications, which are a long way from the decision boundary, and have little effect on it.  This is a large part of the justification for purely discriminative methods like the SVM.


text:
What is "Word Sense Disambiguation"?
I recently came across this article which cites a paper, which apparently won the outstanding paper award in ACL 2019. The theme is that it solved a longstanding problem called Word Sense Disambiguation.
What is Word Sense Disambiguation? How does it affect NLP?
(Moreover, how does the proposed method solve this problem?)

"Word Sense Disambiguation" refers to the idea that words can have different meanings in different contexts. Here are some examples

"I went to the river river bank" vs "I deposited my check at the bank"
"He's mad good at that game" vs "I am so mad at you"

How it effects NLP, comes down to the way we process text. This generally includes the steps of tokenization and embedding them into some form of vector space. These embeddings in many cases are trained either through some self supervised task on some corpora (examples include Word2Vec or Glove) or doing it from scratch under whichever task/data-set is being used.
Now regarding that paper it does not solve the problem but it does extend a new methodology that assists in helping achieve a better learned generalizable representation for this task. The way I interpreted it is that they don't just use a sense-label (which would be a one-hot encoding / what they call discrete) but instead use a continuous sense representation and do the comparison there. This difference allows for words with similar but different senses to not be equidistant from words with completely different senses.

Word Sense Disambiguation (WSD) is the task of associating meanings or senses (from an existing collection of meanings) with words, given the context of the words. (The word sense is a synonym for meaning.)
For example, consider the noun "tie" in the following two sentences

He wore a vest and a tie.
Their record was 3 wins, 6 losses, and one tie.

In these two sentences, the meaning of the word tie is different. In sentence 1, the word tie refers to a necktie, which is a piece of cloth. In sentence 2, the word tie is a synonym for a draw, so it refers to a situation of a game. Therefore, we could associate the meaning (or sense) "neckwear consisting of a long narrow piece of material" to the word tie in the first sentence and the meaning "the finish of a contest in which the winner is undecided" to the same word in the second sentence.
The goal of WSD is thus to predict the appropriate sense or meaning of a word, given the context of the word.
Why is WSD important in NLP? Of course, there are many words that change meaning depending on the context, so WSD is important because you expect NLP algorithms and models to be able to correctly give meanings to words, given their context.


text:
What is the correct name for state explosion from sensor discretization?
The position of a robot on a map contains of an x/y value, for example $position(x=100.23,y=400.78)$. The internal representation of the variable is a 32bit float which is equal to 4 byte in the RAM memory. For storing the absolute position of the robot (x,y) only $4+4=8$ bytes are needed. During the robot movements, the position is updated continuously.
The problem is, that a 32 bit float variable creates a state space of $2^{32}=4294967296$. Which means there are endless amount of possible positions in which the robot can be. A robot control system maps the sensor readings to an action. If the input space is large, then the control system gets more complicated.
What is the term used in the literature for describing the problem of exploding state space of sensor variables? Can it be handled with discretization?

I would refer to your problem as having a continuous state space.
By using a 32-bit float variable you discritize it. However, creating states for every possible value of a 32-bit float variable is probably too much. You should decide on:

the variable range: what is the real range of the position variables (e.g. from 0 m to 10 m),
and what is the resolution you require for your problem (e.g. 0.01 m or 0.1 m).

Note that you should take into account:

the sensor range resolution,
the required range and resolution for the problem.

Then, based on the number of states you could decide to discretize the states or to use a Monte Carlo approach.
See for example the work of Brechtel et al. (2013).


text:
Understanding how continuous bag of words method learns embedded representations
I'm reading notes on word vectors here. Specifically, I'm referring to section 4.2 on page 7. First, regarding points 1 to 6 - here's my understanding:
If we have a vocabulary $V$, the naive way to represent words in it would be via one-hot-encoding, or in other words, as basis vectors of $R^{|V|}$ - say $e_1, e_2,\ldots,e_{|V|}$. We want to map these to $\mathbb{R}^n$, via some linear transformation such that the images of similar words (more precisely, the images of basis vectors corresponding to similar words) have higher inner products. Assuming the matrix representation of the linear transformation given the standard basis of $\mathbb{R}^{|V|}$ is denoted by $\mathcal{V}$, then the "embedding" of the $i$-th vocab word (i.e. the image of the corresponding basis vector $e_i$ of $V$) is given by $\mathcal{V}e_i$.
Now suppose we have a context "The cat ____ over a", CBoW seeks to find a word that would fit into this context. Let the words "the", "cat", "over", "a" be denoted (in the space $V$) by $x_{i_1},x_{i_2},x_{i_3},x_{i_4}$ respectively. We take the image of their linear combination (in particular, their average):
$$\hat v=\mathcal{V}\bigg(\frac{x_{i_1}+x_{i_2}+x_{i_3}+x_{i_4}}{4}\bigg)$$
We then map $\hat v$ back from $\mathbb{R}^n$ to $\mathbb{R}^{|V|}$ via another linear mapping whose matrix representation is $\mathcal{U}$: $$z=\mathcal{U}\hat v$$
Then we turn this score vector $z$ into softmax probabilities $\hat y=softmax(z)$ and compare it to the basis vector corresponding to the actual word, say $e_c$. For example, $e_c$ could be the basis vector corresponding to "jumped".
Here's my interpretation of what this procedure is trying to do: given a context, we're trying to learn maps $\mathcal{U}$ and $\mathcal{V}$ such that given a context like "the cat ____ over a", the model should give a high score to words like "jumped" or "leaped", etc. Not just that - but "similar" contexts should also give rise to high scores for "jumped", "leaped", etc. For example, given a context "that dog ____ above this" wherein "that", "dog", "above", "this" are represented by $x_{j_1},x_{j_2},x_{j_3},x_{j_4}$, let the image of their average be
$$\hat w=\mathcal{V}\bigg(\frac{x_{j_1}+x_{j_2}+x_{j_3}+x_{j_4}}{4}\bigg)$$
This gets mapped to a score vector $z'=\mathcal{U}\hat w$. Ideally, both score vectors $z$ and $z'$ should have similarly high magnitudes in their components corresponding to similar words "jumped" and "leaped".
Now to the questions:

We create two matrices, $\mathcal{V} \in \mathbb{R}^{n\times |V|}$ and $\mathcal{U} \in \mathbb{R}^{|V|\times n}$, where $n$ is an arbitrary size which defines the size of our embedding space. $\mathcal{V}$ is the input word matrix such that the $i$-th column of $\mathcal{V}$ is the $n$-dimensional embedded vector for word $w_i$ when it is an input to this model. We denote this $n\times 1$ vector as $v_i$. Similarly, $\mathcal{U}$ is the output word matrix. The $j$-th row of $\mathcal{U}$ is an $n$-dimensional embedded vector for word $w_j$ when it is an output of the model. We denote this row of $\mathcal{U}$ as $u_j$.

How does minimizing the cross-entropy loss between $e_c$ and $\hat y$ ensure that basis vectors corresponding to similar words $e_i$ and $e_j$ are mapped to vectors in $\mathbb{R}^n$ that have high inner product? I'm not sure of the mechanism how the above procedure ensures that. In other words, how is it ensured that if words no. $i_1$ and $i_2$ are similar, then $\langle v_{i_1}, v_{i_2}\rangle$ and $\langle u_{i_1}, u_{i_2}\rangle$ have high values?

How does the above procedure ensure that linear combinations of words in similar contexts are mapped to "similar" images? Does that even happen? In the above description for example, do $\hat v$ and $\hat w$ corresponding to similar contexts also have a high inner product? If so, how is that ensured?

Maybe my linear algebra is rusty and this is a silly question, but from what I gather, the columns of $\mathcal{V}$ represent the images of OHE vectors (standard basis of $V$) in the standard basis of $\mathbb{R}^n$ - i.e. the embedded representation of vocab words. Also, the rows of $\mathcal{U}$ also somehow represent the embedded representation of vocab words in $\mathbb{R}^n$. It's not obvious to me why $v_i=\mathcal{V}e_i$ should be the same as or even similar to $u_i$. Again, how does the above procedure ensure that?

Hi. You're asking several questions in this post. Ideally, you should ask just one question per post, so that to facilitate the life of potential answerers (who might know the answer to one question but not the others). Furthermore, long posts I think might discourage people like me (who don't have much time) from reading them. I suggest you ask just one question in this post and ask the other questions in their separate post.

@nbro: Thanks for the tip! I'll try to see how to reformulate the above question and break it down into parts.


text:
What are the implications of the "No Free Lunch" theorem for machine learning?
The No Free Lunch (NFL) theorem states (see the paper Coevolutionary Free Lunches by David H. Wolpert and William G. Macready)

any two algorithms are equivalent when their performance is averaged across all possible problems

Is the "No Free Lunch" theorem really true? What does it actually mean? A nice example (in ML context) illustrating this assertion would be nice.
I have seen some algorithms which behave very poorly, and I have a hard time believing that they actually follow the above-stated theorem, so I am trying to understand whether my interpretation of this theorem is correct or not. Or is it just another ornamental theorem like Cybenko's Universal Approximation theorem?

This is a really common reaction after first encountering the No Free Lunch theorems (NFLs). The one for machine learning is especially unintuitive, because it flies in the face of everything that's discussed in the ML community. That said, the theorem is true, but what it means is open to some debate.
To restate the theorem for people who don't know it, the NFL theorem for machine learning is really a special case of the NFL theorem for local search and optimization. The local search version is easier to understand. The theorem makes the following, somewhat radical claim:

Averaged across all possible optimization problems, the average solution quality found by any local search algorithm you choose to use is exactly the same as the average solution quality of a local "search" algorithm that just generates possible solutions by sampling uniformly at random from the space of all solutions.

Another formulation, when people want an even stronger reaction, is to say that if you want to find the best solution to a problem, it's just as good to try things that seem to be making your solution iteratively worse as it is to try things that seem to be making your solution iteratively better. On average, both these approaches are equally good.
Okay, so why is this true? Well, the key is in the details. Wolpert has sometimes described the theorem as a specialization of Hume's work on the problem of induction. The basic statement of the problem of induction is: we have no logical basis for assuming that the future will be like the past. Logically, there's no reason that the laws of physics couldn't all just radically change tomorrow. From a purely logical perspective, it's totally reasonable that the future can be different from the past in any number of ways. Hume's problem is that, in general the future is like the past in a lot of ways. He tried to formulate a philosophical (logical) argument that this needed to be so, but basically failed.
The No Free Lunch theorems say the same thing. If you don't know what your search space looks like, then if you iteratively refine your guess at what a good solution looks like, in response to the observations you've made in the past about what good solutions look like (i.e. learning from data), then it's just as likely that operation you make helps as it is that it hurts. That's why the "averaged over all possible optimization problems" part is key. For any optimization problem where hill climbing is a good strategy after $k$ moves, we can make one that is identical, except that the kth hill climbing move leads to an awful solution. The actual proof is more subtle than that, but that's the basic idea.
A very brief lay summary might be:

A machine learning algorithim can only be made to work better on some kinds of problems by being made to work worse on another kind of problem.

So what does this mean in a practical sense? It means that you need to have some apriori reason for thinking that your algorithm will be effective on a particular problem. Exactly what a good reason looks like is the subject of vigorous debate within the ML community. This is very closely related to the bias/variance tradeoff.
Some common responses are:

When you're looking at a new optimization problem, although it could have any random kind of structure, the problems we actually encounter in the real world are a lot more regular, and certain common themes are present, like the the fact that moving "uphill" (minimizing error) iteratively tends to lead to good solutions. Basically, this school of thought says NFL is an ornamental theorem: most ML algorithms work better on "the kind of problems we see in real life", by working worse on "the kind of problems we don't see in real life".
When you're looking at a new optimization problem in [insert your favourite application domain], although it could have any random kind of structure,  problems tend to look like [whatever you think], which makes [your favorite algorithm] a lot more effective than random guessing.
Wolpert & McCready themselves published an interesting result showing that there actually are specialized optimization processes, based on co-evolution, that are consistently better than random guessing.

Regardless, it's indisputable that some algorithms are better than others, in certain sub-domains (we can see this empirically). NFL tells us that to be better there, they need to be worse somewhere else. The question up for debate is whether the "somewhere else" is real problems, or purely artificial ones.

Great answer. But by algorithm do they include all variations of it? For example backprop maybe implemented by derivatives, or by taking small differences or by double derivatives (as far as I know), so are they same or different? And by performance is it end results or resources too?

Yes. The key idea is that, no matter what strategy you come up with for solving your optimization problem (like minimizing error by taking into account higher derivatives), I can create a version of the problem that looks exactly the same except that, after $k$ iterations, you end up in a bad solution.

For ML, does the size of the training set matters? In other words, if for all problems we had an infinite amount of training data, would some learning algorithm perform better than the other (averaged over all possible data distributions)? I don't know if infinite data makes sense, because in this case the notion of "generalization" is lost.


text:
Can reinforcement learning be utilized for creating a simulation?
According to the definition, the AI agent has to play a game by it's own. A typical domain is the blocksworld problem. The AI determines which action the robot in a game should execute and a possible strategy for determine the action sequence is reinforcement learning. Colloquial spoken, reinforcement learning leads to an AI agent who can play games.
Before a self-learning character can be realized, the simulation has to be programmed first. That is an environment which contains the rules for playing blocksworld or any other game. The environment is the house in which the AI character operates. Can the Q-learning algorithm be utilized to build the simulation itself?

Can the Q-learning algorithm be utilized to build the simulation itself?

Only in the presence of a meta-environment, or meta-simulation where the goals of creating the original simulation are encoded in the states, available actions and rewards.
A special case of this might be in model-learning planning algorithms where there exists a "real" environment to refer to, and the agent benefits from exploring it and constructing a statistical model that it can then use to create an approximate simulation of the outcomes of sequences of actions. The Dyna-Q algorithm, which is a simple extension of Q-learning, is an example of this kind of model building approach. The simulation is very basic - it simply replays previous relevant experience. But you could consider this as an example of the agent constructing a simulation.
Getting an agent to act like a researcher and actually design and/or code a simulation from scratch would require a different kind of meta-environment. It is theoretically possible, but likely very hard to implement in general way - even figuring out the reward scheme to express the goals of such an agent could be a challenge. I'm not aware of any examples, but entirely possible someone has attempted this kind of meta agent, because it is an interesting idea.
Possibly the simplest example would be a gridworld meta-environment where a "designer" agent could select the layout of objects in the maze, with the goal of making a second "explorer" agent's task progressivley more difficult. The designer would be "creating the simulation" only in a very abstract way though, by setting easy-to-manage parameters of the environment, not writing low level code.
There is not much difference between the approach above and having two opposing agents playing a game. It is different from a turn-based game like chess in that each agent would complete a full episode, and then be rewarded by the outcome at the end of the combined two episodes. There are some similarities to GANs for image generation.


text:
How do I determine the most appropriate classifier for a certain problem?
Consider a Bayesian classifier used in spam e-mail filtering. It converts an e-mail to a vector, most of the time using the bag-of-words method. Although it learns first before getting employed, it can be made to work as an online system, i.e. it can be used to filter and learn from examples even after deployment.
Now, on the other hand, now comes the perceptron. It calculates a mean vector of spam and not spam, and then classifies them into the appropriate categories. The model adjusts the mean vectors each time it makes mistakes.
Now, comes neural nets, they too are capable of taking a vector-like bag of words or image pixels of dogs and cats and classify them into yes or no.
So, while designing and implementing them into the system, how to determine which one of the methods (Bayesian classifier, perceptron or neural network) is the most appropriate for a given situation or task? One factor to consider is the time complexity (or speed), but what are other factors, and how to rank them?

This is one of the main skills that separates someone with a deep understanding of, and experience in, machine learning learning, from a neophyte. There are several approaches:

Try several methods, perhaps with automated hyperparameter optimization, and see if there's a big difference in typical model quality. This is pretty common if you don't have a lot of experience, but also something experts may try in a more targeted way.
Visualize the shape of your problem, perhaps by using a dimensionality reduction technique like PCA or tSNE, or maybe an auto-encoder. If you compress the data to 2d, are there clear linear patterns? Maybe try a linear model like logistic regression. Are there several distinct groups? How many lines would you need to draw to separate them? If it's a lot, maybe you're going to need a very non-linear model. If it's just a few, maybe a small multi-layer perceptron can help. If there are spiral bands or circular shapes, maybe and SVM with a non-linear kernel. Knowing how to translate the visualizations into intuition about the kinds of model that can help is an advanced skill. You need to understand what shapes of patterns each kind of model can learn to fit, and how these do or do not translate into a higher dimensional space.
Read the literature. If you're working in computer vision, you should try a CNN. Why? Well, everyone else is using them. They work great on most computer vision problems. They hold most competition records. It'd be silly not to try them. If you're working on spam classification though, CNNs are a bad choice. People use compression classifiers, Bayesian models, and sometimes multi-stage models to build complex features. If you look at the recent literature in your area, you can tell what to use. Reading and understanding this literature well enough to interpret it usually requires more advanced scientific training in ML and/or the application area.

As a final note, the No Free Lunch theorems for ML tell us that this is always going to be an art (at least, that's one interpretation experts argue for), and not a science, so the more practice you get with it, the better you'll become.