问:
Can we consume JMS messages from a Topic through Spark Streaming?
Problem:There is a JMS topic and I've following connection details :
URL:xyz
Connection Factory: jms.xyz
Topic Name: jms.xyz
username:
password:

Is there a working code which creates a subscriber in Spark Scala which consumes the JMS messages from the topic?

I have tried using socketTextStream function of spark streaming, but it has only URL parameter.
I'm looking for a spark streaming function which can have all my 5 parameters :
1) URL
2) Connection Factory
3) Topic Name
4) Username
5) Password

I've tried running in Spark-Shell

I'm looking for a spark streaming function which can have all my 5 parameters and a working spark-scala code which can consume JMS messages from the topic:
1) URL
2) Connection Factory
3) Topic Name
4) Username
5) Password

I am looking for Spark-Shell base commands which I can execute line by line

答:
Q: Can we consume JMS messages from a Topic through Spark Streaming?

Yes. AFAIK there is no silver bullet solution for this.

Based on Messaging provider implementation may vary.
you may need to write a custom receiver from spark docs for that..

See Example 2 which uses jms topic integration with spark-streaming

Example 1 (source):

```<code>import org.apache.log4j.Logger;
import org.apache.spark.storage.StorageLevel;
import org.apache.spark.streaming.receiver.Receiver;

import javax.jms.*;
import javax.naming.Context;
import java.util.Hashtable;

public class JMSReceiver extends Receiver<JMSEvent> implements MessageListener
{
    private static final Logger log = Logger.getLogger(JMSReceiver.class);

    private static final String JNDI_INITIAL_CONTEXT_FACTORY       = "org.apache.qpid.jms.jndi.JmsInitialContextFactory";
    private static final String JNDI_CONNECTION_FACTORY_NAME       = "JMSReceiverConnectionFactory";
    private static final String JNDI_QUEUE_NAME                    = "JMSReceiverQueue";
    private static final String JNDI_CONNECTION_FACTORY_KEY_PREFIX = "connectionfactory.";
    private static final String JNDI_QUEUE_KEY_PREFIX              = "queue.";

    private StorageLevel _storageLevel;

    private String _brokerURL;
    private String _username;
    private String _password;
    private String _queueName;
    private String _selector;

    private Connection _connection;

    public JMSReceiver(String brokerURL, String username, String password, String queueName, String selector, StorageLevel storageLevel)
    {
        super(storageLevel);
        _storageLevel = storageLevel;
        _brokerURL = brokerURL;
        _username = username;
        _password = password;
        _queueName = queueName;
        _selector = selector;

        log.info("Constructed" + this);
    }

    @Override
    public void onMessage(Message message)
    {
        try
        {
            log.info("Received: " + message);
            JMSEvent jmsEvent = new JMSEvent(message);
            store(jmsEvent);
        } catch (Exception exp)
        {
            log.error("Caught exception converting JMS message to JMSEvent", exp);
        }
    }

    @Override
    public StorageLevel storageLevel()
    {
        return _storageLevel;
    }

    public void onStart()
    {

        log.info("Starting up...");

        try
        {

            Hashtable<Object, Object> env = new Hashtable<Object, Object>();
            env.put(Context.INITIAL_CONTEXT_FACTORY, JNDI_INITIAL_CONTEXT_FACTORY);
            env.put(JNDI_CONNECTION_FACTORY_KEY_PREFIX + JNDI_CONNECTION_FACTORY_NAME, _brokerURL);
            env.put(JNDI_QUEUE_KEY_PREFIX + JNDI_QUEUE_NAME, _queueName);
            javax.naming.Context context = new javax.naming.InitialContext(env);

            ConnectionFactory factory = (ConnectionFactory) context.lookup(JNDI_CONNECTION_FACTORY_NAME);
            Destination queue = (Destination) context.lookup(JNDI_QUEUE_NAME);

            if ((_username == null) || (_password == null))
            {
                _connection = factory.createConnection();
            } else
            {
                _connection = factory.createConnection(_username, _password);
            }
            _connection.setExceptionListener(new JMSReceiverExceptionListener());

            Session session = _connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

            MessageConsumer messageConsumer;

            if (_selector != null)
            {
                messageConsumer = session.createConsumer(queue, _selector);
            } else
            {
                messageConsumer = session.createConsumer(queue);
            }
            messageConsumer.setMessageListener(this);

            _connection.start();

            log.info("Completed startup.");
        } catch (Exception exp)
        {
            // Caught exception, try a restart
            log.error("Caught exception in startup", exp);
            restart("Caught exception, restarting.", exp);
        }
    }

    public void onStop()
    {
        // Cleanup stuff (stop threads, close sockets, etc.) to stop receiving data

        log.info("Stopping...");
        try
        {
            _connection.close();
        } catch (JMSException exp)
        {
            log.error("Caught exception stopping", exp);
        }
        log.info("Stopped.");
    }

    private class JMSReceiverExceptionListener implements ExceptionListener
    {
        @Override
        public void onException(JMSException exp)
        {
            log.error("Connection ExceptionListener fired, attempting restart.", exp);
            restart("Connection ExceptionListener fired, attempting restart.");
        }
    }

    @Override
    public String toString()
    {
        return "JMSReceiver{" +
                "brokerURL='" + _brokerURL + '\'' +
                ", username='" + _username + '\'' +
                ", password='" + _password + '\'' +
                ", queueName='" + _queueName + '\'' +
                ", selector='" + _selector + '\'' +
                '}';
    }
}
</code>```

your JMSInputDstream will look like

```<code>import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming._
import org.apache.spark.streaming.dstream._
import org.apache.spark.streaming.receiver.Receiver

private[streaming]
class JMSInputDStream(
                       @transient ssc_ : StreamingContext,
                       brokerURL: String,
                       username: String,
                       password: String,
                       queuename: String,
                       selector: String,
                       storageLevel: StorageLevel
                       ) extends ReceiverInputDStream[JMSEvent](ssc_) {

  override def getReceiver(): Receiver[JMSEvent] = {
    new JMSReceiver(brokerURL, username, password, queuename, selector, storageLevel)
  }
}
</code>```

Example 2 using activemq and JmsTopicReceiver.scala :

```<code>import org.apache.spark.Logging
import org.apache.spark.storage.StorageLevel
import javax.{jms => jms}

/** Simple class of a receiver that can be run on worker nodes to receive the data from JMS Topic.
  *
  * In JMS a Topic implements publish and subscribe semantics.
  * When you publish a message it goes to all the subscribers who are interested - so zero to many subscribers will receive a copy of the message.
  * Only subscribers who had an active subscription at the time the broker receives the message will get a copy of the message.
  *
  * {{{
  *  val sc: SparkContext = SparkContext.getOrCreate(conf)
  *  val ssc: StreamingContext = new StreamingContext(sc, Seconds(...))
  *
  *  val stream: InputDStream[String] = ssc.receiverStream(new JmsTopicReceiver(
  *    topicName = "testTopic",
  *    transformer = { msg => msg.asInstanceOf[javax.jms.TextMessage].getText() },
  *    connectionProvider = { () => {
  *      val cf = new org.apache.activemq.ActiveMQConnectionFactory("tcp://localhost:61616")
  *      cf.setOptimizeAcknowledge(true)
  *      cf.createConnection("username", "password")
  *    }}
  *  ))
  *
  *  ...
  *
  *  ssc.start()
  *  ssc.awaitTermination()
  * }}}
  *
  * @param connectionProvider provides <CODE>javax.jms.Connection</CODE> for the receiver.
  * @param transformer (pre)transforms <CODE>javax.jms.Message</CODE> to appropriate class (it's required to do this before populate the result).
  * @param topicName the name of required <CODE>javax.jms.Topic</CODE>.
  * @param messageSelector only messages with properties matching the message selector expression are delivered.
  * @param storageLevel flags for controlling the storage of an RDD.
  * @tparam T RDD element type.
  */
class JmsTopicReceiver[T] (
  connectionProvider: (() => jms.Connection),
  transformer: (jms.Message => T),
  topicName: String,
  messageSelector: Option[String] = None,
  storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_SER_2
) extends AbstractJmsReceiver[T](
  messageSelector = messageSelector,
  storageLevel = storageLevel
) with Logging {

  override protected def buildConnection(): jms.Connection = connectionProvider()
  override protected def transform(message: jms.Message): T = transformer(message)
  override protected def buildDestination(session: jms.Session): jms.Destination = session.createTopic(topicName)

}

</code>```

Example 3 : Solace used custom receiver of spark : This way I worked long back when spark 1.3 was there,

Solace-JMS-Integration-Spark-Streaming.pdf

Further reading : Processing Data from MQ with Spark Streaming: Part 1 - Introduction to Messaging, JMS & MQ

Body:
Q: Can we consume JMS messages from a Topic through Spark Streaming?

Yes. AFAIK there is no silver bullet solution for this.

Based on Messaging provider implementation may vary.
you may need to write a custom receiver from spark docs for that..

See Example 2 which uses jms topic integration with spark-streaming

Example 1 (source):

import org.apache.log4j.Logger;
import org.apache.spark.storage.StorageLevel;
import org.apache.spark.streaming.receiver.Receiver;

import javax.jms.*;
import javax.naming.Context;
import java.util.Hashtable;

public class JMSReceiver extends Receiver<JMSEvent> implements MessageListener
{
    private static final Logger log = Logger.getLogger(JMSReceiver.class);

    private static final String JNDI_INITIAL_CONTEXT_FACTORY       = "org.apache.qpid.jms.jndi.JmsInitialContextFactory";
    private static final String JNDI_CONNECTION_FACTORY_NAME       = "JMSReceiverConnectionFactory";
    private static final String JNDI_QUEUE_NAME                    = "JMSReceiverQueue";
    private static final String JNDI_CONNECTION_FACTORY_KEY_PREFIX = "connectionfactory.";
    private static final String JNDI_QUEUE_KEY_PREFIX              = "queue.";

    private StorageLevel _storageLevel;

    private String _brokerURL;
    private String _username;
    private String _password;
    private String _queueName;
    private String _selector;

    private Connection _connection;

    public JMSReceiver(String brokerURL, String username, String password, String queueName, String selector, StorageLevel storageLevel)
    {
        super(storageLevel);
        _storageLevel = storageLevel;
        _brokerURL = brokerURL;
        _username = username;
        _password = password;
        _queueName = queueName;
        _selector = selector;

        log.info("Constructed" + this);
    }

    @Override
    public void onMessage(Message message)
    {
        try
        {
            log.info("Received: " + message);
            JMSEvent jmsEvent = new JMSEvent(message);
            store(jmsEvent);
        } catch (Exception exp)
        {
            log.error("Caught exception converting JMS message to JMSEvent", exp);
        }
    }

    @Override
    public StorageLevel storageLevel()
    {
        return _storageLevel;
    }

    public void onStart()
    {

        log.info("Starting up...");

        try
        {

            Hashtable<Object, Object> env = new Hashtable<Object, Object>();
            env.put(Context.INITIAL_CONTEXT_FACTORY, JNDI_INITIAL_CONTEXT_FACTORY);
            env.put(JNDI_CONNECTION_FACTORY_KEY_PREFIX + JNDI_CONNECTION_FACTORY_NAME, _brokerURL);
            env.put(JNDI_QUEUE_KEY_PREFIX + JNDI_QUEUE_NAME, _queueName);
            javax.naming.Context context = new javax.naming.InitialContext(env);

            ConnectionFactory factory = (ConnectionFactory) context.lookup(JNDI_CONNECTION_FACTORY_NAME);
            Destination queue = (Destination) context.lookup(JNDI_QUEUE_NAME);

            if ((_username == null) || (_password == null))
            {
                _connection = factory.createConnection();
            } else
            {
                _connection = factory.createConnection(_username, _password);
            }
            _connection.setExceptionListener(new JMSReceiverExceptionListener());

            Session session = _connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

            MessageConsumer messageConsumer;

            if (_selector != null)
            {
                messageConsumer = session.createConsumer(queue, _selector);
            } else
            {
                messageConsumer = session.createConsumer(queue);
            }
            messageConsumer.setMessageListener(this);

            _connection.start();

            log.info("Completed startup.");
        } catch (Exception exp)
        {
            // Caught exception, try a restart
            log.error("Caught exception in startup", exp);
            restart("Caught exception, restarting.", exp);
        }
    }

    public void onStop()
    {
        // Cleanup stuff (stop threads, close sockets, etc.) to stop receiving data

        log.info("Stopping...");
        try
        {
            _connection.close();
        } catch (JMSException exp)
        {
            log.error("Caught exception stopping", exp);
        }
        log.info("Stopped.");
    }

    private class JMSReceiverExceptionListener implements ExceptionListener
    {
        @Override
        public void onException(JMSException exp)
        {
            log.error("Connection ExceptionListener fired, attempting restart.", exp);
            restart("Connection ExceptionListener fired, attempting restart.");
        }
    }

    @Override
    public String toString()
    {
        return "JMSReceiver{" +
                "brokerURL='" + _brokerURL + '\'' +
                ", username='" + _username + '\'' +
                ", password='" + _password + '\'' +
                ", queueName='" + _queueName + '\'' +
                ", selector='" + _selector + '\'' +
                '}';
    }
}

your JMSInputDstream will look like

import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming._
import org.apache.spark.streaming.dstream._
import org.apache.spark.streaming.receiver.Receiver

private[streaming]
class JMSInputDStream(
                       @transient ssc_ : StreamingContext,
                       brokerURL: String,
                       username: String,
                       password: String,
                       queuename: String,
                       selector: String,
                       storageLevel: StorageLevel
                       ) extends ReceiverInputDStream[JMSEvent](ssc_) {

  override def getReceiver(): Receiver[JMSEvent] = {
    new JMSReceiver(brokerURL, username, password, queuename, selector, storageLevel)
  }
}

Example 2 using activemq and JmsTopicReceiver.scala :

import org.apache.spark.Logging
import org.apache.spark.storage.StorageLevel
import javax.{jms => jms}

/** Simple class of a receiver that can be run on worker nodes to receive the data from JMS Topic.
  *
  * In JMS a Topic implements publish and subscribe semantics.
  * When you publish a message it goes to all the subscribers who are interested - so zero to many subscribers will receive a copy of the message.
  * Only subscribers who had an active subscription at the time the broker receives the message will get a copy of the message.
  *
  * {{{
  *  val sc: SparkContext = SparkContext.getOrCreate(conf)
  *  val ssc: StreamingContext = new StreamingContext(sc, Seconds(...))
  *
  *  val stream: InputDStream[String] = ssc.receiverStream(new JmsTopicReceiver(
  *    topicName = "testTopic",
  *    transformer = { msg => msg.asInstanceOf[javax.jms.TextMessage].getText() },
  *    connectionProvider = { () => {
  *      val cf = new org.apache.activemq.ActiveMQConnectionFactory("tcp://localhost:61616")
  *      cf.setOptimizeAcknowledge(true)
  *      cf.createConnection("username", "password")
  *    }}
  *  ))
  *
  *  ...
  *
  *  ssc.start()
  *  ssc.awaitTermination()
  * }}}
  *
  * @param connectionProvider provides <CODE>javax.jms.Connection</CODE> for the receiver.
  * @param transformer (pre)transforms <CODE>javax.jms.Message</CODE> to appropriate class (it's required to do this before populate the result).
  * @param topicName the name of required <CODE>javax.jms.Topic</CODE>.
  * @param messageSelector only messages with properties matching the message selector expression are delivered.
  * @param storageLevel flags for controlling the storage of an RDD.
  * @tparam T RDD element type.
  */
class JmsTopicReceiver[T] (
  connectionProvider: (() => jms.Connection),
  transformer: (jms.Message => T),
  topicName: String,
  messageSelector: Option[String] = None,
  storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_SER_2
) extends AbstractJmsReceiver[T](
  messageSelector = messageSelector,
  storageLevel = storageLevel
) with Logging {

  override protected def buildConnection(): jms.Connection = connectionProvider()
  override protected def transform(message: jms.Message): T = transformer(message)
  override protected def buildDestination(session: jms.Session): jms.Destination = session.createTopic(topicName)

}

Example 3 : Solace used custom receiver of spark : This way I worked long back when spark 1.3 was there,

Solace-JMS-Integration-Spark-Streaming.pdf

Further reading : Processing Data from MQ with Spark Streaming: Part 1 - Introduction to Messaging, JMS & MQ


来源: stackexchange

问:
Get commit hash for a commit in a branch
In a git branch I store the latest commit hash SHA using

latest_sha=$(git log --pretty=oneline | head -1 | cut -d ' ' -f 1)

After a bunch of commits in this branch, how do I get the next commit SHA after the latest_sha.

Say if there are 5 commits made to this branch after the $latest_sha.

I want to always get the SHA of the first commit after the latest_sha.

```<code>b8eead8ba4ff375911af6
c2452680eb7731e4d36ca
da2e113ca4768f5f34730
95b98d42a6e567ed56fc2
716c4f84a855f48bee55c
6a7223a74269f925cfd9e---I need this one
e945bcfabf3fbafc85084---latest_sha
159df375376ded565bec0
d725350982626f46a8b80
56a4b6ca91d93acc8d751
de584608616b1ed99a554
3cfc15339a98bb286d5baa
6ae834bf36c90fbd81854
fa9bdebd0f814f04ee05ba
cc44c4d9ff14314c1255da
5a6145586a8fdcaa2da659
bfea8cfe121d24a0ff1525

</code>```

Thanks!

答:
git logs have a parent commit field, so you could do something like</p>

```<code>git log --pretty=format:"%P %H" | awk '$1 == "<YOUR_HASH>" {print $2}'
</code>```

ContentLicense: CC BY-SA 4.0

Body:
In a git branch I store the latest commit hash SHA using

latest_sha=$(git log --pretty=oneline | head -1 | cut -d ' ' -f 1)

After a bunch of commits in this branch, how do I get the next commit SHA after the latest_sha.

Say if there are 5 commits made to this branch after the $latest_sha.

I want to always get the SHA of the first commit after the latest_sha.

b8eead8ba4ff375911af6
c2452680eb7731e4d36ca
da2e113ca4768f5f34730
95b98d42a6e567ed56fc2
716c4f84a855f48bee55c
6a7223a74269f925cfd9e---I need this one
e945bcfabf3fbafc85084---latest_sha
159df375376ded565bec0
d725350982626f46a8b80
56a4b6ca91d93acc8d751
de584608616b1ed99a554
3cfc15339a98bb286d5baa
6ae834bf36c90fbd81854
fa9bdebd0f814f04ee05ba
cc44c4d9ff14314c1255da
5a6145586a8fdcaa2da659
bfea8cfe121d24a0ff1525

Thanks!

Body:
git logs have a parent commit field, so you could do something like

git log --pretty=format:"%P %H" | awk '$1 == "<YOUR_HASH>" {print $2}'


问:
Uploading file in a form without page refresh
I have this bit of code:

```<code><form name="myUploadForm" method="post" action="/scripts/upload.do" enctype="multipart/form-data" id="fileUpload">
   <table width="100%" border="0">
      <tr>
         <td>
            <input type="file" name="xlsFile" size="60" value="test.xls">
            <input type="button" value="Upload File" name="upload_xls">
         </td>
      </tr>
   </table>
</form>
</code>```

Right now I can upload the file with Struts but it refreshes the page.  How do I do this without the page refreshing?

答: If you can use jQuery, you can use something like jQuery File Upload.</p>

ContentLicense: CC BY-SA 3.0

Body:
I have this bit of code:

<form name="myUploadForm" method="post" action="/scripts/upload.do" enctype="multipart/form-data" id="fileUpload">

            <input type="file" name="xlsFile" size="60" value="test.xls">
            <input type="button" value="Upload File" name="upload_xls">

</form>

Right now I can upload the file with Struts but it refreshes the page.  How do I do this without the page refreshing?


问:
Setting a value in a nested Python dictionary given a list of indices and value
I'm trying to programmatically set a value in a dictionary, potentially nested, given a list of indices and a value.

So for example, let's say my list of indices is:

```<code>['person', 'address', 'city']
</code>```

and the value is

```<code>'New York'
</code>```

I want as a result a dictionary object like:

```<code>{ 'Person': { 'address': { 'city': 'New York' } }
</code>```

Basically, the list represents a 'path' into a nested dictionary.

I think I can construct the dictionary itself, but where I'm stumbling is how to set the value. Obviously if I was just writing code for this manually it would be:

```<code>dict['Person']['address']['city'] = 'New York'
</code>```

But how do I index into the dictionary and set the value like that programmatically if I just have a list of the indices and the value?

Python

答:
Here's a variant of Bakuriu's answer that doesn't rely on a separate function:</p>
```<code>keys = ['Person', 'address', 'city']
value = 'New York'

nested_dict = {}

# Build nested dictionary up until 2nd to last key
# (Effectively nested_dict['Person']['address'] = {})
sub_dict = nested_dict
for key_ind, key in enumerate(keys[:-1]):
    if not key_ind:
        # Point to newly added piece of dictionary
        sub_dict = nested_dict.setdefault(key, {})
    else:
        # Point to newly added piece of sub-dictionary
        # that is also added to original dictionary
        sub_dict = sub_dict.setdefault(key, {})
# Add value to last key of nested structure of keys
# (Effectively nested_dict['Person']['address']['city'] = value)
sub_dict[keys[-1]] = value

print(nested_dict)

>>> {'Person': {'address': {'city': 'New York'}}}
</code>```

Body:
I'm trying to programmatically set a value in a dictionary, potentially nested, given a list of indices and a value.

So for example, let's say my list of indices is:

['person', 'address', 'city']

and the value is

'New York'

I want as a result a dictionary object like:

{ 'Person': { 'address': { 'city': 'New York' } }

Basically, the list represents a 'path' into a nested dictionary.

I think I can construct the dictionary itself, but where I'm stumbling is how to set the value. Obviously if I was just writing code for this manually it would be:

dict['Person']['address']['city'] = 'New York'

But how do I index into the dictionary and set the value like that programmatically if I just have a list of the indices and the value?

Python

Body:
Here's a variant of Bakuriu's answer that doesn't rely on a separate function:

keys = ['Person', 'address', 'city']
value = 'New York'

nested_dict = {}

# Build nested dictionary up until 2nd to last key
# (Effectively nested_dict['Person']['address'] = {})
sub_dict = nested_dict
for key_ind, key in enumerate(keys[:-1]):
    if not key_ind:
        # Point to newly added piece of dictionary
        sub_dict = nested_dict.setdefault(key, {})
    else:
        # Point to newly added piece of sub-dictionary
        # that is also added to original dictionary
        sub_dict = sub_dict.setdefault(key, {})
# Add value to last key of nested structure of keys
# (Effectively nested_dict['Person']['address']['city'] = value)
sub_dict[keys[-1]] = value

print(nested_dict)

>>> {'Person': {'address': {'city': 'New York'}}}


问:
What is the range of a Windows HANDLE on a 64 bits application?
On WinAPI, the HANDLE type is defined as a void*, thus on a 64 bit application the HANDLE value may range from 0 to 18446744073709551615.
But is that true in practice? Does any documentation specify the integral range of such a HANDLE?

If for instance one wants to store this HANDLE as an int32_t on a 32 bit application that's completely fine, but on a 64 bit application the doubts sticks.

答:
To add to the previous correct answers, let me note that a HWND is also a valid handle across processes. Any other void* handle (like HBRUSH, HBITMAP etc) might be truncatable because only the lower 32 bits are significant all right, but it's not valid outside its own process.

For GDI objects it might work because these are actually indices (see https://learn.microsoft.com/en-us/previous-versions/ms810501(v=msdn.10))

Well, what happens here is that handles to GDI objects are internally
  implemented as offsets into a handle table that resides on the client
  side of the Win32 subsystem. (Remember that the Win32 client is a DLL
  that resides in a Win32-based application's address space and is
  called by the application.) In other words, handle tables are kept on
  a per-process basis, but they are not process-tagged. That means that
  a handle to an object that belongs to process A might coincidentally
  look like a valid handle in process B's context. Thus, a call to
  SelectObject from B might succeed, but B will actually have selected a
  totally different object into its device context—or worse, the right
  one. Selecting the right object may be worse because the objects might
  coincidentally be the same, so you think it works, but the application
  will behave weirdly later on. So, do not pass handles to GDI objects
  between applications; they have totally different meanings in
  different processes.

HWND is a documented exception to this.


问:
web scraping : how to scrape one particular table body out of many table bodies?
I am trying to scrape a particular table in the site - http://stats.espncricinfo.com/ci/engine/player/35320.html?class=2;template=results;type=batting

Now, there are multiple tables that are indistinguishable from each other. And I want to scrape only one particular table from there. How do I do that?

I have tried using the find_all() function. But that only lists ALL the
 tags.

I want to scrape only the highlighted table body.

答:
It is tbody tagged and you can use the following css selector with bs4. Then wrap with table tags and pass to pandas to print nicely. I'm using bs4 4.7.1

You could also use table = soup.select('tbody:contains(year)').

Python:

```<code>from bs4 import BeautifulSoup as bs
import requests
import pandas as pd

r = requests.get('http://stats.espncricinfo.com/ci/engine/player/35320.html?class=2;template=results;type=batting')
soup = bs(r.content, 'lxml')
table = soup.select('tbody:nth-child(7)')
headers = [item.text for item in soup.select('.headlinks th')]
df = pd.read_html('<table>' + str(table) + '</table>')[0]
df.columns = headers
df = df.dropna(how = 'all', axis=0).drop(['Span',''], axis=1)
print(df)
</code>```

df.head()

Body:
It is tbody tagged and you can use the following css selector with bs4. Then wrap with table tags and pass to pandas to print nicely. I'm using bs4 4.7.1

You could also use table = soup.select('tbody:contains(year)').

Python:

from bs4 import BeautifulSoup as bs
import requests
import pandas as pd

r = requests.get('http://stats.espncricinfo.com/ci/engine/player/35320.html?class=2;template=results;type=batting')
soup = bs(r.content, 'lxml')
table = soup.select('tbody:nth-child(7)')
headers = [item.text for item in soup.select('.headlinks th')]
df = pd.read_html('<table>' + str(table) + '</table>')[0]
df.columns = headers
df = df.dropna(how = 'all', axis=0).drop(['Span',''], axis=1)
print(df)

df.head()


问:
How to create a program that converts binary to decimal
I'm trying to convert binary to decimal, how do I change my code to be able to do that? Where did I mess up?

i tried looking at other examples, looking at java api and watching videos but i still can't figure out what mistake i have made.

```<code>package Calculator;
import java.util.Scanner;
public class Calculator {
    public static void main(String[] args){
        Scanner scan = new Scanner(System.in);
            System.out.println("(2) Convert binary to decimal");
            System.out.println("\n\n Please enter your choice: ");
            int choice = scan.nextInt();
            if(choice == 2){
                scan.nextLine();
                //prompt for user input
                System.out.println("Please enter a binary number: ");
                String binary = scan.nextLine();
                char[] binaryArray = binary.toCharArray();
                int i=1;
                int integer=0;
            //potential problem somewhere around here?
            while(i<8){
                if(binaryArray[i]==0) {
                    ++i;
                }else if(binaryArray[i]==1) {
                    switch(i) {
                    case 1:
                        integer+=128;
                        ++i;
                        break;
                    case 2:
                        integer+=64;
                        ++i;
                        break;
                    case 3:
                        integer+=32;
                        ++i;
                        break;
                    case 4:
                        integer+=16;
                        ++i;
                        break;
                    case 5:
                        integer+=8;
                        ++i;
                        break;
                    case 6:
                        integer+=4;
                        ++i;
                        break;
                    case 7:
                        integer+=2;
                        ++i;
                        break;
                    case 8:
                        integer+=1;
                        ++i;
                        break;
                    }
            }
        }
            System.out.println("The decimal value of the binary number is: "+ integer);
            scan.close();
    }
}
</code>```

}

The input is always 0. I've tried 11010110, 11111111,and 01010111. Always 0. I know the problem lies somewhere with my integer value not changing but I can't figure out what it specifically is.

答:
You should get away from all those switch statements.

Say you have "10101101" as input.
set val = 0;
Then either multiply by val by 2 or shift left 1 bit.  They're the same. It is important
you do this before adding the next bit.

Start from the left and if it's a '1', add a 1 to val. Otherwise, add 0.
Then repeat starting at multiply until you've gone thru the string.

val should then have the decimal version when you print it.

Body:
I'm trying to convert binary to decimal, how do I change my code to be able to do that? Where did I mess up?

i tried looking at other examples, looking at java api and watching videos but i still can't figure out what mistake i have made.

package Calculator;
import java.util.Scanner;
public class Calculator {
    public static void main(String[] args){
        Scanner scan = new Scanner(System.in);
            System.out.println("(2) Convert binary to decimal");
            System.out.println("\n\n Please enter your choice: ");
            int choice = scan.nextInt();
            if(choice == 2){
                scan.nextLine();
                //prompt for user input
                System.out.println("Please enter a binary number: ");
                String binary = scan.nextLine();
                char[] binaryArray = binary.toCharArray();
                int i=1;
                int integer=0;
            //potential problem somewhere around here?
            while(i<8){
                if(binaryArray[i]==0) {
                    ++i;
                }else if(binaryArray[i]==1) {
                    switch(i) {
                    case 1:
                        integer+=128;
                        ++i;
                        break;
                    case 2:
                        integer+=64;
                        ++i;
                        break;
                    case 3:
                        integer+=32;
                        ++i;
                        break;
                    case 4:
                        integer+=16;
                        ++i;
                        break;
                    case 5:
                        integer+=8;
                        ++i;
                        break;
                    case 6:
                        integer+=4;
                        ++i;
                        break;
                    case 7:
                        integer+=2;
                        ++i;
                        break;
                    case 8:
                        integer+=1;
                        ++i;
                        break;
                    }
            }
        }
            System.out.println("The decimal value of the binary number is: "+ integer);
            scan.close();
    }
}

}

The input is always 0. I've tried 11010110, 11111111,and 01010111. Always 0. I know the problem lies somewhere with my integer value not changing but I can't figure out what it specifically is.


问:
How to stop rebuilding .dcu from third-party components?
I work with many third party components. They have an installer that compiles the .pas and automatically adds the path to the library path.

The problem is that when I use the component in a project. The compiler is re-creating the .dcu of the third-party component. How can I make the compiler not create the .dcu again?

I found these old posts where they report similar problems. But I do not quite understand how I really say to compiler: "Compiler stops recreating the .dcu of third-party components, and get the .dcu from that place."

How to deal with warnings/hints in third party libraries?

How do I make third party component's DCU files, not to get generated in my application directory?

答:
First: remove the location of the .pas files from the ‘library path’ (IDE setting) and replace it with the location of the .dcu path (note: the dcu files must be in a different folder then the .pas files of the library, or else the .pas files will be build anyway).
Be sure your project itself also does not reference the .pas files using it’s own search path.

Second: when you want to inspect the code of the third party component using ctrl-mouseclick, add the .pas path to the browsing path of your IDE.

Regarding debugging: you might want to build two sets of dcu’s; one with debugging info and one without...


问:
Import Python class that only has `__new__` into Robot Framework
Have a python class that is imported in Robot Framework as follows:

Python: MyClass.py

```<code>class MyClass:
    def __new__(cls, a, b):
        # implementation here
</code>```

Robot Framework: MyTest.robot

```<code>*** Settings ***
Library    MyClass.py    a=1    b=2
</code>```

Getting the error Error in file 'MyTest.robot': Test Library 'MyClass' expected 0 arguments, got 2.

I understand the error is because MyClass doesn't have __init__ defined (if it's defined, there's no import error).

My question is: How can I import MyClass.py, which has __new__ defined but not __init__?

(without going into the details of why I have __new__ and not __init__)

答:
According to the python documentation, __new__ will pass the arguments it receives on to the __init__. Since you didn't create an __init__ that accepts arguments, that's why you get the error.

If __new__() returns an instance of cls, then the new instance’s __init__() method will be invoked like __init__(self[, ...]), where self is the new instance and the remaining arguments are the same as were passed to __new__().

If you don't want the __init__ to be called, you might get some help from  this question: Possible to prevent init from being called?

Body:
Have a python class that is imported in Robot Framework as follows:

Python: MyClass.py

class MyClass:
    def __new__(cls, a, b):
        # implementation here

Robot Framework: MyTest.robot

*** Settings ***
Library    MyClass.py    a=1    b=2

Getting the error Error in file 'MyTest.robot': Test Library 'MyClass' expected 0 arguments, got 2.

I understand the error is because MyClass doesn't have __init__ defined (if it's defined, there's no import error).

My question is: How can I import MyClass.py, which has __new__ defined but not __init__?

(without going into the details of why I have __new__ and not __init__)


问:
Use of Ngfor in label mat-ltab
I am facing a problem when using *ngFor in a mat-tab (included from angular material). I have a button that when clicked duplicates the code with *ngFor. My textArea inside this code has an id = serReal{{j}} where j is the index of the *ngFor. My reference is not working though. When I tried to get the element in my typescript code, it gets only the first one (serReal0). For the others I receive null.

My Html code:

```<code>    <!-- here I call it to duplicate -->
    <button mat-icon-button (click)="addData()" type="button" id="add_icon"></button>

    <!-- here it is my ngfor code which will duplicate -->
    <mat-tab-group>
          <mat-tab *ngFor="let itemServ of itemsServ; let j = index;" [label] = label[j]>
                <div class="areaTexto">
                      <div class="mdc-text-field text-field mdc-text-field--fullwidth mdc-text-field--textarea">
                            <textarea class="campoTextoServReal"
id ="servReal{{j}}"></textarea>
                      </div>
                </div>
          </mat-tab>
    </mat-tab-group>

    <!-- here I call it to read the text areas by the id -->
    <div class="divEncerraRel" id = apply>
          <button mat-raised-button (click) = 'setRelatorio()'>Encerrar Relatório</button>
    </div>
</code>```

My TS code:

```<code>    itemsServ: number[] = [1];

addData() {
  this.itemsServ.push(this.itemsServ.length + 1);
  this.label.push('Titulo');
}

setRelatorio() {
  for (let i = 0; i < this.itemsServ.length; i++) {
     console.log((<HTMLInputElement>document.getElementById("servReal" + i)));
  }
}
</code>```

答:
I think part of your problem is how you are setting your id in your html. Instead of

<textarea class="campoTextoServReal" [id]="servReal{{j}}"></textarea>

try

<textarea class="campoTextoServReal" id="{{'servReal' + j }}"></textarea>

Also, the reason why you are getting nulls when you are expecting the <textarea id="servRealj"> (where j is the expected index) is because it is a tab group. When you are going between tabs, Angular removes the mat-tab-body from the DOM, so when you are searching for them by the element id, they don't exist. You can see this because when you go to tab 1 or tab 2, you will only see servReal1 and servReal2.

Title: Use of Ngfor in label mat-ltab

Body:
I am facing a problem when using *ngFor in a mat-tab (included from angular material). I have a button that when clicked duplicates the code with *ngFor. My textArea inside this code has an id = serReal{{j}} where j is the index of the *ngFor. My reference is not working though. When I tried to get the element in my typescript code, it gets only the first one (serReal0). For the others I receive null.

My Html code:

    <button mat-icon-button (click)="addData()" type="button" id="add_icon"></button>

    <mat-tab-group>
          <mat-tab *ngFor="let itemServ of itemsServ; let j = index;" [label] = label[j]>

                            <textarea class="campoTextoServReal"
id ="servReal{{j}}"></textarea>

          </mat-tab>
    </mat-tab-group>

          <button mat-raised-button (click) = 'setRelatorio()'>Encerrar Relatório</button>

My TS code:

    itemsServ: number[] = [1];

addData() {
  this.itemsServ.push(this.itemsServ.length + 1);
  this.label.push('Titulo');
}

setRelatorio() {
  for (let i = 0; i < this.itemsServ.length; i++) {
     console.log((<HTMLInputElement>document.getElementById("servReal" + i)));
  }
}


问:
Java repeated lines across methods
I have a Java class which needs to provide various methods. All these methods use the same line. Let's say that these methods have to exist because they are required by the user.

```<code>public class Restaurant {

public Drink getDrink() {
    callWaiter();
    ...
}

public Food getFood() {
    callWaiter();
    ...
}

public Bill getBill() {
    callWaiter();
}

}
</code>```

As you can see, callWaiter is repeated for all these methods. Is there a way to do this in Java without repeating the calls to callWaiter?

答:
you can use AOP to do this. </p>

<pre class="lang-java prettyprint-override"><code>@Before("execution(public * get*(..))")
private void callWaiter() {}
</code>```

Title: Java repeated lines across methods

Body:
I have a Java class which needs to provide various methods. All these methods use the same line. Let's say that these methods have to exist because they are required by the user.

public class Restaurant {

public Drink getDrink() {
    callWaiter();
    ...
}

public Food getFood() {
    callWaiter();
    ...
}

public Bill getBill() {
    callWaiter();
}

}

As you can see, callWaiter is repeated for all these methods. Is there a way to do this in Java without repeating the calls to callWaiter?

Body:
you can use AOP to do this.

@Before("execution(public * get*(..))")
private void callWaiter() {}


问:
Unexpected behaviour of 'every' gnuplot keyword inside 'do for' loop
Unexpected behaviour of 'every' inside 'do for' loop compared to its behaviour outside the loop

```<code>set style line 1 linecolor rgb "blue" pointtype 7 pointsize 3 lt 1
#plot 4 points
plot 'every_test.txt' every 1:1:0:0:3:0 u 1:2 with points linestyle 1
# Next 4 lines, when active, draw 2 arrows from 2nd point to 1st & 3rd
#pause 1
#replot '' every 1:1:1:0:1:0 u 3:4:5:6 with vectors filled head linestyle 1
#pause 1
#replot '' every 1:1:2:0:2:0 u 3:4:5:6 with vectors filled head linestyle 1
# Try 'do for' loop instead of preceding 4 line to draw the 2 arrows
do for [k = 1:2] {
   pause 1
   replot '' every 1:1:k:0:k:0 u 3:4:5:6 with vectors filled head linestyle 1
}
# Loop behaves unexpectedly.
# It draws 1st arrow, removes it [how does replot even do that!]
# then draws 2nd arrow. Final result is 2nd arrow only.
replot '' every 1:1:1:0:1:0 u 3:4:5:6 with vectors filled head linestyle 1
# Line above attempts to restore 1st arrow, but only results in error:
# "undefined variable: k" even though it makes no reference to k.
</code>```

Data in 'every_test.txt:

```<code>1  2
2  3 1.9  2.9 -0.8 -0.8
3  2 2.1  2.9  0.8 -0.8
4  1

5  3
6  4
</code>```

答:
I suspect the problem is your use of "replot".
Proposed version with a loop but no "replot"</p>

```<code>do for [k=1:N] {
    plot 'every_test.txt' every 1:1:0:0:3:0 u 1:2 with points linestyle 1, \
    '' every 1:1:1:0:k:0 u 3:4:5:6 with vectors filled head linestyle 1
    pause 1
}
</code>```

Body:
Unexpected behaviour of 'every' inside 'do for' loop compared to its behaviour outside the loop

set style line 1 linecolor rgb "blue" pointtype 7 pointsize 3 lt 1
#plot 4 points
plot 'every_test.txt' every 1:1:0:0:3:0 u 1:2 with points linestyle 1
# Next 4 lines, when active, draw 2 arrows from 2nd point to 1st & 3rd
#pause 1
#replot '' every 1:1:1:0:1:0 u 3:4:5:6 with vectors filled head linestyle 1
#pause 1
#replot '' every 1:1:2:0:2:0 u 3:4:5:6 with vectors filled head linestyle 1
# Try 'do for' loop instead of preceding 4 line to draw the 2 arrows
do for [k = 1:2] {
   pause 1
   replot '' every 1:1:k:0:k:0 u 3:4:5:6 with vectors filled head linestyle 1
}
# Loop behaves unexpectedly.
# It draws 1st arrow, removes it [how does replot even do that!]
# then draws 2nd arrow. Final result is 2nd arrow only.
replot '' every 1:1:1:0:1:0 u 3:4:5:6 with vectors filled head linestyle 1
# Line above attempts to restore 1st arrow, but only results in error:
# "undefined variable: k" even though it makes no reference to k.

Data in 'every_test.txt:

1  2
2  3 1.9  2.9 -0.8 -0.8
3  2 2.1  2.9  0.8 -0.8
4  1

5  3
6  4

Body:
I suspect the problem is your use of "replot".
Proposed version with a loop but no "replot"

do for [k=1:N] {
    plot 'every_test.txt' every 1:1:0:0:3:0 u 1:2 with points linestyle 1, \
    '' every 1:1:1:0:k:0 u 3:4:5:6 with vectors filled head linestyle 1
    pause 1
}


问:
Function returns True instead of False over iteration
I am doing some challenges on edabit and I came across a problem with my output. So i have to Create a function that takes in an array (slot machine outcome) and returns true if all elements in the array are identical, and false otherwise. The array will contain 4 elements. Can someone explain why my code isn't working the right way.

I have tried with other inputs and it's working fine.

```<code>   'use strict';

   function array_Validator(e1,e2,e3,e4)
   {
       let m_Array=[e1,e2,e3,e4];
       for(let i=0;i<m_Array.length-1;i++)
       {
           for(let x=i;x<m_Array.length-1;x++)
           {
               if(m_Array[i]!==m_Array[x+1])
               {
                   return false;
               }
               else
               return true;
           }
       }
   }

  let u_Result=array_Validator("SS","SS","Ss","Ss");

   console.log(u_Result);
</code>```

So when I input Ss it shoes true instead of false.

答:
Note the first iteration of inner loop: you have a return statement there on the if and the else blocks. So the function will always returns on the first iteration of the inner loop.

Why the result is true for your input data "SS","SS","Ss","Ss"?

Because in the first iteration, for x = i = 0, the expression m_array[i] !== m_Array[x+1] evaluates to m_array[0] !== m_Array[1] and that results in "SS" !== "SS". That last expression is then evaluated to false and the else block is taken returning the true value.

There is no need to use nested loops to check if all the elements on an array are equals. Now, you can use Array.every() for this situation.

Example:

function array_Validator(e1, e2, e3, e4)
{
    let m_Array = [e2, e3, e4];
    return m_Array.every(elem => elem === e1);
}

console.log(array_Validator("SS","SS","Ss","Ss"));
console.log(array_Validator("SS","SS","SS","Ss"));
console.log(array_Validator("SS","SS","SS","SS"));```
<pre class="snippet-code-css lang-css prettyprint-override"><code>.as-console {background-color:black !important; color:lime;}
.as-console-wrapper {max-height:100% !important; top:0;}</code>```

Body:
I am doing some challenges on edabit and I came across a problem with my output. So i have to Create a function that takes in an array (slot machine outcome) and returns true if all elements in the array are identical, and false otherwise. The array will contain 4 elements. Can someone explain why my code isn't working the right way.

I have tried with other inputs and it's working fine.

   'use strict';

   function array_Validator(e1,e2,e3,e4)
   {
       let m_Array=[e1,e2,e3,e4];
       for(let i=0;i<m_Array.length-1;i++)
       {
           for(let x=i;x<m_Array.length-1;x++)
           {
               if(m_Array[i]!==m_Array[x+1])
               {
                   return false;
               }
               else
               return true;
           }
       }
   }

  let u_Result=array_Validator("SS","SS","Ss","Ss");

   console.log(u_Result);

So when I input Ss it shoes true instead of false.

Body:
Note the first iteration of inner loop: you have a return statement there on the if and the else blocks. So the function will always returns on the first iteration of the inner loop.

Why the result is true for your input data "SS","SS","Ss","Ss"?

Because in the first iteration, for x = i = 0, the expression m_array[i] !== m_Array[x+1] evaluates to m_array[0] !== m_Array[1] and that results in "SS" !== "SS". That last expression is then evaluated to false and the else block is taken returning the true value.

There is no need to use nested loops to check if all the elements on an array are equals. Now, you can use Array.every() for this situation.

Example:

function array_Validator(e1, e2, e3, e4)
{
    let m_Array = [e2, e3, e4];
    return m_Array.every(elem => elem === e1);
}

console.log(array_Validator("SS","SS","Ss","Ss"));
console.log(array_Validator("SS","SS","SS","Ss"));
console.log(array_Validator("SS","SS","SS","SS"));

.as-console {background-color:black !important; color:lime;}
.as-console-wrapper {max-height:100% !important; top:0;}


问:
Can I change the statefulness of RNN after training?
If I build and train my RNN based model with stateful=False, can I simply do (e.g.):

```<code>model.layers[0].stateful = True
</code>```

And have it take effect as might be expected for use in prediction?  I ask because in other situations (learning rate for example), the exposed parameters are "cosmetic" and don't actually affect behavior.

答:
If your model is not stateful, model resets the state variable after each batch. You can create a stateful model but manually reset the state variable after each batch in prediction using callbacks (model.reset_states after on_batch_end callback).

Also, why do you want to change statefulness after training ?

Body:
If I build and train my RNN based model with stateful=False, can I simply do (e.g.):

model.layers[0].stateful = True

And have it take effect as might be expected for use in prediction?  I ask because in other situations (learning rate for example), the exposed parameters are "cosmetic" and don't actually affect behavior.


问:
React props change unexpectedly when a variable initialized as props changes
I created a variable and set it equal to some props. When I changed my variable, the props also changed. How do I change the variable without changing the props?

```<code>import React from 'react';
import { connect } from 'react-redux';

...

class TestApp extends React.Component {
    render() {
        var test = this.props.test;
        console.log("before change")
        console.log(test.name)
        console.log(this.props.test.name)

        // change the variable
        test.name[0] = 'pakpahan'

        console.log("after change")
        console.log(test.name)
        console.log(this.props.test.name)

        return (
            ...
        )
    }
}

...

const mapStateToProps = function (state) {
    return {
        test : {
            name : ['aldo', 'lino']
        }
    }
};

export default connect(mapStateToProps)(TestApp);
</code>```

I've already tried to use some solutions provided by others

```<code>var test = {...this.props.test};
</code>```

But the result is the same, the props still change.

I expected the variable to change while the props retain the original value. But when I change the variable, the props changes as well:

Screenshot

答:
Expanding on Shubham's answer, only primitives(int, string, bool, ...) are stored in memory. Non-primitives (array, object, function) only store pointers to memory.

So primitives act as you're expecting the variable to because they actually store the value:

```<code>let a = 1;
let b = a;
b = 2;
console.log(a); // 1
console.log(b); // 2
</code>```

While non-primitives really only store a reference:

```<code>let x = [1, 2];
let y = x;
y[0] = 5;
console.log(x); //[5,2]
</code>```

Both x and y are storing pointers to the array's position in memory. So when you change position [0] on y, x sees '5' at position [0] as well.
x -> [5,2] <-y

https://medium.com/@junshengpierre/javascript-primitive-values-object-references-361cfc1cbfb0

Shubham is (I think) creating a new space in memory with the same values. So both variables will have different pointers.

```<code>x = [1,2]; // x -> [1,2]
y = x;     // x -> [1,2] y -> [1,2]
y[0] = 5   // x -> [1,2] y -> [5,2]
</code>```

Another way to do this on an object storing primitives would be to create a new property so you can preserve the old value as well.

```<code>test = {
    name : 'aldo'
}

test2 = test;
test2.newName = 'pakpahan';

console.log(test.name); // aldo
console.log(test2.newName) // pakpahan
</code>```

However, because you have an array inside of an object you run into another reference pointer issue. You need to create a new pointer if you want to edit the new array while preserving the original.

Body:
I created a variable and set it equal to some props. When I changed my variable, the props also changed. How do I change the variable without changing the props?

import React from 'react';
import { connect } from 'react-redux';

...

class TestApp extends React.Component {
    render() {
        var test = this.props.test;
        console.log("before change")
        console.log(test.name)
        console.log(this.props.test.name)

        // change the variable
        test.name[0] = 'pakpahan'

        console.log("after change")
        console.log(test.name)
        console.log(this.props.test.name)

        return (
            ...
        )
    }
}

...

const mapStateToProps = function (state) {
    return {
        test : {
            name : ['aldo', 'lino']
        }
    }
};

export default connect(mapStateToProps)(TestApp);

I've already tried to use some solutions provided by others

var test = {...this.props.test};

But the result is the same, the props still change.

I expected the variable to change while the props retain the original value. But when I change the variable, the props changes as well:

Screenshot

Body:
Expanding on Shubham's answer, only primitives(int, string, bool, ...) are stored in memory. Non-primitives (array, object, function) only store pointers to memory.

So primitives act as you're expecting the variable to because they actually store the value:

let a = 1;
let b = a;
b = 2;
console.log(a); // 1
console.log(b); // 2

While non-primitives really only store a reference:

let x = [1, 2];
let y = x;
y[0] = 5;
console.log(x); //[5,2]

Both x and y are storing pointers to the array's position in memory. So when you change position [0] on y, x sees '5' at position [0] as well.
x -> [5,2] <-y

https://medium.com/@junshengpierre/javascript-primitive-values-object-references-361cfc1cbfb0

Shubham is (I think) creating a new space in memory with the same values. So both variables will have different pointers.

x = [1,2]; // x -> [1,2]
y = x;     // x -> [1,2] y -> [1,2]
y[0] = 5   // x -> [1,2] y -> [5,2]

Another way to do this on an object storing primitives would be to create a new property so you can preserve the old value as well.

test = {
    name : 'aldo'
}

test2 = test;
test2.newName = 'pakpahan';

console.log(test.name); // aldo
console.log(test2.newName) // pakpahan

However, because you have an array inside of an object you run into another reference pointer issue. You need to create a new pointer if you want to edit the new array while preserving the original.


问:
Is there a way to write this “if” list in a more elegant way?
Suppose I have an object POJO style and for some of its attributes I have to check if are equals to a particular value; if so I have to add this attribute to a list and throw an exception (just once if one of the attributes is equal to the particular value). Is there a better way to do it than this way?</p>

```<code>// pseudocode
List<String> list = new ArrayList<String>();
boolean haveToThrowException = false;
if (object.getAttributeA().equals(“0”) {
     list.add(object.getAttributeA());
     haveToThrowException = true;
}
if (object.getAttributeB().equals(“0”) {
    list.add(object.getAttributeB());
    haveToThrowException = true;
}
if (object.getAttributeC().equals(“0”) {
    list.add(object.getAttributeC());
    haveToThrowException = true;
}//and so on

if (haveToThrownException) {
    throw new Exception(list.toString());
}
</code>```

答:
According to the first answer, you could also do something like this:

```<code>Stream.of(object.getAttributeA(), object.getAttributeB(), object.getAttributeC())
    .filter("0"::equals)
    .reduce((var1, var2) -> var1 + var2)
    .ifPresent(string -> {
          throw new RuntimeException(string);
     });
</code>```

And it seems you don't have to have if, at all.

Body:
Suppose I have an object POJO style and for some of its attributes I have to check if are equals to a particular value; if so I have to add this attribute to a list and throw an exception (just once if one of the attributes is equal to the particular value). Is there a better way to do it than this way?

// pseudocode
List<String> list = new ArrayList<String>();
boolean haveToThrowException = false;
if (object.getAttributeA().equals(“0”) {
     list.add(object.getAttributeA());
     haveToThrowException = true;
}
if (object.getAttributeB().equals(“0”) {
    list.add(object.getAttributeB());
    haveToThrowException = true;
}
if (object.getAttributeC().equals(“0”) {
    list.add(object.getAttributeC());
    haveToThrowException = true;
}//and so on

if (haveToThrownException) {
    throw new Exception(list.toString());
}

Body:
According to the first answer, you could also do something like this:

Stream.of(object.getAttributeA(), object.getAttributeB(), object.getAttributeC())
    .filter("0"::equals)
    .reduce((var1, var2) -> var1 + var2)
    .ifPresent(string -> {
          throw new RuntimeException(string);
     });

And it seems you don't have to have if, at all.


问:
Bad return type in lambda expression: BigDecimal cannot be converted to long
I was trying to write a query in java stream, through speedment.
When I try to sum (l_extendedprice * (1 - l_discount)) in select, I get this error:

Bad return type in lambda expression: BigDecimal cannot be converted to long. Operator '-' cannot be applied to 'int', 'java.math.BigDecimal'.

My code is this:

```<code>JoinComponent joinComponent = app.getOrThrow(JoinComponent.class);
Join<Tuple6<Customer, Orders, Lineitem, Supplier, Nation, Region>> join = joinComponent
        .from(CustomerManager.IDENTIFIER)
        .innerJoinOn(Orders.O_CUSTKEY).equal(Customer.C_CUSTKEY)
        .where(Orders.O_ORDERDATE.greaterOrEqual(sqlDate))
        .where(Orders.O_ORDERDATE.lessThan(sqlDate2))
        .innerJoinOn(Lineitem.L_ORDERKEY).equal(Orders.O_ORDERDATE)
        .innerJoinOn(Supplier.S_SUPPKEY ).equal(Customer.C_NATIONKEY)
        .innerJoinOn(Nation.N_NATIONKEY).equal(Supplier.S_NATIONKEY)
        .innerJoinOn(Region.R_REGIONKEY).equal(Nation.N_REGIONKEY)
        .where(Region.R_NAME.equal("ASIA"))
        .build(Tuples::of);

Comparator<Tuple1<String>> comparator = Comparator
        .comparing((Function<Tuple1<String>, String>) Tuple1::get0)
        .thenComparing(Tuple1::get0);

Map<Tuple1<String>, LongSummaryStatistics> grouped = join.stream()
        .collect(groupingBy(t -> Tuples.of(t.get4().getNName()),
                () -> new TreeMap<>(comparator),
                summarizingLong(t->t.get2().getLDiscount()*(1-t.get2().getLDiscount()))
        ));
</code>```

How can I resolve this?

答:
So the problem is that +, -, *, /, ... are not working with BigDecimals. You have to use .add(), .subtract(), .multiply(), .divide(), ... methods for calculations.

If it is possible, you can use BigDecimal.longValue() or BigDecimal.longValueExact() to convert the BigDecimals to long values to use them in your calculation:

```<code>Map<Tuple1<String>, LongSummaryStatistics> grouped = join.stream()
        .collect(Collectors.groupingBy(Tuples::of,
                () -> new TreeMap<>(comparator),
                Collectors.summarizingLong(t -> t.get2().getLDiscount().longValue() *
                        (1 - t.get2().getLDiscount().longValue()))
        ));
</code>```

Alternatively you can du the whole calculation with BigDecimal and convert the value to long at the end:

```<code>Map<Tuple1<String>, LongSummaryStatistics> grouped = join.stream()
        .collect(Collectors.groupingBy(Tuples::of,
                () -> new TreeMap<>(comparator),
                Collectors.summarizingLong(t -> t.get2().getLDiscount()
                        .multiply(BigDecimal.ONE
                        .subtract(t.get2().getLDiscount())).longValue())
        ));
</code>```

If both solutions do not work for you you have to write an own collection for BigDecimalSummaryStatistics or just calculate the values you need directly. You can read this question for summarizing BigDecimal values using Java Stream.

Body:
I was trying to write a query in java stream, through speedment.
When I try to sum (l_extendedprice * (1 - l_discount)) in select, I get this error:

Bad return type in lambda expression: BigDecimal cannot be converted to long. Operator '-' cannot be applied to 'int', 'java.math.BigDecimal'.

My code is this:

JoinComponent joinComponent = app.getOrThrow(JoinComponent.class);
Join<Tuple6<Customer, Orders, Lineitem, Supplier, Nation, Region>> join = joinComponent
        .from(CustomerManager.IDENTIFIER)
        .innerJoinOn(Orders.O_CUSTKEY).equal(Customer.C_CUSTKEY)
        .where(Orders.O_ORDERDATE.greaterOrEqual(sqlDate))
        .where(Orders.O_ORDERDATE.lessThan(sqlDate2))
        .innerJoinOn(Lineitem.L_ORDERKEY).equal(Orders.O_ORDERDATE)
        .innerJoinOn(Supplier.S_SUPPKEY ).equal(Customer.C_NATIONKEY)
        .innerJoinOn(Nation.N_NATIONKEY).equal(Supplier.S_NATIONKEY)
        .innerJoinOn(Region.R_REGIONKEY).equal(Nation.N_REGIONKEY)
        .where(Region.R_NAME.equal("ASIA"))
        .build(Tuples::of);

Comparator<Tuple1<String>> comparator = Comparator
        .comparing((Function<Tuple1<String>, String>) Tuple1::get0)
        .thenComparing(Tuple1::get0);

Map<Tuple1<String>, LongSummaryStatistics> grouped = join.stream()
        .collect(groupingBy(t -> Tuples.of(t.get4().getNName()),
                () -> new TreeMap<>(comparator),
                summarizingLong(t->t.get2().getLDiscount()*(1-t.get2().getLDiscount()))
        ));

How can I resolve this?

Body:
So the problem is that +, -, *, /, ... are not working with BigDecimals. You have to use .add(), .subtract(), .multiply(), .divide(), ... methods for calculations.

If it is possible, you can use BigDecimal.longValue() or BigDecimal.longValueExact() to convert the BigDecimals to long values to use them in your calculation:

Map<Tuple1<String>, LongSummaryStatistics> grouped = join.stream()
        .collect(Collectors.groupingBy(Tuples::of,
                () -> new TreeMap<>(comparator),
                Collectors.summarizingLong(t -> t.get2().getLDiscount().longValue() *
                        (1 - t.get2().getLDiscount().longValue()))
        ));

Alternatively you can du the whole calculation with BigDecimal and convert the value to long at the end:

Map<Tuple1<String>, LongSummaryStatistics> grouped = join.stream()
        .collect(Collectors.groupingBy(Tuples::of,
                () -> new TreeMap<>(comparator),
                Collectors.summarizingLong(t -> t.get2().getLDiscount()
                        .multiply(BigDecimal.ONE
                        .subtract(t.get2().getLDiscount())).longValue())
        ));

If both solutions do not work for you you have to write an own collection for BigDecimalSummaryStatistics or just calculate the values you need directly. You can read this question for summarizing BigDecimal values using Java Stream.


问:
Using multiple values for iQuery method .css() throws error: 'missing )'. However its not missing
Error says I'm missing a parenthesis. However the syntax is identical to w3Schools and Mozilla. .css() works if im only setting one style, but break when setting more than 1.

jQuery

```<code> <script>
      $(document).ready( function(){

          // event-handler to create new element
          function createContainer() {

              // variables store values from <select> element
              var elementType = $('select').val();

              // string containing markup for use with .append()
              var elementTypeStr = $(` <${elementType}> This is a ${elementType} </${elementType}> `);
              // this throws error, while setting single style doesn't
              $('#layout_demo').append( elementTypeStr );
              elementTypeStr.css("background-color":" blue" , "height": "100px");

          } // end createContainer()

          // ---------------------------- event handlers ---------------------

          $('#btn').click( createContainer );
          // button to delete latest element or selected element

      }); // end ready

  </script>

</code>```

HTML:

```<code><section id="form">
  <!-- select element --->
  <label for="container"> Please select type of container to add</label>
  <select id= "container">
    <option value= "div" > &ltdiv&gt </option>
    <option value= "p"> &ltp&gt </option>
    <option value= "section"> &ltsection&gt </option>

  </select>
  <!-- Seperate container to hold now elements -->

  <button id="btn" > Create Container </button>
</section>

<div id="layout_demo">

</div>
</code>```

答:
When using .css() you have two options for passing arguments: comma separated strings or Object literal notation

therefore:

```<code>.css("background-color", " blue").css("height", "100px");
</code>```

or rather:

```<code>.css({backgroundColor:"blue" , height:"100px"});
</code>```

Also, instead of using $('select').val(); use a stricter selector like some ID; 'select' could be any select in your page.

jQuery($ => {

  const createElement = () => {

    const tagName = $('#element').val();

    return $(`<${tagName}/>`, {
      appendTo: '#layout',
      text: `This is a ${tagName}`,
      css: {
        backgroundColor: "blue",
        height: "100px"
      },
    });

  }

  $('#create').on('click', createElement);

});```
<pre class="snippet-code-html lang-html prettyprint-override"><code><section id="form">
  <label>
    Please select type of container to add
    <select id="element">
      <option value="div">&ltdiv&gt</option>
      <option value="p">&ltp&gt</option>
      <option value="section">&ltsection&gt</option>
    </select>
  </label>
  <button id="create">Create Container</button>
</section>
<div id="layout"></div>

<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script></code>```

Body:
Error says I'm missing a parenthesis. However the syntax is identical to w3Schools and Mozilla. .css() works if im only setting one style, but break when setting more than 1.

jQuery

HTML:

   Please select type of container to add
  <select id= "container">
    <option value= "div" > &ltdiv&gt </option>
    <option value= "p"> &ltp&gt </option>
    <option value= "section"> &ltsection&gt </option>

  </select>

  <button id="btn" > Create Container </button>

Body:
When using .css() you have two options for passing arguments: comma separated strings or Object literal notation

therefore:

.css("background-color", " blue").css("height", "100px");

or rather:

.css({backgroundColor:"blue" , height:"100px"});

Also, instead of using $('select').val(); use a stricter selector like some ID; 'select' could be any select in your page.

jQuery($ => {

  const createElement = () => {

    const tagName = $('#element').val();

    return $(`<${tagName}/>`, {
      appendTo: '#layout',
      text: `This is a ${tagName}`,
      css: {
        backgroundColor: "blue",
        height: "100px"
      },
    });

  }

  $('#create').on('click', createElement);

});

<section id="form">
  <label>
    Please select type of container to add
    <select id="element">
      <option value="div">&ltdiv&gt</option>
      <option value="p">&ltp&gt</option>
      <option value="section">&ltsection&gt</option>
    </select>
  </label>
  <button id="create">Create Container</button>
</section>
<div id="layout"></div>

<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>


问:
How to fix added parameter in rest call with ajax?
I'm trying to test Datatables composer with ajax Rest calls to spring RestController. I setup front in wampserver and the back with spring boot.

I'm using the spring tuto to setup the RestController https://spring.io/guides/gs/rest-service/
It works fine, I got Json file while calling the controller. I want to using that resutl and show in Datatables.

The code script.js of the font is :

```<code>    $(document).ready( function () {
        $('#table_id').DataTable( {
                  "processing": true,
                  "serverSide": false,
                  "ajax": {
                      "url": "http://localhost:8080/greeting",
                      "type": 'GET',
                      "dataType": "json",
                      "data": function (data) {
                         // console.log(data);
                         return data = JSON.stringify(data);
                      }
                      },

                  "columns": [
                                   {"data": 'id'},
                                   {"data": 'content'}
                               ]
           });
    });
</code>```

html page

```<code><!DOCTYPE html>
<html lang="en">
<head>
    <script
            src="https://code.jquery.com/jquery-3.4.1.min.js"
            integrity="sha256-CSXorXvZcTkaix6Yvo6HppcZGetbYMGWSFlBw8HfCJo="
            crossorigin="anonymous"></script>

    <link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/1.10.19/css/jquery.dataTables.css">
    <script type="text/javascript" charset="utf8" src="https://cdn.datatables.net/1.10.19/js/jquery.dataTables.js"></script>
    <script type="text/javascript" charset="utf8" src="script.js"></script>
    <meta charset="UTF-8">

    <title>Gtreetings</title>
</head>
<h3>Hi, little one </h3>
<body>

    <table id="table_id" class="display" style="width:100%">
        <thead>
        <tr>
            <th>id</th>
            <th>content</th>
        </tr>
        </thead>
        <tfoot>
        <tr>
            <th>id</th>
            <th>content</th>
        </tr>
        </tfoot>
    </table>

</body>
</html>
</code>```

I got a weird added parameter {}&=1558786054608
and Cross-Origin Request error on http://localhost:8080/greeting?{}&=1558786054608.

I'm not sure if it is a timestamp, I don't know how to explane this.

答:
First, in your JS script, you have an error in the ajax call, the data means the parameter send not from ajax. here is a correct version of the code.

```<code>     $(document).ready( function () {

        $('#table_id').DataTable({
                  processing: true,
                  serverSide: false,
                   dataType: "json",
                   ajax: {
                      url: "http://localhost:8080/greeting",
                      method: 'GET',
                      dataSrc: function (json) {

                         console.log("json",json)
                         return json;
                        },

                    },

                  columns: [
                               {data: 'id'},
                               {data: 'content'}
                           ]
           });
    });
</code>```

Second, in the backend on the controller you have to add annotation @CrossOrigin(origins = "*") thus you avoid the error cross-origin.

Finally, Datatable has to have an array in the retune result, and it is not the case in your controller. It returns an object.
I suggest to wrap your object within a list as follows :

```<code>@CrossOrigin(origins = "*")
@RestController
public class GreetingController {

    private static final String template = "Hello, %s!";
    private final AtomicLong counter = new AtomicLong();

    @RequestMapping("/greeting")
    public List<Greeting> greeting(@RequestParam(value="name", defaultValue="World") String name) {

        List<Greeting> ls = new ArrayList<Greeting>();
        ls.add(new Greeting(counter.incrementAndGet(),
                String.format(template, name)));
        return ls;
    }
}
</code>```

I hope it helps.

Body:
I'm trying to test Datatables composer with ajax Rest calls to spring RestController. I setup front in wampserver and the back with spring boot.

I'm using the spring tuto to setup the RestController https://spring.io/guides/gs/rest-service/
It works fine, I got Json file while calling the controller. I want to using that resutl and show in Datatables.

The code script.js of the font is :

    $(document).ready( function () {
        $('#table_id').DataTable( {
                  "processing": true,
                  "serverSide": false,
                  "ajax": {
                      "url": "http://localhost:8080/greeting",
                      "type": 'GET',
                      "dataType": "json",
                      "data": function (data) {
                         // console.log(data);
                         return data = JSON.stringify(data);
                      }
                      },

                  "columns": [
                                   {"data": 'id'},
                                   {"data": 'content'}
                               ]
           });
    });

html page

<!DOCTYPE html>

    <link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/1.10.19/css/jquery.dataTables.css">

    <meta charset="UTF-8">

    <title>Gtreetings</title>

Hi, little one

            id
            content

            id
            content

I got a weird added parameter {}&=1558786054608
and Cross-Origin Request error on http://localhost:8080/greeting?{}&=1558786054608.

I'm not sure if it is a timestamp, I don't know how to explane this.

Body:
First, in your JS script, you have an error in the ajax call, the data means the parameter send not from ajax. here is a correct version of the code.

     $(document).ready( function () {

        $('#table_id').DataTable({
                  processing: true,
                  serverSide: false,
                   dataType: "json",
                   ajax: {
                      url: "http://localhost:8080/greeting",
                      method: 'GET',
                      dataSrc: function (json) {

                         console.log("json",json)
                         return json;
                        },

                    },

                  columns: [
                               {data: 'id'},
                               {data: 'content'}
                           ]
           });
    });

Second, in the backend on the controller you have to add annotation @CrossOrigin(origins = "*") thus you avoid the error cross-origin.

Finally, Datatable has to have an array in the retune result, and it is not the case in your controller. It returns an object.
I suggest to wrap your object within a list as follows :

@CrossOrigin(origins = "*")
@RestController
public class GreetingController {

    private static final String template = "Hello, %s!";
    private final AtomicLong counter = new AtomicLong();

    @RequestMapping("/greeting")
    public List<Greeting> greeting(@RequestParam(value="name", defaultValue="World") String name) {

        List<Greeting> ls = new ArrayList<Greeting>();
        ls.add(new Greeting(counter.incrementAndGet(),
                String.format(template, name)));
        return ls;
    }
}

I hope it helps.


问:
jQuery autocomplete adding additional data with autocompleted data
I have an input field (text) that I want to perform autocomplete on with multiple names. It acts like the To: field in Gmail, with the user typing in a name and hitting enter and continuing. The problem is that I have a list of names and that's what gets added to the input field and once you submit, as far as I know, only the field string gets sent to the server. The problem is when there are two people with the same name, so: John Smith, Bob, John Smith might be inputted and that string will get sent to the server.

On the backend, all users have a unique id. Is there any way I can have the users' names be the autocomplete source but have their ids sent to the server?

I'm using jQuery 1.5.1 and this plugin: jQuery UI Autocomplete

答:
The standard search will offer up everything you need. You just need to use a $.map() function to get the data you need out from the server response. Example:

```<code>...
success: function (data) {
   response($.map(data.d, function (item) {
      return {
         id: item.ContactId,
         value: item.LastName + ", " + item.FirstName
      }
   }))
}
...
</code>```

Then just store the selected items (including the ids and names) in a $().data() variable on the input. Example:

```<code>...
select: function (event, ui) {
   $(this).data("Selected", item);
}
...
</code>```

Now when you're ready to post your value back to the server, you can simply send the id of the selected entry like so:

```<code>var idToSend = $("#autoComplete").data("Selected").id;
</code>```

I hope that helps. Good luck!

* * EDIT * *

Sorry, the $.map() was a split thought. The data.d in my example is .NET specific, but there are two properties that are native to the Autocomplete plugin (.value and .label).

.value will be what is shown and stored if .label does not exist; otherwise, .value will be treated as the value stored in the input post-selection, and .label will be what is displayed in the dropdown.

Body:
The standard search will offer up everything you need. You just need to use a $.map() function to get the data you need out from the server response. Example:

...
success: function (data) {
   response($.map(data.d, function (item) {
      return {
         id: item.ContactId,
         value: item.LastName + ", " + item.FirstName
      }
   }))
}
...

Then just store the selected items (including the ids and names) in a $().data() variable on the input. Example:

...
select: function (event, ui) {
   $(this).data("Selected", item);
}
...

Now when you're ready to post your value back to the server, you can simply send the id of the selected entry like so:

var idToSend = $("#autoComplete").data("Selected").id;

I hope that helps. Good luck!

* * EDIT * *

Sorry, the $.map() was a split thought. The data.d in my example is .NET specific, but there are two properties that are native to the Autocomplete plugin (.value and .label).

.value will be what is shown and stored if .label does not exist; otherwise, .value will be treated as the value stored in the input post-selection, and .label will be what is displayed in the dropdown.


问:
Viewing a Kafka topic as a Flink Table in Flink SQL CLI client?
I am trying to make a kafka topic to apear as a table in Flink SQL CLI client.

Here is how I call the sql-client.sh:

```<code>./bin/sql-client.sh embedded -l /Users/Behzad.Pirvali/Tools/Data/flink-1.7.2/lib  -d ./conf/sql-client-config-1.yaml
</code>```

I am getting the following exception:

```<code>Caused by: org.apache.flink.table.api.NoMatchingTableFactoryException: Could not find a suitable table factory for 'org.apache.flink.table.factories.StreamTableSourceFactory' in
the classpath.
</code>```

Here is the full stack-trace:

```<code>USHOLBPI1-ML:flink Behzad.Pirvali$ ./bin/sql-client.sh embedded -l /Users/Behzad.Pirvali/Tools/Data/flink-1.7.2/lib  -d ./conf/sql-client-config-1.yaml
Reading default environment from: file:/Users/Behzad.Pirvali/Tools/Data/flink-1.7.2/./conf/sql-client-config-1.yaml
No session environment specified.
Validating current environment...

Exception in thread "main" org.apache.flink.table.client.SqlClientException: The configured environment is invalid. Please check your environment files again.
    at org.apache.flink.table.client.SqlClient.validateEnvironment(SqlClient.java:140)
    at org.apache.flink.table.client.SqlClient.start(SqlClient.java:99)
    at org.apache.flink.table.client.SqlClient.main(SqlClient.java:187)
Caused by: org.apache.flink.table.client.gateway.SqlExecutionException: Could not create execution context.
    at org.apache.flink.table.client.gateway.local.LocalExecutor.getOrCreateExecutionContext(LocalExecutor.java:488)
    at org.apache.flink.table.client.gateway.local.LocalExecutor.validateSession(LocalExecutor.java:316)
    at org.apache.flink.table.client.SqlClient.validateEnvironment(SqlClient.java:137)
    ... 2 more
Caused by: org.apache.flink.table.api.NoMatchingTableFactoryException: Could not find a suitable table factory for 'org.apache.flink.table.factories.StreamTableSourceFactory' in
the classpath.

Reason: No context matches.

The following properties are requested:
connector.properties.0.key=zookeeper.connect
connector.properties.0.value=localhost:2181
connector.properties.1.key=bootstrap.servers
connector.properties.1.value=localhost:9092
connector.properties.2.key=group.id
connector.properties.2.value=group-scanned-tickets
connector.property-version=1
connector.startup-mode=earliest-offset
connector.topic=scanned-tickets
connector.type=kafka
connector.version=2.2.0
format.property-version=1
format.schema=ROW(venueId LONG, eventName STRING, ticketId LONG, eventStartTime TIMESTAMP, eventTime TIMESTAMP)
format.type=json
schema.0.name=venueId
schema.0.type=LONG
schema.1.name=eventName
schema.1.type=STRING
schema.2.name=ticketId
schema.2.type=LONG
schema.3.name=eventStartTime
schema.3.type=TIMESTAMP
schema.4.name=scannedTime
schema.4.rowtime.timestamps.from=eventTime
schema.4.rowtime.timestamps.type=from-field
schema.4.rowtime.watermarks.delay=60000
schema.4.rowtime.watermarks.type=periodic-bounded
schema.4.type=TIMESTAMP
update-mode=append

The following factories have been considered:
org.apache.flink.streaming.connectors.kafka.Kafka011TableSourceSinkFactory
org.apache.flink.formats.json.JsonRowFormatFactory
org.apache.flink.table.sources.CsvBatchTableSourceFactory
org.apache.flink.table.sources.CsvAppendTableSourceFactory
org.apache.flink.table.sinks.CsvBatchTableSinkFactory
org.apache.flink.table.sinks.CsvAppendTableSinkFactory

    at org.apache.flink.table.factories.TableFactoryService$.filterByContext(TableFactoryService.scala:218)
    at org.apache.flink.table.factories.TableFactoryService$.findInternal(TableFactoryService.scala:134)
    at org.apache.flink.table.factories.TableFactoryService$.find(TableFactoryService.scala:100)
    at org.apache.flink.table.factories.TableFactoryService.find(TableFactoryService.scala)
    at org.apache.flink.table.client.gateway.local.ExecutionContext.createTableSource(ExecutionContext.java:236)
    at org.apache.flink.table.client.gateway.local.ExecutionContext.lambda$new$0(ExecutionContext.java:121)
    at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
    at org.apache.flink.table.client.gateway.local.ExecutionContext.<init>(ExecutionContext.java:119)
    at org.apache.flink.table.client.gateway.local.LocalExecutor.getOrCreateExecutionContext(LocalExecutor.java:484)
    ... 4 more
</code>```

Here is the sql-config.yaml:

```<code>tables:
  - name: scanned_tickets
    type: source
    update-mode: append
    schema:
    - name: venueId
      type: LONG
    - name: eventName
      type: STRING
    - name: ticketId
      type: LONG
    - name: eventStartTime
      type: TIMESTAMP
    - name: scannedTime
      type: TIMESTAMP
      rowtime:
        timestamps:
          type: "from-field"
          from: "eventTime"
        watermarks:
          type: "periodic-bounded"
          delay: "60000"
    connector:
      property-version: 1
      type: kafka
      version: 2.2.0
      topic: scanned-tickets
      startup-mode: earliest-offset
      properties:
      - key: zookeeper.connect
        value: localhost:2181
      - key: bootstrap.servers
        value: localhost:9092
      - key: group.id
        value: group-scanned-tickets
    format:
      property-version: 1
      type: json
      schema: "ROW(venueId LONG, eventName STRING, ticketId LONG, eventStartTime TIMESTAMP, eventTime TIMESTAMP)"

#==============================================================================
# Execution properties
#==============================================================================

# Execution properties allow for changing the behavior of a table program.

execution:
  type: streaming              # 'batch' or 'streaming' execution
  result-mode: table           # 'changelog' or 'table' presentation of results
  parallelism: 1               # parallelism of the program
  max-parallelism: 128         # maximum parallelism
  min-idle-state-retention: 0  # minimum idle state retention in ms
  max-idle-state-retention: 0  # maximum idle state retention in ms

#==============================================================================
# Deployment properties
#==============================================================================

# Deployment properties allow for describing the cluster to which table
# programs are submitted to.

deployment:
  type: standalone             # only the 'standalone' deployment is supported
  response-timeout: 5000       # general cluster communication timeout in ms
  gateway-address: ""          # (optional) address from cluster to gateway
  gateway-port: 0              # (optional) port from cluster to gateway

</code>```

Flink version is 1.7.2 and here is the lib directory content:

```<code>-rw-r--r--   1 Behzad.Pirvali  staff     28524 May 30 10:25 flink-json-1.7.2.jar
drwxr-xr-x@  8 Behzad.Pirvali  staff       256 May 30 10:25 .
-rw-r--r--   1 Behzad.Pirvali  staff   1739194 May 30 10:25 flink-connector-kafka-0.11_2.12-1.7.2-sql-jar.jar
drwxr-xr-x@ 12 Behzad.Pirvali  staff       384 Feb 11 06:50 ..
-rw-r--r--@  1 Behzad.Pirvali  staff  84497196 Feb 11 06:50 flink-dist_2.12-1.7.2.jar
-rw-r--r--@  1 Behzad.Pirvali  staff    141942 Feb 11 06:49 flink-python_2.12-1.7.2.jar
-rw-r--r--@  1 Behzad.Pirvali  staff    489884 Feb 11 06:32 log4j-1.2.17.jar
-rw-r--r--@  1 Behzad.Pirvali  staff      9931 Feb 11 06:32 slf4j-log4j12-1.7.15.jar
</code>```

So, it looks like I am missing a jar file, but I cannot quite figure out which one?

答:
I was able to find out the root cause of the problem.
It was this configuration mismatch in the yaml file:

```<code>connector:
      property-version: 1
      type: kafka
      version: 2.2.0
</code>```

I was setting the broker version, but the version has to be 0.11 to match the fink-kafka-connector: flink-connector-kafka-0.11_2.12-1.7.2-sql-jar

So, changing the configuration to this worked:

```<code>connector:
      property-version: 1
      type: kafka
      version: 0.11
</code>```

Body:
I am trying to make a kafka topic to apear as a table in Flink SQL CLI client.

Here is how I call the sql-client.sh:

./bin/sql-client.sh embedded -l /Users/Behzad.Pirvali/Tools/Data/flink-1.7.2/lib  -d ./conf/sql-client-config-1.yaml

I am getting the following exception:

Caused by: org.apache.flink.table.api.NoMatchingTableFactoryException: Could not find a suitable table factory for 'org.apache.flink.table.factories.StreamTableSourceFactory' in
the classpath.

Here is the full stack-trace:

USHOLBPI1-ML:flink Behzad.Pirvali$ ./bin/sql-client.sh embedded -l /Users/Behzad.Pirvali/Tools/Data/flink-1.7.2/lib  -d ./conf/sql-client-config-1.yaml
Reading default environment from: file:/Users/Behzad.Pirvali/Tools/Data/flink-1.7.2/./conf/sql-client-config-1.yaml
No session environment specified.
Validating current environment...

Exception in thread "main" org.apache.flink.table.client.SqlClientException: The configured environment is invalid. Please check your environment files again.
    at org.apache.flink.table.client.SqlClient.validateEnvironment(SqlClient.java:140)
    at org.apache.flink.table.client.SqlClient.start(SqlClient.java:99)
    at org.apache.flink.table.client.SqlClient.main(SqlClient.java:187)
Caused by: org.apache.flink.table.client.gateway.SqlExecutionException: Could not create execution context.
    at org.apache.flink.table.client.gateway.local.LocalExecutor.getOrCreateExecutionContext(LocalExecutor.java:488)
    at org.apache.flink.table.client.gateway.local.LocalExecutor.validateSession(LocalExecutor.java:316)
    at org.apache.flink.table.client.SqlClient.validateEnvironment(SqlClient.java:137)
    ... 2 more
Caused by: org.apache.flink.table.api.NoMatchingTableFactoryException: Could not find a suitable table factory for 'org.apache.flink.table.factories.StreamTableSourceFactory' in
the classpath.

Reason: No context matches.

The following properties are requested:
connector.properties.0.key=zookeeper.connect
connector.properties.0.value=localhost:2181
connector.properties.1.key=bootstrap.servers
connector.properties.1.value=localhost:9092
connector.properties.2.key=group.id
connector.properties.2.value=group-scanned-tickets
connector.property-version=1
connector.startup-mode=earliest-offset
connector.topic=scanned-tickets
connector.type=kafka
connector.version=2.2.0
format.property-version=1
format.schema=ROW(venueId LONG, eventName STRING, ticketId LONG, eventStartTime TIMESTAMP, eventTime TIMESTAMP)
format.type=json
schema.0.name=venueId
schema.0.type=LONG
schema.1.name=eventName
schema.1.type=STRING
schema.2.name=ticketId
schema.2.type=LONG
schema.3.name=eventStartTime
schema.3.type=TIMESTAMP
schema.4.name=scannedTime
schema.4.rowtime.timestamps.from=eventTime
schema.4.rowtime.timestamps.type=from-field
schema.4.rowtime.watermarks.delay=60000
schema.4.rowtime.watermarks.type=periodic-bounded
schema.4.type=TIMESTAMP
update-mode=append

The following factories have been considered:
org.apache.flink.streaming.connectors.kafka.Kafka011TableSourceSinkFactory
org.apache.flink.formats.json.JsonRowFormatFactory
org.apache.flink.table.sources.CsvBatchTableSourceFactory
org.apache.flink.table.sources.CsvAppendTableSourceFactory
org.apache.flink.table.sinks.CsvBatchTableSinkFactory
org.apache.flink.table.sinks.CsvAppendTableSinkFactory

    at org.apache.flink.table.factories.TableFactoryService$.filterByContext(TableFactoryService.scala:218)
    at org.apache.flink.table.factories.TableFactoryService$.findInternal(TableFactoryService.scala:134)
    at org.apache.flink.table.factories.TableFactoryService$.find(TableFactoryService.scala:100)
    at org.apache.flink.table.factories.TableFactoryService.find(TableFactoryService.scala)
    at org.apache.flink.table.client.gateway.local.ExecutionContext.createTableSource(ExecutionContext.java:236)
    at org.apache.flink.table.client.gateway.local.ExecutionContext.lambda$new$0(ExecutionContext.java:121)
    at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
    at org.apache.flink.table.client.gateway.local.ExecutionContext.<init>(ExecutionContext.java:119)
    at org.apache.flink.table.client.gateway.local.LocalExecutor.getOrCreateExecutionContext(LocalExecutor.java:484)
    ... 4 more

Here is the sql-config.yaml:

tables:
  - name: scanned_tickets
    type: source
    update-mode: append
    schema:
    - name: venueId
      type: LONG
    - name: eventName
      type: STRING
    - name: ticketId
      type: LONG
    - name: eventStartTime
      type: TIMESTAMP
    - name: scannedTime
      type: TIMESTAMP
      rowtime:
        timestamps:
          type: "from-field"
          from: "eventTime"
        watermarks:
          type: "periodic-bounded"
          delay: "60000"
    connector:
      property-version: 1
      type: kafka
      version: 2.2.0
      topic: scanned-tickets
      startup-mode: earliest-offset
      properties:
      - key: zookeeper.connect
        value: localhost:2181
      - key: bootstrap.servers
        value: localhost:9092
      - key: group.id
        value: group-scanned-tickets
    format:
      property-version: 1
      type: json
      schema: "ROW(venueId LONG, eventName STRING, ticketId LONG, eventStartTime TIMESTAMP, eventTime TIMESTAMP)"

#==============================================================================
# Execution properties
#==============================================================================

# Execution properties allow for changing the behavior of a table program.

execution:
  type: streaming              # 'batch' or 'streaming' execution
  result-mode: table           # 'changelog' or 'table' presentation of results
  parallelism: 1               # parallelism of the program
  max-parallelism: 128         # maximum parallelism
  min-idle-state-retention: 0  # minimum idle state retention in ms
  max-idle-state-retention: 0  # maximum idle state retention in ms

#==============================================================================
# Deployment properties
#==============================================================================

# Deployment properties allow for describing the cluster to which table
# programs are submitted to.

deployment:
  type: standalone             # only the 'standalone' deployment is supported
  response-timeout: 5000       # general cluster communication timeout in ms
  gateway-address: ""          # (optional) address from cluster to gateway
  gateway-port: 0              # (optional) port from cluster to gateway

Flink version is 1.7.2 and here is the lib directory content:

-rw-r--r--   1 Behzad.Pirvali  staff     28524 May 30 10:25 flink-json-1.7.2.jar
drwxr-xr-x@  8 Behzad.Pirvali  staff       256 May 30 10:25 .
-rw-r--r--   1 Behzad.Pirvali  staff   1739194 May 30 10:25 flink-connector-kafka-0.11_2.12-1.7.2-sql-jar.jar
drwxr-xr-x@ 12 Behzad.Pirvali  staff       384 Feb 11 06:50 ..
-rw-r--r--@  1 Behzad.Pirvali  staff  84497196 Feb 11 06:50 flink-dist_2.12-1.7.2.jar
-rw-r--r--@  1 Behzad.Pirvali  staff    141942 Feb 11 06:49 flink-python_2.12-1.7.2.jar
-rw-r--r--@  1 Behzad.Pirvali  staff    489884 Feb 11 06:32 log4j-1.2.17.jar
-rw-r--r--@  1 Behzad.Pirvali  staff      9931 Feb 11 06:32 slf4j-log4j12-1.7.15.jar

So, it looks like I am missing a jar file, but I cannot quite figure out which one?

Body:
I was able to find out the root cause of the problem.
It was this configuration mismatch in the yaml file:

connector:
      property-version: 1
      type: kafka
      version: 2.2.0

I was setting the broker version, but the version has to be 0.11 to match the fink-kafka-connector: flink-connector-kafka-0.11_2.12-1.7.2-sql-jar

So, changing the configuration to this worked:

connector:
      property-version: 1
      type: kafka
      version: 0.11