问:
How to address an illegal port number in PySpark?
Using: OSX 10.11.2, SPARK version 1.5.2, Python version 2.7.10 , iPython 4.0.1

Opening SPARK from terminal in an iPython Notebook, I use this command:

```<code>IPYTHON=1 $SPARK_HOME/bin/pyspark
</code>```

My goal is to parallelize a vector of integers, then apply the count, first, and take functions on that vector.

My workflow is the following (after spark is already opened within an IPython notebook):

```<code>rdd=sc.parallelize([1,2 ,3])
rdd.collect()
</code>```

Then when I attempt to run the count function like so,

```<code>rdd.count()
</code>```

I receive the following error:
(Please note, I realize the source of the error is the illegal port number, however I could use help in solving this error by changing the port number used. My suspicion is that the error source originates from the interaction between iPython and Spark, but looking around, it seems like the developers already addressed this error.)

```<code>In [3]: rdd.count()
15/12/21 14:27:39 INFO SparkContext: Starting job: count at <ipython-input-3-a0443394e570>:1
15/12/21 14:27:39 INFO DAGScheduler: Got job 1 (count at <ipython-input-3-a0443394e570>:1) with 4 output partitions
15/12/21 14:27:39 INFO DAGScheduler: Final stage: ResultStage 1(count at <ipython-input-3-a0443394e570>:1)
15/12/21 14:27:39 INFO DAGScheduler: Parents of final stage: List()
15/12/21 14:27:39 INFO DAGScheduler: Missing parents: List()
15/12/21 14:27:39 INFO DAGScheduler: Submitting ResultStage 1 (PythonRDD[1] at count at <ipython-input-3-a0443394e570>:1), which has no missing parents
15/12/21 14:27:39 INFO MemoryStore: ensureFreeSpace(4200) called with curMem=2001, maxMem=555755765
15/12/21 14:27:39 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.1 KB, free 530.0 MB)
15/12/21 14:27:39 INFO MemoryStore: ensureFreeSpace(2713) called with curMem=6201, maxMem=555755765
15/12/21 14:27:39 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.6 KB, free 530.0 MB)
15/12/21 14:27:39 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:64049 (size: 2.6 KB, free: 530.0 MB)
15/12/21 14:27:39 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:861
15/12/21 14:27:39 INFO DAGScheduler: Submitting 4 missing tasks from ResultStage 1 (PythonRDD[1] at count at <ipython-input-3-a0443394e570>:1)
15/12/21 14:27:39 INFO TaskSchedulerImpl: Adding task set 1.0 with 4 tasks
15/12/21 14:27:39 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 4, localhost, PROCESS_LOCAL, 2071 bytes)
15/12/21 14:27:39 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 5, localhost, PROCESS_LOCAL, 2090 bytes)
15/12/21 14:27:39 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 6, localhost, PROCESS_LOCAL, 2090 bytes)
15/12/21 14:27:39 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 7, localhost, PROCESS_LOCAL, 2090 bytes)
15/12/21 14:27:39 INFO Executor: Running task 0.0 in stage 1.0 (TID 4)
15/12/21 14:27:39 INFO Executor: Running task 1.0 in stage 1.0 (TID 5)
15/12/21 14:27:39 INFO Executor: Running task 2.0 in stage 1.0 (TID 6)
15/12/21 14:27:39 INFO Executor: Running task 3.0 in stage 1.0 (TID 7)
15/12/21 14:27:39 ERROR Executor: Exception in task 2.0 in stage 1.0 (TID 6)
java.lang.IllegalArgumentException: port out of range:1315905645
    at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143)
    at java.net.InetSocketAddress.<init>(InetSocketAddress.java:188)
    at java.net.Socket.<init>(Socket.java:244)
    at org.apache.spark.api.python.PythonWorkerFactory.createSocket$1(PythonWorkerFactory.scala:75)
    at org.apache.spark.api.python.PythonWorkerFactory.liftedTree1$1(PythonWorkerFactory.scala:90)
    at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:89)
    at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:62)
    at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:135)
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:101)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
15/12/21 14:27:39 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 4)
java.lang.IllegalArgumentException: port out of range:1315905645
    at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143)
    at java.net.InetSocketAddress.<init>(InetSocketAddress.java:188)
    at java.net.Socket.<init>(Socket.java:244)
    at org.apache.spark.api.python.PythonWorkerFactory.createSocket$1(PythonWorkerFactory.scala:75)
    at org.apache.spark.api.python.PythonWorkerFactory.liftedTree1$1(PythonWorkerFactory.scala:90)
    at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:89)
    at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:62)
    at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:135)
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:101)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
15/12/21 14:27:39 ERROR Executor: Exception in task 3.0 in stage 1.0 (TID 7)
java.lang.IllegalArgumentException: port out of range:1315905645
    at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143)
    at java.net.InetSocketAddress.<init>(InetSocketAddress.java:188)
    at java.net.Socket.<init>(Socket.java:244)
    at org.apache.spark.api.python.PythonWorkerFactory.createSocket$1(PythonWorkerFactory.scala:75)
    at org.apache.spark.api.python.PythonWorkerFactory.liftedTree1$1(PythonWorkerFactory.scala:90)
    at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:89)
    at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:62)
    at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:135)
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:101)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
15/12/21 14:27:39 ERROR Executor: Exception in task 1.0 in stage 1.0 (TID 5)
java.lang.IllegalArgumentException: port out of range:1315905645
    at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143)
    at java.net.InetSocketAddress.<init>(InetSocketAddress.java:188)
    at java.net.Socket.<init>(Socket.java:244)
    at org.apache.spark.api.python.PythonWorkerFactory.createSocket$1(PythonWorkerFactory.scala:75)
    at org.apache.spark.api.python.PythonWorkerFactory.liftedTree1$1(PythonWorkerFactory.scala:90)
    at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:89)
    at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:62)
    at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:135)
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:101)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
odule named dateutil.tz
?615/12/21 14:27:39 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 4, localhost): java.lang.IllegalArgumentException: port out of range:1315905645
    at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143)
    at java.net.InetSocketAddress.<init>(InetSocketAddress.java:188)
    at java.net.Socket.<init>(Socket.java:244)
    at org.apache.spark.api.python.PythonWorkerFactory.createSocket$1(PythonWorkerFactory.scala:75)
    at org.apache.spark.api.python.PythonWorkerFactory.liftedTree1$1(PythonWorkerFactory.scala:90)
    at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:89)
    at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:62)
    at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:135)
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:101)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

15/12/21 14:27:39 ERROR TaskSetManager: Task 0 in stage 1.0 failed 1 times; aborting job
15/12/21 14:27:39 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
15/12/21 14:27:39 INFO TaskSetManager: Lost task 2.0 in stage 1.0 (TID 6) on executor localhost: java.lang.IllegalArgumentException (port out of range:1315905645) [duplicate 1]
15/12/21 14:27:39 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
15/12/21 14:27:39 INFO TaskSetManager: Lost task 3.0 in stage 1.0 (TID 7) on executor localhost: java.lang.IllegalArgumentException (port out of range:1315905645) [duplicate 2]
15/12/21 14:27:39 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
15/12/21 14:27:39 INFO TaskSetManager: Lost task 1.0 in stage 1.0 (TID 5) on executor localhost: java.lang.IllegalArgumentException (port out of range:1315905645) [duplicate 3]
15/12/21 14:27:39 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
15/12/21 14:27:39 INFO TaskSchedulerImpl: Cancelling stage 1
15/12/21 14:27:39 INFO DAGScheduler: ResultStage 1 (count at <ipython-input-3-a0443394e570>:1) failed in 0.537 s
15/12/21 14:27:39 INFO DAGScheduler: Job 1 failed: count at <ipython-input-3-a0443394e570>:1, took 0.564707 s
---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
<ipython-input-3-a0443394e570> in <module>()
----> 1 rdd.count()

/Users/jason/spark-1.5.2-bin-hadoop2.6/python/pyspark/rdd.pyc in count(self)
   1004         3
   1005         """
-> 1006         return self.mapPartitions(lambda i: [sum(1 for _ in i)]).sum()
   1007
   1008     def stats(self):

/Users/jason/spark-1.5.2-bin-hadoop2.6/python/pyspark/rdd.pyc in sum(self)
    995         6.0
    996         """
--> 997         return self.mapPartitions(lambda x: [sum(x)]).fold(0, operator.add)
    998
    999     def count(self):

/Users/jason/spark-1.5.2-bin-hadoop2.6/python/pyspark/rdd.pyc in fold(self, zeroValue, op)
    869         # zeroValue provided to each partition is unique from the one provided
    870         # to the final reduce call
--> 871         vals = self.mapPartitions(func).collect()
    872         return reduce(op, vals, zeroValue)
    873

/Users/jason/spark-1.5.2-bin-hadoop2.6/python/pyspark/rdd.pyc in collect(self)
    771         """
    772         with SCCallSiteSync(self.context) as css:
--> 773             port = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
    774         return list(_load_from_socket(port, self._jrdd_deserializer))
    775

/Users/jason/spark-1.5.2-bin-hadoop2.6/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py in __call__(self, *args)
    536         answer = self.gateway_client.send_command(command)
    537         return_value = get_return_value(answer, self.gateway_client,
--> 538                 self.target_id, self.name)
    539
    540         for temp_arg in temp_args:

/Users/jason/spark-1.5.2-bin-hadoop2.6/python/pyspark/sql/utils.pyc in deco(*a, **kw)
     34     def deco(*a, **kw):
     35         try:
---> 36             return f(*a, **kw)
     37         except py4j.protocol.Py4JJavaError as e:
     38             s = e.java_exception.toString()

/Users/jason/spark-1.5.2-bin-hadoop2.6/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    298                 raise Py4JJavaError(
    299                     'An error occurred while calling {0}{1}{2}.\n'.
--> 300                     format(target_id, '.', name), value)
    301             else:
    302                 raise Py4JError(

Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 4, localhost): java.lang.IllegalArgumentException: port out of range:1315905645
    at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143)
    at java.net.InetSocketAddress.<init>(InetSocketAddress.java:188)
    at java.net.Socket.<init>(Socket.java:244)
    at org.apache.spark.api.python.PythonWorkerFactory.createSocket$1(PythonWorkerFactory.scala:75)
    at org.apache.spark.api.python.PythonWorkerFactory.liftedTree1$1(PythonWorkerFactory.scala:90)
    at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:89)
    at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:62)
    at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:135)
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:101)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
    at scala.Option.foreach(Option.scala:236)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1837)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1850)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1921)
    at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:909)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
    at org.apache.spark.rdd.RDD.collect(RDD.scala:908)
    at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:405)
    at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
    at py4j.Gateway.invoke(Gateway.java:259)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:207)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: port out of range:1315905645
    at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143)
    at java.net.InetSocketAddress.<init>(InetSocketAddress.java:188)
    at java.net.Socket.<init>(Socket.java:244)
    at org.apache.spark.api.python.PythonWorkerFactory.createSocket$1(PythonWorkerFactory.scala:75)
    at org.apache.spark.api.python.PythonWorkerFactory.liftedTree1$1(PythonWorkerFactory.scala:90)
    at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:89)
    at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:62)
    at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:135)
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:101)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    ... 1 more
</code>```

Any ideas?

答: pyspark read some bytes at the bengining of python std to get worker port value when star daemon.py, maybe you have some configuraion added some outputs to std. </p>

Body:
Using: OSX 10.11.2, SPARK version 1.5.2, Python version 2.7.10 , iPython 4.0.1

Opening SPARK from terminal in an iPython Notebook, I use this command:

IPYTHON=1 $SPARK_HOME/bin/pyspark

My goal is to parallelize a vector of integers, then apply the count, first, and take functions on that vector.

My workflow is the following (after spark is already opened within an IPython notebook):

rdd=sc.parallelize([1,2 ,3])
rdd.collect()

Then when I attempt to run the count function like so,

rdd.count()

I receive the following error:
(Please note, I realize the source of the error is the illegal port number, however I could use help in solving this error by changing the port number used. My suspicion is that the error source originates from the interaction between iPython and Spark, but looking around, it seems like the developers already addressed this error.)

In [3]: rdd.count()
15/12/21 14:27:39 INFO SparkContext: Starting job: count at <ipython-input-3-a0443394e570>:1
15/12/21 14:27:39 INFO DAGScheduler: Got job 1 (count at <ipython-input-3-a0443394e570>:1) with 4 output partitions
15/12/21 14:27:39 INFO DAGScheduler: Final stage: ResultStage 1(count at <ipython-input-3-a0443394e570>:1)
15/12/21 14:27:39 INFO DAGScheduler: Parents of final stage: List()
15/12/21 14:27:39 INFO DAGScheduler: Missing parents: List()
15/12/21 14:27:39 INFO DAGScheduler: Submitting ResultStage 1 (PythonRDD[1] at count at <ipython-input-3-a0443394e570>:1), which has no missing parents
15/12/21 14:27:39 INFO MemoryStore: ensureFreeSpace(4200) called with curMem=2001, maxMem=555755765
15/12/21 14:27:39 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.1 KB, free 530.0 MB)
15/12/21 14:27:39 INFO MemoryStore: ensureFreeSpace(2713) called with curMem=6201, maxMem=555755765
15/12/21 14:27:39 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.6 KB, free 530.0 MB)
15/12/21 14:27:39 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:64049 (size: 2.6 KB, free: 530.0 MB)
15/12/21 14:27:39 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:861
15/12/21 14:27:39 INFO DAGScheduler: Submitting 4 missing tasks from ResultStage 1 (PythonRDD[1] at count at <ipython-input-3-a0443394e570>:1)
15/12/21 14:27:39 INFO TaskSchedulerImpl: Adding task set 1.0 with 4 tasks
15/12/21 14:27:39 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 4, localhost, PROCESS_LOCAL, 2071 bytes)
15/12/21 14:27:39 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 5, localhost, PROCESS_LOCAL, 2090 bytes)
15/12/21 14:27:39 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 6, localhost, PROCESS_LOCAL, 2090 bytes)
15/12/21 14:27:39 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 7, localhost, PROCESS_LOCAL, 2090 bytes)
15/12/21 14:27:39 INFO Executor: Running task 0.0 in stage 1.0 (TID 4)
15/12/21 14:27:39 INFO Executor: Running task 1.0 in stage 1.0 (TID 5)
15/12/21 14:27:39 INFO Executor: Running task 2.0 in stage 1.0 (TID 6)
15/12/21 14:27:39 INFO Executor: Running task 3.0 in stage 1.0 (TID 7)
15/12/21 14:27:39 ERROR Executor: Exception in task 2.0 in stage 1.0 (TID 6)
java.lang.IllegalArgumentException: port out of range:1315905645
    at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143)
    at java.net.InetSocketAddress.<init>(InetSocketAddress.java:188)
    at java.net.Socket.<init>(Socket.java:244)
    at org.apache.spark.api.python.PythonWorkerFactory.createSocket$1(PythonWorkerFactory.scala:75)
    at org.apache.spark.api.python.PythonWorkerFactory.liftedTree1$1(PythonWorkerFactory.scala:90)
    at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:89)
    at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:62)
    at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:135)
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:101)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
15/12/21 14:27:39 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 4)
java.lang.IllegalArgumentException: port out of range:1315905645
    at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143)
    at java.net.InetSocketAddress.<init>(InetSocketAddress.java:188)
    at java.net.Socket.<init>(Socket.java:244)
    at org.apache.spark.api.python.PythonWorkerFactory.createSocket$1(PythonWorkerFactory.scala:75)
    at org.apache.spark.api.python.PythonWorkerFactory.liftedTree1$1(PythonWorkerFactory.scala:90)
    at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:89)
    at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:62)
    at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:135)
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:101)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
15/12/21 14:27:39 ERROR Executor: Exception in task 3.0 in stage 1.0 (TID 7)
java.lang.IllegalArgumentException: port out of range:1315905645
    at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143)
    at java.net.InetSocketAddress.<init>(InetSocketAddress.java:188)
    at java.net.Socket.<init>(Socket.java:244)
    at org.apache.spark.api.python.PythonWorkerFactory.createSocket$1(PythonWorkerFactory.scala:75)
    at org.apache.spark.api.python.PythonWorkerFactory.liftedTree1$1(PythonWorkerFactory.scala:90)
    at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:89)
    at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:62)
    at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:135)
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:101)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
15/12/21 14:27:39 ERROR Executor: Exception in task 1.0 in stage 1.0 (TID 5)
java.lang.IllegalArgumentException: port out of range:1315905645
    at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143)
    at java.net.InetSocketAddress.<init>(InetSocketAddress.java:188)
    at java.net.Socket.<init>(Socket.java:244)
    at org.apache.spark.api.python.PythonWorkerFactory.createSocket$1(PythonWorkerFactory.scala:75)
    at org.apache.spark.api.python.PythonWorkerFactory.liftedTree1$1(PythonWorkerFactory.scala:90)
    at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:89)
    at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:62)
    at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:135)
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:101)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
odule named dateutil.tz
?615/12/21 14:27:39 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 4, localhost): java.lang.IllegalArgumentException: port out of range:1315905645
    at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143)
    at java.net.InetSocketAddress.<init>(InetSocketAddress.java:188)
    at java.net.Socket.<init>(Socket.java:244)
    at org.apache.spark.api.python.PythonWorkerFactory.createSocket$1(PythonWorkerFactory.scala:75)
    at org.apache.spark.api.python.PythonWorkerFactory.liftedTree1$1(PythonWorkerFactory.scala:90)
    at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:89)
    at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:62)
    at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:135)
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:101)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

15/12/21 14:27:39 ERROR TaskSetManager: Task 0 in stage 1.0 failed 1 times; aborting job
15/12/21 14:27:39 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
15/12/21 14:27:39 INFO TaskSetManager: Lost task 2.0 in stage 1.0 (TID 6) on executor localhost: java.lang.IllegalArgumentException (port out of range:1315905645) [duplicate 1]
15/12/21 14:27:39 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
15/12/21 14:27:39 INFO TaskSetManager: Lost task 3.0 in stage 1.0 (TID 7) on executor localhost: java.lang.IllegalArgumentException (port out of range:1315905645) [duplicate 2]
15/12/21 14:27:39 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
15/12/21 14:27:39 INFO TaskSetManager: Lost task 1.0 in stage 1.0 (TID 5) on executor localhost: java.lang.IllegalArgumentException (port out of range:1315905645) [duplicate 3]
15/12/21 14:27:39 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
15/12/21 14:27:39 INFO TaskSchedulerImpl: Cancelling stage 1
15/12/21 14:27:39 INFO DAGScheduler: ResultStage 1 (count at <ipython-input-3-a0443394e570>:1) failed in 0.537 s
15/12/21 14:27:39 INFO DAGScheduler: Job 1 failed: count at <ipython-input-3-a0443394e570>:1, took 0.564707 s
---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
<ipython-input-3-a0443394e570> in <module>()
----> 1 rdd.count()

/Users/jason/spark-1.5.2-bin-hadoop2.6/python/pyspark/rdd.pyc in count(self)
   1004         3
   1005         """
-> 1006         return self.mapPartitions(lambda i: [sum(1 for _ in i)]).sum()
   1007
   1008     def stats(self):

/Users/jason/spark-1.5.2-bin-hadoop2.6/python/pyspark/rdd.pyc in sum(self)
    995         6.0
    996         """
--> 997         return self.mapPartitions(lambda x: [sum(x)]).fold(0, operator.add)
    998
    999     def count(self):

/Users/jason/spark-1.5.2-bin-hadoop2.6/python/pyspark/rdd.pyc in fold(self, zeroValue, op)
    869         # zeroValue provided to each partition is unique from the one provided
    870         # to the final reduce call
--> 871         vals = self.mapPartitions(func).collect()
    872         return reduce(op, vals, zeroValue)
    873

/Users/jason/spark-1.5.2-bin-hadoop2.6/python/pyspark/rdd.pyc in collect(self)
    771         """
    772         with SCCallSiteSync(self.context) as css:
--> 773             port = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
    774         return list(_load_from_socket(port, self._jrdd_deserializer))
    775

/Users/jason/spark-1.5.2-bin-hadoop2.6/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py in __call__(self, *args)
    536         answer = self.gateway_client.send_command(command)
    537         return_value = get_return_value(answer, self.gateway_client,
--> 538                 self.target_id, self.name)
    539
    540         for temp_arg in temp_args:

/Users/jason/spark-1.5.2-bin-hadoop2.6/python/pyspark/sql/utils.pyc in deco(*a, **kw)
     34     def deco(*a, **kw):
     35         try:
---> 36             return f(*a, **kw)
     37         except py4j.protocol.Py4JJavaError as e:
     38             s = e.java_exception.toString()

/Users/jason/spark-1.5.2-bin-hadoop2.6/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    298                 raise Py4JJavaError(
    299                     'An error occurred while calling {0}{1}{2}.\n'.
--> 300                     format(target_id, '.', name), value)
    301             else:
    302                 raise Py4JError(

Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 4, localhost): java.lang.IllegalArgumentException: port out of range:1315905645
    at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143)
    at java.net.InetSocketAddress.<init>(InetSocketAddress.java:188)
    at java.net.Socket.<init>(Socket.java:244)
    at org.apache.spark.api.python.PythonWorkerFactory.createSocket$1(PythonWorkerFactory.scala:75)
    at org.apache.spark.api.python.PythonWorkerFactory.liftedTree1$1(PythonWorkerFactory.scala:90)
    at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:89)
    at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:62)
    at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:135)
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:101)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
    at scala.Option.foreach(Option.scala:236)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1837)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1850)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1921)
    at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:909)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
    at org.apache.spark.rdd.RDD.collect(RDD.scala:908)
    at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:405)
    at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
    at py4j.Gateway.invoke(Gateway.java:259)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:207)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: port out of range:1315905645
    at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143)
    at java.net.InetSocketAddress.<init>(InetSocketAddress.java:188)
    at java.net.Socket.<init>(Socket.java:244)
    at org.apache.spark.api.python.PythonWorkerFactory.createSocket$1(PythonWorkerFactory.scala:75)
    at org.apache.spark.api.python.PythonWorkerFactory.liftedTree1$1(PythonWorkerFactory.scala:90)
    at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:89)
    at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:62)
    at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:135)
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:101)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    ... 1 more

Any ideas?


来源: stackexchange

问:
How to read browser url in sitecore jss
I tried to read browser url(this.props.location) in sitecore jss scaffold component but not able to get that.

Can anybody help with this to read browser url please ?

or

Is it possible to read window url in sitecore jss like normal javascript/reactjs?

答:
document.location might be what you are looking for. Assuming you are talking about client-side JS.

https://developer.mozilla.org/en-US/docs/Web/API/Document/location

ContentLicense: CC BY-SA 4.0

Title: How to read browser url in sitecore jss


问:
Number of bytes used for Unicode characters in varchar

A common misconception is to think that CHAR(n) and VARCHAR(n), the n defines the number of characters. But in CHAR(n) and VARCHAR(n) the n defines the string length in bytes (0-8,000). n never defines numbers of characters that can be stored

According to this statement from Microsoft, I assume, n is the data length of a string and when we store unicode characters in varchar, a single character should take 2 bytes. But, when I try with a sample as below, I see varchar data taking 1 byte instead of 2 bytes.

```<code>declare @varchar varchar(6), @nvarchar nvarchar(6)

set @varchar = 'Ø'

select @varchar as VarcharString, len(@varchar) as VarcharStringLength, DATALENGTH(@varchar) as VarcharStringDataLength

</code>```

Could someone explain the reason behind it?

答:
Found time to test the assumptions of my first answer:

Create UTF8-enabled database

CREATE DATABASE [test-sc] COLLATE Latin1_General_100_CI_AI_KS_SC_UTF8

Create table with all kinds of N/VARCHAR columns

CREATE TABLE [dbo].[UTF8Test](
    [Id] [int] IDENTITY(1,1) NOT NULL,
    [VarcharText] [varchar](50) COLLATE Latin1_General_100_CI_AI NULL,
    [VarcharTextSC] [varchar](50) COLLATE Latin1_General_100_CI_AI_KS_SC NULL,
    [VarcharUTF8] [varchar](50) COLLATE Latin1_General_100_CI_AI_KS_SC_UTF8 NULL,
    [NVarcharText] [nvarchar](50) COLLATE Latin1_General_100_CI_AI_KS NULL,
    [NVarcharTextSC] [nvarchar](50) COLLATE Latin1_General_100_CI_AI_KS_SC NULL,
    [NVarcharUTF8] [nvarchar](50) COLLATE Latin1_General_100_CI_AI_KS_SC_UTF8 NULL)

Insert test data from various Unicode ranges

INSERT INTO [dbo].[UTF8Test] ([VarcharText],[VarcharTextSC],[VarcharUTF8],[NVarcharText],[NVarcharTextSC],[NVarcharUTF8])
     VALUES ('a','a','a','a','a','a')
INSERT INTO [dbo].[UTF8Test] ([VarcharText],[VarcharTextSC],[VarcharUTF8],[NVarcharText],[NVarcharTextSC],[NVarcharUTF8])
     VALUES ('ö','ö','ö',N'ö',N'ö',N'ö')
-- U+56D7
INSERT INTO [dbo].[UTF8Test] ([VarcharText],[VarcharTextSC],[VarcharUTF8],[NVarcharText],[NVarcharTextSC],[NVarcharUTF8])
     VALUES (N'囗',N'囗',N'囗',N'囗',N'囗',N'囗')
-- U+2000B
INSERT INTO [dbo].[UTF8Test] ([VarcharText],[VarcharTextSC],[VarcharUTF8],[NVarcharText],[NVarcharTextSC],[NVarcharUTF8])
     VALUES (N'',N'',N'',N'',N'',N'')

SELECT lengths

SELECT TOP (1000) [Id]
      ,[VarcharText]
      ,[VarcharTextSC]
      ,[VarcharUTF8]
      ,[NVarcharText]
      ,[NVarcharTextSC]
      ,[NVarcharUTF8]
  FROM [test-sc].[dbo].[UTF8Test]
SELECT TOP (1000) [Id]
      ,LEN([VarcharText]) VT
      ,LEN([VarcharTextSC]) VTSC
      ,LEN([VarcharUTF8]) VU
      ,LEN([NVarcharText]) NVT
      ,LEN([NVarcharTextSC]) NVTSC
      ,LEN([NVarcharUTF8]) NVU
  FROM [test-sc].[dbo].[UTF8Test]
SELECT TOP (1000) [Id]
      ,DATALENGTH([VarcharText]) VT
      ,DATALENGTH([VarcharTextSC]) VTSC
      ,DATALENGTH([VarcharUTF8]) VU
      ,DATALENGTH([NVarcharText]) NVT
      ,DATALENGTH([NVarcharTextSC]) NVTSC
      ,DATALENGTH([NVarcharUTF8]) NVU
  FROM [test-sc].[dbo].[UTF8Test]

I was surprised to find that the old mantra "a VARCHAR only stores single byte characters" needs to be revised when using UTF8 collations.

Note that only table columns are associated with collations, but not T-SQL variables:

SELECT @VarcharText = [VarcharText],
    @NVarcharText = [NVarcharText]
  FROM [test-sc].[dbo].[UTF8Test]
   WHERE [Id] = 4
SELECT @VarcharText, Len(@VarcharText), DATALENGTH(@VarcharText), @NVarcharText, Len(@NVarcharText), DATALENGTH(@NVarcharText)
SELECT @VarcharText = [VarcharTextSC],
    @NVarcharText = [NVarcharTextSC]
  FROM [test-sc].[dbo].[UTF8Test]
   WHERE [Id] = 4
SELECT @VarcharText, Len(@VarcharText), DATALENGTH(@VarcharText), @NVarcharText, Len(@NVarcharText), DATALENGTH(@NVarcharText)
SELECT @VarcharText = [VarcharUTF8],
    @NVarcharText = [NVarcharUTF8]
  FROM [test-sc].[dbo].[UTF8Test]
   WHERE [Id] = 4
SELECT @VarcharText, Len(@VarcharText), DATALENGTH(@VarcharText), @NVarcharText, Len(@NVarcharText), DATALENGTH(@NVarcharText)

Body:
A common misconception is to think that CHAR(n) and VARCHAR(n), the n defines the number of characters. But in CHAR(n) and VARCHAR(n) the n defines the string length in bytes (0-8,000). n never defines numbers of characters that can be stored

According to this statement from Microsoft, I assume, n is the data length of a string and when we store unicode characters in varchar, a single character should take 2 bytes. But, when I try with a sample as below, I see varchar data taking 1 byte instead of 2 bytes.

declare @varchar varchar(6), @nvarchar nvarchar(6)

set @varchar = 'Ø'

select @varchar as VarcharString, len(@varchar) as VarcharStringLength, DATALENGTH(@varchar) as VarcharStringDataLength

Could someone explain the reason behind it?


问:
Why Request doesn't work on a specific URL?
I have a question re: requests module in Python.

So far I have been using this to scrape and it's been working well.

However when I do it against one particular website (code below - and refer to the Jupyter Notebook snapshot), it just doesn't want to complete the task (showing [*] forever).

```<code>from bs4 import BeautifulSoup
import requests
import pandas as pd
import json

page = requests.get('https://www.stoneisland.com/ca/stone-island-shadow-project/coats-jackets', verify = False)

soup = BeautifulSoup(page.content, 'html.parser')
</code>```

Some users also suggest using headers such as below to speed it up but it doesnt work for me as well:

```<code>url = 'https://www.stoneisland.com/ca/stone-island-shadow-project/coats-jackets'

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.3'}

req = requests.get(url = url, headers = headers)
</code>```

Not sure what's going on (this is the first time for me) but I might be missing on something obvious. If someone can explain why this is not working? Or if it's working in your machine, please do let me know!

答:
The page attempts to add a cookie the first time you visit it. By using the requests module and not defining a cookie will prevent you from being able to connect to the page.

I've modified your script to include my cookie which should work - if it doesn't, copy your cookie (for this host domain) from the browser to the script.

```<code>url = 'https://www.stoneisland.com/ca/stone-island-shadow-project/coats-jackets'

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.3'}

cookies = {
    'TS01e58ec0': '01a1c9e334eb0b8b191d36d0da302b2bca8927a0ffd2565884aff3ce69db2486850b7fb8e283001c711cc882a8d1f749838ff59d3d'
}

req = requests.get(url = url, headers = headers, cookies=cookies)
</code>```

Body:
I have a question re: requests module in Python.

So far I have been using this to scrape and it's been working well.

However when I do it against one particular website (code below - and refer to the Jupyter Notebook snapshot), it just doesn't want to complete the task (showing [*] forever).

from bs4 import BeautifulSoup
import requests
import pandas as pd
import json

page = requests.get('https://www.stoneisland.com/ca/stone-island-shadow-project/coats-jackets', verify = False)

soup = BeautifulSoup(page.content, 'html.parser')

Some users also suggest using headers such as below to speed it up but it doesnt work for me as well:

url = 'https://www.stoneisland.com/ca/stone-island-shadow-project/coats-jackets'

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.3'}

req = requests.get(url = url, headers = headers)

Not sure what's going on (this is the first time for me) but I might be missing on something obvious. If someone can explain why this is not working? Or if it's working in your machine, please do let me know!

Body:
The page attempts to add a cookie the first time you visit it. By using the requests module and not defining a cookie will prevent you from being able to connect to the page.

I've modified your script to include my cookie which should work - if it doesn't, copy your cookie (for this host domain) from the browser to the script.

url = 'https://www.stoneisland.com/ca/stone-island-shadow-project/coats-jackets'

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.3'}

cookies = {
    'TS01e58ec0': '01a1c9e334eb0b8b191d36d0da302b2bca8927a0ffd2565884aff3ce69db2486850b7fb8e283001c711cc882a8d1f749838ff59d3d'
}

req = requests.get(url = url, headers = headers, cookies=cookies)


问:
How to do/workaround a conditional join in python Pandas?
I am trying to calculate time-based aggregations in Pandas based on date values stored in a separate tables.

The top of the first table table_a looks like this:

```<code>    COMPANY_ID  DATE            MEASURE
    1   2010-01-01 00:00:00     10
    1   2010-01-02 00:00:00     10
    1   2010-01-03 00:00:00     10
    1   2010-01-04 00:00:00     10
    1   2010-01-05 00:00:00     10
</code>```

Here is the code to create the table:

```<code>    table_a = pd.concat(\
    [pd.DataFrame({'DATE': pd.date_range("01/01/2010", "12/31/2010", freq="D"),\
    'COMPANY_ID': 1 , 'MEASURE': 10}),\
    pd.DataFrame({'DATE': pd.date_range("01/01/2010", "12/31/2010", freq="D"),\
    'COMPANY_ID': 2 , 'MEASURE': 10})])
</code>```

The second table, table_b, looks like this:

```<code>        COMPANY     END_DATE
        1   2010-03-01 00:00:00
        1   2010-06-02 00:00:00
        2   2010-03-01 00:00:00
        2   2010-06-02 00:00:00
</code>```

and the code to create it is:

```<code>    table_b = pd.DataFrame({'END_DATE':pd.to_datetime(['03/01/2010','06/02/2010','03/01/2010','06/02/2010']),\
                    'COMPANY':(1,1,2,2)})
</code>```

I want to be able to get the sum of the 'measure' column for each 'COMPANY_ID' for each 30-day period prior to the 'END_DATE' in table_b.

This is (I think) the SQL equivalent:

```<code> select
       b.COMPANY_ID,
       b.DATE
       sum(a.MEASURE) AS MEASURE_TO_END_DATE
 from table_a a, table_b b
 where a.COMPANY = b.COMPANY and
       a.DATE < b.DATE and
       a.DATE > b.DATE - 30
 group by b.COMPANY;
</code>```

答:
There is a very easy, and practical (or maybe the only direct way) to do conditional join in pandas. Since there is no direct way to do conditional join in pandas, you will need an additional library, and that is, pandasql

Install the library pandasql from pip using the command pip install pandasql. This library allows you to manipulate the pandas dataframes using the SQL queries.

```<code>import pandas as pd
from pandasql import sqldf

df = pd.read_excel(r'play_data.xlsx')
df

    id    Name  Amount
0   A001    A   100
1   A002    B   110
2   A003    C   120
3   A005    D   150
</code>```

Now let's just do a conditional join to compare the Amount of the IDs

```<code># Make your pysqldf object:
pysqldf = lambda q: sqldf(q, globals())

# Write your query in SQL syntax, here you can use df as a normal SQL table
cond_join= '''
    select
        df_left.*,
        df_right.*
    from df as df_left
    join df as df_right
    on
        df_left.[Amount] > (df_right.[Amount]+10)

'''

# Now, get your queries results as dataframe using the sqldf object that you created
pysqldf(cond_join)

    id  Name    Amount  id    Name  Amount
0   A003    C   120    A001   A   100
1   A005    D   150    A001   A   100
2   A005    D   150    A002   B   110
3   A005    D   150    A003   C   120
</code>```

Body:
I am trying to calculate time-based aggregations in Pandas based on date values stored in a separate tables.

The top of the first table table_a looks like this:

    COMPANY_ID  DATE            MEASURE
    1   2010-01-01 00:00:00     10
    1   2010-01-02 00:00:00     10
    1   2010-01-03 00:00:00     10
    1   2010-01-04 00:00:00     10
    1   2010-01-05 00:00:00     10

Here is the code to create the table:

    table_a = pd.concat(\
    [pd.DataFrame({'DATE': pd.date_range("01/01/2010", "12/31/2010", freq="D"),\
    'COMPANY_ID': 1 , 'MEASURE': 10}),\
    pd.DataFrame({'DATE': pd.date_range("01/01/2010", "12/31/2010", freq="D"),\
    'COMPANY_ID': 2 , 'MEASURE': 10})])

The second table, table_b, looks like this:

        COMPANY     END_DATE
        1   2010-03-01 00:00:00
        1   2010-06-02 00:00:00
        2   2010-03-01 00:00:00
        2   2010-06-02 00:00:00

and the code to create it is:

    table_b = pd.DataFrame({'END_DATE':pd.to_datetime(['03/01/2010','06/02/2010','03/01/2010','06/02/2010']),\
                    'COMPANY':(1,1,2,2)})

I want to be able to get the sum of the 'measure' column for each 'COMPANY_ID' for each 30-day period prior to the 'END_DATE' in table_b.

This is (I think) the SQL equivalent:

 select
       b.COMPANY_ID,
       b.DATE
       sum(a.MEASURE) AS MEASURE_TO_END_DATE
 from table_a a, table_b b
 where a.COMPANY = b.COMPANY and
       a.DATE  b.DATE - 30
 group by b.COMPANY;

Body:
There is a very easy, and practical (or maybe the only direct way) to do conditional join in pandas. Since there is no direct way to do conditional join in pandas, you will need an additional library, and that is, pandasql

Install the library pandasql from pip using the command pip install pandasql. This library allows you to manipulate the pandas dataframes using the SQL queries.

import pandas as pd
from pandasql import sqldf

df = pd.read_excel(r'play_data.xlsx')
df

    id    Name  Amount
0   A001    A   100
1   A002    B   110
2   A003    C   120
3   A005    D   150

Now let's just do a conditional join to compare the Amount of the IDs

# Make your pysqldf object:
pysqldf = lambda q: sqldf(q, globals())

# Write your query in SQL syntax, here you can use df as a normal SQL table
cond_join= '''
    select
        df_left.*,
        df_right.*
    from df as df_left
    join df as df_right
    on
        df_left.[Amount] > (df_right.[Amount]+10)

'''

# Now, get your queries results as dataframe using the sqldf object that you created
pysqldf(cond_join)

    id  Name    Amount  id    Name  Amount
0   A003    C   120    A001   A   100
1   A005    D   150    A001   A   100
2   A005    D   150    A002   B   110
3   A005    D   150    A003   C   120


问:
htaccess rewriteCond ignored
I want all traffic redirected from app.foo.bar.com to maintenance.foo.bar.com, except urls containing the string api, ref/mail and track.php.

```<code>#maintenance routing
RewriteCond %{REQUEST_URI} !api
RewriteCond %{REQUEST_URI} !ref/mail
RewriteCond %{REQUEST_URI} !track.php
RewriteCond %{HTTP_HOST} ^(?:www\.)?(?:admin\.)?(?:app\.)?(.+)$
RewriteRule ^.*$ http://maintenance.%1/ [QSA,L]

# basic framework routing
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ index.php?url=$1 [QSA,L]
</code>```

This is not working.

Working example:

Example url: http://app.foo.bar.com/

Result: http://maintenance.foo.bar.com/

Not working example:

Example url: http://app.foo.bar.com/api/objects?token=foobarfoobarfoobar

Result: http://maintenance.foo.bar.com/?url=api/objects&token=foobarfoobarfoobar

What's missing or wrong?

答: I had to add RewriteCond %{QUERY_STRING} !api to solve the problem. At the result, 'api' is in the query string instead of the request uri.

Title: htaccess rewriteCond ignored

Body:
I want all traffic redirected from app.foo.bar.com to maintenance.foo.bar.com, except urls containing the string api, ref/mail and track.php.

#maintenance routing
RewriteCond %{REQUEST_URI} !api
RewriteCond %{REQUEST_URI} !ref/mail
RewriteCond %{REQUEST_URI} !track.php
RewriteCond %{HTTP_HOST} ^(?:www\.)?(?:admin\.)?(?:app\.)?(.+)$
RewriteRule ^.*$ http://maintenance.%1/ [QSA,L]

# basic framework routing
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ index.php?url=$1 [QSA,L]

This is not working.

Working example:

Example url: http://app.foo.bar.com/

Result: http://maintenance.foo.bar.com/

Not working example:

Example url: http://app.foo.bar.com/api/objects?token=foobarfoobarfoobar

Result: http://maintenance.foo.bar.com/?url=api/objects&token=foobarfoobarfoobar

What's missing or wrong?


问:
Getting error The type arguments for method '......' cannot be inferred from the usage
i am currently trying to use the method bellow in my async Task but im getting the error when im trying to use the StartSTATask:

The type arguments for method 'PrintService.StartSTATask(Func)' cannot be inferred from the usage. Try specifying the type arguments explicitly.

This is the StartSTATask method

```<code>private Task<T> StartSTATask<T>(Func<T> func)
        {
            var tcs = new TaskCompletionSource<T>();
            Thread thread = new Thread(() =>
            {
                try
                {
                    tcs.SetResult(func());
                }
                catch (Exception e1)
                {
                    tcs.SetException(e1);
                }
            });
            thread.SetApartmentState(ApartmentState.STA);
            thread.Start();
            return tcs.Task;
        }
</code>```

this is where i try to use StartSTATask:

```<code>public async Task printLabelEindcontroleAsync(LabelEindControleDto label)
        {
            await StartSTATask(() => {

                var config = new MapperConfiguration(cfg => cfg.CreateMap<LabelEindControleDto, LabelEindcontrole>());
                var mapper = new Mapper(config);
                LabelEindcontrole labelEindcontrole = mapper.Map<LabelEindcontrole>(label);

                new SuplaconPrint.Queries.LabelEindcontrole(labelEindcontrole);
            });
        }
</code>```

答:
You are not returning anything. Therefore, it can't infer how you want to use the generic parameter and returning type of StartSTATask in the lambda .

The following would work, as it can infer the types, in this case an int

```<code>await StartSTATask(
        () =>
           {
              return 0; // example
           });
</code>```

I am guessing you'd also want a variant without a generic type (for such an occasion)

```<code>private Task StartSTATask(Action action)
{
   var tcs = new TaskCompletionSource<object>();
   Thread thread = new Thread(() =>
                                 {
                                    try
                                    {
                                       action();
                                       tcs.SetResult(null);
                                    }
                                    catch (Exception e1)
                                    {
                                       tcs.SetException(e1);
                                    }
                                 });
   thread.SetApartmentState(ApartmentState.STA);
   thread.Start();
   return tcs.Task;
}
</code>```

For which this would compile and work

```<code>await StartSTATask(
        () =>
           {
                //
           });
</code>```

Body:
i am currently trying to use the method bellow in my async Task but im getting the error when im trying to use the StartSTATask:

The type arguments for method 'PrintService.StartSTATask(Func)' cannot be inferred from the usage. Try specifying the type arguments explicitly.

This is the StartSTATask method

private Task<T> StartSTATask<T>(Func<T> func)
        {
            var tcs = new TaskCompletionSource<T>();
            Thread thread = new Thread(() =>
            {
                try
                {
                    tcs.SetResult(func());
                }
                catch (Exception e1)
                {
                    tcs.SetException(e1);
                }
            });
            thread.SetApartmentState(ApartmentState.STA);
            thread.Start();
            return tcs.Task;
        }

this is where i try to use StartSTATask:

public async Task printLabelEindcontroleAsync(LabelEindControleDto label)
        {
            await StartSTATask(() => {

                var config = new MapperConfiguration(cfg => cfg.CreateMap<LabelEindControleDto, LabelEindcontrole>());
                var mapper = new Mapper(config);
                LabelEindcontrole labelEindcontrole = mapper.Map<LabelEindcontrole>(label);

                new SuplaconPrint.Queries.LabelEindcontrole(labelEindcontrole);
            });
        }

Body:
You are not returning anything. Therefore, it can't infer how you want to use the generic parameter and returning type of StartSTATask in the lambda .

The following would work, as it can infer the types, in this case an int

await StartSTATask(
        () =>
           {
              return 0; // example
           });

I am guessing you'd also want a variant without a generic type (for such an occasion)

private Task StartSTATask(Action action)
{
   var tcs = new TaskCompletionSource<object>();
   Thread thread = new Thread(() =>
                                 {
                                    try
                                    {
                                       action();
                                       tcs.SetResult(null);
                                    }
                                    catch (Exception e1)
                                    {
                                       tcs.SetException(e1);
                                    }
                                 });
   thread.SetApartmentState(ApartmentState.STA);
   thread.Start();
   return tcs.Task;
}

For which this would compile and work

await StartSTATask(
        () =>
           {
                //
           });


问:
manual object constructor call
Can you please tell me if it is possible to call object constructor manually? I know it's wrong and I would never do something like that in my own code and I know I can fix this problem by creating and calling initialization function, however the problem is that I stumbled at a case where there are thousands of lines of code in object's and its parents' constructors...

```<code>class MyClass()
{
    MyClass() { }
    virtual ~MyClass();

    void reset()
    {
         this->~MyClass();
         this->MyClass::MyClass(); //error: Invalid use of MyClass
    }
};
</code>```

答:
A constructor is called using placement new

```<code>new (address) MyClass();
</code>```

This constructs a MyClass in an empty space at address.

Would never do this inside the class though!

Edit

If you already have an object of the right type, and want to assign it default values, an alternative is

```<code>*this = MyClass();
</code>```

which creates a new object with default values and assigns that value to your existing object.

ContentLicense: CC BY-SA 3.0

Title: manual object constructor call

Body:
Can you please tell me if it is possible to call object constructor manually? I know it's wrong and I would never do something like that in my own code and I know I can fix this problem by creating and calling initialization function, however the problem is that I stumbled at a case where there are thousands of lines of code in object's and its parents' constructors...

class MyClass()
{
    MyClass() { }
    virtual ~MyClass();

    void reset()
    {
         this->~MyClass();
         this->MyClass::MyClass(); //error: Invalid use of MyClass
    }
};

Body:
A constructor is called using placement new

new (address) MyClass();

This constructs a MyClass in an empty space at address.

Would never do this inside the class though!

Edit

If you already have an object of the right type, and want to assign it default values, an alternative is

*this = MyClass();

which creates a new object with default values and assigns that value to your existing object.


问:
VBA - Split from right to left (REVERSED)
I want to extract the last folder from a folder path:

```<code>path = C:\Users\z204685\tecware\RESULTS\D1369_3 (R=0) Surface Failure
</code>```

I want to extract the last part, after the "\", in a new string:

```<code>newString = "D1369_3 (R=0) Surface Failure"
</code>```

Maybe reversing the path string, then using the Split function with "\" and then reversing it again...Any better ideas?

答:
Another solution, according to @TimWilliams comment, You can achieve this using Split and UBound

```<code>Sub test()

Dim fpath As String
Dim newString As String
Dim temp_arr As Variant

fpath = "C:\Users\z204685\tecware\RESULTS\D1369_3 (R=0) Surface Failure"
temp_arr = Split(fpath, "\")
newString = temp_arr(UBound(temp_arr))

End Sub
</code>```

Body:
I want to extract the last folder from a folder path:

path = C:\Users\z204685\tecware\RESULTS\D1369_3 (R=0) Surface Failure

I want to extract the last part, after the "\", in a new string:

newString = "D1369_3 (R=0) Surface Failure"

Maybe reversing the path string, then using the Split function with "\" and then reversing it again...Any better ideas?

Body:
Another solution, according to @TimWilliams comment, You can achieve this using Split and UBound

Sub test()

Dim fpath As String
Dim newString As String
Dim temp_arr As Variant

fpath = "C:\Users\z204685\tecware\RESULTS\D1369_3 (R=0) Surface Failure"
temp_arr = Split(fpath, "\")
newString = temp_arr(UBound(temp_arr))

End Sub


问:
Using Apache Solr to index Nutch data
I am attempting to set up Solr to index the results from my Nutch crawler. The tutorials I have found online require the file conf/schema.xml to be copied from Nutch into Solr as part of the configuration, however this file does not exist in my version of Nutch. Is there another way to do this or is there something I am missing?
cheers</p>

答:
See the "official" Nutch tutorial:

Note: due to NUTCH-2745 the schema.xml is not contained in the binary package. Please download the schema.xml from the source repository.

NUTCH-2745 applies only to Nutch version 1.16 and will be fixed in 1.17.

Title: Using Apache Solr to index Nutch data


问:
Regex recursion captured string
I have a problem with a regex that has to capture a substring that it's already captured...

I have this regex:

(?<domain>\w+\.\w+)($|\/|\.)

And I want to capture every subdomain recursively. For example, in this string:

test1.test2.abc.def

This expression captures test1.test2 and abc.def but I need to capture:
test1.test2
test2.abc
abc.def

Do you know if there is any option to do this recursively?

Thanks!

答:
You can use a positive look ahead to capture the next group.

```<code>/(\w+)\.(?=(\w+))/g
</code>```

Demonstration.

Edit: JvdV's regex is more correct.

Note that \w+ is will fail to match domains like regex-tester.com and will match invalid regex_tester.com. [a-zA-Z0-9-]+ is closer to correct. See this answer for a complete regex.

It's simpler and more robust to do this by splitting on . and iterating through the pieces in pairs. For example, in Ruby...

```<code>"test1.test2.abc.def".split(".").each_cons(2) { |a|
  puts a.join(".")
}

test1.test2
test2.abc
abc.def
</code>```

Title: Regex recursion captured string

Body:
You can use a positive look ahead to capture the next group.

/(\w+)\.(?=(\w+))/g

Demonstration.

Edit: JvdV's regex is more correct.

Note that \w+ is will fail to match domains like regex-tester.com and will match invalid regex_tester.com. [a-zA-Z0-9-]+ is closer to correct. See this answer for a complete regex.

It's simpler and more robust to do this by splitting on . and iterating through the pieces in pairs. For example, in Ruby...

"test1.test2.abc.def".split(".").each_cons(2) { |a|
  puts a.join(".")
}

test1.test2
test2.abc
abc.def


问:
Get the average
Can you make a query that

My data looks like this:

acccount_id | value | timestamp
-------------------------------
a12         | 122   | jan 1
a13         | 133   | jan 1
a14         | 443   | jan 1
a12         | 251   | jan 2
a13         | 122   | jan 2
a14         | 331   | jan 2
a12         | 412   | jan 3
a13         | 323   | jan 3
a14         | 432   | jan 3
```

<p>I would like to get the average for all previous days for each account.</p>

<pre class="lang-none prettyprint-override"><code>acccount_id | value | timestamp | Average
-----------------------------------------
a12         |  122  | jan 1     | 122
a13         |  133  | jan 1     | 133
a14         |  443  | jan 1     | 443
a12         |  251  | jan 2     | 188.5
a13         |  122  | jan 2     | 222.5
a14         |  331  | jan 2     | 387
a12         |  412  | jan 3     | 261.6
a13         |  323  | jan 3     | 192.6
a14         |  432  | jan 3     | 402
</code>```

I am not a pro with Postgres but I was looking into window functions and it seems like the way to go but i couldn't get it to work correctly. I am obviously open to better options as well.

For more background, I will actually be generating new values every day and have a lambda function that is computing them. So a statement that will append where values do not exist would be exactly what I am trying to do.

答:
You want a running average, and indeed this is done using window functions. When you supply an ORDER BY to the partition definition of the aggregate you get the running average

```<code>select account_id, value, "timestamp",
       avg(value) over (partition by account_id order by "timestamp") as average
from the_table
order by account_id, "timestamp";
</code>```

Title: Get the average

Body:
Can you make a query that

My data looks like this:

acccount_id | value | timestamp
-------------------------------
a12         | 122   | jan 1
a13         | 133   | jan 1
a14         | 443   | jan 1
a12         | 251   | jan 2
a13         | 122   | jan 2
a14         | 331   | jan 2
a12         | 412   | jan 3
a13         | 323   | jan 3
a14         | 432   | jan 3

I would like to get the average for all previous days for each account.

acccount_id | value | timestamp | Average
-----------------------------------------
a12         |  122  | jan 1     | 122
a13         |  133  | jan 1     | 133
a14         |  443  | jan 1     | 443
a12         |  251  | jan 2     | 188.5
a13         |  122  | jan 2     | 222.5
a14         |  331  | jan 2     | 387
a12         |  412  | jan 3     | 261.6
a13         |  323  | jan 3     | 192.6
a14         |  432  | jan 3     | 402

I am not a pro with Postgres but I was looking into window functions and it seems like the way to go but i couldn't get it to work correctly. I am obviously open to better options as well.

For more background, I will actually be generating new values every day and have a lambda function that is computing them. So a statement that will append where values do not exist would be exactly what I am trying to do.

Body:
You want a running average, and indeed this is done using window functions. When you supply an ORDER BY to the partition definition of the aggregate you get the running average

select account_id, value, "timestamp",
       avg(value) over (partition by account_id order by "timestamp") as average
from the_table
order by account_id, "timestamp";


问:
Using ggplot2, how do I make multiple lines show up in my grid graph?
I'm new to plotting graphs in R. I'm working with code someone else has written and I'm updating/modifying it to update figures for an annual report. I have the following data but it has samples for every month from 2016 - 2019:

```<code># Groups:   Year, Month, Site, Reach, ReachID [33,264]
Year  Month Site      Reach           ReachID       Size          Mussels_m2
<fct> <ord> <fct>     <fct>           <ord>         <fct>              <dbl>
1 2019  Feb   Below SFD Downstream Weir Property Line Total mussels         2
2 2019  Feb   Below SFD Downstream Weir Property Line >10 mm                2
3 2019  Feb   Below SFD Downstream Weir Property Line 5-10 mm               2
4 2019  Feb   Below SFD Downstream Weir Property Line <5 mm                 2
5 2019  Feb   Below SFD Upstream Weir   Release Pool  Total mussels         2
6 2019  Feb   Below SFD Upstream Weir   Release Pool  >10 mm                2
7 2019  Feb   Below SFD Upstream Weir   Release Pool  5-10 mm               2
8 2019  Feb   Below SFD Upstream Weir   Release Pool  <5 mm                 2
9 2019  Mar   Below SFD Upstream Weir   Release Pool  Total mussels       174.
10 2019  Mar   Below SFD Upstream Weir   Release Pool  >10 mm              174.
</code>```

The code I'm using is:

```<code>ggplot(releasereach2019, aes(x = Month, y = Mussels_m2, color = Size, group =
Size)) +
  geom_point() +
  geom_line() +
  facet_grid(ReachID ~ .) +
  theme(axis.text.x = element_text(angle = 45, hjust = 1)) +
  scale_y_continuous(breaks = seq(0,8000,2000)) +
  labs(x = NULL,
      y = "Mussel Density (m^2)",
      title = "Mussel Density Below SFD in 2019")
</code>```

With this code, only the <5 mm value will plot in the graph, not the other sizes. I'm wondering what I can do to get the rest of the size values to plot. I've tried group = Size in geom_line() but it is still the same. The legend seems fine. Ideally I would like a graph similar to what is attached to this post with all the lines showing.

答:
With the data you just provided is hard to work with, so I just created a whole new data frame, thats why there is only two faceted grids (pool and line). Your code is ok, but you could improve some things:

To order your month you should provide R with Date data or it will order them alphabetically, to solve this you can use from dplyr library mutate() and factor()
funtion.

This is just because I love to complicate myself in the axis text, to put a superscript you can use expression() and ^  for example a² would be expression("a"^"2").

The code it would be like this:

```<code>library(dplyr)
library(ggplot2)

releasereach2019%>%
  arrange(Mussels_m2)%>% mutate(Month=factor(Month,levels=c("Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec")))%>%
ggplot(aes(x = Month, y = Mussels_m2, color = Size,group=Size)) +
geom_point()+
geom_line()+
facet_grid(ReachID ~ .) +
theme(axis.text.x = element_text(angle = 45, hjust = 1)) +
scale_y_continuous(breaks = seq(0,8000,2000)) +
labs(x = NULL,
     y = expression(paste("Mussel Density (m"^"2",")")),
     title = "Mussel Density Below SFD in 2019")
</code>```

Body:
I'm new to plotting graphs in R. I'm working with code someone else has written and I'm updating/modifying it to update figures for an annual report. I have the following data but it has samples for every month from 2016 - 2019:

# Groups:   Year, Month, Site, Reach, ReachID [33,264]
Year  Month Site      Reach           ReachID       Size          Mussels_m2
<fct> <ord> <fct>     <fct>           <ord>         <fct>              <dbl>
1 2019  Feb   Below SFD Downstream Weir Property Line Total mussels         2
2 2019  Feb   Below SFD Downstream Weir Property Line >10 mm                2
3 2019  Feb   Below SFD Downstream Weir Property Line 5-10 mm               2
4 2019  Feb   Below SFD Downstream Weir Property Line <5 mm                 2
5 2019  Feb   Below SFD Upstream Weir   Release Pool  Total mussels         2
6 2019  Feb   Below SFD Upstream Weir   Release Pool  >10 mm                2
7 2019  Feb   Below SFD Upstream Weir   Release Pool  5-10 mm               2
8 2019  Feb   Below SFD Upstream Weir   Release Pool  <5 mm                 2
9 2019  Mar   Below SFD Upstream Weir   Release Pool  Total mussels       174.
10 2019  Mar   Below SFD Upstream Weir   Release Pool  >10 mm              174.

The code I'm using is:

ggplot(releasereach2019, aes(x = Month, y = Mussels_m2, color = Size, group =
Size)) +
  geom_point() +
  geom_line() +
  facet_grid(ReachID ~ .) +
  theme(axis.text.x = element_text(angle = 45, hjust = 1)) +
  scale_y_continuous(breaks = seq(0,8000,2000)) +
  labs(x = NULL,
      y = "Mussel Density (m^2)",
      title = "Mussel Density Below SFD in 2019")

With this code, only the <5 mm value will plot in the graph, not the other sizes. I'm wondering what I can do to get the rest of the size values to plot. I've tried group = Size in geom_line() but it is still the same. The legend seems fine. Ideally I would like a graph similar to what is attached to this post with all the lines showing.

Body:
With the data you just provided is hard to work with, so I just created a whole new data frame, thats why there is only two faceted grids (pool and line). Your code is ok, but you could improve some things:

To order your month you should provide R with Date data or it will order them alphabetically, to solve this you can use from dplyr library mutate() and factor()
funtion.

This is just because I love to complicate myself in the axis text, to put a superscript you can use expression() and ^  for example a² would be expression("a"^"2").

The code it would be like this:

library(dplyr)
library(ggplot2)

releasereach2019%>%
  arrange(Mussels_m2)%>% mutate(Month=factor(Month,levels=c("Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec")))%>%
ggplot(aes(x = Month, y = Mussels_m2, color = Size,group=Size)) +
geom_point()+
geom_line()+
facet_grid(ReachID ~ .) +
theme(axis.text.x = element_text(angle = 45, hjust = 1)) +
scale_y_continuous(breaks = seq(0,8000,2000)) +
labs(x = NULL,
     y = expression(paste("Mussel Density (m"^"2",")")),
     title = "Mussel Density Below SFD in 2019")


问:
Return JSON from WCF
my WCF service is in .net framework 3.5,this WCF service will be used by PHP, so i have used basicHttpBinding (please suggest if any other better one).

i am using below code/attribut on function to return JSON.

```<code>[WebInvoke(Method = "POST", BodyStyle = WebMessageBodyStyle.Wrapped, ResponseFormat = WebMessageFormat.Json)]
</code>```

but its not returning JSON (its returning object).
Please help me.

NOTE: i found a solution but its for .NET Framework 4.0. but i need solution for 3.5.
WCF 4.0 : WebMessageFormat.Json not working with WCF REST Template
Thanks

答:
Have you added the AspNetCompatability attribute to your service implementation class?

```<code>[AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
public class ServiceImpl{
}
</code>```

Also, change you might need to change your binding to webHttpBinding as basicHttpBinding is for SOAP rather than Json

Title: Return JSON from WCF

Body:
my WCF service is in .net framework 3.5,this WCF service will be used by PHP, so i have used basicHttpBinding (please suggest if any other better one).

i am using below code/attribut on function to return JSON.

[WebInvoke(Method = "POST", BodyStyle = WebMessageBodyStyle.Wrapped, ResponseFormat = WebMessageFormat.Json)]

but its not returning JSON (its returning object).
Please help me.

NOTE: i found a solution but its for .NET Framework 4.0. but i need solution for 3.5.
WCF 4.0 : WebMessageFormat.Json not working with WCF REST Template
Thanks

Body:
Have you added the AspNetCompatability attribute to your service implementation class?

[AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
public class ServiceImpl{
}

Also, change you might need to change your binding to webHttpBinding as basicHttpBinding is for SOAP rather than Json


问:
remove elements from CopyOnWriteArrayList
I am getting an exception when I try to remove elements from CopyOnWriteArrayList using an iterator.
I have noticed that it is documented

Element-changing operations on iterators themselves (remove, set, and add) are not supported. These methods throw UnsupportedOperationException.

(from http://download.oracle.com/javase/6/docs/api/java/util/concurrent/CopyOnWriteArrayList.html)

Now, surprisingly i can iterate it with foreach and use the remove() function . But then I get the famous bug - when trying to remove an item from a list using a for loop - you skip the element next to the removed element.
any suggestions then?

答:
If you want to delete all use just clear(). If you want to keep elements put them in a temporary ArrayList and get them back from there.</p>

```<code>List<Object> tKeepThese= new ArrayList<>();
for(ListIterator<Object> tIter = theCopyOnWriteArrayList; tIter.hasNext();)
{
    tObject = tIter.next();
    if(condition to keep element)
        tKeepThese.add(tObject);
}
theCopyOnWriteArrayList.clear();
theCopyOnWriteArrayList.addAll(tKeepThese);
</code>```

Body:
If you want to delete all use just clear(). If you want to keep elements put them in a temporary ArrayList and get them back from there.

List<Object> tKeepThese= new ArrayList<>();
for(ListIterator<Object> tIter = theCopyOnWriteArrayList; tIter.hasNext();)
{
    tObject = tIter.next();
    if(condition to keep element)
        tKeepThese.add(tObject);
}
theCopyOnWriteArrayList.clear();
theCopyOnWriteArrayList.addAll(tKeepThese);


问:
How to plot single pixel values from 3d NumPy array?
I have a stack of 7 images of 288 x 288 pixels that I have converted to a 3d NumPy array

newarray.shape = (288, 288, 7)
```

I want to plot a particular pixel value from each of the 7 images and plot it as a graph with y axis showing pixel values and x axis showing the image number.

答:
from matplotlib import pyplot as plt
import numpy as np

# NumPy array storing images
images = np.random.randint(0, 255, (288, 288, 7), np.uint8)

# Get pixel values across all images of pixel of interest
x, y = (8, 3)
pixels = images[y, x, :]

# Plot
plt.plot(np.arange(images.shape[2]), pixels)
plt.ylim(0, 255)
plt.title('Pixel values for x=' + str(x) + ', y=' + str(y))
plt.tight_layout()
plt.show()
```

<p>Output:</p>

<p><img src="https://i.stack.imgur.com/ThoDR.png" alt="Output"></p>

<p>Hope that helps!</p>

<pre class="lang-none prettyprint-override"><code>----------------------------------------
System information
----------------------------------------
Platform:    Windows-10-10.0.16299-SP0
Python:      3.8.1
Matplotlib:  3.2.0rc3
NumPy:       1.18.1
----------------------------------------
</code>```

Body:
I have a stack of 7 images of 288 x 288 pixels that I have converted to a 3d NumPy array

newarray.shape = (288, 288, 7)

I want to plot a particular pixel value from each of the 7 images and plot it as a graph with y axis showing pixel values and x axis showing the image number.

Body:
from matplotlib import pyplot as plt
import numpy as np

# NumPy array storing images
images = np.random.randint(0, 255, (288, 288, 7), np.uint8)

# Get pixel values across all images of pixel of interest
x, y = (8, 3)
pixels = images[y, x, :]

# Plot
plt.plot(np.arange(images.shape[2]), pixels)
plt.ylim(0, 255)
plt.title('Pixel values for x=' + str(x) + ', y=' + str(y))
plt.tight_layout()
plt.show()

Output:

Hope that helps!

----------------------------------------
System information
----------------------------------------
Platform:    Windows-10-10.0.16299-SP0
Python:      3.8.1
Matplotlib:  3.2.0rc3
NumPy:       1.18.1
----------------------------------------


问:
Unable to Open .txt File
```<code>std::ifstream infile;
infile.open("example.txt");

if (!infile.is_open()) {
   std::cout << "can't open" << std::endl;
}
</code>```

I read this: unable to open file stream c++ and have tried using the full path infile.open("~/projects/example.txt");

The text file is in the same folder where the .cpp / .hpp files reside.

What else could be the problem?

答:
I once had the same problem on Windows. The problem was that the actual file name was "example.txt.txt", but Window Explorer was configured (by default) to hide file extensions.

A simple solution is to use c++17 filesystem library. For example,   current_path will return what its name promises. Or write a simple function that will list the current path contents (see the example on cppreference, https://en.cppreference.com/w/cpp/experimental/fs/directory_iterator ) . Or create a file of some similar name, e.g. __example__.txt and see where it was created and under which name it is reported by your filesystem manager.

If this is Linux, try running your program under strace (strace myprog myoptions).

Title: Unable to Open .txt File

Body:
std::ifstream infile;
infile.open("example.txt");

if (!infile.is_open()) {
   std::cout << "can't open" << std::endl;
}

I read this: unable to open file stream c++ and have tried using the full path infile.open("~/projects/example.txt");

The text file is in the same folder where the .cpp / .hpp files reside.

What else could be the problem?


问:
Rendering MVC view in code
I'm trying to render a strongly typed viewpage in code and get the resulting HTML output.  Here's the code I am using:

```<code>public static string RenderPartialToString(ViewPage vp) {
    StringBuilder sb = new StringBuilder();
    using (StringWriter sw = new StringWriter(sb)) {
        using (HtmlTextWriter tw = new HtmlTextWriter(sw)) {
            ViewContext vc = new ViewContext();
            vc.ViewData = vp.ViewData;
            vp.RenderView(vc);
        }
    }
    vp.Dispose();
    vp = null;

    string s = sb.ToString();
    sb = null;

    return s;
}
</code>```

Right now, I am getting a MethodNotImplemented error on the vp.RenderView() line.  Is it not possible to render MVC viewpages at this time (MVC 3)?

答:
I would take a look at how they do it in the MvcMailer application which renders MVC views into emails:

https://github.com/smsohan/MvcMailer

http://www.codeproject.com/KB/aspnet/MvcMailerNuGet.aspx

Hope this helps.

Title: Rendering MVC view in code

Body:
I'm trying to render a strongly typed viewpage in code and get the resulting HTML output.  Here's the code I am using:

public static string RenderPartialToString(ViewPage vp) {
    StringBuilder sb = new StringBuilder();
    using (StringWriter sw = new StringWriter(sb)) {
        using (HtmlTextWriter tw = new HtmlTextWriter(sw)) {
            ViewContext vc = new ViewContext();
            vc.ViewData = vp.ViewData;
            vp.RenderView(vc);
        }
    }
    vp.Dispose();
    vp = null;

    string s = sb.ToString();
    sb = null;

    return s;
}

Right now, I am getting a MethodNotImplemented error on the vp.RenderView() line.  Is it not possible to render MVC viewpages at this time (MVC 3)?


问:
What are all these deprecated "loop" parameters in asyncio?
A lot of the functions in asyncio have deprecated loop parameters, scheduled to be removed in Python 3.10. Examples include as_completed(), sleep(), and wait().

I'm looking for some historical context on these parameters and their removal.

What problems did loop solve? Why would one have used it in the first place?

What was wrong with loop? Why is it being removed en masse?

What replaces loop, now that it's gone?

答:
What problems did loop solve? Why would one have used it in the first place?

Prior to Python 3.6, asyncio.get_event_loop() was not guaranteed to return the event loop currently running when called from an asyncio coroutine or callback. It would return whatever event loop was previously set using set_event_loop(some_loop), or the one automatically created by asyncio. But sync code could easily create a different loop with another_loop = asyncio.new_event_loop() and spin it up using another_loop.run_until_complete(some_coroutine()). In this scenario, get_event_loop() called inside some_coroutine and the coroutines it awaits would return some_loop rather than another_loop. This kind of thing wouldn't occur when using asyncio casually, but it had to be accounted for by async libraries which couldn't assume that they were running under the default event loop. (For example, in tests or in some usages involving threads, one might want to spin up an event loop without disturbing the global setting with set_event_loop.) The libraries would offer the explicit loop argument where you'd pass another_loop in the above case, and which you'd use whenever the running loop differed from the loop set up with asyncio.set_event_loop().

This issue would be fixed in Python 3.6 and 3.5.3, where get_event_loop() was modified to reliably return the running loop if called from inside one, returning another_loop in the above scenario. Python 3.7 would additionally introduced get_running_loop() which completely ignores the global setting and always returns the currently running loop, raising an exception if not inside one. See this thread for the original discussion.

Once get_event_loop() became reliable, another problem was that of performance. Since the event loop was needed for some very frequently used calls, most notably call_soon, it was simply more efficient to pass around and cache the loop object. Asyncio itself did that, and many libraries followed suit. Eventually get_event_loop() was accelerated in C and was no longer a bottleneck.

These two changes made the loop arguments redundant.

What was wrong with loop? Why is it being removed en masse?

As any other redundancy, it complicates the API and opens up possibilities for errors. Async code should almost never just randomly communicate with a different loop, and now that get_event_loop() is both correct and fast, there is no reason not to use it.

Also, passing the loop through all the layers of abstraction of a typical application is simply tedious. With async/await becoming mainstream in other languages, it has become clear that manually propagating a global object is not ergonomic and should not be required from programmers.

What replaces loop, now that it's gone?

Just use get_event_loop() to get the loop when you need it. Alternatively, you can use get_running_loop() to assert that a loop is running.

The need for accessing the event loop is somewhat reduced in Python 3.7, as some functions that were previously only available as methods on the loop, such as create_task, are now available as stand-alone functions.


问:
nil check, best way?
I'm just learning lua, and see these two ways to check for nil

```<code>local stats = game.ServerStorage.PlayerMoney:FindFirstChild(player.Name)

if (stats ~= nil) then
    -- do stuff
end

if (stats) then
    -- do stuff
end
</code>```

Are the if statements equivalent? If so, is there any advantage to including the extra "~= nil" part?

答:
Statement "~= nil" works also if stats = false.

You can read in docs:

The condition expression of a control structure can return any value.
  Both false and nil are considered false. All values different from nil
  and false are considered true (in particular, the number 0 and the
  empty string are also true).

Title: nil check, best way?

Body:
I'm just learning lua, and see these two ways to check for nil

local stats = game.ServerStorage.PlayerMoney:FindFirstChild(player.Name)

if (stats ~= nil) then
    -- do stuff
end

if (stats) then
    -- do stuff
end

Are the if statements equivalent? If so, is there any advantage to including the extra "~= nil" part?


问:
Calling function from dll with QT
I created by the first time a dll to use with C++ (and then C#, I have plans to share this dll between a C++ and C# applications) using QT creator but when I try to use it I get SEGFAULT error.

Here's my files:

mydll.h

```<code>#ifndef MYDLL_H
#define MYDLL_H

#include "mydll_global.h"

class MYDLLSHARED_EXPORT MyDll
{

public:
    MyDll();
    int getAnswer();
};

MYDLLSHARED_EXPORT int getNumber();

#endif // MYDLL_H
</code>```

mydll_global.h

```<code>#ifndef MYDLL_GLOBAL_H
#define MYDLL_GLOBAL_H

#include <QtCore/qglobal.h>

#if defined(MYDLL_LIBRARY)
#  define MYDLLSHARED_EXPORT Q_DECL_EXPORT
#else
#  define MYDLLSHARED_EXPORT Q_DECL_IMPORT
#endif

#endif // MYDLL_GLOBAL_H
</code>```

mydll.cpp

```<code>#include "mydll.h"

MyDll::MyDll()
{
}

int MyDll::getAnswer()
{
    return 42;
}

int getNumber()
{
    return 10;
}
</code>```

So I build it and created mydll.dll: then I went to other C++ project wheere I want to use this dll and put in the .pro file:

```<code>LIBS += "C:\path\to\mydll.h"
</code>```

and in the main.cpp

```<code>#include "mydll.h"
</code>```

and when I use function from dll like this:

```<code> qDebug() << getNumber();
</code>```

I get a SEGFAULT error.

I thought that the header to provide the compiler type information and the dll to compiler to provide the function body was all I needed but as I'm getting a SEGFAULT I'm acessing NULL or someone else memory or so (I can't see the value on debug).

What am I missing?

答:
I had this same issue. Almost every example I looked at had a class as their example, but I wanted to export functions like getNumber(). It's in the qt5 docs as you probably know by now.

This line needs to be in your header file:

```<code>extern "C" MYDLLSHARED_EXPORT int getNumber();
</code>```

and it is either the mydll_global.h or mydll.h depending on what you are using.

mydll_global.h would be what qtcreator creates and puts export definitions in if you create a new Shared Library project.

I was thrown off guard by the extern "C" part as I wasn't using C and wasn't as familiar with how compilers export symbols.

Trying to add a public static function to a class definition using static MYDLLSHARED_EXPORT int getNumber(); or simply using a namespace (function defined with or without static) will give a not found for architecture x86_64 error using clang on osx.

Title: Calling function from dll with QT

Body:
I created by the first time a dll to use with C++ (and then C#, I have plans to share this dll between a C++ and C# applications) using QT creator but when I try to use it I get SEGFAULT error.

Here's my files:

mydll.h

#ifndef MYDLL_H
#define MYDLL_H

#include "mydll_global.h"

class MYDLLSHARED_EXPORT MyDll
{

public:
    MyDll();
    int getAnswer();
};

MYDLLSHARED_EXPORT int getNumber();

#endif // MYDLL_H

mydll_global.h

#ifndef MYDLL_GLOBAL_H
#define MYDLL_GLOBAL_H

#include <QtCore/qglobal.h>

#if defined(MYDLL_LIBRARY)
#  define MYDLLSHARED_EXPORT Q_DECL_EXPORT
#else
#  define MYDLLSHARED_EXPORT Q_DECL_IMPORT
#endif

#endif // MYDLL_GLOBAL_H

mydll.cpp

#include "mydll.h"

MyDll::MyDll()
{
}

int MyDll::getAnswer()
{
    return 42;
}

int getNumber()
{
    return 10;
}

So I build it and created mydll.dll: then I went to other C++ project wheere I want to use this dll and put in the .pro file:

LIBS += "C:\path\to\mydll.h"

and in the main.cpp

#include "mydll.h"

and when I use function from dll like this:

 qDebug() << getNumber();

I get a SEGFAULT error.

I thought that the header to provide the compiler type information and the dll to compiler to provide the function body was all I needed but as I'm getting a SEGFAULT I'm acessing NULL or someone else memory or so (I can't see the value on debug).

What am I missing?

Body:
I had this same issue. Almost every example I looked at had a class as their example, but I wanted to export functions like getNumber(). It's in the qt5 docs as you probably know by now.

This line needs to be in your header file:

extern "C" MYDLLSHARED_EXPORT int getNumber();

and it is either the mydll_global.h or mydll.h depending on what you are using.

mydll_global.h would be what qtcreator creates and puts export definitions in if you create a new Shared Library project.

I was thrown off guard by the extern "C" part as I wasn't using C and wasn't as familiar with how compilers export symbols.

Trying to add a public static function to a class definition using static MYDLLSHARED_EXPORT int getNumber(); or simply using a namespace (function defined with or without static) will give a not found for architecture x86_64 error using clang on osx.