title: How to Configure Spark 3.2 (with Hadoop) to fix the Broken HTML Web UI on Port 4040
body: <p>I am running Spark 3.2 on OS X Monterey (12.1 Beta).  Spark is configured to use Yarn.
  My application is running properly, however, the Web UI on 4040 is broken.</p>
  <p>This blog describes the problem:</p>
  <p><a href="https://laujohn.com/2018/09/22/Apache-Spark-Web-UI-on-Amazon-EMR/" rel="nofollow noreferrer">https://laujohn.com/2018/09/22/Apache-Spark-Web-UI-on-Amazon-EMR/</a></p>
  <p>Here's a screen snapshot:
  <a href="https://i.stack.imgur.com/pBQmP.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/pBQmP.png" alt="enter image description here" /></a></p>
  <p>Here's my command to spark-shell:</p>
  <pre><code>~/spark/bin/spark-shell --class com.cloudera.datascience.lsa.RunLSA --master yarn --deploy-mode client  --jars target/ch06-lsa-2.0.0-jar-with-dependencies.jar --executor-cores 2 --driver-memory 6G --conf spark.yarn.populateHadoopClasspath=true  
  </code></pre>
  <p>I tried adding --conf spark.ui.proxyBase=&quot;&quot;, but it didn't help</p>
  <p>Here's my conf/spark-defaults.conf file:</p>
  <pre><code># Example:
  spark.master                     spark://127.0.0.1:7077
  spark.eventLog.enabled           true
  spark.eventLog.dir               file:///Users/davidlaxer/spark/events
  spark.serializer                 org.apache.spark.serializer.KryoSerializer
  spark.driver.memory              9g
  spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value -Dnumbers=&quot;one two three&quot;
  spark.ui.proxyBase        &quot;&quot;
  (base) davidlaxer@x86_64-apple-darwin13 spark-3.2.0-bin-hadoop3.2 % 
  
  </code></pre>
  <p>Here's the output from the spark-shell job:</p>
  <pre><code>~/spark/bin/spark-shell --class com.cloudera.datascience.lsa.RunLSA --master yarn --deploy-mode client  --jars target/ch06-lsa-2.0.0-jar-with-dependencies.jar --executor-cores 2 --driver-memory 6G --conf spark.yarn.populateHadoopClasspath=true                         
  WARNING: An illegal reflective access operation has occurred
  WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/Users/davidlaxer/spark-3.2.0-bin-hadoop3.2/jars/spark-unsafe_2.12-3.2.0.jar) to constructor java.nio.DirectByteBuffer(long,int)
  WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
  WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
  WARNING: All illegal access operations will be denied in a future release
  21/12/01 07:40:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  21/12/01 07:40:29 INFO SparkContext: Running Spark version 3.2.0
  21/12/01 07:40:29 INFO ResourceUtils: ==============================================================
  21/12/01 07:40:29 INFO ResourceUtils: No custom resources configured for spark.driver.
  21/12/01 07:40:29 INFO ResourceUtils: ==============================================================
  21/12/01 07:40:29 INFO SparkContext: Submitted application: Spark shell
  21/12/01 07:40:29 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -&gt; name: cores, amount: 2, script: , vendor: , memory -&gt; name: memory, amount: 4096, script: , vendor: , offHeap -&gt; name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -&gt; name: cpus, amount: 1.0)
  21/12/01 07:40:29 INFO ResourceProfile: Limiting resource is cpus at 2 tasks per executor
  21/12/01 07:40:29 INFO ResourceProfileManager: Added ResourceProfile id: 0
  21/12/01 07:40:29 INFO SecurityManager: Changing view acls to: davidlaxer
  21/12/01 07:40:29 INFO SecurityManager: Changing modify acls to: davidlaxer
  21/12/01 07:40:29 INFO SecurityManager: Changing view acls groups to: 
  21/12/01 07:40:29 INFO SecurityManager: Changing modify acls groups to: 
  21/12/01 07:40:29 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(davidlaxer); groups with view permissions: Set(); users  with modify permissions: Set(davidlaxer); groups with modify permissions: Set()
  21/12/01 07:40:29 INFO Utils: Successfully started service 'sparkDriver' on port 52069.
  21/12/01 07:40:29 INFO SparkEnv: Registering MapOutputTracker
  21/12/01 07:40:29 INFO SparkEnv: Registering BlockManagerMaster
  21/12/01 07:40:29 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
  21/12/01 07:40:29 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
  21/12/01 07:40:29 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
  21/12/01 07:40:29 INFO DiskBlockManager: Created local directory at /private/var/folders/3n/56fpv14n4wj0c1l1sb106pzw0000gn/T/blockmgr-41b2b803-e486-4737-88f3-08583729665e
  21/12/01 07:40:29 INFO MemoryStore: MemoryStore started with capacity 3.4 GiB
  21/12/01 07:40:29 INFO SparkEnv: Registering OutputCommitCoordinator
  21/12/01 07:40:29 INFO Utils: Successfully started service 'SparkUI' on port 4040.
  21/12/01 07:40:29 INFO SparkUI: Bound SparkUI to 127.0.0.1, and started at http://localhost:4040
  21/12/01 07:40:29 INFO DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at /0.0.0.0:8032
  21/12/01 07:40:30 INFO Client: Requesting a new application from cluster with 1 NodeManagers
  21/12/01 07:40:30 INFO Configuration: resource-types.xml not found
  21/12/01 07:40:30 INFO ResourceUtils: Unable to find 'resource-types.xml'.
  21/12/01 07:40:30 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
  21/12/01 07:40:30 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
  21/12/01 07:40:30 INFO Client: Setting up container launch context for our AM
  21/12/01 07:40:30 INFO Client: Setting up the launch environment for our AM container
  21/12/01 07:40:30 INFO Client: Preparing resources for our AM container
  21/12/01 07:40:30 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
  21/12/01 07:40:33 INFO Client: Uploading resource file:/private/var/folders/3n/56fpv14n4wj0c1l1sb106pzw0000gn/T/spark-0124ebbd-6b25-461c-be6e-12fbb3bb4997/__spark_libs__9757210747505021224.zip -&gt; hdfs://localhost:9000/user/davidlaxer/.sparkStaging/application_1638294072748_0006/__spark_libs__9757210747505021224.zip
  21/12/01 07:40:34 INFO Client: Uploading resource file:/Users/davidlaxer/aas/ch06-lsa/target/ch06-lsa-2.0.0-jar-with-dependencies.jar -&gt; hdfs://localhost:9000/user/davidlaxer/.sparkStaging/application_1638294072748_0006/ch06-lsa-2.0.0-jar-with-dependencies.jar
  21/12/01 07:40:34 INFO Client: Uploading resource file:/private/var/folders/3n/56fpv14n4wj0c1l1sb106pzw0000gn/T/spark-0124ebbd-6b25-461c-be6e-12fbb3bb4997/__spark_conf__17958787253444057624.zip -&gt; hdfs://localhost:9000/user/davidlaxer/.sparkStaging/application_1638294072748_0006/__spark_conf__.zip
  21/12/01 07:40:35 INFO SecurityManager: Changing view acls to: davidlaxer
  21/12/01 07:40:35 INFO SecurityManager: Changing modify acls to: davidlaxer
  21/12/01 07:40:35 INFO SecurityManager: Changing view acls groups to: 
  21/12/01 07:40:35 INFO SecurityManager: Changing modify acls groups to: 
  21/12/01 07:40:35 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(davidlaxer); groups with view permissions: Set(); users  with modify permissions: Set(davidlaxer); groups with modify permissions: Set()
  21/12/01 07:40:35 INFO Client: Submitting application application_1638294072748_0006 to ResourceManager
  21/12/01 07:40:35 INFO YarnClientImpl: Submitted application application_1638294072748_0006
  21/12/01 07:40:36 INFO Client: Application report for application_1638294072748_0006 (state: ACCEPTED)
  21/12/01 07:40:36 INFO Client: 
       client token: N/A
       diagnostics: AM container is launched, waiting for AM container to Register with RM
       ApplicationMaster host: N/A
       ApplicationMaster RPC port: -1
       queue: default
       start time: 1638373235022
       final status: UNDEFINED
       tracking URL: http://BlueDiamond.local:8088/proxy/application_1638294072748_0006/
       user: davidlaxer
  21/12/01 07:40:37 INFO Client: Application report for application_1638294072748_0006 (state: ACCEPTED)
  21/12/01 07:40:38 INFO Client: Application report for application_1638294072748_0006 (state: ACCEPTED)
  21/12/01 07:40:39 INFO Client: Application report for application_1638294072748_0006 (state: ACCEPTED)
  21/12/01 07:40:40 INFO Client: Application report for application_1638294072748_0006 (state: ACCEPTED)
  21/12/01 07:40:41 INFO Client: Application report for application_1638294072748_0006 (state: ACCEPTED)
  21/12/01 07:40:41 INFO YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -&gt; BlueDiamond.local, PROXY_URI_BASES -&gt; http://BlueDiamond.local:8088/proxy/application_1638294072748_0006), /proxy/application_1638294072748_0006
  21/12/01 07:40:42 INFO Client: Application report for application_1638294072748_0006 (state: RUNNING)
  21/12/01 07:40:42 INFO Client: 
       client token: N/A
       diagnostics: N/A
       ApplicationMaster host: 10.0.1.26
       ApplicationMaster RPC port: -1
       queue: default
       start time: 1638373235022
       final status: UNDEFINED
       tracking URL: http://BlueDiamond.local:8088/proxy/application_1638294072748_0006/
       user: davidlaxer
  21/12/01 07:40:42 INFO YarnClientSchedulerBackend: Application application_1638294072748_0006 has started running.
  21/12/01 07:40:42 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 52095.
  21/12/01 07:40:42 INFO NettyBlockTransferService: Server created on localhost:52095
  21/12/01 07:40:42 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
  21/12/01 07:40:42 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, localhost, 52095, None)
  21/12/01 07:40:42 INFO BlockManagerMasterEndpoint: Registering block manager localhost:52095 with 3.4 GiB RAM, BlockManagerId(driver, localhost, 52095, None)
  21/12/01 07:40:42 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, localhost, 52095, None)
  21/12/01 07:40:42 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, localhost, 52095, None)
  21/12/01 07:40:42 INFO ServerInfo: Adding filter to /metrics/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
  21/12/01 07:40:42 INFO SingleEventLogFileWriter: Logging events to file:/Users/davidlaxer/spark/events/application_1638294072748_0006.inprogress
  21/12/01 07:40:42 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM)
  21/12/01 07:40:46 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (127.0.0.1:52104) with ID 1,  ResourceProfileId 0
  21/12/01 07:40:46 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.1.26:52105 with 2004.6 MiB RAM, BlockManagerId(1, 10.0.1.26, 52105, None)
  21/12/01 07:40:59 INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000000000(ns)
  21/12/01 07:41:00 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 187.7 KiB, free 3.4 GiB)
  21/12/01 07:41:01 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 32.3 KiB, free 3.4 GiB)
  21/12/01 07:41:01 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:52095 (size: 32.3 KiB, free: 3.4 GiB)
  21/12/01 07:41:01 INFO SparkContext: Created broadcast 0 from newAPIHadoopFile at AssembleDocumentTermMatrix.scala:52
  21/12/01 07:41:02 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir.
  21/12/01 07:41:02 INFO SharedState: Warehouse path is 'file:/Users/davidlaxer/aas/ch06-lsa/spark-warehouse'.
  21/12/01 07:41:02 INFO ServerInfo: Adding filter to /SQL: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
  21/12/01 07:41:02 INFO ServerInfo: Adding filter to /SQL/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
  21/12/01 07:41:02 INFO ServerInfo: Adding filter to /SQL/execution: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
  21/12/01 07:41:02 INFO ServerInfo: Adding filter to /SQL/execution/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
  21/12/01 07:41:02 INFO ServerInfo: Adding filter to /static/sql: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
  21/12/01 07:41:03 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 15.1 KiB, free 3.4 GiB)
  21/12/01 07:41:03 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 919.0 B, free 3.4 GiB)
  21/12/01 07:41:03 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:52095 (size: 919.0 B, free: 3.4 GiB)
  21/12/01 07:41:03 INFO SparkContext: Created broadcast 1 from broadcast at AssembleDocumentTermMatrix.scala:90
  21/12/01 07:41:03 INFO CodeGenerator: Code generated in 150.307621 ms
  21/12/01 07:41:03 INFO FileInputFormat: Total input files to process : 1
  21/12/01 07:41:03 INFO SparkContext: Starting job: count at CountVectorizer.scala:233
  21/12/01 07:41:03 INFO DAGScheduler: Registering RDD 11 (flatMap at CountVectorizer.scala:212) as input to shuffle 0
  21/12/01 07:41:03 INFO DAGScheduler: Got job 0 (count at CountVectorizer.scala:233) with 155 output partitions
  21/12/01 07:41:03 INFO DAGScheduler: Final stage: ResultStage 1 (count at CountVectorizer.scala:233)
  21/12/01 07:41:03 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
  21/12/01 07:41:03 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
  21/12/01 07:41:03 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[11] at flatMap at CountVectorizer.scala:212), which has no missing parents
  21/12/01 07:41:04 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 37.1 KiB, free 3.4 GiB)
  21/12/01 07:41:04 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 16.7 KiB, free 3.4 GiB)
  21/12/01 07:41:04 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:52095 (size: 16.7 KiB, free: 3.4 GiB)
  21/12/01 07:41:04 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1427
  21/12/01 07:41:04 INFO DAGScheduler: Submitting 155 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[11] at flatMap at CountVectorizer.scala:212) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
  21/12/01 07:41:04 INFO YarnScheduler: Adding task set 0.0 with 155 tasks resource profile 0
  21/12/01 07:41:04 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (10.0.1.26, executor 1, partition 0, PROCESS_LOCAL, 4550 bytes) taskResourceAssignments Map()
  21/12/01 07:41:04 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1) (10.0.1.26, executor 1, partition 1, PROCESS_LOCAL, 4550 bytes) taskResourceAssignments Map()
  21/12/01 07:41:04 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.0.1.26:52105 (size: 16.7 KiB, free: 2004.6 MiB)
  21/12/01 07:41:07 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.1.26:52105 (size: 32.3 KiB, free: 2004.6 MiB)
  21/12/01 07:41:09 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.1.26:52105 (size: 919.0 B, free: 2004.6 MiB)
  21/12/01 08:12:07 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2) (10.0.1.26, executor 1, partition 2, PROCESS_LOCAL, 4550 bytes) taskResourceAssignments Map()
  21/12/01 08:12:07 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 1863054 ms on 10.0.1.26 (executor 1) (1/155)
  21/12/01 08:17:47 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3) (10.0.1.26, executor 1, partition 3, PROCESS_LOCAL, 4550 bytes) taskResourceAssignments Map()
  21/12/01 08:17:47 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 2203259 ms on 10.0.1.26 (executor 1) (2/155)
  21/12/01 08:46:00 INFO TaskSetManager: Starting task 4.0 in stage 0.0 (TID 4) (10.0.1.26, executor 1, partition 4, PROCESS_LOCAL, 4550 bytes) taskResourceAssignments Map()
  21/12/01 08:46:00 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 2033542 ms on 10.0.1.26 (executor 1) (3/155)
  21/12/01 09:01:48 INFO TaskSetManager: Starting task 5.0 in stage 0.0 (TID 5) (10.0.1.26, executor 1, partition 5, PROCESS_LOCAL, 4550 bytes) taskResourceAssignments Map()
  21/12/01 09:01:48 INFO TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 2641608 ms on 10.0.1.26 (executor 1) (4/155)
  21/12/01 09:36:05 INFO TaskSetManager: Starting task 6.0 in stage 0.0 (TID 6) (10.0.1.26, executor 1, partition 6, PROCESS_LOCAL, 4550 bytes) taskResourceAssignments Map()
  21/12/01 09:36:05 INFO TaskSetManager: Finished task 4.0 in stage 0.0 (TID 4) in 3005295 ms on 10.0.1.26 (executor 1) (5/155)
  21/12/01 09:36:58 INFO TaskSetManager: Starting task 7.0 in stage 0.0 (TID 7) (10.0.1.26, executor 1, partition 7, PROCESS_LOCAL, 4550 bytes) taskResourceAssignments Map()
  21/12/01 09:36:58 INFO TaskSetManager: Finished task 5.0 in stage 0.0 (TID 5) in 2109400 ms on 10.0.1.26 (executor 1) (6/155)
  21/12/01 10:09:55 INFO TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8) (10.0.1.26, executor 1, partition 8, PROCESS_LOCAL, 4550 bytes) taskResourceAssignments Map()
  21/12/01 10:09:55 INFO TaskSetManager: Finished task 7.0 in stage 0.0 (TID 7) in 1977417 ms on 10.0.1.26 (executor 1) (7/155)
  21/12/01 10:18:35 INFO TaskSetManager: Starting task 9.0 in stage 0.0 (TID 9) (10.0.1.26, executor 1, partition 9, PROCESS_LOCAL, 4550 bytes) taskResourceAssignments Map()
  21/12/01 10:18:35 INFO TaskSetManager: Finished task 6.0 in stage 0.0 (TID 6) in 2549559 ms on 10.0.1.26 (executor 1) (8/155)
  21/12/01 10:42:02 INFO TaskSetManager: Starting task 10.0 in stage 0.0 (TID 10) (10.0.1.26, executor 1, partition 10, PROCESS_LOCAL, 4550 bytes) taskResourceAssignments Map()
  21/12/01 10:42:02 INFO TaskSetManager: Finished task 9.0 in stage 0.0 (TID 9) in 1406945 ms on 10.0.1.26 (executor 1) (9/155)
  21/12/01 10:54:55 INFO TaskSetManager: Starting task 11.0 in stage 0.0 (TID 11) (10.0.1.26, executor 1, partition 11, PROCESS_LOCAL, 4550 bytes) taskResourceAssignments Map()
  21/12/01 10:54:55 INFO TaskSetManager: Finished task 8.0 in stage 0.0 (TID 8) in 2699743 ms on 10.0.1.26 (executor 1) (10/155)
  21/12/01 11:12:09 INFO TaskSetManager: Starting task 12.0 in stage 0.0 (TID 12) (10.0.1.26, executor 1, partition 12, PROCESS_LOCAL, 4550 bytes) taskResourceAssignments Map()
  21/12/01 11:12:09 INFO TaskSetManager: Finished task 10.0 in stage 0.0 (TID 10) in 1806639 ms on 10.0.1.26 (executor 1) (11/155)
  21/12/01 11:24:13 INFO TaskSetManager: Starting task 13.0 in stage 0.0 (TID 13) (10.0.1.26, executor 1, partition 13, PROCESS_LOCAL, 4550 bytes) taskResourceAssignments Map()
  21/12/01 11:24:13 INFO TaskSetManager: Finished task 11.0 in stage 0.0 (TID 11) in 1757634 ms on 10.0.1.26 (executor 1) (12/155)
  21/12/01 11:46:33 INFO TaskSetManager: Starting task 14.0 in stage 0.0 (TID 14) (10.0.1.26, executor 1, partition 14, PROCESS_LOCAL, 4550 bytes) taskResourceAssignments Map()
  21/12/01 11:46:33 INFO TaskSetManager: Finished task 12.0 in stage 0.0 (TID 12) in 2064644 ms on 10.0.1.26 (executor 1) (13/155)
  21/12/01 11:48:49 INFO TaskSetManager: Starting task 15.0 in stage 0.0 (TID 15) (10.0.1.26, executor 1, partition 15, PROCESS_LOCAL, 4550 bytes) taskResourceAssignments Map()
  21/12/01 11:48:49 INFO TaskSetManager: Finished task 13.0 in stage 0.0 (TID 13) in 1476664 ms on 10.0.1.26 (executor 1) (14/155)
  [Stage 0:=====&gt;                                                  (14 + 2) / 155]
  ...
  </code></pre>
  <p>Code snippet:</p>
  <pre><code>object RunLSA {
    def main(args: Array[String]): Unit = {
  
      sys.props.update(&quot;spark.ui.proxyBase&quot;, &quot;&quot;)
  
      val k = if (args.length &gt; 0) args(0).toInt else 100
      val numTerms = if (args.length &gt; 1) args(1).toInt else 20000
  
      val spark = SparkSession.builder().config(&quot;spark.serializer&quot;, classOf[KryoSerializer].getName).getOrCreate()
      val assembleMatrix = new AssembleDocumentTermMatrix(spark)
      import assembleMatrix._
  
  </code></pre>