Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Desactivate Spark-UI in the Tests #12

Open
ahoy-jon opened this issue May 26, 2019 · 1 comment
Open

Desactivate Spark-UI in the Tests #12

ahoy-jon opened this issue May 26, 2019 · 1 comment

Comments

@ahoy-jon
Copy link
Member

When we run sbt test, we have a couple of warnings/errors related to the SparkUI.

19/05/26 17:53:55 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/05/26 17:53:55 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/05/26 17:53:55 WARN AbstractLifeCycle: FAILED Spark@53f74852{HTTP/1.1}{0.0.0.0:4040}: java.net.BindException: Address already in use
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.spark_project.jetty.server.ServerConnector.open(ServerConnector.java:321)
        at org.spark_project.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
        at org.spark_project.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
        at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.spark_project.jetty.server.Server.doStart(Server.java:366)
        at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:365)
        at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:375)
        at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:375)
        at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:2213)
        at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
        at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2204)
        at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:375)
        at org.apache.spark.ui.WebUI.bind(WebUI.scala:130)
        at org.apache.spark.SparkContext$$anonfun$10.apply(SparkContext.scala:460)
        at org.apache.spark.SparkContext$$anonfun$10.apply(SparkContext.scala:460)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:460)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2320)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
        at io.univalence.sparkzio.SparkEnvImplicitClassTest.<init>(SparkEnvImplicitClassTest.scala:11)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:435)
        at sbt.TestRunner.runTest$1(TestFramework.scala:113)
        at sbt.TestRunner.run(TestFramework.scala:124)
        at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:282)
        at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:246)
        at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282)
        at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282)
        at sbt.TestFunction.apply(TestFramework.scala:294)
        at sbt.Tests$.processRunnable$1(Tests.scala:347)
        at sbt.Tests$.$anonfun$makeSerial$1(Tests.scala:353)
        at sbt.std.Transform$$anon$3.$anonfun$apply$2(System.scala:46)
        at sbt.std.Transform$$anon$4.work(System.scala:67)
        at sbt.Execute.$anonfun$submit$2(Execute.scala:269)
        at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
        at sbt.Execute.work(Execute.scala:278)
        at sbt.Execute.$anonfun$submit$1(Execute.scala:269)
        at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
        at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
19/05/26 17:53:55 WARN AbstractLifeCycle: FAILED org.spark_project.jetty.server.Server@512d4ee7: java.net.BindException: Address already in use
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.spark_project.jetty.server.ServerConnector.open(ServerConnector.java:321)
        at org.spark_project.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
        at org.spark_project.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
        at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.spark_project.jetty.server.Server.doStart(Server.java:366)
        at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:365)
        at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:375)
        at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:375)
        at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:2213)
        at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
        at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2204)
        at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:375)
        at org.apache.spark.ui.WebUI.bind(WebUI.scala:130)
        at org.apache.spark.SparkContext$$anonfun$10.apply(SparkContext.scala:460)
        at org.apache.spark.SparkContext$$anonfun$10.apply(SparkContext.scala:460)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:460)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2320)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
        at io.univalence.sparkzio.SparkEnvImplicitClassTest.<init>(SparkEnvImplicitClassTest.scala:11)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:435)
        at sbt.TestRunner.runTest$1(TestFramework.scala:113)
        at sbt.TestRunner.run(TestFramework.scala:124)
        at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:282)
        at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:246)
        at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282)
        at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282)
        at sbt.TestFunction.apply(TestFramework.scala:294)
        at sbt.Tests$.processRunnable$1(Tests.scala:347)
        at sbt.Tests$.$anonfun$makeSerial$1(Tests.scala:353)
        at sbt.std.Transform$$anon$3.$anonfun$apply$2(System.scala:46)
        at sbt.std.Transform$$anon$4.work(System.scala:67)
        at sbt.Execute.$anonfun$submit$2(Execute.scala:269)
        at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
        at sbt.Execute.work(Execute.scala:278)
        at sbt.Execute.$anonfun$submit$1(Execute.scala:269)
        at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
        at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
19/05/26 17:53:55 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
19/05/26 17:53:57 WARN AbstractLifeCycle: FAILED Spark@656dcca8{HTTP/1.1}{0.0.0.0:4040}: java.net.BindException: Address already in use
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.spark_project.jetty.server.ServerConnector.open(ServerConnector.java:321)
        at org.spark_project.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
        at org.spark_project.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
        at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.spark_project.jetty.server.Server.doStart(Server.java:366)
        at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:365)
        at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:375)
        at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:375)
        at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:2213)
        at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
        at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2204)
        at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:375)
        at org.apache.spark.ui.WebUI.bind(WebUI.scala:130)
        at org.apache.spark.SparkContext$$anonfun$10.apply(SparkContext.scala:460)
        at org.apache.spark.SparkContext$$anonfun$10.apply(SparkContext.scala:460)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:460)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2320)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
        at io.univalence.centrifuge.ExecutorTest.<init>(ExecutorTest.scala:24)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:435)
        at sbt.TestRunner.runTest$1(TestFramework.scala:113)
        at sbt.TestRunner.run(TestFramework.scala:124)
        at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:282)
        at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:246)
        at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282)
        at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282)
        at sbt.TestFunction.apply(TestFramework.scala:294)
        at sbt.Tests$.processRunnable$1(Tests.scala:347)
        at sbt.Tests$.$anonfun$makeSerial$1(Tests.scala:353)
        at sbt.std.Transform$$anon$3.$anonfun$apply$2(System.scala:46)
        at sbt.std.Transform$$anon$4.work(System.scala:67)
        at sbt.Execute.$anonfun$submit$2(Execute.scala:269)
        at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
        at sbt.Execute.work(Execute.scala:278)
        at sbt.Execute.$anonfun$submit$1(Execute.scala:269)
        at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
        at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
19/05/26 17:53:57 WARN AbstractLifeCycle: FAILED org.spark_project.jetty.server.Server@7e49b80f: java.net.BindException: Address already in use
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.spark_project.jetty.server.ServerConnector.open(ServerConnector.java:321)
        at org.spark_project.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
        at org.spark_project.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
        at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.spark_project.jetty.server.Server.doStart(Server.java:366)
        at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:365)
        at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:375)
        at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:375)
        at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:2213)
        at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
        at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2204)
        at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:375)
        at org.apache.spark.ui.WebUI.bind(WebUI.scala:130)
        at org.apache.spark.SparkContext$$anonfun$10.apply(SparkContext.scala:460)
        at org.apache.spark.SparkContext$$anonfun$10.apply(SparkContext.scala:460)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:460)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2320)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
        at io.univalence.centrifuge.ExecutorTest.<init>(ExecutorTest.scala:24)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:435)
        at sbt.TestRunner.runTest$1(TestFramework.scala:113)
        at sbt.TestRunner.run(TestFramework.scala:124)
        at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:282)
        at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:246)
        at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282)
        at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282)
        at sbt.TestFunction.apply(TestFramework.scala:294)
        at sbt.Tests$.processRunnable$1(Tests.scala:347)
        at sbt.Tests$.$anonfun$makeSerial$1(Tests.scala:353)
        at sbt.std.Transform$$anon$3.$anonfun$apply$2(System.scala:46)
        at sbt.std.Transform$$anon$4.work(System.scala:67)
        at sbt.Execute.$anonfun$submit$2(Execute.scala:269)
        at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
        at sbt.Execute.work(Execute.scala:278)
        at sbt.Execute.$anonfun$submit$1(Execute.scala:269)
        at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
        at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
19/05/26 17:53:57 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
19/05/26 17:53:57 WARN AbstractLifeCycle: FAILED Spark@106deb8a{HTTP/1.1}{0.0.0.0:4041}: java.net.BindException: Address already in use
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.spark_project.jetty.server.ServerConnector.open(ServerConnector.java:321)
        at org.spark_project.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
        at org.spark_project.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
        at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.spark_project.jetty.server.Server.doStart(Server.java:366)
        at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:365)
        at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:375)
        at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:375)
        at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:2213)
        at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
        at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2204)
        at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:375)
        at org.apache.spark.ui.WebUI.bind(WebUI.scala:130)
        at org.apache.spark.SparkContext$$anonfun$10.apply(SparkContext.scala:460)
        at org.apache.spark.SparkContext$$anonfun$10.apply(SparkContext.scala:460)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:460)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2320)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
        at io.univalence.centrifuge.ExecutorTest.<init>(ExecutorTest.scala:24)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:435)
        at sbt.TestRunner.runTest$1(TestFramework.scala:113)
        at sbt.TestRunner.run(TestFramework.scala:124)
        at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:282)
        at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:246)
        at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282)
        at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282)
        at sbt.TestFunction.apply(TestFramework.scala:294)
        at sbt.Tests$.processRunnable$1(Tests.scala:347)
        at sbt.Tests$.$anonfun$makeSerial$1(Tests.scala:353)
        at sbt.std.Transform$$anon$3.$anonfun$apply$2(System.scala:46)
        at sbt.std.Transform$$anon$4.work(System.scala:67)
        at sbt.Execute.$anonfun$submit$2(Execute.scala:269)
        at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
        at sbt.Execute.work(Execute.scala:278)
        at sbt.Execute.$anonfun$submit$1(Execute.scala:269)
        at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
        at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
19/05/26 17:53:57 WARN AbstractLifeCycle: FAILED org.spark_project.jetty.server.Server@71a82e82: java.net.BindException: Address already in use
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.spark_project.jetty.server.ServerConnector.open(ServerConnector.java:321)
        at org.spark_project.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
        at org.spark_project.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
        at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.spark_project.jetty.server.Server.doStart(Server.java:366)
        at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:365)
        at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:375)
        at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:375)
        at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:2213)
        at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
        at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2204)
        at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:375)
        at org.apache.spark.ui.WebUI.bind(WebUI.scala:130)
        at org.apache.spark.SparkContext$$anonfun$10.apply(SparkContext.scala:460)
        at org.apache.spark.SparkContext$$anonfun$10.apply(SparkContext.scala:460)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:460)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2320)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
        at io.univalence.centrifuge.ExecutorTest.<init>(ExecutorTest.scala:24)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:435)
        at sbt.TestRunner.runTest$1(TestFramework.scala:113)
        at sbt.TestRunner.run(TestFramework.scala:124)
        at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:282)
        at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:246)
        at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282)
        at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282)
        at sbt.TestFunction.apply(TestFramework.scala:294)
        at sbt.Tests$.processRunnable$1(Tests.scala:347)
        at sbt.Tests$.$anonfun$makeSerial$1(Tests.scala:353)
        at sbt.std.Transform$$anon$3.$anonfun$apply$2(System.scala:46)
        at sbt.std.Transform$$anon$4.work(System.scala:67)
        at sbt.Execute.$anonfun$submit$2(Execute.scala:269)
        at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
        at sbt.Execute.work(Execute.scala:278)
        at sbt.Execute.$anonfun$submit$1(Execute.scala:269)
        at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
        at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
19/05/26 17:53:57 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.
@ahoy-jon
Copy link
Member Author

We could use SparkTest on the other libs ! #eating your own dog food

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ahoy-jon