Re: Build customized resource manager

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Build customized resource manager

Klaus Ma
Any suggestions?

- Klaus

On Mon, Nov 4, 2019 at 5:04 PM Klaus Ma <[hidden email]> wrote:
Hi team,

AFAIK, we built k8s/yarn/mesos as resource manager; but I'd like to did some enhancement to them, e.g. integrate with Volcano in k8s. Is that possible to do that without fork the whole spark project? For example, enable customized resource manager with configuration, e.g. replace `org.apache.spark.deploy.k8s.submit.KubernetesClientApplication` with `MyK8SClient`, so I can only maintain the resource manager instead of the whole project.

-- Klaus
Reply | Threaded
Open this post in threaded view
|

Re: Build customized resource manager

Tom Graves-2
I don't know if it all works but some work was done to make cluster manager pluggable, see SPARK-13904.

Tom

On Wednesday, November 6, 2019, 07:22:59 PM CST, Klaus Ma <[hidden email]> wrote:


Any suggestions?

- Klaus

On Mon, Nov 4, 2019 at 5:04 PM Klaus Ma <[hidden email]> wrote:
Hi team,

AFAIK, we built k8s/yarn/mesos as resource manager; but I'd like to did some enhancement to them, e.g. integrate with Volcano in k8s. Is that possible to do that without fork the whole spark project? For example, enable customized resource manager with configuration, e.g. replace `org.apache.spark.deploy.k8s.submit.KubernetesClientApplication` with `MyK8SClient`, so I can only maintain the resource manager instead of the whole project.

-- Klaus
Reply | Threaded
Open this post in threaded view
|

Re: Build customized resource manager

Klaus Ma
hm.... that'll be better to me if we can build customized resource manager out of core; otherwise, we have to go through the long discussion in the community :)
But if we support that, why still mesos/yarn/k8s resource manager there in the tree?

On Fri, Nov 8, 2019 at 10:18 PM Tom Graves <[hidden email]> wrote:
I don't know if it all works but some work was done to make cluster manager pluggable, see SPARK-13904.

Tom

On Wednesday, November 6, 2019, 07:22:59 PM CST, Klaus Ma <[hidden email]> wrote:


Any suggestions?

- Klaus

On Mon, Nov 4, 2019 at 5:04 PM Klaus Ma <[hidden email]> wrote:
Hi team,

AFAIK, we built k8s/yarn/mesos as resource manager; but I'd like to did some enhancement to them, e.g. integrate with Volcano in k8s. Is that possible to do that without fork the whole spark project? For example, enable customized resource manager with configuration, e.g. replace `org.apache.spark.deploy.k8s.submit.KubernetesClientApplication` with `MyK8SClient`, so I can only maintain the resource manager instead of the whole project.

-- Klaus