The problem is when spark initailizes I see the following error:
io.fabric8.kubernetes.client.KubernetesClientException: pods is forbidden:
User "system:serviceaccount:default:default" cannot watch resource "pods" in
API group "" in the namespace "spark"
If I deploy my "driver" notebook pod in the spark namespace then things improve slightly:
" Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:spark:default" cannot list resource "pods"
Again, I don't want spark:default I want spark:spark for the service account. Why aren't my configuration parameters taking?
For the pour soul that reads this thread and runs into the same issue, the fix is to set the serviceAccount in your deployment for the pod to "spark". I'm not sure why this has to be done but it works.