I tried everything including sudo, but it still did not work using the local directory.
However, I finally got it working by getting the history server to log into hdfs.
I first created a directory in hdfs like the following:
./ephemeral-hdfs/bin/hadoop fs -mkdir /spark_logs
Then I started the started the history server like the following.