=================================================================================

For establishing simple clusters in Apache Spark, the Spark Standalone cluster manager is often the most suitable choice. It's specifically designed to manage Spark clusters and is relatively straightforward to set up for smaller or simpler deployments. Cluster parallelism increases due to the direct consequence of increasing the number of executors and cores in a distributed computing environment.

We can use the command --total-executor-cores followed by the number of cores to specify the number of executor cores for an Apache Spark standalone cluster for an application.

The command --executor-cores followed by the number of cores specifies the number of executor cores for an Apache Spark Standalone cluster per executor process.

Figure 3309 shows, in Spark, a cluster consists of one or more nodes, with each node possibly running multiple worker processes. These workers are responsible for executing the tasks of a Spark job. Each worker has a defined number of cores, which can be set through Spark's configuration settings. The number of cores assigned to each worker determines how many tasks it can handle concurrently. Limiting the number of cores per worker can help in managing the workload and preventing a single worker from becoming a bottleneck due to excessive simultaneous tasks. Therefore, workers in the cluster contain a limited number of cores.

Figure 3309. Spark cluster which contains multiple nodes and workers (code).

===========================================