Salesforce details Spark resource optimization for 8M daily jobs
Salesforce's Data 360 team runs about eight million Apache Spark jobs daily and found most workloads use more computing resources than needed. By comparing requested resources to actual usage, they could spot when CPUs and memory sat idle despite jobs finishing successfully. To gather this data without changing application code, they used their Data Processing Controller platform as a central point for Spark telemetry. The result: more efficient workloads without risking job failures.
- Data 360 runs about 8M Spark jobs daily
- Many workloads over-allocate resources without failing
- Telemetry collected via central Data Processing Controller
- No app code changes needed for efficiency gains
Sources covering this
More in Enterprise
Windows 11 23H2 Enterprise support ends in 30 days
Microsoft just announced that Windows 11 version 23H2 Enterprise, Education, and IoT Enterprise will stop getting security updates in…
Developer releases open-source clones of Office apps
Brandon Thomas, creator of open-source alternatives to Adobe's creative software, has released new free clones of Microsoft Word, Excel,…
Enterprises shift focus from AI experimentation to measurable returns
A report from consultancy ISG finds that while enterprise investment in AI remains strong, companies are prioritizing clear returns and…
AI pushes data centers to rethink power and cooling
The rapid rise of AI is surpassing what traditional data centers can handle, putting strain on everything from power supply and cooling…