Update nexus wiki content

wiki/entities/Apache-Spark.md

@@ -0,0 +1,51 @@
+---
+title: "Apache Spark"
+type: entity
+tags: [data-engineering, big-data, processing-engine]
+sources: [engineering-data-engineer]
+last_updated: 2026-05-02
+---
+
+## Overview
+
+Apache Spark 是统一的大规模数据处理引擎，支持批处理、流处理、机器学习和 SQL 查询。Data Engineer Agent 使用 PySpark（Spark 的 Python API）作为核心计算平台，构建 Bronze→Silver→Gold ETL/ELT 管道。
+
+## Key Capabilities for Data Engineering
+
+### PySpark Data Pipeline
+```python
+from pyspark.sql import SparkSession
+from pyspark.sql.functions import col, current_timestamp, sha2, concat_ws, lit
+from delta.tables import DeltaTable
+
+spark = SparkSession.builder \
+    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
+    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
+    .getOrCreate()
+```
+
+### Delta Lake Integration
+- Spark + Delta Lake 是 Medallion Architecture 的标准实现组合
+- 支持 `mergeSchema=true` 处理 schema 演进
+- 支持 `MERGE INTO` 实现幂等 upsert
+
+### Streaming
+- Spark Structured Streaming + Kafka：构建 Exactly-Once 语义的实时管道
+- 触发模式：Continuous（连续处理）或 Micro-batch（微批次）
+
+## Performance Features
+
+- **Adaptive Query Execution (AQE)**：动态分区合并、Broadcast Join 优化
+- **Z-Ordering**：多维聚类加速复合过滤查询
+- **Bloom Filters**：高基数字符串列（ID、邮箱）的文件跳过
+
+## Managed Platforms
+- [[Databricks]]（Unity Catalog、DLT、Workflows）
+- [[Amazon-RDS]] / EMR（AWS Spark 托管）
+- Google Dataproc（GCP Spark 托管）
+
+## Related Concepts
+- [[Medallion Architecture]]
+- [[Delta Lake]]
+- [[Apache Kafka]]
+- [[CDC (Change Data Capture)]]