spark-python-data-source
Skill from databricks/databricks-agent-skills
Install
npx skills add databricks/databricks-agent-skills --skill spark-python-data-source
Runs the open skills CLI and installs the skill for your agent (Claude Code, Codex, Cursor …). Alternatively copy the skill folder into .claude/skills/ or upload it in Claude.ai under Settings → Skills.
What does the spark-python-data-source skill do?
Description from the SKILL.md in the source repository.
Works with Databricks
Questions about spark-python-data-source
What does the spark-python-data-source skill do?
Build custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers/writers for external systems. Use this skill whenever someone wants to connect Spark to an external system (database, API, message queue, custom protocol), build a Spark connector or plugin in Python, implement a DataSourceReader or DataSourceWriter, pull data from or push data to a system via Spark, or work with the PySpark DataSource API in any way. Even if they just say "read from X in Spark" or "write DataFrame to Y" and there's no native connector, this skill applies.
How do I install spark-python-data-source?
In a terminal: npx skills add databricks/databricks-agent-skills --skill spark-python-data-source. The skills CLI asks which agents to install it for. In Claude.ai you can upload the skill folder as a ZIP under Settings → Skills.
Which agents can use spark-python-data-source?
Skills use the open SKILL.md format, which Claude Code, Claude.ai, the Claude API, OpenAI Codex, Cursor, Gemini CLI and other agents support.
Is spark-python-data-source safe to use?
Our automatic check found no risky patterns (piped shell scripts, hidden characters, secret access, prompt overrides). It is not a full audit – read the SKILL.md and any scripts before installing.
Which tool is spark-python-data-source for?
spark-python-data-source comes from the makers of Databricks and helps your agent work with it.
More skills from databricks/databricks-agent-skills
Databricks Model Serving endpoint lifecycle and ops. Use when asked to: CRUD serving endpoints (CLI or MLflow Deployments client); configure traffic routing for A/B / canary deploys and zero-downtime version swaps; retrieve …
Migrate Databricks workloads from classic compute to serverless compute. Use when migrating notebooks, jobs, pipelines, or Scala JARs (`spark_jar_task`) from classic clusters to serverless, checking if existing code is serverless-compatible, or writing …
Databricks Sandbox: on-demand serverless compute for running untrusted or agent-generated code and background tasks. Create/start/stop/delete sandboxes, run commands (synchronous execute_command_sync in all SDKs; streaming execute_command in the Python SDK only), and manage …
Databricks development guidance including Python SDK, Databricks Connect, CLI, and REST API. Use when working with databricks-sdk, databricks-connect, or Databricks APIs.
Develop Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables) on Databricks. Use when building batch or streaming data pipelines with Python or SQL, including Auto CDC from event streams or periodic complete …
MLflow GenAI evaluation and agent improvement on Databricks. Use when running mlflow.genai.evaluate(), building scorers or datasets, aligning judges with MemAlign, optimizing prompts, or monitoring production quality.
Train ML models on Databricks. Use for: classification/regression/deep-learning (XGBoost, scikit-learn, LightGBM, PyTorch) with Optuna, @prod/@challenger aliases, batch scoring (spark_udf for plain models, fe.score_batch for feature-store-backed), custom PyFunc, custom ResponsesAgent (LangGraph + UC …
Serverless compute for Databricks jobs, Lakeflow pipelines and Declarative Automation Bundles (DABs), with STANDARD or PERFORMANCE_OPTIMIZED and classic only where serverless cannot run the workload.
Related skills in Development
Scan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.
Diagnosis loop for hard bugs and performance regressions. Use when the user says "diagnose"/"debug this", or reports something broken/throwing/failing/slow.
Review the changes since a fixed point (commit, branch, tag, or merge-base) along two axes: Standards (does the code follow this repo's documented coding standards?) and Spec (does the code match what …
Investigate a question against high-trust primary sources and capture the findings as a Markdown file in the repo.
Guidance for instrumenting webapps with Azure Application Insights. Provides telemetry patterns, SDK setup, and configuration references.
Analyze Azure resource groups and generate detailed Mermaid architecture diagrams showing the relationships between individual resources. WHEN: create architecture diagram, visualize Azure resources, show resource relationships, generate Mermaid diagram, analyze resource group …
Configure Azure API Management as an AI Gateway for AI models, MCP tools, and agents. WHEN: semantic caching, token limit, content safety, load balancing, AI model governance, MCP rate limiting, jailbreak detection …
Troubleshoot and resolve issues with Azure Messaging SDKs for Event Hubs and Service Bus. Covers connection failures, authentication errors, message processing issues, and SDK configuration problems.