Data Engineer / Senior Data Engineer

About the Role

Cyble's threat intelligence platform ingests cybercrime data at petabyte scale: stealer logs, breach dumps, dark web forums and marketplaces, Telegram and Discord channels, ransomware leak sites, and domain intelligence. That data feeds our AI agents and analyst tooling, and it is only useful when it is clean, fresh, deduplicated, and retrievable.

We're hiring a Data Engineer /Senior Data Engineer to build and run the Iceberg-based data lake that powers Cyble's AI agents. You'll own pipelines end to end, from raw scraped and leaked data landing in S3 to curated lake tables, OpenSearch indexes, and retrieval-ready datasets that agents and products query. You'll work in a small, low-process team where engineers own their domains, ship directly, and use AI coding tools as a normal part of the workflow.

What You’ll Do at Cyble:

• Build and operate the data lake powering Cyble's AI agents: ingestion, Data modeling, deduplication, freshness, lineage, and retrieval-ready outputs.
• Design and run PySpark pipelines on EMR over Apache Iceberg on S3 (Glue Data Catalog), including schema evolution, partitioning, compaction, snapshot expiry, and small-file control.
• Ingest scraped, crawled, and leaked data from heterogeneous sources (HTML, JSON, CSV, text dumps, archives, feeds, APIs) and handle the realities: encoding issues, schema drift, malformed records, duplicates, and re-crawled content.
• Build and tune OpenSearch/Elasticsearch indexing pipelines: mappings, analyzers, shard and index lifecycle strategy, bulk ingestion, and query performance.
• Build the data side of RAG and agent workloads: chunking, embedding generation, vector and hybrid (keyword + kNN) indexing, and datasets the agents can trust.
• Run streaming and event-driven pipelines on Kafka; orchestrate batch workloads with Airflow.
• Deploy and operate data workloads on Kubernetes (EKS) and ECS, including Spark jobs, Trino, workers, and supporting services.
• Own lake and pipeline reliability: monitoring, alerting, data quality checks, backfills and reprocessing, retention, and storage/compute cost control.
• Use AI coding assistants and agents (Claude Code, Cursor, Copilot, or similar) daily to build, refactor, test, debug, and document, and review AI-generated code critically before it reaches production.
• Work with security researchers, data scientists, and product engineers on data models for new intelligence products.

What You’ll Need:

• Bachelor's degree in computer science, engineering, or equivalent experience.
• 3+ years building production data pipelines in Python and Spark.
• Deep Spark knowledge on AWS EMR: shuffle, skew, partitioning, memory tuning, AQE, and cost-aware cluster sizing.
• Hands-on experience with Apache Iceberg (or Delta/Hudi with willingness to go deep on Iceberg): table design, partition evolution, compaction, snapshot management, time travel.
• Experience building and operating a data lake or lakehouse that serves downstream AI/ML, search, or agent workloads.
• Strong OpenSearch/Elasticsearch experience: pipeline integration, index design, and performance tuning at scale.
• Experience ingesting and normalizing scraped or semi-structured data at scale, including deduplication and schema-drift handling.
• Working knowledge of the AWS data stack (S3, Glue, EMR, Lambda, IAM) and containerized deployment on EKS and/or ECS.
• Hands-on with PostgreSQL (indexing, query plans, large-table performance) and MongoDB, with the judgment to choose between relational, document, and search stores.
• Working understanding of RAG architectures: embeddings, vector stores (OpenSearch kNN, pgvector, or similar), chunking strategies, hybrid retrieval, reranking.
• Experience with Kafka (or similar) and Airflow in production.
• Hands-on experience using AI coding tools in real data engineering work, with a track record of catching their mistakes.
• Ability to explain technical trade-offs clearly to non-technical stakeholders, and comfort owning a domain end to end in a lean team.

Bonus Points If You Have:

• Trino/Presto on Iceberg, especially on Kubernetes.
• AI/agent-based context extraction in pipelines: LLMs or NER models (e.g., GLiNER) for entity extraction, classification, and enrichment, and their cost and accuracy at scale.
• LLM serving and inference (vLLM or similar) and evaluating retrieval or extraction quality (recall, precision/recall on labeled samples).
• OCR and document extraction (Tesseract, PaddleOCR, Textract, or vision-language models) for images, scans, and PDFs.
• Building data access layers for agents (MCP servers, retrieval APIs).
• Graph processing (GraphFrames, Neo4j) for entity relationship mapping.
• Web scraping frameworks (Scrapy, Playwright) and anti-bot handling, or working downstream of such systems.
• Infrastructure-as-code (Terraform) and CI/CD for data pipelines.
• Prior exposure to cybersecurity or threat intelligence data.

If you like working in an inclusive environment, you want to advance your career quickly, and your opinion is valued, look no further than Cyble, Inc. We are young, hungry, and ready to impact the cyber security landscape!

Cyble, Inc. takes into consideration an individual’s skillset, experience and location in making final salary determination.

All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, protected Veteran status age, or genetics, or any other characteristic protected by law.