Data Discovery Engineer

Data Discovery Engineer

We are looking for a strong hands-on data/platform engineer who can be trained on BigID. Prior BigID experience is preferred but not required. The ideal candidate will already have a strong technical foundation across databases, SharePoint/Microsoft 365, APIs, Linux, networking, authentication, cloud platforms, and enterprise troubleshooting.

This assignment will support enterprise data discovery, classification, labeling, and remediation activities using BigID, with significant focus on Microsoft 365 and SharePoint environments. The engineer will be expected to work across structured and unstructured data sources, support BigID-to-Purview labeling workflows, troubleshoot scanning and connectivity issues, analyze scan results, and help operationalize BigID as an enterprise data discovery platform.
 

Key Responsibilities

• Onboard and configure enterprise data sources for BigID scanning and discovery;
• Configure and execute scans across structured, semi-structured, and unstructured data sources;
• Support SharePoint Online and Microsoft 365 scanning use cases, including site/library/folder scoping, permissions, metadata, and file-based discovery;
• Troubleshoot data source connectivity, authentication, permissions, scanner, network, API, and performance issues;
• Analyze scan results and identify opportunities to improve coverage, accuracy, and performance;
• Support configuration and tuning of classifiers, scan profiles, and sensitivity classification logic;
• Support Microsoft Purview / Microsoft Information Protection labeling integrations;
• Assist with BigID-to-Purview label mapping and labeling scan configuration;
• Validate labeling outcomes and troubleshoot labeling failures;
• Work with SharePoint, Purview, security, infrastructure, and data source owners to resolve technical issues;
• Validate service accounts, application registrations, roles, permissions, certificates, secrets, and connectivity required for scanning and labeling;
• Review logs and technical errors to isolate root causes;
• Support automation through REST APIs and scripting;
• Produce clear technical documentation, runbooks, troubleshooting notes, and scan reports;
• Support BigID platform operations and coordinate with BigID support when necessary;
• Participate in knowledge transfer and BigID training to become a hands-on BigID engineer.

Required Skills / Experience

• 3+ years of hands-on experience in data engineering, platform engineering, cloud engineering, Microsoft 365 engineering, database engineering, or a similar technical role;
• Strong SQL skills and solid understanding of relational database concepts;
• Hands-on experience with enterprise data platforms and databases such as:
  • SQL Server, Oracle, PostgreSQL, MySQL, Snowflake, Databricks etc.
• Strong experience with SharePoint Online and/or Microsoft 365.
• Good understanding of:
  •   SharePoint sites, document libraries, folders, and files;
  • SharePoint permissions and access models;
  • Microsoft Entra ID / Azure AD;
  • Application registrations and service principals;
  • OAuth and API-based authentication;
  • Microsoft Graph and/or SharePoint APIs.
• Strong understanding of:
  • Schemas, tables, views, columns, and metadata;
  • Database roles and permissions;
  • Service accounts;
  • Query execution and performance;
  • JDBC/ODBC connectivity.
• Experience working with unstructured data sources such as:
  • SharePoint Online, OneDrive, SMB/NFS, S3, Azure Blob Storage, Enterprise file shares
• Experience with Linux and command-line troubleshooting;
• Working knowledge of networking concepts including:
  • DNS  – TCP/IP, Ports, Firewalls, Proxies, TLS/certificates, Connectivity troubleshooting.
• Experience working with REST APIs, JSON, HTTP methods/status codes, and API authentication.
• Ability to use tools such as curl and/or Postman.
• Scripting experience, preferably Python; Bash and/or PowerShell also valuable.
• Strong troubleshooting and problem-solving skills.
• Ability to systematically isolate issues across application, database, SharePoint, infrastructure, network, authentication, and data layers.
• Strong communication and documentation skills
• Microsoft Purview / Labeling Experience.

Strong preference for candidates with exposure to Microsoft Purview or Microsoft Information Protection, including: Sensitivity labels- Label publishing, Label policies, DLP concepts, File labeling, Auto-labeling concepts, SharePoint and OneDrive labeling behavior, Microsoft Information Protection / MIP
Understanding how classification findings can drive downstream labeling or protection actions

Direct Purview administration experience is preferred but not mandatory if the candidate has a strong Microsoft 365 engineering foundation and can learn quickly.

BigID Experience
Prior BigID experience is a plus, but is not required.

Candidates with strong engineering fundamentals and the ability to learn new platforms quickly will be considered. BigID-specific training will be provided.

Ideal Candidate Profile
A hands-on engineer who is comfortable working across both structured and unstructured data environments.

The ideal candidate does not need to know BigID on day one, but should have enough technical depth to quickly understand how BigID connects to enterprise systems, scans data, applies classification logic, interacts with Microsoft 365, and supports downstream labeling and remediation workflows.