Overview

Data stores provide the foundation for data storage, organization, and retrieval in DOCC Platform. The platform supports two main approaches for accessing cloud storage, each optimized for different use cases and organizational requirements.

Storage Characteristics

• Multiple Approaches: Traditional mounting and modern governance options
• Cloud Provider Support: AWS S3, Azure Blob Storage, Google Cloud Storage
• Security & Compliance: Enterprise-grade access controls and audit trails
• Scalable Architecture: From development to enterprise-scale deployments

Storage Approaches

Choose the approach that best fits your organizational needs and governance requirements.

DOCCFS Mount

Traditional approach for direct storage mounting. Ideal for development environments and simple use cases.

| Feature | Description | |---------|-------------| | **Setup Complexity** | Simple - single configuration step | | **Credential Management** | Per-mount credentials | | **Access Control** | Basic read/write permissions | | **Best For** | Development, testing, small teams | **Key Benefits:** - Quick setup and immediate access - Simple configuration in one place - Direct mapping between mount point and storage - Perfect for development and experimentation **Configuration Example:** ```json { "name": "analytics-data", "basePath": "/doccfs/analytics", "provider": "S3", "config": { "bucketName": "company-analytics", "region": "us-east-1", "prefix": "datasets/" }, "authConfig": { "accessKeyId": "AKIA...", "secretAccessKey": "..." }, "policyConfig": { "allowRead": true, "allowWrite": true, "quotaGB": 100 } } ```

DOCC Catalog Volume

Modern governance approach with enterprise-grade controls. Perfect for production environments.

| Feature | Description | |---------|-------------| | **Setup Complexity** | Moderate - multi-step governance setup | | **Credential Management** | Centralized credential repository | | **Access Control** | Workspace-level permissions | | **Best For** | Production, enterprise, compliance | **Key Benefits:** - Centralized credential management - Enterprise-grade governance and audit trails - Workspace-level access controls - Zero-downtime credential rotation - Comprehensive compliance features **Architecture Components:** 1. **Storage Credentials:** Centralized, reusable authentication 2. **External Locations:** Governed pointers to storage paths **Setup Process:** 1. **Prepare Infrastructure:** Configure cloud storage and IAM 2. **Create Storage Credentials:** Set up centralized authentication 3. **Define External Locations:** Configure governed storage access 4. **Set Governance Policies:** Implement access controls and compliance

Feature Comparison

Detailed comparison to help you choose the right approach.

| Feature | DOCCFS Mount | DOCC Catalog Volume | |---------|--------------|-------------------| | **Setup Complexity** | Simple | Moderate | | **Credential Management** | Per-mount | Centralized | | **Access Control** | Basic | Workspace-level | | **Audit Capabilities** | Limited | Comprehensive | | **Governance** | Basic | Enterprise-grade | | **Scalability** | Small-medium | Enterprise-scale | | **Security** | Basic encryption | Advanced security patterns | | **Compliance** | Limited | Full compliance support | | **Best For** | Development, Testing | Production, Compliance |

Decision Matrix

Setup Guides

Step-by-step instructions for configuring each storage approach.

DOCCFS Mount Setup

Quick setup for development and simple use cases.

**Setup Steps:** 1. **Configure Credentials:** Set up cloud storage access keys 2. **Create Mount Configuration:** Define mount point and storage location 3. **Test Connectivity:** Validate access and permissions 4. **Start Using Data:** Begin data operations **Example Commands:** ```bash # Create mount configuration curl -X POST /api/v1/mounts \ -H "Content-Type: application/json" \ -d '{ "name": "dev-data", "provider": "S3", "config": { "bucketName": "dev-bucket", "region": "us-east-1" } }' ``` → Complete DOCCFS Setup Guide

DOCC Catalog Volume Setup

Comprehensive setup for enterprise environments.

**Setup Steps:** 1. **Prepare AWS Infrastructure:** Create S3 bucket and IAM roles 2. **Choose Authentication Method:** IAM User or Assume Role 3. **Create Storage Credential:** Configure centralized authentication 4. **Create External Location:** Set up governed storage access 5. **Configure Governance:** Implement policies and controls **Example: Create Storage Credential** ```bash curl -X POST /api/v1/catalog/storage-credentials \ -H "Content-Type: application/json" \ -d '{ "name": "s3-production-credentials", "credentialType": "ASSUME_ROLE", "providerType": "AWS_S3", "credentialConfig": { "roleArn": "arn:aws:iam::123456789012:role/DataPlatformRole", "externalId": "unique-external-id", "region": "us-east-1" } }' ``` **Example: Create External Location** ```bash curl -X POST /api/v1/catalog/external-locations \ -H "Content-Type: application/json" \ -d '{ "name": "s3-data-lake-bronze", "url": "s3://company-datalake/bronze/", "storageCredentialId": 1, "governanceConfig": { "auditLevel": "FULL", "complianceTags": ["PII", "GDPR"] } }' ``` → Complete Catalog Volume Setup Guide

Security & Best Practices

Essential security measures and optimization guidelines.

Security Guidelines

| Category | Recommendation | Implementation | |----------|----------------|----------------| | **Authentication** | Use IAM roles over access keys | Configure assume role patterns | | **Network Security** | Enable VPC endpoints | Set up private networking | | **Data Encryption** | Encrypt at rest and in transit | Enable TLS 1.2+ and storage encryption | | **Access Control** | Implement least privilege | Use workspace-level permissions | | **Credential Rotation** | Regular rotation schedule | Automate credential lifecycle |

Security Checklist

Performance Optimization

**Key Areas:** - **Storage Location:** Choose regions close to compute resources - **File Formats:** Use optimized formats like Parquet for analytics - **Compression:** Enable appropriate compression algorithms - **Caching:** Implement intelligent caching strategies - **Resource Sizing:** Right-size storage allocations

Cost Optimization

**Strategies:** - Use appropriate storage tiers (Hot, Cool, Archive) - Implement automated lifecycle policies - Monitor and optimize data transfer costs - Right-size storage allocations - Regular cost reviews and optimization

Migration Path

Guidance for migrating from DOCCFS Mount to DOCC Catalog Volume as your needs evolve.

Migration Benefits

• Enhanced security and governance capabilities
• Centralized credential management
• Better audit and compliance features
• Improved scalability for enterprise use
• Advanced authentication patterns

Migration Timeline

| Deployment Size | Estimated Timeline | Key Considerations | |-----------------|-------------------|-------------------| | **Small** | 1-2 weeks | Limited mounts, simple access patterns | | **Medium** | 1-2 months | Multiple teams, moderate complexity | | **Large Enterprise** | 3-6 months | Complex governance, multiple workspaces |

Migration Process

1. **Assessment:** Inventory existing mounts and access patterns 2. **Credential Consolidation:** Create storage credentials for common patterns 3. **Location Mapping:** Convert mount paths to external locations 4. **Access Policy Migration:** Map mount permissions to workspace controls 5. **Testing & Validation:** Validate access patterns in non-production 6. **Gradual Rollout:** Migrate applications incrementally 7. **Decommission:** Remove legacy mounts after validation

Support & Resources