# ForgePack (`.fp`)

## Modern High-Performance Archive Container

Author: Dawid Bautsch  
Status: Early concept / architecture draft  
Target: Cross-platform, extensible, modern archive format and SDK  

---

## 1) Vision

ForgePack (`.fp`) is designed as a modern container format focused on:

- high performance
- extensibility
- portability
- deterministic behavior
- rich metadata
- flexible compression pipelines
- forward-compatible architecture

It is **not**:

- another ZIP clone
- a RAR reimplementation
- a single-algorithm archive format

The target is a stable container with:

- pluggable compression methods
- advanced metadata handling
- streaming support
- random access
- optional integrity, authentication, and recovery features
- data-driven per-file codec selection and tuning

---

## 2) Core Philosophy

### 2.1 Container is separate from compression

The archive format must not be tightly coupled to one codec.

The container is responsible for:

- metadata organization
- chunk layout
- table/index management
- offsets
- signatures
- encryption
- recovery data
- streaming
- integrity
- indexing

Compression algorithms are independent, replaceable modules.

### 2.2 Deterministic design

Behavior must be stable for identical inputs:

- explicit layout rules
- fixed parsing semantics
- clear extension points

---

## 3) Initial Development Strategy

### 3.1 V1 Alpha Goals

For early versions, prioritize:

- correctness
- testability (`CTest`)
- clean architecture
- stable metadata format
- deterministic output
- stable container ABI

Compression ratio is not the primary goal in V1.

### 3.2 Initial codec baseline

Start with a single stable backend:

- `zlib` (or explicitly pinned version if needed for reproducibility)

Rationale:

- battle-tested
- portable
- minimal dependencies
- fast to integrate
- stable behavior

The format must be designed so codecs can be added/removed without redesigning the container.

---

## 4) Container Feature Set

### 4.1 Filesystem behavior

- UTF-8 paths
- Unicode paths
- long path support (>\~255 chars)
- Unix-style path semantics
- Windows compatibility
- optional symlinks
- optional hardlinks

### 4.2 Archive behavior

- chunk-based internal structure
- streaming operations
- random access
- partial extraction
- deterministic archives
- append mode support
- optional solid compression mode
- mmap-friendly layout option

### 4.3 Security

- optional AES-256-GCM encryption
- optional ChaCha20-Poly1305 in future versions
- optional signed manifests
- optional archive locking/policy protection
- integrity verification

### 4.4 Recovery

Planned optional recovery features:

- parity blocks
- Reed-Solomon style repair
- segment-level repair
- configurable recovery level (tradeoff: safety vs size)

---

## 5) Readonly / Locked Archive Modes

Archive-level immutability is a policy flag, not DRM.

- metadata flags can be signed
- archive policy can be cryptographically verified
- useful for tamper detection and integrity policy enforcement

### Example

```cpp
enum class ArchiveFlags : uint64_t
{
    ReadOnly  = 1u << 0,
    AppendOnly = 1u << 1,
    NoDelete  = 1u << 2,
    NoRepack  = 1u << 3,
};
```

---

## 6) Archive Authenticity

Planned authenticity features:

- signed manifests
- signed indexes
- archive authenticity verification

Potential algorithms:

- Ed25519
- SHA-256
- BLAKE3 (preferred initial choice)

Possible key material metadata:

- `AuthorKey`
- `PublisherKey`
- `TimestampKey`

---

## 7) Recovery record policy

Recovery data should be optional and version-aware.

Possible modes:

- `none`
- `light`
- `medium`
- `heavy`

Alternative form can be added later: explicit percentage (`0..100`) or equivalent.

For V1, support archive formats prepared for future two-part recovery:

- `.fp` contains primary archive
- `.fpr` contains recovery payload
- optional concatenation/splitting workflow

This does not need full implementation in V1, but structural placeholders should exist.

Example:

```bash
fp pack data/ backup.fp --recovery 10%
```

Goals:

- repair damaged archives
- recover damaged chunks
- minimize full archive rebuild requirements

---

## 8) 64-bit-first format design

All core archive values must be 64-bit from day one:

- huge archives
- huge file entries
- future growth and extension

Examples:

- `uint64_t fileSize`
- `uint64_t compressedSize`
- `uint64_t archiveOffset`
- `uint64_t timestampNs`

---

## 9) Metadata architecture

Metadata must **not** be a decompression bottleneck.

- listing should be fast
- metadata lookups should not require full decode

Example:

```bash
fp list archive.fp
```

should be fast for large archives.

---

## 10) Metadata compression strategy

Use a two-tier approach.

### 10.1 Hot metadata (latency critical)

Priority:

- low parse latency
- fast scanning

Examples:

- file tables
- offsets
- chunk maps

Allowed codecs:

- raw
- LZ4
- other ultra-fast codecs

### 10.2 Cold metadata (size critical)

Priority:

- stronger compression ratio
- less frequent access

Examples:

- comments
- thumbnails
- extended attributes
- optional manifests

Allowed codecs:

- Zstd
- Brotli
- LZMA

For V1, prefer plain zlib for compatibility and simplicity.

---

## 11) Codec-agnostic compression blocks

Each compressed block must contain:

- codec type
- codec version
- codec parameters (descriptor)
- compressed size
- original size

The container does not need to know codec internals.

---

## 12) Compression method IDs

```cpp
enum class CompressionMethod : uint16_t
{
    Raw     = 0,
    Zlib    = 1,
    LZ4     = 2,  // optional backend
    Zstd    = 3,  // optional backend
    LZX     = 4,  // postponed for somewhere in the future
    Brotli  = 5,  // optional backend
    Lzma2   = 6   // optional backend
};
```

---

## 13) Self-describing compression blocks

Each codec block must be decodable without global assumptions.

```cpp
struct CompressionDescriptorHeader
{
    uint16_t method;
    uint16_t version;
    uint32_t descriptorSize;
    uint64_t compressedSize;
    uint64_t originalSize;
};
```

---

## 14) Variable-size codec descriptors

Codec descriptors may have different sizes and fields.

Examples:

- zlib descriptor: 24 B
- LZX descriptor: postponed for somewhere in the future

Common header remains stable:

```cpp
struct CompressionDescriptorHeader
{
    uint16_t method;
    uint16_t version;
    uint32_t descriptorSize;
    uint64_t compressedSize;
    uint64_t originalSize;
};

struct ZlibDescriptor
{
    CompressionDescriptorHeader header;
    uint32_t windowBits;
    uint32_t strategy;
};

struct LzxDescriptor
{
    CompressionDescriptorHeader header;
    uint32_t dictionarySize;
    uint32_t parserFlags;
    uint32_t entropyMode;
};
```

---

## 15) Forward compatibility requirements

The parser must not assume fixed descriptor size.

Unknown codecs must remain:

- skippable
- listable
- partially extractable
- non-fatal for forward-compatible readers

---

## 16) Chunk-based architecture

ForgePack must process at block/chunk granularity.

Not:

- `1 file = 1 stream`

But:

- `file -> chunks -> compression blocks`

---
