# Storage Operations

:::tip[Looking for uploads?]
This page covers data set management, retrieval, and lifecycle operations. For uploading data (from simple one-liner to manual store/pull/commit), see the **[Upload Pipeline](/developer-guides/storage/upload-pipeline/)**.
:::

## Key Concepts

**Data Set**: A logical container of pieces stored with one provider. When a data set is created, a payment rail is established with that provider. All pieces in the data set share this single payment rail and are verified together via PDP proofs.

**PieceCID**: Content-addressed identifier for your data (format: `bafkzcib...`). Automatically calculated during upload and used to retrieve data from any provider.

**Metadata**: Optional key-value pairs for organization:

- **Data Set Metadata**: Max 10 keys (e.g., `project`, `environment`)
- **Piece Metadata**: Max 3 keys per piece (e.g., `filename`, `contentType`)

**Copies and Durability**: By default, `upload()` stores your data with 2 independent providers. Each provider maintains its own data set with separate PDP proofs and payment rails. If one provider goes down, your data is still available from the other.

**Storage Manager**: The main entry point for storage operations (`synapse.storage`). Handles provider selection, multi-copy orchestration, data set management, and provider-agnostic downloads.

## Context Creation

Storage contexts represent a connection to a specific provider and data set. The SDK creates and manages contexts internally during `upload()`, but you can create them explicitly for data set management, retrieval, or [split operations](/developer-guides/storage/upload-pipeline/#split-operations).

### Single Context

```ts twoslash
// @lib: esnext,dom
import { Synapse } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: "my-app" })

// All options are optional
await synapse.storage.createContext({
  providerId: 1n,           // specific provider (optional)
  dataSetId: 42n,           // specific data set (optional)
  metadata: { source: "my-app" },        // data set metadata for matching/creation
  withCDN: true,            // enable fast-retrieval (paid, optional)
  excludeProviderIds: [3n], // skip specific providers (optional)
})
```

### Multiple Contexts

For multi-copy replication, use `createContexts()`:

```ts twoslash
// @lib: esnext,dom
import { Synapse } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: "my-app" })
// ---cut---
const contexts = await synapse.storage.createContexts({
  copies: 3,                   // number of contexts (default: 2)
  providerIds: [1n, 2n, 3n],   // specific providers (mutually exclusive with dataSetIds)
  dataSetIds: [10n, 20n, 30n], // specific data sets (mutually exclusive with providerIds)
})
const [primary, secondary] = contexts
```

[**View creation options for `createContext()`**](/reference/filoz/synapse-sdk/synapse/interfaces/storageserviceoptions/)

[**View creation options for `createContexts()`**](/reference/filoz/synapse-sdk/synapse/interfaces/contextcreatecontextsoptions/)

### Data Set Matching

:::tip[Metadata Matching for Cost Efficiency]
**The SDK reuses existing data sets when metadata matches exactly**, avoiding duplicate payment rails. To maximize reuse:

- Use consistent metadata keys and values across uploads
- Avoid changing metadata unnecessarily
- Group related content with the same metadata

**Example**: If you create a data set with `{Application: "MyApp", Version: "1.0"}`, all subsequent uploads with the same metadata will reuse that data set and its payment rail.
:::

The SDK intelligently manages data sets to minimize on-chain transactions. The selection behavior depends on the parameters you provide:

**Selection Scenarios**:

1. **Explicit data set ID**: If you specify `dataSetId`, that exact data set is used (must exist and be accessible)
2. **Specific provider**: If you specify `providerId`, the SDK searches for matching data sets only within that provider's existing data sets
3. **Automatic selection**: Without specific parameters, the SDK searches across all your data sets with any approved provider

**Exact Metadata Matching**: In scenarios 2 and 3, the SDK will reuse an existing data set only if it has **exactly** the same metadata keys and values as requested. This ensures data sets remain organized according to your specific requirements.

**Selection Priority**: When multiple data sets match your criteria:

- Data sets with existing pieces are preferred over empty ones
- Within each group (with pieces vs. empty), the oldest data set (lowest ID) is selected

**Provider Selection** (when no matching data sets exist):

- If you specify a provider (via `providerId`), that provider is used
- Otherwise, the SDK selects from endorsed providers for the primary copy and any approved provider for secondaries
- Before finalizing selection, the SDK verifies the provider is reachable via a ping test
- If a provider fails the ping test, the SDK tries the next candidate
- A new data set will be created automatically during the first commit

## Retrieval

Download from any provider that has the piece. The SDK resolves the provider automatically:

```ts twoslash
// @lib: esnext,dom
import { Synapse } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: 'my-app' })

// Download using PieceCID from a previous upload
const pieceCid = "bafkzcib..." // from upload result
const bytes = await synapse.storage.download({ pieceCid })
const text = new TextDecoder().decode(bytes)
console.log("Downloaded:", text)
```

### CDN-Accelerated Downloads

```ts twoslash
// @lib: esnext,dom
import { Synapse } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

// Enable CDN globally
const synapse = Synapse.create({
  account: privateKeyToAccount("0x..."),
  source: 'my-app',
  withCDN: true,
})

const bytes = await synapse.storage.download({ pieceCid: "bafkzcib..." })

// Or per-download:
const bytes2 = await synapse.storage.download({
  pieceCid: "bafkzcib...",
  withCDN: true,
})
```

### Context-Specific Downloads

When using a StorageContext, downloads are restricted to that specific provider:

```ts twoslash
// @lib: esnext,dom
import { Synapse, type PieceCID } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: "my-app" })
const ctx = await synapse.storage.createContext({
  metadata: { source: "my-app" },
})
const pieceCid = null as unknown as PieceCID;
// Downloads from the provider associated with this context
const data = await ctx.download({ pieceCid })
```

### CDN Option Inheritance

The `withCDN` option follows a clear inheritance hierarchy:

1. **Synapse level**: Default setting for all operations
2. **StorageContext level**: Can override Synapse's default
3. **Method level**: Can override instance settings

```ts twoslash
// @lib: esnext,dom
import { Synapse, type PieceCID } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: "my-app" })
const ctx = await synapse.storage.createContext({ withCDN: false })
const pieceCid = null as unknown as PieceCID;
// ---cut---
await synapse.storage.download({ pieceCid })                // Uses Synapse's withCDN: true
await ctx.download({ pieceCid })                            // Uses context's withCDN: false
await synapse.storage.download({ pieceCid, withCDN: false }) // Method override: CDN disabled
```

When `withCDN: true` is set, it adds `{ withCDN: '' }` to the data set's metadata, ensuring CDN-enabled and non-CDN data sets remain separate.

## Data Set Management

:::note[When You Need This]
These APIs are useful when you want to inspect existing data sets, query stored pieces, or retrieve metadata. For basic upload/download, you don't need these.
:::

### Getting All Data Sets

Retrieve all data sets owned by your account to inspect piece counts, CDN status, and metadata:

```ts twoslash
// @lib: esnext,dom
import { Synapse } from "@filoz/synapse-sdk";
import { privateKeyToAccount } from 'viem/accounts'

const synapse = Synapse.create({ account: privateKeyToAccount('0x...'), source: 'my-app' });
const dataSets = await synapse.storage.findDataSets();

for (const ds of dataSets) {
  console.log(`Dataset ${ds.pdpVerifierDataSetId}:`, {
    live: ds.isLive,
    cdn: ds.withCDN,
    pieces: ds.activePieceCount,
    metadata: ds.metadata
  });
}
```

### Getting Data Set Pieces

List all pieces stored in a specific data set by iterating through a context:

```ts twoslash
// @lib: esnext,dom
import { Synapse } from "@filoz/synapse-sdk";
import { privateKeyToAccount } from 'viem/accounts'

const synapse = Synapse.create({ account: privateKeyToAccount('0x...'), source: 'my-app' });
const dataSetId = 1n;
// ---cut---
const context = await synapse.storage.createContext({ dataSetId });

const pieces = [];
for await (const piece of context.getPieces()) {
  pieces.push(piece);
}
console.log(`Found ${pieces.length} pieces`);
```

### Getting Piece Metadata

Access custom metadata attached to individual pieces:

```ts twoslash
// @lib: esnext,dom
const dataSetId = 1n;
const piece = null as unknown as { pieceCid: string; pieceId: bigint };
// ---cut---
import { WarmStorageService } from "@filoz/synapse-sdk/warm-storage";
import { privateKeyToAccount } from 'viem/accounts'

const warmStorage =  WarmStorageService.create({ account: privateKeyToAccount('0x...') });

const metadata = await warmStorage.getPieceMetadata({ dataSetId, pieceId: piece.pieceId });
console.log("Piece metadata:", metadata);
```

### Getting Piece Size

Extract the size directly from a PieceCID using Synapse Core:

```ts twoslash
// @lib: esnext,dom
import * as Piece from "@filoz/synapse-core/piece";

const pieceCid = "bafkzcib...";
const size = Piece.from(pieceCid).size;
console.log(`Piece size: ${size} bytes`);
```

## Storage Information

Query service-wide pricing, available providers, and network parameters:

```ts twoslash
// @lib: esnext,dom
import { Synapse } from "@filoz/synapse-sdk";
import { privateKeyToAccount } from 'viem/accounts'

const synapse = Synapse.create({ account: privateKeyToAccount('0x...'), source: 'my-app' });
// ---cut---
const info = await synapse.storage.getStorageInfo();
console.log("Price/TiB/month:", info.pricing.noCDN.perTiBPerMonth);
console.log("Providers:", info.providers.length);

const providerInfo = await synapse.getProviderInfo("0x...");
console.log("PDP URL:", providerInfo.pdp.serviceURL);
```

## Lifecycle Management

### Terminating the Storage Service

:::warning[Irreversible Operation]
**Termination cannot be undone.** Once initiated:

- The termination transaction is irreversible
- When the service ends, the provider may delete all data
- Payment rails associated with the data set will be terminated
- You cannot cancel the termination

Only terminate when you're certain you no longer need the data.
:::

To end the storage service for a data set and discontinue its payments, call
`context.terminate()`, or `synapse.storage.terminateService({ dataSetId })`
when the data set ID is known (a context is not necessary).

There are two ways to terminate, chosen with the `skipProvider` option. The
provider-mediated path is the recommended default: you sign an authorization,
the provider relays it on chain, and if your account can settle in full the
service and payment terminate immediately.

The direct on-chain path is an independent fallback. It is useful if the
provider-mediated request is unavailable, account doesn't have available funds to settle in full or you
need to submit the transaction yourself. It does not require provider
cooperation, but it uses the standard lockup wind-down: the service and payment continues
until `endEpoch`, typically about 30 days, while the provider settles and winds
down the data set before cleanup.

| | Recommended default (via provider) | Fallback (`skipProvider: true`) |
| --- | --- | --- |
| Who submits the transaction | The provider relays your signed authorization | You |
| Effect | Can terminate immediately and allow cleanup to begin | Service runs to the end of the lockup period, typically ~30 days |
| Cost to you | Small termination fee (USDFC, drawn from your account) | Transaction gas (FIL) |
| Requires | Provider online, your account able to settle in full | A funded wallet |
| Use when | Preferred path for ordinary termination | Provider-mediated termination cannot be completed, or you need independent submission |

Provider-relayed termination never silently degrades: it either takes effect
immediately or fails. The most common failure is an account that cannot cover
full settlement; deposit funds or use `skipProvider: true` in that case. The
result's `endEpoch` reports the actual epoch at which the service stops; on
the on-chain path this can be sooner than 30 days if your account stopped
settling some time ago.

In both cases termination ends the service and its payments. The data set's
remaining on-chain state is cleaned up afterwards by the provider; that
cleanup is not part of this call.

```ts twoslash
// @lib: esnext,dom
import { Synapse } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: "my-app" })
const ctx = await synapse.storage.createContext({
  metadata: { source: "my-app" },
})
// Via context: relayed through this context's provider, immediate
const result = await ctx.terminate()
console.log("Service terminated, ends at epoch", result.endEpoch)

// Or directly by data set ID
const result2 = await synapse.storage.terminateService({ dataSetId: 42n })

// Or submit independently on chain (winds down over the lockup period)
const result3 = await synapse.storage.terminateService({
  dataSetId: 42n,
  skipProvider: true,
  onSubmitted: (txHash) => console.log("Submitted:", txHash),
})
console.log("Service ends at epoch", result3.endEpoch)
```

### Deleting a Piece

To delete an individual piece from the data set, call `context.deletePiece()`.
This method submits an on-chain transaction to initiate the deletion process.

**Important:** Piece deletion is irreversible and cannot be canceled once initiated.

```ts twoslash
// @lib: esnext,dom
import { Synapse } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: "my-app" })
const ctx = await synapse.storage.createContext({
  metadata: { source: "my-app" },
})
// List all pieces in the data set
const pieces = []
for await (const piece of ctx.getPieces()) {
  pieces.push(piece)
}

if (pieces.length > 0) {
  await ctx.deletePiece({ piece: pieces[0].pieceId })
  console.log(
    `Piece ${pieces[0].pieceCid} (ID: ${pieces[0].pieceId}) deleted successfully`
  )
}

// Delete by PieceCID
await ctx.deletePiece({ piece: "bafkzcib..." })
```

## Next Steps

- **[Upload Pipeline](/developer-guides/storage/upload-pipeline/)** - Upload data with multi-copy durability, from simple one-liner to manual store/pull/commit.

- **[Storage Costs](/developer-guides/storage/storage-costs/)** - Calculate your monthly costs and understand funding requirements.

- **[Payment Management](/developer-guides/payments/payment-operations/)** - Manage deposits, approvals, and payment rails.