This chapter approaches the Swarm features introduced in the previous chapters from the user’s perspective. In 6.1, we discuss the developer interface for configuring uploads including aspects of postage, pinning, erasure coding, as well as tracking the progress of chunk propagation into the network using upload tags. Then we turn to the storage API, specifically uploading collections, in 6.2. Finally, we provide an overview of the communication options available in 6.3.
6.1 Configuring and tracking uploads
The upload interface is the cornerstone of Swarm’s functionality, allowing users to easily transfer their data into the network. Section 6.1.1 presents the request headers (or alternatively, query parameters) used for configuring the upload APIs. In 6.1.2, we introduce upload tags that allow tracking the upload process through progress bars together with an estimate of how long it will take for the process to complete. Additionally, tags also record partial root hashes, enabling the resumption of interrupted uploads. In 6.1.3, we sketch the scenarios relating to payment for upload and dispersal into the network, including how users can purchase and attach stamps to chunks. Finally, 6.1.4 runs through optional parameters such as encryption, pinning, and erasure codes.
6.1.1 Upload options
The local HTTP proxy offers the bzz URL scheme as a storage API.
Requests can specify Swarm-specific options such as:
-
tag - use this upload tag] generated and returned in response header if not specified.
-
stamp - upload using this postage subscription] if not given, the one used most recently is used.
-
encryption - encrypt content if set] if set to a 64-byte hex value encoding a 256-bit integer, then that is used instead of a randomly generated key.
-
pin - pin chunks] pin all chunks of the upload if set.
-
parities - apply RS erasure coding] use this number of parities per child batch for all intermediate chunks
These options can be used as URL query parameters or specified as
headers. The name of the header is obtained prefixing the parameter
name with SWARM-, written in uppercase, e.g., SWARM-PARITIES.
6.1.2 Upload tags and progress bar
When uploading a file or collection, it is useful for the user to know when the upload is complete, meaning that all newly created chunks are synced to the network and have arrived at the neighbourhood where they can be retrieved. At this point, the uploader can "disappear", i.e. they can quit their client. A publisher can disseminate the root hash and be assured that the file or collection is retrievable from every node on Swarm.
To track the progress of an upload, we utilise the push-sync protocol’s statements of custody receipts for individual chunks. By collecting and counting these receipts, we can track the ratio of sent chunks and returned receipts, which provides the data to a progress bar. An upload tag is an object that represents an upload and tracks its progress by counting how many chunks have reached a particular state. The states are:
- split
- Number of chunks split; count chunk instances.
- stored
- Number of chunks stored locally; count chunk instances.
- seen
- Count of chunks previously stored (duplicates).
- sent
- Number of distinct chunks sent with push-sync.
- synced
- Number of distinct chunks for which the statement of custody arrived.
With the help of these counts, it is possible to monitor the progress of 1) chunking 2) storing 3) push-syncing, and 4) receipts. If the upload tag is not specified in the header, a random tag will be generated and returned as a response header once the file is fully chunked. In order to monitor the progress of chunking and storing during the upload, the tag needs to be created beforehand and supplied in the request header. Thus, the tag can be queried concurrently while the uploaded content is being processed.
Known vs. unknown file sizes
If the file size is known, it is possible to calculate the total number of chunks that will be generated. This allows for meaningful progress tracking of both chunking and storage from the beginning, in proportion to the total number of chunks.
However, if the size and total number of chunks split are not known prior to the upload, the progress of chunk splitting is undefined. After the chunker has finished splitting the file, the total count can be set to the split count. From that point on, a percentage progress and estimated time of arrival (ETA) are available for the rest of the counts.
Note that if the upload also includes a manifest, the total count will only serve as an estimation until it is set to the split count. This estimation converges to the correct value as the size of the file increases during the upload process.
Duplicate chunks
Duplicate chunks refer to chunks that occur multiple times within an upload or across different uploads. In order to have a locally verifiable definition, we define a chunk as a duplicate (or seen) if and only if it already exists in the local store. Since chunks enter the local store via upload and are push-synced, there is no need for seen chunks to be push-synced again.
In other words, only newly stored chunks need to be counted when assessing the estimated time of syncing an upload. If we want to track progress based on sent and synced counts, they must reflect the completeness of stored distinct chunks in proportion to the overall progress.
Tags API
The HTTP server’s bzz URL scheme provides the tags endpoint for the
tags API. This endpoint supports the creation, listing, and viewing of
individual tags. Most importantly, it offers an option to track the changes
of a tag in real time using HTTP streams.
6.1.3 Postage
To impose a cost on uploads and efficiently allocate storage resources in the network, all uploads must be paid for. This concept may be unfamiliar in the context of the web, so a novel user experience needs to be developed. The closest and most relatable metaphor is that of a subscription.
Postage subscriptions
The user can create a subscription for a specific duration and storage capacity (e.g. 1 month for 100 megabytes) and make the corresponding payment based on the price provided by the client software (similar to how transaction fees are determined for the blockchain). Price estimates can be obtained from the postage lottery contract. Subscriptions are named, but these names are only meaningful locally. The API will offer a few default options for the chosen storage period, on a logarithmic scale, e.g.:
- minimal (a few hours)
- Useful for immediate delivery of pss messages or single owner chunks that are part of ephemeral chat or other temporary files.
- temporary (week)
- Files or mailboxed messages that are not meant to be stored for long, but rather picked up by third parties asynchronously.
- long term (year)
- The default option for long-term storage.
- forever (10 years)
- Important content that should not be lost/forgotten; intended to survive the network’s growth and subsequent increase of batch depth even if the uploader remains completely offline.
When uploading files, the user can indicate which subscription they
would like to use by specifying the value of the stamp upload parameter.
If no subscription is specified, the most recent one is used as the default.
If the size of the upload is known, the user can receive a warning if it
exceeds the available postage capacity.
Postage reuse strategies
As mentioned in 3.3, encryption can be used to mine chunks to make sure they align with the collision slots of postage batches. This strategy of mining chunks into batches makes sense for users who only want to upload a particular file/collection for a particular period or want to keep the option open to change the storage period for the file/collection independently of other uploads.
The alternative is to prepay a large set of postage batches and maintain a fixed number of them open for every period. This ensures that there is always an available collision slot in one of the batches for any chunk. The postage batches are ordered based on the time of purchase, and when attaching a stamp to a chunk, the first batch with a free slot for the chunk address is utilised. This profitable strategy is used most effectively by power users including insurers who can afford to tie up liquidity for longer time periods. The choice of the preferred strategy must be specified when creating a subscription.
The postage subscription API
In order to manage subscriptions, the bzz URL-scheme provides the
stamp endpoint for the postage API. This API allows users to
perform various actions such as creating postage subscriptions, listing
them, viewing details, topping them up, or draining and expiring
them.
When checking their subscription(s), the user is informed how
much data has already been uploaded to that subscription and how
long it can be stored given the current price (e.g.
6.1.4 Additional upload features
Additional features such as swap costs, encryption, pinning, and erasure coding can be specified during upload by using relevant request headers.
Swap costs
Uploading content incurs swap costs, which are associated with the push-sync protocol and reflected in the SWAP accounting system (3.2.1). When the balance tilts beyond the effective payment threshold on a peer connection, a cheque is issued and sent to the peer. If, however, the checkbook contract does not exist or lacks sufficient funds to cover the outstanding debt, the peer connection is blocked. This may lead to unsaturated Kademlia and during that time, the node is considered a light node. While the download process can still continue with potentially fewer peers, the average cost of retrieving a chunk may be higher since some chunks may need to be sent to peers that are not closer to the request address than the node itself.
To provide transparency to the user, information such as the average number of peers, average swap balance, and the number of unavailable peer connections due to insufficient funds can be accessed through the SWAP API.
Encryption
Encrypted upload can be enabled by setting the option encryption to a
non-zero value. If the value provided is a hexadecimal representation of a
32-byte seed, that seed is used for encryption. Since encryption
and decryption are handled by Swarm itself, it must only be used
in situations where the transport between the HTTP client and
Swarm’s proxy server can be guaranteed to be private. As such, it
should not be used on public gateways, but rather through local
Swarm nodes or private environments with well-configured TLS
and self-signed certificates to ensure the necessary privacy and
security.
Pinning
When uploading content, users have the option to specify if they want the
content to be pinned locally by setting the pin upload option
to a non-zero value. Beyond this, the local HTTP server’s bzz
URL scheme provides the pin endpoint for the pinning API, which
allows users to manage pinned content. By making a GET request on
the pinning endpoint, users can retrieve a list of pinned files or
connections. The API also accepts PUT and DELETE requests on a hash (or
domain name that resolves to a hash) to pin and unpin content,
respectively.
When a file or collection is pinned, it is supposed to be retrieved and stored. Pinning triggers the traversal of the hash tree and increments the reference count on each chunk. If a chunk is found to be missing locally, it will be retrieved. After pinning, the root hash is saved locally for future access. Unpinning triggers the traversal of the hash and decrements the reference count of each chunk. If a chunk is missing locally, it is ignored and a warning is included in the response. After unpinning, the hash is removed from the list of pinned hashes.
Erasure coding
Erasure coding (see 5.1) can be enabled during an upload by setting the
parities upload option to specify the number of parity chunks among
the children of each intermediate chunk. It is worth noting that erasure
coded files cannot be retrieved using the default Swarm downloader.
Therefore, when using erasure coding, it is important to indicate the
erasure coding settings in the encapsulating manifest entry by setting the
rs attribute to the number of parity chunks.
6.2 Storage
In this section, we will introduce the storage API provided through the
bzz family of URL schemes by Swarm’s local HTTP proxy.
6.2.1 Uploading files
The bzz scheme allows for uploading files directly through the file API.
The file is expected to be encoded in the request body or in a multipart
form. All the query paramaters (or corresponding headers) introduced in
6.1.1 can be used with this scheme. The POST request will chunk and
upload the file. A manifest entry is created, which includes the reference
to the uploaded content and reflects the configuration of the upload, e.g.
the number of parities per batch for erasure codes specified by the
rs attribute. The manifest entry is provided as a response. The
upload tag. which allows us to monitor the status of the upload, will
contain the reference to the uploaded file as well as the manifest
entry.
Appending to existing files
The PUT request is used to append data to pre-existing data in the swarm
and requires the URL to point to the specific file. If the file is directly
referenced, the settings, such as the number of parities for erasure coding,
are taken from the upload headers. Otherwise, if the file is not directly
referenced, the settings obtained from the enclosing manifest entry will be
used. The response follows the same format as in the case of a POST
request.
Resuming incomplete uploads
As a special case, append is used when resuming uploads after a crash or user-initiated abort. In order to facilitate this, it is necessary to track partial uploads by periodically recording root hashes on the upload tag. When the specified upload tag in the header is not complete, it is assumed that the request is intended to resume the same upload. The last recorded root hash in the tag is used as an append target: the right edge of the existing file is retrieved to initialise the state of the chunker. The file sent with the request is read from where the partial upload ended, with the offset set to the span of the root chunk recorded on the tag.
6.2.2 Collections and manifests
As described in Section 4.1.2, a manifest can represent a generic index that maps string paths to entries. This means that a manifest can serve as the routing table for a virtual website, the directory tree of a file collection, or even as a key–value store.
Manifest entries vs.singleton manifests
Manifest entries are essential not only for collections, but also for single files as they contain key information about the file, including content type and erasure coding details necessary for correct retrieval. A manifest with a single entry for a file located on the empty path is called a singleton manifest. In this case, the manifest contains no additional information beyond the entry itself.
To facilitate the storage of collections, the HTTP server provides the bzz
URL scheme, which implements the collection storage API. When a
single file is uploaded via a POST request using the bzz scheme, a manifest
entry is created and stored, and the response contains the reference to the
created entry.
Uploading and updating collections
The bzz URL scheme provides an API that supports uploading and
updating collections. When utilising the POST and PUT requests with
a multipart payload containing a tar stream, the directory tree
structure encoded within the tar stream is translated into a manifest.
Simultaneously, the files within the collection are chunked and uploaded,
with their respective Swarm references included in the corresponding
manifest entries. The POST request is used to upload the resulting
manifest, and the response includes the Swarm reference associated with
it. The PUT request requires the request URL to reference an existing
manifest that will be updated with the contents of the tar stream. The
update process involves merging the new paths with the paths of the
existing manifest. If there are identical paths, the entry coming from
the upload will replace the corresponding entry in the existing
manifest.
The API enables inserting a path into a manifest, updating a path in a
manifest (PUT), and deleting (DELETE) a path from a manifest. In case of
a PUT request, a file is expected in the request body, which is uploaded
and its manifest entry is inserted at the path present in the URL. If the
path already exists in the manifest, the existing entry is replaced with
the entry generated with the upload, effectively performing an
update. If the path is new, the upload inserts a new entry into the
manifest.
The collection API supports the same headers as the file upload endpoint, including those that configure postage subscription, tags, encryption, erasure coding, and pinning.
Updating manifests directly
Manipulating manifests is also supported directly: the bzz URL scheme
manifest endpoint supports the PUT and the DELETE methods and
behaves similarly to the collection endpoint. However, unlike the
collection endpoint, the manifest endpoint does not handle the files
referenced in the manifest entries. When using a manifest endpoint, the
URL path is expected to reference a manifest with a path PUT, the
request body should contain a manifest entry which will be placed at
path DELETE method expects empty request body and
removes the entry on the path from the manifest: i.e., it creates a
new manifest where the referenced path in the URL is no longer
present.
The POST request made directly on the manifest API endpoint installs a
manifest entry. Practically, calling a manifest POST on the output of a file
POST is equivalent to making a POST request on the generic storage
endpoint.
Given manifest references POST request on
manifest/merge/<a>/<b> merges
In case the URL path references a manifest, another manifest can be included in the request body, which is then merged into the referenced manifest. In case of conflicts, the one that is uploaded wins.
6.2.3 Access control
Access control, as described in 4.2, is facilitated through the bzz URL
scheme, which provides the access API endpoint. This API offers a
convenient way for users to apply access control to files, collections, or
sites, as well as manage grantees.
If the URL path references a collection manifest, then a POST request
with access control (AC) settings sent as a JSON-encoded request body
will encrypt the manifest reference and wrap it with the submitted AC
settings in a so-called root access manifest. This manifest is then
uploaded, and the unencrypted reference to it is returned as the response
body.
If the URL path references a root access manifest and the access control
settings specify an ACT, then the ACT can be created or updated using
POST, PUT, and DELETE requests. All requests expect a JSON array of
grantee public keys or URLs in the request body. If a grantee is
referenced by a URL, the resolver is used to extract the owner’s public
key through ENS.
The POST request will create the ACT with the list of grantees. PUT will
update an existing ACT by merging the grantees specified in the
request body. DELETE removes all the grantees listed in the request
body. The new ACT root hash is then updated in the root access
manifest, which is uploaded, and its Swarm address is returned in the
response.
6.2.4 Download
Downloading is supported by the bzz URL scheme. This URL scheme
assumes that the domain part of the URL is referencing a manifest as the
entry point.
It is worth noting that the processes involved in downloading are the
same, even if a file is only partially retrieved. As shown in 4.1.1, random
access to a file at an arbitrary offset is supported at the lowest level.
Therefore, GET requests on a URL pointing to a file can include
range queries in the header. These range queries will trigger the
retrieval of only the necessary chunks of the file that cover the desired
range.
Retrieval costs
Downloading files in Swarm incurs costs in terms of SWAP accounting (as described in Section 3.2.1). These costs are associated with retrieving chunks from the network. When a node’s balance tilts beyond the effective payment threshold on a peer connection, a cheque is issued and sent to that peer. If, however, the checkbook contract does not exist or lacks sufficient funds to cover the outstanding debt, the peer connection is blocked. This may lead to non-saturated Kademlia and during that time, the node will count as a light node. The download can continue potentially using a smaller number of peers, however, since some chunks will need to be sent to peers that are not closer to the request address than the node itself, the average cost of retrieving a chunk will be higher.
To provide transparency to the user, it is valuable to display the average
number of retrieve requests, the average swap balance, and the number of
peer connections not unavailable due to insufficient funds. This
information can be accessed through the SWAP API, which is available
on the swap endpoint of the bzz URL scheme.
Domain Name Resolution
The domain part of the URL can be a human-readable domain or
subdomain with a top level domain (TLD) extension. Depending on the
TLD, various name resolvers can be invoked. The TLD eth is linked
to the Ethereum Name Service contract on the Ethereum main
chain. If you register a Swarm hash to an ENS domain, Swarm is
able to resolve that by calling the ENS contract as a nameserver
would.
Authentication for access control
If the resolved domain references a root access manifest, the retrieved
URL is subject to access control (see 4.2). Depending on the credentials
used, the user may be prompted for a password or a pair of keys (key 1
for lookup and key 2 for decrypting the access key). The Diffie–Hellmann
shared secret is hashed with
Next the manifest entry corresponding to the path of the URL is retrieved. The retrieval process considers the following attributes:
-
rs - structure with attributes needed to use RS erasure coding for retrieval, e.g. the number of parity chunks required.
-
sw3 - structure with attributes needed for litigation, which includes challenging insurer regarding a missing chunk.
If the rs attribute is provided, the user can set the prefetching strategy
for redundancy decoding (see 5.1.4).
The default strategy choice is race, if the user wants to prioritise saving
on downloads, then the fallback strategy can be enforced by setting the
header SWARM-RS-STRATEGY=fallback. Alternatively, the RS strategy
can be completely disabled by setting SWARM-RS-STRATEGY=disabled.
The number of parities for an erasure coded batch is taken from the rs
attribute in the enclosing manifest. However, this can be overridden with
the header SWARM-RS-PARITIES.
Missing chunks
If a requested chunk is found to be missing, we can fall back on the missing chunk notification protocol (see 5.2). The reference to the root of the data structure representing the set of recovery targets can be found at the latest update of the recovery feed.
When a chunk request times out, the client can initiate the creation of a recovery message using the set of pinner hosts. Once the recovery message is created, it is sent to the host. If this request also times out, the next recovery chunk is tried using a different pinner host node. This process is repeated until the recovery chunk is successfully retrieved or all the pinners and insurers have been exhausted.
6.3 Communication
Somewhat surprisingly, Swarm’s network layer can serve as a highly efficient communication platform with robust privacy features. This section aims to showcase how a small set of primitives can be utilised as the fundamental building blocks for a comprehensive communication infrastructure. The capabilities cover the full range of communication modalities including real-time anonymous chat, sending and receiving messages from previously unconnected and potentially anonymous senders, mailboxing for asynchronous delivery, long-term notifications, and publish/subscribe interfaces.
Swarm core offers the lowest-level entry-points for communication-related functionality:
- The pss module provides an API for sending and receiving Trojan chunks
- The bzz module offers a way to upload single owner chunks. The retrieval of single owner chunks only requires the functionality provided by the storage APIs, without any additional requirements
Since Trojan message handling has distinct characteristics from storage
operations, Swarm introduces its own URL-scheme called pss. The pss
scheme provides an API specifically for sending messages. When sending
a POST request, the URL is interpreted as referencing the X3DH pre-key
bundle feed update chunk. This chunk contains the necessary public
key for encryption as well as the destination targets. One of the
destination targets should match the address for the Trojan message (see
4.4.1).
Destination targets are represented as the buzz serialisation of the target prefixes as a file. The URL path points to this file or is a top-level domain. The topic is specified as a query parameter and the message as the request body.
Receiving messages is supported only by registering topic handlers internally. In the context of the API, this typically involves push notifications via web-sockets.