Skip navigation
Syndie - distributed forums

# Syndie archives

Each Syndie instance operates with its own archive of messages, storing the data extracted from the signed messages within its local database (by default stored under `$dataRoot/db/`), an archive of those signed messages in the `$dataRoot/archive/` file hierarchy, and an archive of locally created but not yet distributed messages in the `$dataRoot/outbound/` file hierarchy. The contents of `$dataRoot/archive/` can be wiped out without any loss of functionality, though doing so prevents the Syndie instance from sharing the authenticated messages with other people.

Within the `$dataRoot/archive/` directory is a periodically rebuilt file `shared-index.dat`, containing a listing of files in the archive that should be reachable, some metadata surrounding each of those channel and message files, and some general metadata for the archive itself. This file is only accessed when serving up the archive on the web - never for those not publishing an archive. In general, it contain a subset of the available files underneath the archive directory, so as to hide some anonymity-related facts (such as delaying the listing of locally created messages until they have been published elsewhere, etc).

The `shared-index.dat` file uses the following format:

        $archiveFlags       // 2 bytes - see below for meaning
        $archiveAdminChannel// 4 bytes - index into the channels where the archive admin posts
        $numAltURIs         // 1 byte
        for (i = 0; i < $numAltURIs; i++)
          $url              // UTF-8 encoded SyndieURI of some other archives
        $numChannels        // 4 bytes
        for (i = 0; i < $numChannels; i++)
          $channelHash      // 32 byte SHA256 value
          $channelEdition   // 8 byte unsigned integer
          $channelFlags     // 1 byte - see below
        $numMessages        // 4 bytes
        for (i = 0; i < $numMessages; i++)
          $messageId        // 8 byte unsigned integer
          $scopeChannel     // 4 byte unsigned integer - index into $numChannels above
          $targetChannel    // 4 byte unsigned integer - index into $numChannels above
          $msgFlags         // 1 byte - see below

    $archiveFlags:
      first byte:
        high bit: if true, the archive wants to receive only recent messages
        bit 6   : if true, the archive wants to receive only messages in known channels
        bit 5   : if true, the archive will accept passphrase encrypted messages
        bit 4   : if true, the archive will accept private reply messages
        bit 3   : if true, the archive requires a passphrase to post to it
        bit 2-0 : along with bits 7 and 6 of the second byte, configures the hashcash requirements to post
      second byte:
        high bit: see above
        bit 6   : see above
        bits 5-4: index rebuild frequency
                  0 = hourly, 1 = every 6 hours, 2 = every 12 hours, 3 = every 24 hours
        bits 3-0: maximum message size the archive will receive, in kilobytes, bitshifted
                  0 = 1KB, 1 = 2KB, 2 = 4KB, ..., 14 = 8MB, 15 = 16MB

    $channelFlags:
      high bit: if true, the channel info is passphrase encrypted
      bit    6: if true, the channel info publishes its read key
      bit    5: if true, the channel info has been updated "recently"
      bit    4: if true, the archive wants to receive updates to this channel's metadata
      bit    3: if true, the archive wants to receive new messages in this channel
      bits 2-0: undefined

    $messageFlags:
      high bit: if true, the message is passphrase encrypted
      bit    6: if true, the message is encrypted to the forum owner
      bit    5: if true, the message publishes its read key
      bit    4: if true, the archive considers this message "new"
      bits 3-0: message size shift, in kilobytes, starting at 4KB
                0 = 4KB, 1 = 8KB, 2 = 16KB, ..., 14 = 32MB, 15 = 64MB

Individual posts are found under `$dataRoot/archive/$scope/$messageId.syndie`, and metadata under `$dataRoot/archive/$scope/meta.syndie`. The externally referenced posts are found under their original scope path, not the targetted channel path - `$dataRoot/archive/$scopeHash/$messageId.syndie` and not `$dataRoot/archive/$channelHash/$messageId.syndie`

Given the simple file-based archive hierarchy and index, running a public Syndie archive is trivial - simply publish your `$dataRoot/archive/` in a webserver and tell people the URL. They will then be able to load up their Syndie instance and use the getindex and fetch commands to pull posts from the archive into their local Syndie instance.

Syndie also includes a built-in HTTP server (run with the httpserv command), which both serves up the content and accepts HTTP posts from others (if allowed), limiting the data accessible to those messages and channel metadata files published in the `shared-index.dat` file.

The server accepts posts (uploaded through schedule and put), writing them to a temporary directory under the syndie root, and running bulkimport on them after they've all been read.

The HTTP post received is not a mime-encoded post, but a fairly simple concatenation of all of the data to be uploaded (with metadata messages first):

      POST /import.cgi\r\n
      Content-length: $total\r\n
      \r\n
      $headerSize	// 2 bytes
      $header	// $headerSize bytes, for authorization/etc
      foreach (msg)
        $msgFlags	// 1 byte.  0x0 for normal posts, 0x1 for metadata posts
        $msgSize	// 4 bytes
        $msgData	// $msgSize bytes