update documentation for the download status page #1169

New issue

Open

opened 2010-08-12 04:58:50 +00:00 by zooko · 10 comments

zooko commented

2010-08-12 04:58:50 +00:00

Brian's New Downloader (#287, #288, #798, #800, #990), and subsequently Brian's New Visualizer (#1200) also offers a download status page with a great deal more information than the old one. Although it is really good to have all this information available, it is rather cryptic. This ticket is to update the documentation in source:docs/frontends/download-status.rst explaining what all the fields mean.

See also #1265 which is about adding labels and documentation onto the web page itself.

zooko added the

labels 2010-08-12 04:58:50 +00:00

zooko added this to the soon milestone 2010-08-12 04:58:50 +00:00

zooko commented

2010-08-12 04:59:37 +00:00

Author

Attachment down-0.html (716868 bytes) added

**Attachment** down-0.html (716868 bytes) added

down-0.html

700 KiB

warner commented

2010-08-12 20:10:51 +00:00

First, what's involved in a download?:

downloads are triggered by read() calls, each with a starting offset
(defaults to 0) and a length (defaults to the whole file). A regular
webapi GET request will result in a whole-file read() call
each read() call turns into an ordered sequence of
get_segment() calls. A whole-file read will fetch all segments, in
order, but partial reads or multiple simultaneous reads will result in
random-access of segments. Segment reads always return ciphertext: the
layer above that (in read()) is responsible for decryption.
before we can satisfy any segment reads, we need to find some shares.
("DYHB" is an abbreviation for "Do You Have Block", and is the message we
send to storage servers to ask them if they have any shares for us. The
name is historical, from Mnet/MV, but nicely distinctive. Tahoe's actual
message name is remote_get_buckets().). Responses come back
eventually, or don't.
Once we get enough positive DYHB responses, we have enough shares to start
downloading. We send "block requests" for various pieces of the share.
Responses come back eventually, or don't.
When we get enough block-request responses for a given segment, we can
decode the data and satisfy the segment read.
When the segment read completes, some or all of the segment data is used
to satisfy the read() call (if the read call started or ended in the
middle of a segment, we'll only use part of the data, otherwise we'll use
all of it).

With that background, here is the data currently on the download-status page:

"DYHB Requests": this shows every Do-You-Have-Block query sent to storage

servers and their results. Each line shows the following:

the serverid to which the request was sent
the time at which the request was sent. Note that all timestamps are
relative to the start of the first read() call and indicated with a
"+" sign
the time at which the response was received (if ever)
the share numbers that the server has, if any
the elapsed time taken by the request
also, each line is colored according to the serverid. This color is also
used in the "Requests" section below.
"Read Events": this shows all the FileNode read() calls and their

overall results. Each line shows:

the range of the file that was requested (as OFFSET:+LENGTH). A
whole-file GET will start at 0 and read the entire file.
the time at which the read() was made
the time at which the request finished, either because the last byte of
data was returned to the read() caller, or because they cancelled
the read by calling stopProducing (i.e. closing the HTTP connection)
the number of bytes returned to the caller so far
the time spent on the read, so far
the total time spent in AES decryption
total time spend paused by the client (pauseProducing), generally
because the HTTP connection filled up, which most streaming media players
will do to limit how much data they have to buffer
effective speed of the read(), not including paused time
"Segment Events": this shows each get_segment() call and its

resolution. This table is not well organized, and my post-1.8.0 work will
clean it up a lot. In its present form, it records "request" and "delivery"
events separately, indicated by the "type" column.

Each request shows the segment number being requested and the time at
which the get_segment() call was made
Each delivery shows:
segment number
range of file data (as OFFSET:+SIZE) delivered
elapsed time spent doing ZFEC decoding
overall elapsed time fetching the segment
effective speed of the segment fetch
"Requests": this shows every block-request sent to the storage servers.

Each line shows:

the server to which the request was sent
which share number it is referencing
the portion of the share data being requested (as OFFSET:+SIZE)
the time the request was sent
the time the response was received (if ever)
the amount of data that was received (which might be less than SIZE if we
tried to read off the end of the share)
the elapsed time for the request (RTT=Round-Trip-Time)

Also note that each Request line is colored according to the serverid it was
sent to. And all timestamps are shown relative to the start of the first
read() call: for example the first DYHB message was sent at +0.001393s
about 1.4 milliseconds after the read() call started everything off.

I'm slowly (i.e. post-1.8.0, maybe in a 1.8.1) changing the data on this page and adding visualizations, so I don't want to put too much energy into documenting a transient/unstable data structure quite yet. But here's a quick summary of what's on the new downloader status page. First, what's involved in a download?: * downloads are triggered by `read()` calls, each with a starting offset (defaults to 0) and a length (defaults to the whole file). A regular webapi GET request will result in a whole-file `read()` call * each `read()` call turns into an ordered sequence of `get_segment()` calls. A whole-file read will fetch all segments, in order, but partial reads or multiple simultaneous reads will result in random-access of segments. Segment reads always return ciphertext: the layer above that (in `read()`) is responsible for decryption. * before we can satisfy any segment reads, we need to find some shares. ("DYHB" is an abbreviation for "Do You Have Block", and is the message we send to storage servers to ask them if they have any shares for us. The name is historical, from Mnet/MV, but nicely distinctive. Tahoe's actual message name is `remote_get_buckets()`.). Responses come back eventually, or don't. * Once we get enough positive DYHB responses, we have enough shares to start downloading. We send "block requests" for various pieces of the share. Responses come back eventually, or don't. * When we get enough block-request responses for a given segment, we can decode the data and satisfy the segment read. * When the segment read completes, some or all of the segment data is used to satisfy the `read()` call (if the read call started or ended in the middle of a segment, we'll only use part of the data, otherwise we'll use all of it). With that background, here is the data currently on the download-status page: * "DYHB Requests": this shows every Do-You-Have-Block query sent to storage > servers and their results. Each line shows the following: * the serverid to which the request was sent * the time at which the request was sent. Note that all timestamps are relative to the start of the first `read()` call and indicated with a "`+`" sign * the time at which the response was received (if ever) * the share numbers that the server has, if any * the elapsed time taken by the request * also, each line is colored according to the serverid. This color is also used in the "Requests" section below. * "Read Events": this shows all the FileNode `read()` calls and their > overall results. Each line shows: * the range of the file that was requested (as `OFFSET:+LENGTH`). A whole-file GET will start at 0 and read the entire file. * the time at which the `read()` was made * the time at which the request finished, either because the last byte of data was returned to the `read()` caller, or because they cancelled the read by calling `stopProducing` (i.e. closing the HTTP connection) * the number of bytes returned to the caller so far * the time spent on the read, so far * the total time spent in AES decryption * total time spend paused by the client (`pauseProducing`), generally because the HTTP connection filled up, which most streaming media players will do to limit how much data they have to buffer * effective speed of the `read()`, not including paused time * "Segment Events": this shows each `get_segment()` call and its > resolution. This table is not well organized, and my post-1.8.0 work will > clean it up a lot. In its present form, it records "request" and "delivery" > events separately, indicated by the "type" column. * Each request shows the segment number being requested and the time at which the `get_segment()` call was made * Each delivery shows: * segment number * range of file data (as `OFFSET:+SIZE`) delivered * elapsed time spent doing ZFEC decoding * overall elapsed time fetching the segment * effective speed of the segment fetch * "Requests": this shows every block-request sent to the storage servers. > Each line shows: * the server to which the request was sent * which share number it is referencing * the portion of the share data being requested (as `OFFSET:+SIZE`) * the time the request was sent * the time the response was received (if ever) * the amount of data that was received (which might be less than SIZE if we tried to read off the end of the share) * the elapsed time for the request (RTT=Round-Trip-Time) Also note that each Request line is colored according to the serverid it was sent to. And all timestamps are shown relative to the start of the first read() call: for example the first DYHB message was sent at `+0.001393s` about 1.4 milliseconds after the read() call started everything off.

zooko commented

2010-08-12 20:45:00 +00:00

Author

Thanks for the documentation! We should add this into the source:docs directory, and/or I wonder if it would be too disruptive to make it be statically served up by the wui and add a hyperlink on the download status page to it.

warner commented

2010-08-12 21:41:16 +00:00

heh, nice keyword :)

eh, I'm -0.. it'd be nice if folks could find out what the table means without searching for this ticket, but also we've got a lot of random little displays and it'd be a long process to provide (and maintain!) docs for each of them.

That said, it wouldn't be very hard to copy this into source:src/allmydata/web/ and include a static link to it on that page. Somebody (me?) would have to remember to change it once the changes I'm working on finally land. As a curious user poking around, I'd probably appreciate seeing that link.

heh, nice keyword :) eh, I'm -0.. it'd be nice if folks could find out what the table means without searching for this ticket, but also we've got a lot of random little displays and it'd be a long process to provide (and maintain!) docs for each of them. That said, it wouldn't be very hard to copy this into source:src/allmydata/web/ and include a static link to it on that page. Somebody (me?) would have to remember to change it once the changes I'm working on finally land. As a curious user poking around, I'd probably appreciate seeing that link.

zooko commented

2010-08-14 06:02:30 +00:00

Author

Copied Brian's docs from comment:380378 into source:trunk/docs/frontends/download-status.txt in changeset:36f698b6371f82e8. I'm going to change this ticket to be about the idea of linking that document into the WUI itself with a "What does this mean?" link (as per comment:380379 and comment:380380).

Copied Brian's docs from [comment:380378](/tahoe-lafs/trac/issues/1169#issuecomment-380378) into source:trunk/docs/frontends/download-status.txt in changeset:36f698b6371f82e8. I'm going to change this ticket to be about the idea of linking that document into the WUI itself with a "What does this mean?" link (as per [comment:380379](/tahoe-lafs/trac/issues/1169#issuecomment-380379) and [comment:380380](/tahoe-lafs/trac/issues/1169#issuecomment-380380)).

zooko modified the milestone from soon to undecided

2010-08-14 06:02:30 +00:00

zooko changed title from ~~documentation for the new download status page~~ to include documentation for the new download status page linked from the page itself

2010-08-14 06:02:30 +00:00

daira modified the milestone from undecided to 1.9.0

2010-08-21 22:02:26 +00:00

zooko commented

2010-09-18 18:06:22 +00:00

Author

Warning: we're going to integrate the new visualization tool, see #1200, which will change what is displayed on the download status page and/or will change how it is displayed. So you might want to fix this ticket to add help/documentation for the new version instead of the current version.

zooko commented

2011-09-28 19:31:12 +00:00

Author

I think now that the New Visualizer has landed we need to update source:trunk/docs/frontends/download-status.rst.

zooko changed title from ~~include documentation for the new download status page linked from the page itself~~ to update documentation for the new download status page linked from the page itself

2011-09-28 19:31:12 +00:00

zooko changed title from ~~update documentation for the new download status page linked from the page itself~~ to update documentation for the download status page

2011-09-28 19:44:11 +00:00

warner commented

2011-10-28 04:48:28 +00:00

not happening in 1.9

warner modified the milestone from 1.9.0 to 1.10.0

2011-10-28 04:48:28 +00:00

warner commented

2011-10-31 03:16:52 +00:00

FYI the docs in source:docs/frontends/download-status.rst are correct. The timeline is an alternative way of viewing that same data, but the first link the user can get to is a big HTML table, which download-status.rst (mostly) correctly describes. It might be nice to make those docs available as a hyperlinked "about this table" page, but that's not super critical.

zooko commented

2012-05-01 23:20:44 +00:00

Author

Okay, this ticket is just about making sure that source:docs/frontends/download-status.rst is a correct description of the "download status" page which is the big HTML table, not the Javascript visualization. Brian, would you please review that document and fix anything that is incorrect or stale? Thank you!