Dropbox Memcache

老师请问您在Dropbox的系统设计图中为什么要在Metadata base上加上一个Memcached呢?

我看到

这里的这个报告的原因我感觉跟咱们涉及的架构是有出入的,所以这个原因不是很信服。类似说的

  1. 它这个架构中metadata server和block server之间要talk,但是距离遥远,它会在metadata server附近加一个Memcached,为了节省round trip。
  2. 为了数据的备份和分片

它都通过了加一个memcached来实现。

但是咱们这种架构中metadata server 和block server之间并没有直接联系,这里加这个Memcached的目的是什么呢? 我查了您脚注里给的那个dropbox的那篇文章。它说,这个Memcached家在metadata db上的目的是存储额外信息,为了防止上传客户端上传一半不想传了或者各种其他造成commit失败,而记录的一些失败的经历。

文章里提了一个在进行优化之后,需要发生的变化有:

Protocol changes

Turns out the uploading client doesn’t need to change its behavior. However, the downloading client needs to hear about non-SFJ changes. We handled this by adding additional output to list. List now returns new SFJ rows as well as new Streaming Sync prefetchable blocklists.
这两个黑体字都是什么意思。list是哪个list,在之前您画的若干图里,这个prefetchable blocklist是什么?

Metaserver changes

We decided to store the additional state in memcache rather than a persisted table. It is not vital that this data persists, and we chose not to incur the additional cost of writing to a table on a failed commit. The memcache entry looks very similar to an SFJ row, except it is versionless. Thus there is no need for a JID.

Writes to memcache occur on failed calls to commit
Reads from memcache occur on list
Deletes occur on successful calls to commit (or memcache evictions)

Memcached这些动作有什么意义呢?这里只是对它进行增删,对解决错误传送怎么起到作用呢。

Client changes

Clients must now maintain a “prefetch cache” of blocks which do not correspond to files in SFJ. Upon list, a client queues up prefetches, which go into this prefetch cache. On new versions of the client, you can find this in the “.dropbox.cache/prefetch_cache/” directory.

黑体字的意思是什么意思呢。看来这个prefetch cache是在客户端上

We need to make sure that even if the file never completes commit, we don’t break the process. Imagine starting an upload and changing your mind midway. We need to make sure that the server side memcache expires entries and does not thrash. Furthermore, we need the DL client’s prefetch cache to be purged periodically and intelligently so it does not grow unbounded.

When accessing blocks, we check validity against the memcache table, but since memcache entry may be mutated, unavailable, expired, or even evicted during the prefetch, we need to arrange fallback behavior on the client when the server cannot verify that a block is eligible for streaming sync. This involved adding special return codes to the store protocol to indicate this condition.

这里面第二句黑体字尤其不明白,这都是metadata上的memcached能解决的问题么?因为没有图,完全不知道在说什么。