Day23 Lab 2 - Object storage的RAID实作2

在我们的Lab架构中，Object的分割是在API层做的，其实我们的API层要做的事情满多的，最好的情况是还能把API层拆成两部分，这部分可以留给有兴趣的读者做

在API层接收到Object之後，我们的流程是

1. 对Object制造奇偶校验
2. 分割成六个components
3. 把这六个components分别随机送到不同的data server上
4. data server接收到components後便存在server里

程序逻辑如下

PARTITION = 6
def splitData(data):
   rsc = RSCodec(len(data))
   res = rsc.encode(data)  # 第1步
   n = len(res)//PARTITION
   big_num = len(res)%PARTITION
   left_start = big_num*(n+1)

   return [res[i*(n+1):(i+1)*(n+1)] for i in range(big_num)] \ 
					+ [res[left_start+i*n:left_start+(i+1)*n] \ 
						 for i in range(0, PARTITION-big_num)] # 第2步

def produce_object(content):
   if 'name' not in content:
      return False
   size = len(content['obj'])
   components = splitData(content['obj'])
   servers = getDataServers()
   locats = []
   for i, comp in enumerate(components):
      while servers:
         idx = random.randint(0,len(servers)-1)
         server = servers[idx]
         res = Producer.getInstance().send(
						server, value={f"{content['hash']}-{i}": comp}) # 第3步
         if res:  # 第4步
            locats.append((server, i, len(comp)))
            break

      del servers[idx]

   if len(locats) == len(components):
      DB.addMetadata(f"{content['name']}", content['version'], content['hash'], size, locats)
      return True
   else:
      print(f"Numer of components: {num(components)} and number of locates: {len(locats)} are not match.", flush=True)

   return False

https://github.com/kaichiachen/python-simple-object-storage/blob/master/distributed_obj_system/backend/api/obj.py#L35

由於我们使用hash当作档案名称存在data server，所以读Object的时候，只要有hash就可以读出来，程序逻辑如下

   for server, idx, size in locate:
      try:
         res = requests.get(f"http://{server}:{PORT}/partition/{hash}-{idx}", timeout=1)
         if res.status_code == 201:
            return -1
         else:
            data += res.content
      except Exception as e:
         print('request error: %s' % e, flush=True)
         data += b'x'*size # 如果server有问题则随意填充
   print(data, flush=True)
   try:
      res = combineData(data)
      return res

https://github.com/kaichiachen/python-simple-object-storage/blob/master/distributed_obj_system/backend/api/obj.py#L42

Encode和decode的演算法读者可以自行变换使用