-
Notifications
You must be signed in to change notification settings - Fork 0
Meetings
Quinton Hoole edited this page Apr 19, 2019
·
1 revision
- India Team
- Quinton
- Donghui Zhuo
- Reviewed proposed OMS API and Kernel Server API. But there are
- Cancelled Terry's plan (or proposal) to reduce DCAP scope. Quinton emphasized that we will strive to deliver full features set as we planned out earlier, including object health check, full multi-language support, and diamond integration.
- Cancelled the plan to sapphirize face recognition application. Later on, after another discussion, Quinton agreed to proceed on the condition that it won't distract the attention or affect the productivity of core developers who are working on Sapphire core and multiple language support.
- India Team
- Haibin Michael Xie
- Donghui Zhuo
- Quinton
- Sungwook
- Meeting Memo: https://docs.google.com/document/d/13o5cH8nhYLPOftGG69Gn8Uk9ONCxmjfdh4LZJO8Vpp4/edit#heading=h.8273yuqmw09t
- Yulin Sun
- Ying Xiong
- Vincent Li
- Donghui Zhuo
- Some Diamond features can be readily used in edge computing, e.g. replicated storage service, reliable transaction, reactive transaction etc.
- Code offloading may potentially be used to handle facial detection peak load. We first encapsulate facial detection into one sapphire object. The sapphire object fetches images as input. During peak hour, if the resource (e.g. cpu or memory) of an edge device runs low, the facial detection sapphire object can be offloaded to neighbor edge devices to improve performance.
- Weixuan Fan
- Yue Yu
- Ying Xiong
- Donghui Zhuo
- DCAP should focus on edge computing
- Forget about DCAP's current architecture. Reevaluate DCAP capabilities to see which capability can be used on edge computing platform
- DCAP Seattle Team
- Since we will open source DCAP in the future, we will keep using github repository for development, and periodically sync PRs from github to futurewei repositories
- Keep Sapphire, Diamond, and Tapir in separate repositories. Keep deployment code and Sapphire core in separate repositories.
- Three use cases for Diamond and Tapir. First, applications can use Diamond/Tapir directly. Second, we create a Sapphire object that provides key value store service, and this Sapphire object is backed up by Diamond and Tapir. Third, we use Diamond and Tapir in some of our DMs, e.g. Checkpoint DM.
- Review high level design of code offloading DM in the next week.
- Irene Zhang (UW)
- DCAP Seattle Team
-
Who should be in charge of creating new Sapphire object in case of object failure, OMS or DM? According to Irene, DM is probably more appropriate, because sometimes only DM knows how to construct the Sapphire object instance (e.g. reload a checkpoint). OMS monitors the health of object. If an object goes down, OMS can notify the coordinator the DM (the group policy), and the coordinator can spin up a new Sapphire object.
-
How about running DM logics in a separate process? According to Irene, it is okay to run DM in separate processes. Performance and resource consumption may be potential downsides though. It is wasteful to launch a JVM just to run one line of DM code. If we are willing to put a restrict such that DMs can only be written in C++, then we can create DMs and Kernel Servers in C++, and use the C++ Kernel Server to host Java runtime, Python runtime or Go runtime. We can take a look at Diamond to figure out the details.
-
Irene mentioned that the main functionality of Franklin is to provide a reliable RPC mechanism between services. The scope of Franklin is smaller than Sapphire. We can create a reliable RPC DM for Sapphire.
- Irene Zhang (UW)
- Niel Lebeck (UW)
- Adriana Szekeres (UW)
- DCAP Settle Team
- Quinton disclosed the plan to open source DCAP project in a key note presentation at KubeCon China on Nov 14-15.
- Irene mentioned that Microsoft is working on a similar project named Franklin. But we are not able to get any public information yet.
- Irene mentioned that UW syslab is testing TPC-C on Eris
- Irene mentioned that fine granularity on objects causes serious performance issue in Orlean. She will bridge us with Orlean people for further discussion.
- Discussed Multi-DM design at high level. Response from Irene is positive.
- Discussed Side effect handling in Sapphire. But it does not seem to have a good solution.
- Zhenqing Liao
- Weixuan Fan
- Lichang Mu
- Quinton
- Seattle Team
- Demoed License Plate Mobile App
- Demoed Go Board Game App
- Demoed LLVM Code Injection
- Feedback
- Feedbacks are positive
- There are innovative elements in DCAP Sapphire. We should spend more time and keep exploring in this area.
- Weixuan Fan
- Yue Yu
- Lichang Mu
- Krishna
- Sanil
- Donghui Zhuo
Below are some points I collected during my stay in China. They are not from one specific meeting, but a summary from many meetings.
- Everyone agrees DCAP is a direction worth exploration.
- People understand that DCAP will be a long term project which may not be able to generate value in near term.
- Many people suggest DCAP team to work hard to find one use case to apply DCAP technology.
- Weixuan suggests have more discussions with UW faculties and startup companies to identify top 3 problems that can be efficiently solved by DCAP. Try not to implement too many features in the beginning. Focus on solving key problems.
- I talked to DDM team, Consumer Cloud team, ServiceComb team. ServiceComb team says they have interest in Leader Election DM.
- Ying suggest we create one collaboration project with UW, e.g. to solve DCAP multi-language issue.
- Krishna
- Sanil
- Donghui Zhuo
- Find real, concrete use case for DCAP is very important
- Here is one imaginary use case: Suppose we want to do facial recognition on every security camera. Assuming the model size is very big that can not fit on one camera. (This needs to be verified). Then we can split this big model into many smaller models. Each camera runs a smaller model. Given an image, the smaller model may not be able to recognize the image. So the camera needs to replicate the image to nearby cameras for result. This can be easily achieved with DCAP. But it will be difficult to achieve with serverless Similar, it is possible that the camera on one intersection is super busy and does not have enough capacity to process all requests. In this case, the camera can distribute some requests to nearby cameras.
- Ning Jiang
- Donghui Zhuo
- Saga agents need leader election logic. We can create leader election DM and use it in Saga.
- Yue Yu
- Qi Zhang
- Longchun Zhang
- Xiaoning Ding
- Donghui Zhuo
- Code Offloading Demo: Proposed code offloading ideas look ok. We can go ahead to implement the license plate recognition app.
- High Availability Demo: The proposed demo demonstrates DCAP's high availability capability, not DCAP's value. Since the key value of DCAP is to simplify the development of distributed applications, our demo should demonstrate how the application can be simplified by adopting DCAP. Ideally, we should demonstrate this value by porting a real existing distributed application onto DCAP, and do a side-by-side comparison. The team had an extensive discussion on this topic, and didn't agree on which application to port. The conclusion was we should use DCAP to build a high available in-memory key value pair storage service. This storage service will be used by code offloading mobile application to save data. In short, we create one complete application to demo both features.
- Today DCAP relies on object serialization to save object state. For RPCs which mutate states outside of a Sapphire object (e.g. update a database table), their effects cannot be completely rolled backed by DCAP. Team expresses concerns on this limitation. This limitation will not be resolved before March, but it will eventually be resolved.
- License plate recognition and facial recognition turn out to be pretty fast on cell phones. We are not confident that we will be able to demonstrate significant performance improvement by code offloading. This risk was called out in the meeting. We will keep looking for other ways to demo the value of code offloading.
- Design and implement the license plate recognition app to demo the DCAP code offloading capability.
- Design and implement an in-memory key value pair data store with consensus DM to demo DCAP high availability capability.
- The code offload application should use the data store to save data.
- The architect team will revisit DCAP project plan by the end of March.
- Ying Xiong
- Quinton
- Yue Yu
- Donghui Zhuo
-
We need to demo the following two capabilities by the end of March
- Demo a mobile application with the capability to offload facial recognition or voice recognition to remote server. Users should experience performance improvement by doing code offloading.
- Demo the capability to provide high availability in a distributed environment by using DCAP.
-
Deep Dive on Tapir
- Verify the proof of Tapir
- Evaluate the current implementation of Tapir
- Ken Liu
- Qi Zhang
- Terry Zhuo
-
Ken mentioned that software upgrade is a big challenge for most mobile applications. The cost of upgrading device side components is very high. Software upgrade can be a potential use case for DCAP. Mobile app developers only need to upgrade components on server side, DCAP can take care of client side component upgrade.
-
No one use Java to write mobile games. In order to support game development, DCAP needs to support game SDKs like Unity.
- Ken will ask him team to take a look at Sapphire paper and Diamond paper. We will have more discussions with Ken's team down the road.