Demo App
- Sapphire Coding Experience Demo
- DCAP Code Offloading Demo with Raspberry Pis
- Go Game
- License Plate Recognization
Identifying potential customers and convincing customers to use DCAP is important and challenging. To facilitate onboarding process, we need to develop a couple of demo applications to demonstrate the following two DCAP features by the end of March.
- Code Offloading Capability: Automatically offload some computing intensive jobs between device and server. Ying and Yue suggested use facial recognition and voice recognition.
- High Availability Capability: Demonstrate the high availability of replicated Sapphire objects. Failures on minority replicas will not affect the availability of the service provided by the Sapphire object.
We can demo code offloading for the following use cases:
- Go: This is a mobile app for board game Go. The app has two buttons: play-on-device and play-on-cloud. If play-on-device, then the next move will be calculated on device; if play-on-cloud, then the next move will be calculated on cloud.
- License Plate Recognition: This is a mobile app which allows users to pick a picture (which contains one or many license plates) and then press the analyze button to trigger license plate recognition. If the picture has one license plate, then the recognition will be performed on device; if the picture has many license plates, then the recognition will be performed on cloud.
- Face Recognition: This is a mobile app to train a Face Recognition model using Face images as input. For compute intensive parts of training the images, the code can be offloaded to cloud/fog/edge devices (with considerable resources). In a real deployable use case like a surveillance system, ideally first the Face Detection is done and then the Face Recognition is performed. From the live stream video, few frames at a time after some pre-processing can be sent to cloud/fog/edge devices (with considerable resources) for processing and identification and recognition of the face images. While training, some compute intensive parts can be offloaded to cloud or while using the system, a stream of pre-processed images can also be sent to cloud/fog/edge devices (with considerable resources) for recognising.
Game of Go program is well considered as an area of AI/neural network application. Neural network AI typically requires intensive computation resources and presents the value of migrating computation to more capable places.
Pachi is one of the open-source desktop project able to leverage neural network for Go playing. With neural network in place, Pachi is about amateur level 1D (based on personal evaluation). Android Pachi is an open-source Android project; its Pachi game engine is a bit out of date without neural network AI ability yet.
Our goal is to convert the original Pachi android application (after bringing it up to date by porting the neural-network-enabled engine to ARM arch) to the one being able to render go plays on android device and cloud host both, to demonstrate the performance gain made by shifting computation to remote cloud host - all is done in transparent means which is one of the features of Sapphire model.
Very minimum system - 1 OMS node + 1 cloud host node + 1 android device
Amazon Kindle Fire tablet 2 ARM cores 1GB RAM
Huawei Mate 9 (for comparison purpose) 4 ARM cores 4GB RAM
Go game engine built for armeabi
running on Lenovo ideapad 320 (4 AMD cores, 4GB RAM)
Ubuntu Linux 17.10
sharing the laptop of OMS node
Go game engine built for x86_64
Neural network simulation: 5000-15000; search thread number: 4 (same on android device & cloud host)
| Kindle tablet | Huawei Mate 9 phone | Cloud Host | |
|---|---|---|---|
| time to take for best move | 19 seconds | 6-7 seconds | 5-6 seconds |
The target android device must be ARM based build the android-pachi project and locate the application package (app-debug.apk); run following adb commands
adb push app-debug.apk /data/local/tmp/
adb pm install -t /data/local/tmp/app-debug.apkEnsure the cloud node is of x86_64 (or AMD) arch, Ubuntu 17.x or later. OMS and the cloud node are packaged as docker images for the sake of simplicity to run demo reliably. To start OMS run:
docker run -p 22343:22343 -p 35459:35459 -e OMS_HOST=<OMS-machine-ip-addr> pachi-demo-oms:0.2 bash -c 'cd /data && /data/start-oms.go2.sh'To start the cloud node,
docker run -p 22345:22345 -p 35455:35455 -e OMS_HOST=<OMS-machine-ip-addr> -e CLOUD_HOST=<Cloud-node-ip-addr> pachi-demo-cloud:1.1 bash -c 'cd /data && ./start-ubuntu-ks.go2.sh'
License Plate App (OpenAlpr) was successfully sapphirized to work both on device and server, all via Sapphire kernel.
Letter size paper which displays 50 vehicle license plates from various states in US.
- Kindle: Amazon Kindle device
- Huawei phone: Huawei Mate 9 (flagship device)
- Cloud: AWS EC2 instance with 4GB memory
All the pictures were resized not to have bigger than 1600px. Note that Kindle has 1600px so resize time was not skipped after simple size checking.
For processing at Cloud environment, the resized image is uploaded via Sapphire RMI call. Therefore, total time at Cloud had additional time for file transfer but not when it is processed at the device.
All the time is in seconds. Native call time indicates time it takes from calling the native (C++) code method for processing the image.
m: misc time
r: resize time
n: native call time
u: file upload time
- Total time at device = m + r + n
- Total time at cloud = m + r + u + n
For a device with average power (Kindle), Sapphire showed server side processing can be at least 4x faster even including the file uploading time. For a powerful device (Huawei Mate), Sapphire showed server side processing can still be faster (~50%) despite a few seconds of file uploading time.
- Kindle
| Kindle | Kindle w/Cloud | |
|---|---|---|
| Total Time | 43.61 | 10.67 |
| Resize Time | 0.17 | 0.18 |
| Upload Time | 0 | 2.29 |
| Native call time | 35.8 | 3.19 |
| # of Plates found | 45 | 47 |
- Huawei Mate 9
| Huawei phone | Huawei Phone w/Cloud | |
|---|---|---|
| Total Time | 11.82 | 7.78 |
| Resize Time | 0.37 | 0.38 |
| Upload Time | 0 | 1.64 |
| Native call time | 9.06 | 3.61 |
| # of Plates found | 45 | 47 |
Deployment manager that will decide where to process depending on the configuration.
- (3/13/2018) Code was merged for device and server. Image is resized. Device can run Sapphire kernel.
- (2/28/2018) License plate works both on device and server.
- (2/2/2018) We moved on to use OpenAlpr as EasyPR is limited to a small box and Chinese plates only. OpenAlpr provides rich set of configuration with an ability to process multiple plates in a full size picture.
- (1/30/2018) We decided to go first with EasyPR which was tested to work. Processing speed is a core concern (takes short time) and needs to be tested with a large picture.
- Short processing time: If processing takes very short time (e.g., less than 500 milliseconds), demo may not be meaningful. Bigger size of image can be considered for testing if processing time is reasonably slow.
- Code offloading for OpenCV: It may not be feasible to serialize the object to push to the server due to its dependency on C++ library. In such a case, the server will have to start without preserved state.
OpenALPR is an open source Automatic License Plate Recognition library written in C++ with bindings in C#, Java, Node.js, and Python. This project ports this library to Android. This is the app used in our final experiment.
This app provides many tweaking configurations that might affect accuracy and processing time including number of thread cores, max detection width/height, LBP frame increase and strictness. It currently supports US and EU plates but this may be configurable for other countries as well.
Configuration: assets/runtime-datea/openalpr.conf
- Device used: Huawei Mate 9 (launched at Jan 2017)
- Camera resolution: 20M pixels (5120*3840 - unclear whether the app uses this resolution)
- 1 Plate image URL
- 50 Plates image URL
- single plate, small center, ~1 sec
- 50 plates, full fit, ~13 secs (misses 20 plates)
- single plate, small center, ~5 sec
- 50 plates, full fit, ~20 secs
- single plate, small center, 9 sec
- 50 plates, small center, 20 sec (misses 23 plates)
- 50 plates, medium center, 25 sec (misses 4 plates)
- 50 plates, full fit, 25 sec (misses 6 plates)
- single plate, small center, 11 sec
- single plate, full fit, 25 sec
- 50 plates, medium center, 28 sec (misses 4 plates)
- 50 plates, full fit, 27 sec (misses 3 plates)
- Number of cores did not affect processing time.
- Detection strictness did not affect processing time.
Face Recognition involves recognizing of a person whose images have been used before for Training. Usually, a lot of face images, in varying pose, illumination conditions, expressions are taken and fed into a Face Recognition system which learns the characteristics of the face and becomes capable of recognizing the person, given face images of different people.
Face Recognition system consists of overall two parts to it namely, Training of the face images and then the Testing of the face image, where given an image of person, the system will recognize the person. Within training phase itself, usually the face image goes through multiple phases namely, Pre-Processing, Feature Extraction, Feature Selection and Classifier Training. All these phases can be employed in various combinations, and in some cases, some of the phase may not be required/employed. Although these When the test image is given also, the face image goes through all the similar phases and then the classifier is used to recognize the person.
In a real deployable use case like a surveillance system at edge, ideally first the Face Detection is done and then the Face Recognition is performed. From the live stream video, few frames at a time after some pre-processing can be sent to cloud/fog/edge devices (with considerable resources) for processing and identification and recognition of the face images. While training, some compute intensive parts can be offloaded to cloud or while using the system, a stream of pre-processed images can also be sent to cloud/fog/edge devices (with considerable resources) for recognising.
Employed Caltech Faces database. Created a sub database out of the original database. The sub face database consisted of 10 subjects and 10 images per subject, so a total of 100 face images.
Employed TanTriggs Algorithm as part of Pre-Processing and Feature Extraction & Random Forest Algorithm for Classification. Used Training to Testing ratio as 7:3, i.e used 7 face images per subject for training and 3 face images per person for testing and calculating the Face Recognition Rate.
As part of this experiment, sapphirized Random Forest class using Explicit Migration DM. As training Random Forest model is very time taking and compute intensive.
- Laptop: HP EliteBook
- Android Mobile: Huawei Honor 8 Pro (one of the Huawei Honor Flagship)
All the below mentioned measurements are done in milliseconds i.e abbreviated as ms
- Total Pre-Processing and Feature Extraction time as part of Training - 31056 ms
- Total Classifier Learning time as part of Training - 47952 ms
- Total Execution time for Training - 79008 ms
- Total Pre-Processing and Feature Extraction time as part of Testing - 15166 ms
- Overall Face Recognition Rate - 86.67%
- Total Execution Time for Testing - 15173 ms
- Total Pre-Processing and Feature Extraction time as part of Training - 32242 ms
- Total Classifier Learning time as part of Training - 14043 ms
- Total Execution time for Training - 13711 ms
- Total Pre-Processing and Feature Extraction time as part of Testing - 13711 ms
- Overall Face Recognition Rate - 86.67%
- Total Execution Time for Testing - 15173 ms
From the experiment results, it can be observed that, Total Classifier Learning time as part of Training reduced from 47952 ms to 14043 ms, i.e an improvement by 3.5 times.
Employed a dataset on Forest Cover Type from UCI Machine Learning Repository. Employed K Nearest Neighbours algorithm to train the Classifier which can be used to identify the cover type of forest, given some parameters about the forest like Elevation, Aspect, Slope, Soil Type, etc.
- Training Data - 200000 (2 Lakh) diferrent data elements
- Test Data - 100 data elements
Sapphirized the KNearestNeighbour class successfully using ExplicitMigration DM, so that operations on this class can be offloaded to more powerful compute device.
- Laptop: HP EliteBook
- Android Mobile: Huawei Honor 8 Pro (one of the Huawei Honor Flagship)
All the following results are the average of 5 iterations. Before DCAP, the Laptop was used, whereas after DCAP, Android Mobile was used.
- Total Test Time - 84s
- Accuracy - 82%
- Total Migration Time - 1s
- Total Test Time - 4.4s
- Accuracy - 82%
From the experiment results, it can be observed that, after employing DCAP, the Total Test Time reduces overall from 84s to 4s, i.e an overall improvement by 21 times.
(Discarded) An easy, flexible, and accurate plate recognition project for Chinese licenses in unconstrained situations. The instruction is written all in Chinese. Image process is almost instant; however, it can only recognize the image in a small box windows and only supports Chinese license plate.
Adding an image took 0.47 second according to an author. Searching through images takes more than one second for 30,000+ images.
- Limitation: will need Java wrapper as it was only written in C++ with no support for Android.
Pastec is an open source index and search engine for image recognition based on OpenCV. It can recognize flat objects such as covers, packaged goods or artworks. It has, however, not been designed to recognize faces, 3D objects, barcodes, or QR codes. Pastec can be, for example, used to recognize DVD covers in a mobile app or detect near duplicate images in a big database. Pastec does not store the pixels of the images in its database. It stores a signature of each image thanks to the technique of visual words.
Pastec offers a tiny HTTP API using JSON to add, remove, and search for images in the index. https://hosunghwang.wordpress.com/2015/06/28/pastec-test-for-performance/ https://github.com/magwyz/pastec
Written in Python. No support for Android. https://cmusatyalab.github.io/openface/
Processing time is 60ms. https://github.com/gicheonkang/fast_face_android
Blurring is very common in image effect; thus, can be an intuitive use case. It also consumes CPU and memory heavily depending on the size of the image. It does not require model training or OpenCV; therefore, it is achievable in the small time frame.
Blurry is a easy blur library for Android. It uses Android library in Java; therefore, it can easily be sapphirized or used for other apps. Source is available at GitHub
This is a simple benchmark and showcase app on what's possible with blurring in Android 2016. Noteably this app uses Android's Renderscript v8 support library for fast blurring. This is available at Google Play. Source code is available at GitHub
- 1/26/2018 (Fri): Understand about image/voice recognition and look into open source apps we can find for this use case.
- 1/30/2018 (Tue): Sync-up to write a plan to check with Ying/Quinton.
OpenCV is C++ library used in our sister team in Seattle which demonstrates facial recognition; therefore, it can be one area to look at (POC: Keith & Chris). For Java - https://docs.opencv.org/2.4/doc/tutorials/introduction/desktop_java/java_dev_intro.html
One observation is that a single image processing tends to take very short time (e.g., millisecond). In such a case, it may not be meaningful to offload the work to server as data transmission time may take more than processing time. One idea to demonstrate computation heavy scenario is downloading pictures from cloud (e.g, Google photo) and process them in batches in the server vs device to find a person of interest based on training data. Output might be a few selected pictures that include person of interest.
Image blurring looks to consume a bit of time (1sec-30secs depending on algorithm and size). Combining this in facial recognition can add up some computation with a useful feature.
Notes: We did not choose facial recognition for two reasons:
- OpenCV is written in C/C++. We are not able to offload C/C++ code.
- Facial recognition models are built inside C/C++ code. As far as we know, OpenCV does not provide the capability to export models. We have to train the model and use the model on the same host which makes model offloading impossible.
- If we want to use a model/app that uses OpenCV, the app will have to have a trained model at both device and server so that the app does not have to pass the model to the server. When the method is invoked on the server, it will call OpenCV in the server and process the request based on the model it has installed (e.g., docker container initiated them). It will resemble the API call where device simply passes the input parameter and get the result back.
- More details: Sapphire-Objects-in-OpenCV-Android-apps
- Image blur: Image blur is a computing intensive job. The app will present the user with a list of pictures. The user can select one or more pictures, then press blur button to trigger image blur process. If the user selects only one picture, the blur work will be done on the device. If the user selects multiple pictures, the app will offload some workload to the remote server.
This doc is to describe the challenges to apply Sapphire model to Android applications using OpenCV native calls.
OpenCV is written in C for optimal performance. Android applications make use of OpenCV via JNI. Its most interesting aspects for us is the image processing algorithms and ML. OpenCV image processing is intensive stateless computation, ML is comprehensive process typically of discrete separate training/evaluation/predication, an essentially stateful computation.
The relevant scenario of the interests is code offloading by Sapphire model between android device and cloud (Linux host).
It is technically trivial to offload code and move the computation around for the stateless objects.
The heavy lifts of training and evaluating are done entirely inside of native calls; glue layer is presented as a very thin wrapper of java static classes with the minimum code - typical mere passing of parameters/results.
Generally, applications keeps the training data in some form (as files of certain file system, or records in some database). These data feeds directly to OpenCV function for training and building the ML model internally. On predication, application passes the input data directly into proper OpenCV function, and gets back the evaluation result - usually the best match index, or the one of minimum distance.
Simply turning the glue java classes into Sapphire objects won't be able to move the internal model around. Due to the limitation of current Sapphire implementation, it seems infeasible to apply Sapphire model to native binary code generated by C and move the OpenCV internal model data in memory with the java glue objects. Code offloading of the glue layer class without consideration of the ML internal model state may not lead to the desired result.
For the Sapphire model to make sense in such ML-related applications for code offloading scenario, the internal OpenCV ML model needs to be treated carefully.
- Proposal 1
to establish the internal ML model from the training data set. The training data set may be different in number or quality, e.g. data set on cell phone could be much smaller than that on a beefy Linux server. Applications needs to ensure the ML training process well done when the glue layer object is moved by certain Sapphire DM policy.
- Proposal 2
For some ML too expensive to build the model on demand, an already-trained model (e.g. opencv official list of trained models) can be maintained on the places that might be the migration target of the glue objects.
opencv official list of trained models
- Ported Apps in GitHub
- Requirements and Suggestions
- Top Candidates
- issues
- Candidate List
- References
- Sapphirization
- Porting Issues
Master branch has the original repo (except MovieGuide & EasyPR). For Sapphirization, look for non-master branch.
Master branch has the FavoriteStoreManager sapphirized. getReviewSuccessful branch has Sapphirization of GetReview API (tested successfully working). NetworkSapphire branch has Sapphirization of NetworkModule (which fails at serialization).
(by Quinton) Ideally we want an app with a front-end and a backend, and parts that could run on either. Preferably with some significant processing, and sharing of data between multiple clients.
(Sungwook) Many open source apps are either 1) client-only or 2) server side code is missing or 3) requires merge of client and server side code. As one of the value propositions is easiness of development using distributed computing, I think making client-only app work with server can be valuable showcasing. In some cases, client-only app was not intended. If we can have coding framework to make the existing client-only app to support multi-user environment via Sapphire, that may also be useful. For example, we can provide general UserManager that is Sapphire object. This may be useful when number of user objects may spike and needs to have them in server side with auto-scaling. If this becomes true, we can port many client-only applications to have it.
(Quinton) I agree with the above. This is what we plan to do for OmniNotes below.
OmniNotes (Quinton, Hong)
Quinton: OmniNotes looks promising. I'm hoping that we can put the data on the server, and allow multiple clients to edit and view.
Hong: I verified that omni notes source code compiled and played successfully in android studio. It has decent gui design - not a toy but serious app; I think people would love it if we add group-sharing feature on its top. Basic metrics are pretty straight: 141 java files (classes in same order I assume), 20222 lines in total. I am spending time to understand the code structures of omni notes, trying to identify the classes that can be "Sapphirized".
- Basic add, modify, archive, trash and delete notes actions
- Share, merge and search notes
- Image, audio and generic file attachments
- Manage your notes using tags and categories
- To-do list
- Sketch-note mode
- Notes shortcut on home screen
- Export/import notes to backup
- Google Now integration: just tell "write a note" followed by the content
- Multiple widgets, DashClock extension, Android 4.2 lockscreen compatibility
- Multilanguage: 30 languages supported: https://crowdin.com/project/omni-notes
pictures/omninotes-classes.png
- ConsistentCache (for shared data like notes, attachments, todo's)
- ExplicitCheckpoint (for persisting data on the server, perhaps).
Minimal-Todo (Terry)
- Too simple. Most classes are
Activityclasses and Domain classes. It is a pure client side application. It does not worth the effort to port to DCAP.
MovieGuide (Terry/Fanlong/Sungwook)
- The App works in simulator.
- MovieGuide uses REST API to fetch movie information from themoviedb.org
- We can create a Sapphire object to fetch movie information and apply some cache DM on the Sapphire object.
[I] terryz@Donghuis-MacBook-Pro ~/W/mobile-apps> cloc MovieGuide/
171 text files.
171 unique files.
8 files ignored.
github.com/AlDanial/cloc v 1.74 T=0.94 s (174.0 files/s, 8297.6 lines/s)
--------------------------------------------------------------------------------
Language files blank comment code
--------------------------------------------------------------------------------
XML 81 80 2 4405
Java 54 505 166 2179
Bourne Again Shell 1 20 21 123
Groovy 3 19 11 102
DOS Batch 1 24 2 64
Markdown 1 10 0 36
Prolog 1 5 0 25
JSON 22 0 0 22
--------------------------------------------------------------------------------
SUM: 164 663 202 6956
--------------------------------------------------------------------------------*** Use case
The favorites info could be shared between users after Sapphiring favoritesstore. User 2 will see the movie A is favorited when click the Movie A since user1 Favorited movie A.
*** Classes added
1 FavoritesStoreStart as entrypoint
2 FavoritesStoreManager as manager to manage the favoritessotre or other Sapphire Obejct
3 FavoritesStore as SapphireObject
Detail see MovieGuide with Sapphire Object
*** Lessons learned
1 Imported the sapphire-core as module into android app project
2 Conflict with Dagger Dagger uses some tag or annotation to generate code automatically, for example, dagger generated FavoritesStore_Factory FavoritesModule_ProvideFavouritesInteractorFactory in favorites folder.
21 Decouple the class sapphired with original way, for example, FavoritesInteractorImpl got the FavotitesStore through:
class FavoritesInteractorImpl implements FavoritesInteractor
{
private static FavoritesStore favoritesStore;
FavoritesInteractorImpl()
{
new getfstore().execute("192.168.10.34", "22346", "10.0.2.15", "22345");
}
private class getfstore extends AsyncTask<String, Void, String> {
protected String doInBackground(String... params) {
String response = null;
Registry registry;
try {
String[] args = new String[]{ "192.168.10.34", "22346", "10.0.2.15", "22345" };
registry = LocateRegistry.getRegistry(args[0], Integer.parseInt(args[1]));
OMSServer server = (OMSServer) registry.lookup("SapphireOMS");
System.out.println(server);
KernelServer nodeServer = new KernelServerImpl(new InetSocketAddress(args[2], Integer.parseInt(args[3])), new InetSocketAddress(args[0],
Integer.parseInt(args[1])));
favoritesStoreManager = (FavoritesStoreManager) server.getAppEntryPoint();
favoritesStore=favoritesStoreManager.getFavoritesStore("movieguide");
22 Remove the class generated when to generate Sapphire Stubs
3 Couldn’t find the related classes
31 Add the all jar files as parameter for OMS and kernel server
4 couldn't convert the class which implements Parcelable into Sapphire Object
41 The method public void setFavorite(Movie movie) in FavoritesStore use Movie object which implements Parcelable and there was unmarshalling error.
workaround: convert the Movie object into String and pass it:
Moshi moshi = new Moshi.Builder().build();
JsonAdapter<Movie> jsonAdapter = moshi.adapter(Movie.class);
String movieJson = jsonAdapter.toJson(movie);
favoritesStore.setFavorite(movie.getId(),movieJson);
Pros: could cover most of objects
cons: change original method
42 Couldn't find way to convert Movie into Sapphire Object
Network module consists of several widely used open source projects for Android: retrofit2 (Square), OkHttp (Square), ReactiveX, Dagger 2 (Google dependency injection).
- As retrofit2 fails in serialization, Sapphirization was done avoiding this. GetReview added to demonstrate Sapphirization of network API call at below branch. Note that this still uses OkHttp but it does not depend on Retrofit2, ReactiveX or Dagger2.
https://github.com/sungwook-moon/MovieGuide/tree/2018-1-29-getReviewSuccessful
- Below branch is an attempt to Sapphirize the entire network path including all of the above mentioned dependent packages for GetReviews. It fails with deserialization of retrofit2 but was pushed to a separate branch for possible revisit in the future.
https://github.com/sungwook-moon/MovieGuide/tree/2018-1-24-networkSapphire
- Scanned through a few games at https://gist.github.com/roachhd/d579b58148d7e36a6b72
- Most games are written in HTML5 and javascript
- Most games only run locally
- We can potentially wrap some local objects as Sapphire objects and enable code offloading. But we have to support Javascript.
Tox (the protocol) seems quite complicated by itself. a few things makes it risky for the adventure - it has a lot of code in scala; the code has about 200 classes (excluding the auto generated ones); there seems to exists an central tox chat server (to be verified) besides the android app.
AnTox (the reference implementation android app) presents a few concerns/risks: 1. it has external server, not part of the project. I have not found out the server source code; I guess the server might be a simple registry-like service - could be wrong. to be verified. 2. most of its code is in scala; we may need more resource to make scala code working w/ DCAP DM model. not technical blockage; just labor resource, in my understanding. 3. anTox is claimed a peer-to-peer communication tool; we may need to modify its code structure to make it fit into DCAP model in order to leverage app/DM separation in a sensible way - this also implies more resources.
AnTox, however, in my understanding, seems a very typical p2p multiple player app, which could showcase Sapphire P2P related DM.
InstaMaterial had no backend. It's all UI. Ideally we want an app with a front-end and a backend, and parts that could run on either. Preferably with some significant processing, and sharing of data between multiple clients.
Telegram looked interesting except for this:
Q: Why not open source everything?
All code will be released eventually. We started with the most useful parts — a well-documented API that allows developers to build new Telegram apps, and open source clients that can be verified by security specialists.
Q: Can I run Telegram using my own server?
Our architecture does not support federation yet. Telegram is a unified cloud service, so creating forks where two users might end up on two different Telegram clouds is unacceptable. To enable you to run your own Telegram server while retaining both speed and security is a task in itself. At the moment, we are undecided on whether or not Telegram should go in this direction.
VLC is client side only, so no good.
K9 Mail is only an IMAP client - servers are all generic IMAP servers. Not worth porting, IMO.
LibreOffice is client only, and doesn't support editing very well. Not a good candidate.
SoundRecorder (Sungwook)
Good: Simple app with clean straightforward code - confirmed works as expected. Forked 300+ times. Bad: It is client side only and uses internal DB; therefore, not a good candidate for showcasing DCAP.
Sapphirization: DoPrint was created and sapphirized to demonstrate the portability. Other components cannot be sapphirized unless they are completely modified since they all bind to Android specific APIs (e.g, UI, SQL Lite). Below is a link to the example: https://github.com/sungwook-moon/SoundRecorder
Google IoSched (Sungwook) (Terry)
- 15,000+ star rating/forked > 5000 times.
- Developed by Google in 2017 for its conference.
- Support data sharing - sync schedule between devices
- Has both front end and back end
- Server side seems Google provided set-up; hence, it is not much different from client only app unless we modify this app and add our own server side code.
- Relative big. 57800 lines of Java code
- The App fetches news feeds from remote Firebase Database (see
FeedActivity) - The App uses pub/sub message service from Firebase Messaging for synchronization (see
FcmUtilities)
- View the conference agenda and edit your personal schedule
- Sync your schedule between all of your devices and the I/O website
- View detailed session, code lab, office hours, and speaker information, including speaker bios, photos, and Google+ profiles
- Participate in public #io15 conversations on Google+
- Guide yourself using the vector-based conference map
- Get a reminder a few minutes before sessions in your schedule are due to start
- Play "I/O Live" session video streams
- Send feedback on sessions, from your phone/tablet.
[I] terryz@Donghuis-MacBook-Pro ~/W/mobile-apps> cloc iosched/
1232 text files.
1183 unique files.
122 files ignored.
github.com/AlDanial/cloc v 1.74 T=4.07 s (273.0 files/s, 36626.7 lines/s)
--------------------------------------------------------------------------------
Language files blank comment code
--------------------------------------------------------------------------------
Java 546 11735 21429 57854
JSON 18 0 0 14996
XML 442 5936 9534 14213
HTML 26 551 292 4213
Markdown 34 722 0 2814
JavaScript 4 161 527 1407
Groovy 20 146 216 747
CSS 4 56 22 295
Bourne Again Shell 2 39 41 244
Maven 2 22 0 175
Bourne Shell 3 31 52 153
DOS Batch 2 47 4 123
Prolog 2 6 0 41
Dockerfile 1 0 6 21
YAML 2 6 13 20
Ant 1 17 59 16
make 2 13 32 15
--------------------------------------------------------------------------------
SUM: 1111 19488 32227 97347
--------------------------------------------------------------------------------Droidfish (Akhil)
DroidFish is an Android port of Stockfish chess engine (this allows user to set the strength of the AI logic)
It also has been used for evaluation of COSMOS proposed in research paper, COSMOS: Computation Offloading as a Service for Mobile Devices. It can be used for showcasing the offloading capability of DCAP.
The code consists of mostly Java but it also has some C++ code (looking into the possibility of porting it to DCAP), following is summary of the code,
akhil@ubuntu:~/StudioProjects/sapphire/porting/mobile_apps$ cloc droidfish/
845 text files.
669 unique files.
399 files ignored.
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
Java 173 4946 5979 40607
XML 104 112 66 25577
C 26 3359 2646 15750
C++ 29 2413 1957 7210
C/C++ Header 62 1772 2822 5967
HTML 13 209 0 3521
-------------------------------------------------------------------------------
SUM: 407 12811 13470 98632
-------------------------------------------------------------------------------- Code Offloading / Shift Policy (for offloading the compute intensive tasks)
Pocketshphinx-Android (Akhil)
This is a wrapper for Pocketsphinx for Android (developed by CMU) providing high-level interface for recognising the microphone input. This uses pocketshinx (this is a speech recognition engine developed by CMU for performing audio recognition)
This is also a mix of both Java and C++ code. evaluation of COSMOS proposed in research paper, COSMOS: Computation Offloading as a Service for Mobile Devices. It can be used for showcasing the offloading capability of DCAP.
Looking into the possibility of porting this to DCAP. It looks like, using Java's Native Interface API, it is possible to wrap existing C or C++ code with a Java interface and then export this interface remotely through RMI - https://docstore.mik.ua/orelly/java-ent/jenut/ch03_07.htm (checking more about native method calls using Java RMI)
- Code Offloading / Shift Policy (for offloading the audio recognition part which is a compute intensive task)
XRace (Akhil)
XRace is a 3D car race game for the Android mobile phone platform, using OpenGL ES for rendering. It is very compute intensive, so all the compute intensive parts (3D racing scenraio, collision detection, etc.) can be offloaded using Sapphire.
Whole of the code base is in Java. This game has been used for evaluation of a proposal in a research paper, Refactoring Android Java Code for On-Demand Computation Offloading
- Code Offloading / Shift Policy (for offloading all the compute intensive tasks of the game)
TeamNewPipe (Srini)
A free lightweight YouTube frontend for Android.
NewPipe does not use any Google framework libraries, or the YouTube API. It only parses the website in order to gain the information it needs. Therefore this app can be used on devices without Google Services installed. Also, you don't need a YouTube account to use NewPipe, and it's FLOSS.
- Search videos
- Display general information about a video
- Watch YouTube videos
- Listen to YouTube videos
- Popup mode (floating player)
- Select the streaming player to watch the video with
- Download videos
- Download audio only
- Open a video in Kodi
- Show Next/Related videos
- Search YouTube in a specific language
- Watch/Block age restricted material
- Display general information about channels
- Search channels
- Watch videos from a channel
- Orbot/Tor support (not yet directly)
- 1080p/2k/4k support
- View history
- Subscribe to channels
- Search history
- Search/Watch Playlists
- Watch as queues Playlists
- Queuing videos
- Multiservice support (eg. SoundCloud)
- Bookmarks
- Subtitles support
- livestream support ... and many more
TravelMate (Srini)
NewChess (Akhil)
This is a Chess app with the back-end engine also in the code base. Currently it is Single Player. Whole of the code base is in Java.
Sapphirization
At first tried sapphirizing a separate class named SamplePrint, could sapphirize using ShiftPolicy and could see the portability of it across servers successfully.
Further on, I tried sapphirizing the back-end named, SimpleEngine class in the code, it takes care of most of the logic, where it is decided which piece is to be moved by the computer player once the human player makes the move. Could sapphirize this also successfully using a new DM named ExplicitMigrationPolicy. ( Few of the parts of SimpleEngine are not compute intensive, these can be separated out to a new class which need not be sapphirized. There is one more class named Position, that can also be tried to Sapphirize.)
Currently I have put up the code in my fork, whose link is - https://github.com/mbssaiakhil/NewChess/tree/backend_chess_engine_explicit_migration_dm
(Currently build files have been committed, I am in the process of removing build related files in all of the commits in the branch, will soon update it)
Extension Possibilities
-
Currently there is no option to go back by a move after moving user's pieces, using Checkpoint DM, we can provide an option to the user to go back by one move (many of the games generally in the beginner mode, provides these options).
-
By using Sapphire, we can also make this distributed and also make the computer player side to a manual player type and make it Two-Player Game.
-
The code looks to be good, looks to be very extensible. Any new more compute intensive back-end chess engine can be taken and attached to this app. For example, it can be explored to use a ported chess engine of Stockfish i.e Droidfish (this app I have shared above).
Collection of Android samples by Google in GitHub. You can sort by language (e.g., Java). Good starting point for Android developers, therefore, impact can be big if DCAP ports a few of them.
Good list of Android open source projects in GitHub with level of difficulty and description of each project.
(Sungwook) If a class depends on Android package, it may not work on the server with direct conversion (e.g., parcelable). In such a case, you may need to find a way to run the similar package on the server or run the Sapphire object on the device (so it can still access Android specific package) or update code to use non-Android code library.
(Sungwook) HTTP API calls may be Sapphirized. OkHttp is commonly used open source developed by Square. If this is Sapphirized, it can also work for many other apps. Example app: MovieGuide
(Sungwook) SQL Lite is commonly used in Android including SoundRecorder. As this is an open source app that can work on various environments, it might be able to be ported.
Reason: not sure.
Workaround: to get the kernel server from oms, and consult the kernel server for KernelServer stub, invoke RMI call of startApp("appentryClassName") via the stub.
Reason: not sure.
TODO: to investigate the root cause.
- Unmarshall exception can occur when object has different serialVersionUID. Note this can happen when you update Android plugin or Gradle version as well.
- Android SDK prior to version 21 may have this issue. Increase minimum SDK version to 21.
Reason: android gradle build seems not to generate jar files that can be used directly by the existing stub generation tool
Workaround: manually pack relevant jar file, and may need to tweak the tool before the stub files generated
Reason: not sure
Workaround: manually edit the generated stub code
TODO: to investigate the root cause and fix the stub generation bug
(Sungwook) Please look into Terry (Donghui)'s pull request which fixes a bug for Windows.
Android sometimes forbids users to do network I/O operations on Android main thread. When we convert a Java object into a Sapphire object, any method invocation on Sapphire object becomes a remote method invocation which involves network I/O. In other words, you may accidentally do network I/O operations on Android main thread by using Sapphire object. If you run into this error, please consider calling Sapphire object in a AsyncTask.