Practical part of the LambdaThesis project
Generates random events of different types, prints them out to the console, and sends them to Kafka.
Those events can be then processed by Storm and Spark.
Takes events from avro files, and sends them to kafka.
We don't use this project any more.
Our main project, that contains data processing using Storm.
Our second main project, that contains data processing using Spark.
Takes parquet file, parses it using SparkPlayingField library, and saves data to avro files.
We don't use this project any more.
Simple test project to try Spark via word counting.
Test project to try playing with Redis and Jedis.
It is already quite obsolete, and can be easily deleted.
Old Storm project, that is now recreated as MenthalSLStormProcessor.
It is now obsolete, and can be deleted.
- app_install
- app_session
- call_missed
- call_outgoing
- call_received
- dreaming_started
- dreaming_stopped
- phone_shutdown
- screen_off
- screen_on
- screen_unlock
- sms_received
- sms_sent
- window_state_changed
In the final Redis database, that plays the role of real-time views of the speed layer, we have following keys.
Each key is associated with the list, that has two elements. The first one is the date-time of the beginning of counting. The second one is the value itself.
- app:$app_name:$user_id:sessions:* counters
- app:$app_name:$user_id:total_time:* durations
- user:$user_id:$app_name:app_usage:* counters
- user:allUsers:$app_name:app_usage:* counters
- user:$user_id:$app_name:app_starts:* durations
- user:allUsers:$app_name:app_starts:* durations
- user:$user_id:screen_lock:* counters
- user:$user_id:$phone_hash:incoming_msg_count:* counters
- user:allUsers:$phone_hash:incoming_msg_count:* counters
- user:$user_id:$phone_hash:incoming_msg_length:* lengths
- user:allUsers:$phone_hash:incoming_msg_length:* lengths
- user:$user_id:$phone_hash:outgoing_msg_count:* counters
- user:allUsers:$phone_hash:outgoing_msg_count:* counters
- user:$user_id:$phone_hash:outgoing_msg_length:* lengths
- user:allUsers:$phone_hash:outgoing_msg_length:* lengths
- user:$user_id:$phone_hash:outgoing_call_count:* counters
- user:allUsers:$phone_hash:outgoing_call_count:* counters
- user:$user_id:$phone_hash:outgoing_call_duration:* durations
- user:allUsers:$phone_hash:outgoing_call_duration:* durations
- user:$user_id:$phone_hash:incoming_call_count:* counters
- user:allUsers:$phone_hash:incoming_call_count:* counters
- user:$user_id:$phone_hash:incoming_call_duration:* durations
- user:allUsers:$phone_hash:incoming_call_duration:* durations
- user:$user_id:$phone_hash:missed_call_count:* counters
- user:allUsers:$phone_hash:missed_call_count:* counters
Each time * means, that there are 4 counters with that base name and additional part, for example
app:$app_name:$user_id:sessions:* counters - means that there are exactly the following keys
- app:$app_name:$user_id:sessions:count:hourly
- app:$app_name:$user_id:sessions:count:daily
- app:$app_name:$user_id:sessions:count:weekly
- app:$app_name:$user_id:sessions:count:monthly
- app:$app_name:$user_id:total_time:duration:hourly
- app:$app_name:$user_id:total_time:duration:daily
- app:$app_name:$user_id:total_time:duration:weekly
- app:$app_name:$user_id:total_time:duration:monthly
- user:$user_id:$phone_hash:incoming_msg_length:length:hourly
- user:$user_id:$phone_hash:incoming_msg_length:length:daily
- user:$user_id:$phone_hash:incoming_msg_length:length:weekly
- user:$user_id:$phone_hash:incoming_msg_length:length:monthly