We aim at systematic and automated use of the logging mechanisms, which are existing system data collection processes in software engineering practice, to improve system performance and reliability. SmartLog is supported by 2015 Microsoft Research Asia Collaborative Research Program (Project No. FY16-RESTHEME-005). Details please find here.

Performance Monitoring and Reliability Enhancement with Log Data Analysis for Large Scale Distributed Systems

• Cost-Effective Logging Mechanism Design
• Log Collection
• Log Parsing
• Log Analysis for System Troubleshooting

We propose the first steps towards the “Where to Log” question. LogAdvisor automatically recommend whether to log or not in a code snippet.

We design a profiling technique to selectively collect logs of mobile apps for performance bug diagnosis, and make them open-source as DiagDroid.

POP is a parallel log parsing framework built on top of Spark. It can parse 200m lines of HDFS log messages in 7 min with 16 executors.

We systematically evaluate and benchmark the effectiveness and efficiency of current anomaly detection methods, and make them open-source on Github as Loglizer.