An  Overview of Tools for Collecting Data on Software Development and Debugging Processes from Integrated Development Environments

O. O. Zhevaho

doi:10.15802/stp2021/242042

Authors

O. O. Zhevaho Dnipro National University of Railway Transport named after Academician V. Lazaryan, Ukraine https://orcid.org/0000-0003-0019-8320

DOI:

https://doi.org/10.15802/stp2021/242042

Keywords:

software development process, debugging, integrated development environment, literature review

Abstract

Purpose. This paper presents the findings of a review of the literature published in the twenty-first century in order to identify and analyze the current state of tools that track developer interactions with integrated development environments, as well as to recommend future research directions based on the actual state. Methodology. By systematically searching in five digital libraries we conducted a systematic review of the literature on data collection tools from integrated development environments published in the twenty-first century. Fifty-five papers were selected as primary studies. Findings. 55 articles were analyzed and the findings show that using an integrated development environment to collect usage data provides more insight into developer activities than it was previously possible. Usage data allows us to analyze how developers spend their time. With usage data, you can learn more about how developers create mental models, investigate code, conduct mini-experiments through trial and error, and what can help everyone improve performance. The research community continues to be highly active in developing tools to track developer activity. The findings indicate that more research is needed in this area to better understand and measure programmer behavior. Originality. For the first time, systematization and analysis of tools for tracking programmer's behavior in an integrated development environment have been carried out. Practical value. Our study contributes to a better understanding of the current state of research on programmer behavior in integrated development environments. An analysis of the study can help define a research agenda as a starting point for the creation of a novel practical tool.

References

Amann, S., Proksch, S., & Nadi, S. (2016). FeedBaG: An interaction tracker for Visual Studio. In 2016 IEEE 24th International Conference on Program Comprehension (ICPC) (pp. 1-3). Austin, USA. DOI: http://doi.org/10.1109/icpc.2016.7503741 (in English)

Antal, G., Végh, Á. Z., & Bilicki, V. (2014). A methodology for measuring software development productivity using Eclipse IDE. In Proceedings of the 9th International Conference on Applied Informatics (Vol. 2, pp. 252-262). Eger, Hungary. DOI: http://doi.org/10.14794/icai.9.2014.2.255 (in English)

Bao, L., Ye, D., Xing, Z., Xia, X., & Wang, X. (2015). ActivitySpace: A Remembrance Framework to Support Interapplication Information Needs. In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) (pp. 864-869). Lincoln, USA. DOI: http://doi.org/10.1109/ase.2015.90 (in English)

Beller, M., Gousios, G., Panichella, A., & Zaidman, A. (2015). When, how, and why developers (do not) test in their IDEs. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (pp. 179-190). DOI: http://doi.org/10.1145/2786805.2786843 (in English)

Beller, M., Spruit, N., Spinellis, D., & Zaidman, A. (2018). On the dichotomy of debugging behavior among pro-grammers. In Proceedings of the 40th International Conference on Software Engineering (pp. 572-583). DOI: http://doi.org/10.1145/3180155.3180175 (in English)

Biehl, J. T., Czerwinski, M., Smith, G., & Robertson, G. G. (2007). FASTDash: a visual dashboard for fostering awareness in software teams. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1313-1322). DOI: http://doi.org/10.1145/1240624.1240823 (in English)

Bieliková, M., Polášek, I., Barla, M., Kuric, E., Rástočný, K., Tvarožek, J., & Lacko, P. (2014). Platform Inde-pendent Software Development Monitoring: Design of an Architecture. In SOFSEM 2014: Theory and Practice of Computer Science (Vol. 8327, pp. 126-137). DOI: http://doi.org/10.1007/978-3-319-04298-5_12 (in English)

Caldeira, J., Brito e Abreu, F., Cardoso, J., Ribeiro, R., & Werner, C. (2021). Profiling Software Developers with Process Mining and N-Gram Language Models. Arxiv. Retrieved from http://arxiv.org/abs/2101.06733v1 (in English)

Damevski, K., Shepherd, D. C., Schneider, J., & Pollock, L. (2017). Mining Sequences of Developer Interactions in Visual Studio for Usage Smells. IEEE Transactions on Software Engineering (Vol. 43, Iss. 4, P. 359-371). DOI: http://doi.org/10.1109/tse.2016.2592905 (in English)

Dias, M., Cassou, D., & Ducasse, S. (2013). Representing code history with development environment events. IWST-2013 - 5th International Workshop on Smalltalk Technologies, 1-7. (in English)

Ebraert, P., Vallejos, J., Costanza, P., Van Paesschen, E., & D'Hondt, T. (2007). Change-oriented software engi-neering. In Proceedings of the 2007 international conference on Dynamic languages: in conjunction with the 15th International Smalltalk Joint Conference 2007 (pp. 3-24). DOI: http://doi.org/10.1145/1352678.1352680 (in English)

Foster, S. R., Griswold, W. G., & Lerner, S. (2012). WitchDoctor: IDE support for real-time auto-completion of refactorings. In 2012 34th International Conference on Software Engineering (ICSE) (pp. 22-232). Zurich, Switzerland. DOI: http://doi.org/10.1109/icse.2012.6227191 (in English)

Frost, R. (2007). Jazz and the Eclipse Way of Collaboration. In IEEE Software (Vol. 24, Iss. 6, pp. 114-117). DOI: http://doi.org/10.1109/ms.2007.170 (in English)

Ge, X., DuBose, Q. L., & Murphy-Hill, E. (2012). Reconciling manual and automatic refactoring. In 2012 34th International Conference on Software Engineering (ICSE) (pp. 211-221). Zurich, Switzerland. DOI: http://doi.org/10.1109/icse.2012.6227192 (in English)

Gu, Z., Schleck, D., Barr, E. T., & Su, Z. (2012). Capturing and Exploiting IDE Interactions. In Proceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software (pp. 83-94). DOI: http://doi.org/10.1145/2661136.2661144 (in English)

Guarnera, D. T., Bryant, C. A., Mishra, A., Maletic, J. I., & Sharif, B. (2018). iTrace: Eye tracking infrastructure for development environments. In Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications (No 105, pp. 1-3). DOI: http://doi.org/10.1145/3204493.3208343 (in English)

Guzzi, A., Pinzger, M., & van Deursen, A. (2010). Combining micro-blogging and IDE interactions to support developers in their quests. In 2010 IEEE International Conference on Software Maintenance (pp. 1-5). Timisoara, Romania. DOI: http://doi.org/10.1109/icsm.2010.5609683 (in English)

Hattori, L., & Lanza, M. (2010). Syde: a tool for collaborative software development. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering (Vol. 2, pp. 235-238). DOI: http://doi.org/10.1145/1810295.1810339 (in English)

Hegde, R., & Dewan, P. (2008). Connecting Programming Environments to Support Ad-Hoc Collaboration. In 2008 23rd IEEE/ACM International Conference on Automated Software Engineering (pp. 178-187). L'Aquila, Italy. DOI: http://doi.org/10.1109/ase.2008.28 (in English)

Hundhausen, C. D., & Carter, A. S. (2014). Supporting Social Interactions and Awareness in Educational Pro-gramming Environments. In Proceedings of the 5th Workshop on Evaluation and Usability of Program-ming Languages and Tools (pp. 55-56). DOI: http://doi.org/10.1145/2688204.2688215 (in English)

Ioannou, C., Burattin, A., & Weber, B. (2018). Mining Developers’ Workflows from IDE Usage. Advanced In-formation Systems Engineering Workshops, 316, 167-179. DOI: http://doi.org/10.1007/978-3-319-92898-2_14 (in English)

Jaspan, C., Jorde, M., Egelman, C., Green, C., Holtz, B., Smith, E., … & Murphy-Hill, E. (2020). Enabling the Study of Software Development Behavior With Cross-Tool Logs. IEEE Software (Vol. 37, Iss. 6, pp. 44-51). DOI: http://doi.org/10.1109/ms.2020.3014573 (in English)

Jekutsch, S. ElectroCodeoGram: An Environment for Studying Programming. Retrieved from http://www.mi.fu-berlin.de/wiki/pub/SE/ElectroCodeoGram/lancaster.pdf (in English)

Johnson, P. M., Hongbing Kou, Agustin, J., Chan, C., Moore, C., Miglani, J., … & Doane. (2003). Beyond the Personal Software Process: Metrics collection and analysis for the differently disciplined. In 25th Interna-tional Conference on Software Engineering, 2003. Proceedings (pp. 641-646). Portland, USA. DOI: http://doi.org/10.1109/icse.2003.1201249 (in English)

Kitchenham, B. A. & Charters, S. Guidelines for performing Systematic Literature Reviews in Software Engi-neering. Retrieved from https://userpages.uni-koblenz.de/~laemmel/esecourse/slides/slr.pdf (in English)

Kobayashi, T., Kato, N., & Agusa, K. (2012). Interaction histories mining for software change guide. In 2012 Third International Workshop on Recommendation Systems for Software Engineering (RSSE) (pp. 73-77). Zurich, Switzerland. DOI: http://doi.org/10.1109/rsse.2012.6233415 (in English)

Lijie Z., Godfrey, M. W., & Hassan, A. E. (2007). Detecting Interaction Coupling from Task Interaction Histo-ries. In 15th IEEE International Conference on Program Comprehension (ICPC '07) (pp. 135-144). Banff, Canada. DOI: http://doi.org/10.1109/icpc.2007.18 (in English)

Lyulina, E., Birillo, A., Kovalenko, V., & Bryksin, T. (2021). TaskTracker-tool: A Toolkit for Tracking of Code Snapshots and Activity Data During Solution of Programming Tasks. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (pp. 495-501). DOI: http://doi.org/10.1145/3408877.3432534 (in English)

Maalej, W., & Happel, H.-J. (2008). A Lightweight Approach for Knowledge Sharing in Distributed Software Teams. In Practical Aspects of Knowledge Management (Vol. 5345, pp. 14-25). DOI: http://doi.org/10.1007/978-3-540-89447-6_4 (in English)

Maruyama, K., Hayashi, S., & Omori, T. (2018). ChangeMacroRecorder: Recording fine-grained textual chang-es of source code. In 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER) (pp. 537-541). Campobasso, Italy. DOI: http://doi.org/10.1109/saner.2018.8330255 (in English)

Matsuzawa, Y., Okada, K., & Sakai, S. (2013). Programming process visualizer: A proposal of the tool for stu-dents to observe their programming process. In Proceedings of the 18th ACM conference on Innovation and technology in computer science education (pp. 46-51). DOI: http://doi.org/10.1145/2462476.2462493 (in English)

McIntyre, M. M., & Walker, R. J. (2007). Assisting potentially-repetitive small-scale changes via semi-automated heuristic search. In Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering (pp. 497-500). DOI: http://doi.org/10.1145/1321631.1321718 (in English)

McKeogh, J., & Exton, C. (2004). Eclipse plug-in to monitor the programmer behaviour. In Proceedings of the 2004 OOPSLA workshop on eclipse technology eXchange (pp. 93-97). DOI: http://doi.org/10.1145/1066129.1066148 (in English)

Minelli, R., & Lanza, M. (2013). Visualizing the workflow of developers. In 2013 First IEEE Working Confer-ence on Software Visualization (VISSOFT) (pp. 1-4). Eindhoven, Netherlands. DOI: http://doi.org/10.1109/vissoft.2013.6650531 (in English)

Murphy, G. C., Kersten, M., & Findlater, L. (2006). How are Java software developers using the Eclipse IDE? IEEE Software (Vol. 23, Iss. 4, pp. 76-83). DOI: http://doi.org/10.1109/ms.2006.105 (in English)

Negara, S., Vakilian, M., Chen, N., Johnson, R. E., & Dig, D. (2012). Is It Dangerous to Use Version Control His-tories to Study Source Code Evolution? In ECOOP 2012-Object-Oriented Programming (Vol. 7313, pp. 79-103). Berlin, Heidelberg. DOI: http://doi.org/10.1007/978-3-642-31057-7_5 (in English)

Norris, C., Barry, F., Fenwick Jr., J. B., Reid, K., & Rountree, J. (2008). ClockIt: collecting quantitative data on how beginning software developers really work. ACM SIGCSE Bulletin, 40(3), 37-41. DOI: http://doi.org/10.1145/1597849.1384284 (in English)

Omori, T., & Maruyama, K. (2008). A change-aware development environment by recording editing operations of source code. In Proceedings of the 2008 international working conference on Mining software reposi-tories (pp. 31-34). DOI: http://doi.org/10.1145/1370750.1370758 (in English)

Pärtel, M., Luukkainen, M., Vihavainen, A., & Vikberg, T. (2013). Test My Code. International Journal of Technology Enhanced Learning (IJTEL), 5(3/4), 271. DOI: http://doi.org/10.1504/ijtel.2013.059495 (in English)

Parnin, C., & Gorg, C. Building Usage Contexts During Program Comprehension. In 14th IEEE International Conference on Program Comprehension (ICPC'06) (pp. 13-22). Athens, Greece. DOI: http://doi.org/10.1109/icpc.2006.14 (in English)

Petrillo, F., Soh, Z., Khomh, F., Pimenta, M., Freitas, C., & Gueheneuc, Y.-G. (2016). Towards Understanding Interactive Debugging. In 2016 IEEE International Conference on Software Quality, Reliability and Secu-rity (QRS) (pp. 152-163). Vienna, Austria. DOI: http://doi.org/10.1109/qrs.2016.27 (in English)

Proksch, S., Nadi, S., Amann, S., & Mezini, M. (2017). Enriching in-IDE process information with fine-grained source code history. In 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER) (pp. 250-260). Klagenfurt, Austria. DOI: http://doi.org/10.1109/saner.2017.7884626 (in English)

Robbes, R., & Lanza, M. (2008). SpyWare: a change-aware development toolset. In Proceedings of the 30th international conference on Software engineering (pp. 847-850). DOI: http://doi.org/10.1145/1368088.1368219 (in English)

Rothlisberger, D., Nierstrasz, O., Ducasse, S., Pollet, D., & Robbes, R. (2009). Supporting task-oriented navigation in IDEs with configurable HeatMaps. In 2009 IEEE 17th International Conference on Program Compre-hension (pp. 253-257). Vancouver, Canada. DOI: http://doi.org/10.1109/icpc.2009.5090052 (in English)

Salinger, S., Oezbek, C., Beecher, K., & Schenk, J. (2010). Saros: an eclipse plug-in for distributed party pro-gramming. In Proceedings of the 2010 ICSE Workshop on Cooperative and Human Aspects of Software Engineering (pp. 48-55). DOI: http://doi.org/10.1145/1833310.1833319 (in English)

Sharon, Y. EclipsEye Spying onEclipse. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=064DE0D8481B3C9FD39F875EA79AD59C?doi=10.1.1.709.9455&rep=rep1&type=pdf (in English)

Shin, H., Choi, H.-J., & Baik, J. Jasmine: A PSP Supporting Tool. In Software Process Dynamics and Agility (Vol. 4470, pp. 73-83). DOI: http://doi.org/10.1007/978-3-540-72426-1_7 (in English)

Shynkarenko, V., & Zhevago, O. (2019). Visualization of program development process. In 2019 IEEE 14th International Conference on Computer Sciences and Information Technologies (CSIT) (pp. 142-145). Lviv, Ukraine. DOI: http://doi.org/10.1109/stc-csit.2019.8929774 (in English)

Shynkarenko, V., & Zhevaho, O. (2020). Development of a toolkit for analyzing software debugging processes using the constructive approach. Eastern-european Journal of Enterprise Technologies, 5(2(107)), 29-38. DOI: http://doi.org/10.15587/1729-4061.2020.215090 (in English)

Sillitti, J., Succi, & Vernazza. (2003). Collecting, integrating and analyzing software metrics and personal soft-ware process data. In 2003 Proceedings 29th Euromicro Conference (pp. 336-342). Belek-Antalya, Tur-key. DOI: http://doi.org/10.1109/eurmic.2003.1231611 (in English)

Sison, R. (2005). Personal software process (PSP) assistant. In 12th Asia-Pacific Software Engineering Confer-ence (APSEC'05) (pp. 8). Taipei, Taiwan. DOI: http://doi.org/10.1109/apsec.2005.87 (in English)

Snipes, W., Nair, A. R., & Murphy-Hill, E. (2014). Experiences gamifying developer adoption of practices and tools. In Companion Proceedings of the 36th International Conference on Software Engineering (pp. 105-114). DOI: http://doi.org/10.1145/2591062.2591171 (in English)

Soetens, Q. D., & Demeyer, S. (2012). ChEOPSJ: Change-Based Test Optimization. In 2012 16th European Conference on Software Maintenance and Reengineering (pp. 535-538). Szeged, Hungary. DOI: http://doi.org/10.1109/csmr.2012.70 (in English)

Steinert, B., Cassou, D., & Hirschfeld, R. (2012). CoExist: Overcoming Aversion to Change Preserving Immedi-ate Access to Source Code and Run-time Information of Previous Development States. ACM SIGPLAN Notices, 48(2), 107-118. DOI: http://doi.org/10.1145/2480360.2384591 (in English)

Yoon, Y., & Myers, B. A. (2011). Capturing and analyzing low-level events from the code editor. In Proceedings of the 3rd ACM SIGPLAN workshop on Evaluation and usability of programming languages and tools (pp. 25-30). DOI: http://doi.org/10.1145/2089155.2089163 (in English)

Yuan, X., Vega, P., Yu, H., & Li, Y. (2005). A Personal Software Process Tool for Eclipse Environment. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.121.1638&rep=rep1&type=pdf (in English)