ACM Digital Library home
ACM home
Advanced Search
10.1145/3464298.3476133acmconferencesArticle/Chapter ViewAbstractPublication PagesmiddlewareConference Proceedingsconference-collections
research-article

SeBS: a serverless benchmark suite for function-as-a-service computing

Middleware '21: Proceedings of the 22nd International Middleware ConferenceDecember 2021 Pages 64–78https://doi.org/10.1145/3464298.3476133
Published:02 October 2021Publication History
  • 17citation
  • 633
  • Downloads
Metrics
Total Citations17
Total Downloads633
Last 12 Months475
Last 6 weeks63

Middleware '21: Proceedings of the 22nd International Middleware Conference

SeBS: a serverless benchmark suite for function-as-a-service computing
Pages 64–78
PreviousChapterNextChapter

ABSTRACT

Function-as-a-Service (FaaS) is one of the most promising directions for the future of cloud services, and serverless functions have immediately become a new middleware for building scalable and cost-efficient microservices and appli cations. However, the quickly moving technology hinders reproducibility, and the lack of a standardized benchmarking suite leads to ad-hoc solutions and microbenchmarks being used in serverless research, further complicating meta-analysis and comparison of research solutions. To address this challenge, we propose the Serverless Benchmark Suite: the first benchmark for FaaS computing that systematically covers a wide spectrum of cloud resources and applications. Our benchmark consists of the specification of representative workloads, the accompanying implementation and evaluation infrastructure, and the evaluation methodology that facilitates reproducibility and enables interpretability. We demonstrate that the abstract model of a FaaS execution environment ensures the applicability of our benchmark to multiple commercial providers such as AWS, Azure, and Google Cloud. Our work facilities experimental evaluation of serverless systems, and delivers a standardized, reliable and evolving evaluation methodology of performance, efficiency, scalability and reliability of middleware FaaS platforms.

References

  1. 2014. AWS Lambda. https://aws.amazon.com/lambda/. Accessed: 2020-01-20.Google ScholarGoogle Scholar
  2. 2016a. Azure: Bandwidth Pricing. https://azure.microsoft.com/en-us/pricing/details/bandwidth/. Accessed: 2020-08-20.Google ScholarGoogle Scholar
  3. 2016b. Azure Functions. https://azure.microsoft.com/en-us/services/functions/. Accessed: 2020-01-20.Google ScholarGoogle Scholar
  4. 2016. IBM Cloud Functions. https://cloud.ibm.com/functions/. Accessed: 2020-01-20.Google ScholarGoogle Scholar
  5. 2017. Google Cloud Functions. https://cloud.google.com/functions/. Accessed: 2020-01-20.Google ScholarGoogle Scholar
  6. 2018. Firecracker. https://github.com/firecracker-microvm/firecracker. Accessed: 2020-01-20.Google ScholarGoogle Scholar
  7. 2019. DNAvisualization.org. https://github.com/Benjamin-Lee/DNAvisualization.org. Accessed: 2020-01-20.Google ScholarGoogle Scholar
  8. 2019. MinIO Object Storage. min.io. Accessed: 2020-01-20.Google ScholarGoogle Scholar
  9. 2019. <i>SLURM Generic Resource (GRES) Scheduling</i>. Accessed: 2020-01-20.Google ScholarGoogle Scholar
  10. 2020. AWS API Pricing. https://aws.amazon.com/api-gateway/pricing/. Accessed: 2020-08-20.Google ScholarGoogle Scholar
  11. 2020. AWS Lambda Limits. https://docs.aws.amazon.com/lambda/latest/dg/limits.html. Accessed: 2020-01-20.Google ScholarGoogle Scholar
  12. 2020. Azure Functions scale and hosting. https://docs.microsoft.com/en-us/azure/azure-functions/functions-scale. Accessed: 2020-01-20.Google ScholarGoogle Scholar
  13. 2020a. Faasdom. https://github.com/faas-benchmarking/faasdom. Accessed: 2020-08-01.Google ScholarGoogle Scholar
  14. 2020b. FaaSTest. https://github.com/nuweba/faasbenchmark. Accessed: 2020-08-01.Google ScholarGoogle Scholar
  15. 2020a. Google Cloud Functions Pricing. https://cloud.google.com/functions/pricing. Accessed: 2020-08-20.Google ScholarGoogle Scholar
  16. 2020b. Google Cloud Functions Quotas. https://cloud.google.com/functions/quotas. Accessed: 2020-08-20.Google ScholarGoogle Scholar
  17. 2020. nuclio. https://nuclio.io/. Accessed: 2020-01-20.Google ScholarGoogle Scholar
  18. 2020. Serverless Framework. https://github.com/serverless/serverless. Accessed: 2020-08-01.Google ScholarGoogle Scholar
  19. 2020. Standard Performance Evaluation Corporation (SPEC) Benchmarks. https://www.spec.org/benchmarks.html. Accessed: 2020-08-01.Google ScholarGoogle Scholar
  20. Istemi Ekin Akkus, Ruichuan Chen, Ivica Rimac, Manuel Stein, Klaus Satzke, Andre Beck, Paarijaat Aditya, and Volker Hilt. 2018. SAND: Towards High-Performance Serverless Computing. In <i>Proceedings of the 2018 USENIX Conference on Usenix Annual Technical Conference</i> (Boston, MA, USA) <i>(USENIX ATC '18)</i>. USENIX Association, USA, 923–935.Google ScholarGoogle Scholar
  21. Lixiang Ao, Liz Izhikevich, Geoffrey M. Voelker, and George Porter. 2018. Sprocket: A Serverless Video Processing Framework. In <i>Proceedings of the ACM Symposium on Cloud Computing</i> (Carlsbad, CA, USA) <i>(SoCC '18)</i>. Association for Computing Machinery, New York, NY, USA, 263–274. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Michael Armbrust, Armando Fox, Rean Griffith, Anthony D. Joseph, Randy H. Katz, Andrew Konwinski, Gunho Lee, David A. Patterson, Ariel Rabkin, and Matei Zaharia. 2009. <i>Above the Clouds: A Berkeley View of Cloud Computing</i>. Technical Report. EECS Department, University of California, Berkeley.Google ScholarGoogle Scholar
  23. Timon Back and Vasilios Andrikopoulos. 2018. Using a Microbenchmark to Compare Function as a Service Solutions. In <i>Service-Oriented and Cloud Computing</i>. Springer International Publishing, 146–160. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Daniel Barcelona-Pons, Marc Sánchez-Artigas, Gerard París, Pierre Sutra, and Pedro García-López. 2019. On the FaaS Track: Building Stateful Distributed Applications with Serverless Architectures. In <i>Proceedings of the 20th International Middleware Conference</i> (Davis, CA, USA) <i>(Middleware '19)</i>. Association for Computing Machinery, New York, NY, USA, 41–54. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Scott Beamer, Krste Asanović, and David Patterson. 2013. Direction-optimizing breadth-first search. <i>Scientific Programming</i> 21, 3-4 (2013), 137–148.Google ScholarGoogle Scholar
  26. Scott Beamer, Krste Asanović, and David Patterson. 2015. The GAP benchmark suite. <i>arXiv preprint arXiv:1508.03619</i> (2015).Google ScholarGoogle Scholar
  27. Tal Ben-Nun, Maciej Besta, Simon Huber, Alexandros Nikolaos Ziogas, Daniel Peter, and Torsten Hoefler. 2019. A modular benchmarking infrastructure for high-performance and reproducible deep learning. In <i>2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS)</i>. IEEE, 66–77.Google ScholarGoogle Scholar
  28. Pavel Berkhin. 2005. A survey on PageRank computing. <i>Internet mathematics</i> 2, 1 (2005), 73–120.Google ScholarGoogle Scholar
  29. Maciej Besta et al. 2019a. Slim Graph: Practical Lossy Graph Compression for Approximate Graph Processing, Storage, and Analytics. (2019).Google ScholarGoogle Scholar
  30. Maciej Besta, Armon Carigiet, Kacper Janda, Zur Vonarburg-Shmaria, Lukas Gianinazzi, and Torsten Hoefler. 2020. High-performance parallel graph coloring with strong guarantees on work, depth, and quality. In <i>Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis</i>. 1–17.Google ScholarGoogle Scholar
  31. Maciej Besta, Marc Fischer, Tal Ben-Nun, Johannes De Fine Licht, and Torsten Hoefler. 2019b. Substream-Centric Maximum Matchings on FPGA. In <i>ACM/SIGDA FPGA</i>. 152–161.Google ScholarGoogle Scholar
  32. Maciej Besta and Torsten Hoefler. 2015. Accelerating irregular computations with hardware transactional memory and active messages. In <i>ACM HPDC</i>.Google ScholarGoogle Scholar
  33. Maciej Besta, Florian Marending, Edgar Solomonik, and Torsten Hoefler. 2017a. Slimsell: A vectorizable graph representation for breadth-first search. In <i>IEEE IPDPS</i>. 32–41.Google ScholarGoogle Scholar
  34. Maciej Besta, Emanuel Peter, Robert Gerstenberger, Marc Fischer, Michał Podstawski, Claude Barthels, Gustavo Alonso, and Torsten Hoefler. 2019c. Demystifying Graph Databases: Analysis and Taxonomy of Data Organization, System Designs, and Graph Queries. <i>arXiv preprint arXiv:1910.09017</i> (2019).Google ScholarGoogle Scholar
  35. Maciej Besta, Michał Podstawski, Linus Groner, Edgar Solomonik, and Torsten Hoefler. 2017b. To push or to pull: On reducing communication and synchronization in graph computations. In <i>Proceedings of the 26th International Symposium on High-Performance Parallel and Distributed Computing</i>. 93–104.Google ScholarGoogle Scholar
  36. Maciej Besta, Dimitri Stanojevic, Tijana Zivic, Jagpreet Singh, Maurice Hoerold, and Torsten Hoefler. 2018. Log (graph): a near-optimal high-performance graph representation.. In <i>PACT</i>. 7–1.Google ScholarGoogle Scholar
  37. Carsten Binnig, Donald Kossmann, Tim Kraska, and Simon Loesing. 2009. How is the Weather Tomorrow?: Towards a Benchmark for the Cloud. In <i>Proceedings of the Second International Workshop on Testing Database Systems</i> (Providence, Rhode Island) <i>(DBTest '09)</i>. ACM, New York, NY, USA, Article 9, 6 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Stephen M. Blackburn, Robin Garner, Chris Hoffmann, Asjad M. Khang, Kathryn S. McKinley, Rotem Bentzur, Amer Diwan, Daniel Feinberg, Daniel Frampton, Samuel Z. Guyer, Martin Hirzel, Antony Hosking, Maria Jump, Han Lee, J. Eliot B. Moss, Aashish Phansalkar, Darko Stefanović, Thomas VanDrunen, Daniel von Dincklage, and Ben Wiedermann. 2006. The DaCapo Benchmarks: Java Benchmarking Development and Analysis. <i>SIGPLAN Not.</i> 41, 10 (Oct. 2006), 169–190. 0362-1340 Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Jean-Yves Le Boudec. 2011. <i>Performance Evaluation of Computer and Communication Systems</i>. EFPL Press.Google ScholarGoogle Scholar
  40. Ulrik Brandes and Christian Pich. 2007. Centrality estimation in large networks. <i>International Journal of Bifurcation and Chaos</i> 17, 07 (2007), 2303–2318.Google ScholarGoogle Scholar
  41. Joao Carreira, Pedro Fonseca, Alexey Tumanov, Andrew Zhang, and Randy Katz. 2019. Cirrus: A Serverless Framework for End-to-End ML Workflows. In <i>Proceedings of the ACM Symposium on Cloud Computing</i> (Santa Cruz, CA, USA) <i>(SoCC '19)</i>. Association for Computing Machinery, New York, NY, USA, 13–24. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Anirban Das, Stacy Patterson, and Mike Wittie. 2018. EdgeBench: Benchmarking Edge Computing Platforms. In <i>2018 IEEE/ACM International Conference on Utility and Cloud Computing Companion (UCC Companion)</i>. IEEE. Google ScholarGoogle ScholarCross RefCross Ref
  43. Jack J Dongarra, Hans W Meuer, Erich Strohmaier, et al. 1997. TOP500 supercomputer sites. <i>Supercomputer</i> 13 (1997), 89–111.Google ScholarGoogle Scholar
  44. L. Feng, P. Kudva, D. Da Silva, and J. Hu. 2018. Exploring Serverless Computing for Neural Network Training. In <i>2018 IEEE 11th International Conference on Cloud Computing (CLOUD)</i>. 334–341. 2159-6190 Google ScholarGoogle ScholarCross RefCross Ref
  45. Enno Folkerts, Alexander Alexandrov, Kai Sachs, Alexandru Iosup, Volker Markl, and Cafer Tosun. 2013. Benchmarking in the Cloud: What It Should, Can, and Cannot Be. In <i>Selected Topics in Performance Evaluation and Benchmarking</i>, Raghunath Nambiar and Meikel Poess (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 173–188.Google ScholarGoogle Scholar
  46. Lester Randolph Ford and Delbert R Fulkerson. 2009. Maximal flow through a network. In <i>Classic papers in combinatorics</i>. Springer, 243–248.Google ScholarGoogle Scholar
  47. Sadjad Fouladi, Francisco Romero, Dan Iter, Qian Li, Shuvo Chatterjee, Christos Kozyrakis, Matei Zaharia, and Keith Winstein. 2019. From Laptop to Lambda: Outsourcing Everyday Jobs to Thousands of Transient Functional Containers. In <i>2019 USENIX Annual Technical Conference (USENIX ATC 19)</i>. USENIX Association, Renton, WA, 475–488. https://www.usenix.org/conference/atc19/presentation/fouladiGoogle ScholarGoogle Scholar
  48. Sadjad Fouladi, Riad S. Wahby, Brennan Shacklett, Karthikeyan Vasuki Balasubramaniam, William Zeng, Rahul Bhalerao, Anirudh Sivaraman, George Porter, and Keith Winstein. 2017. Encoding, Fast and Slow: Low-Latency Video Processing Using Thousands of Tiny Threads. In <i>Proceedings of the 14th USENIX Conference on Networked Systems Design and Implementation</i> (Boston, MA, USA) <i>(NSDI'17)</i>. USENIX Association, USA, 363–376.Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Yu Gan, Brendon Jackson, Kelvin Hu, Meghna Pancholi, Yuan He, Brett Clancy, Chris Colen, Fukang Wen, Catherine Leung, Siyuan Wang, Leon Zaruvinsky, Yanqi Zhang, Mateo Espinosa, Rick Lin, Zhongling Liu, Jake Padilla, Christina Delimitrou, Dailun Cheng, Ankitha Shetty, Priyal Rathi, Nayan Katarki, Ariana Bruno, Justin Hu, and Brian Ritchken. 2019a. An Open-Source Benchmark Suite for Microservices and Their Hardware-Software Implications for Cloud & Edge Systems. In <i>Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems - ASPLOS 19</i>. ACM Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Yu Gan, Yanqi Zhang, Dailun Cheng, Ankitha Shetty, Priyal Rathi, Nayan Katarki, Ariana Bruno, Justin Hu, Brian Ritchken, Brendon Jackson, and et al. 2019b. An Open-Source Benchmark Suite for Microservices and Their Hardware-Software Implications for Cloud & Edge Systems. In <i>Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems</i> (Providence, RI, USA) <i>(ASPLOS ’19)</i>. Association for Computing Machinery, New York, NY, USA, 3–18. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Lukas Gianinazzi, Pavel Kalvoda, Alessandro De Palma, Maciej Besta, and Torsten Hoefler. 2018. Communication-avoiding parallel minimum cuts and connected components. In <i>ACM SIGPLAN Notices</i>, Vol. 53. ACM, 219–232.Google ScholarGoogle Scholar
  52. Oleksandr Grygorash, Yan Zhou, and Zach Jorgensen. 2006. Minimum spanning tree based clustering algorithms. In <i>2006 18th IEEE International Conference on Tools with Artificial Intelligence (ICTAI'06)</i>. IEEE, 73–81.Google ScholarGoogle Scholar
  53. Joseph M. Hellerstein, Jose M. Faleiro, Joseph E. Gonzalez, Johann Schleier-Smith, Vikram Sreekanti, Alexey Tumanov, and Chenggang Wu. 2018. Serverless Computing: One Step Forward, Two Steps Back. <i>CoRR</i> abs/1812.03651 (2018). [arxiv]1812.03651 http://arxiv.org/abs/1812.03651Google ScholarGoogle Scholar
  54. Torsten Hoefler and Roberto Belli. 2015a. Scientific Benchmarking of Parallel Computing Systems. ACM, 73:1–73:12. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC15).Google ScholarGoogle Scholar
  55. Torsten Hoefler and Roberto Belli. 2015b. Scientific benchmarking of parallel computing systems: twelve ways to tell the masses when reporting performance results. In <i>Proceedings of the international conference for high performance computing, networking, storage and analysis</i>. 1–12.Google ScholarGoogle Scholar
  56. Torsten Hoefler and Roberto Belli. 2015c. Scientific Benchmarking of Parallel Computing Systems: Twelve Ways to Tell the Masses When Reporting Performance Results. In <i>Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis</i> (Austin, Texas) <i>(SC '15)</i>. Association for Computing Machinery, New York, NY, USA, Article 73, 12 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. T. Hoefler, T. Schneider, and A. Lumsdaine. 2008. Accurately measuring collective operations at massive scale. In <i>2008 IEEE International Symposium on Parallel and Distributed Processing</i>. 1–8.Google ScholarGoogle Scholar
  58. Alexandru Iosup, Tim Hegeman, Wing Lung Ngai, Stijn Heldens, Arnau Prat-Pérez, Thomas Manhardto, Hassan Chafio, Mihai Capotă, Narayanan Sundaram, Michael Anderson, et al. 2016. LDBC Graphalytics: A benchmark for large-scale graph analysis on parallel and distributed platforms. <i>Proceedings of the VLDB Endowment</i> 9, 13 (2016), 1317–1328.Google ScholarGoogle Scholar
  59. V. Ishakian, V. Muthusamy, and A. Slominski. 2018. Serving Deep Learning Models in a Serverless Platform. In <i>2018 IEEE International Conference on Cloud Engineering (IC2E)</i>. 257–262. null Google ScholarGoogle ScholarCross RefCross Ref
  60. Eric Jonas, Johann Schleier-Smith, Vikram Sreekanti, Chia-che Tsai, Anurag Khandelwal, Qifan Pu, Vaishaal Shankar, Joao Carreira, Karl Krauth, Neeraja Jayant Yadwadkar, Joseph E. Gonzalez, Raluca Ada Popa, Ion Stoica, and David A. Patterson. 2019. Cloud Programming Simplified: A Berkeley View on Serverless Computing. <i>CoRR</i> abs/1902.03383 (2019). [arxiv]1902.03383 http://arxiv.org/abs/1902.03383Google ScholarGoogle Scholar
  61. Eric Jonas, Shivaram Venkataraman, Ion Stoica, and Benjamin Recht. 2017. Occupy the Cloud: Distributed Computing for the 99%. <i>CoRR</i> abs/1702.04024 (2017). [arxiv]1702.04024 http://arxiv.org/abs/1702.04024Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. Svilen Kanev, Juan Pablo Darago, Kim Hazelwood, Parthasarathy Ranganathan, Tipp Moseley, Gu-Yeon Wei, and David Brooks. 2015. Profiling a Warehouse-Scale Computer. <i>SIGARCH Comput. Archit. News</i> 43, 3S (June 2015), 158–169. 0163-5964 Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. N. Kaviani and M. Maximilien. 2018. CF Serverless: Attempts at a Benchmark for Serverless Computing. https://docs.google.com/document/d/1e7xTz1P9aPpb0CFZucAAI16Rzef7PWSPLN71pNDa5jg. Accessed: 2020-01-20.Google ScholarGoogle Scholar
  64. Jeongchul Kim and Kyungyong Lee. 2019a. FunctionBench: A Suite of Workloads for Serverless Cloud Function Service. In <i>2019 IEEE 12th International Conference on Cloud Computing (CLOUD)</i>. IEEE. Google ScholarGoogle ScholarCross RefCross Ref
  65. Jeongchul Kim and Kyungyong Lee. 2019b. Practical Cloud Workloads for Serverless FaaS. In <i>Proceedings of the ACM Symposium on Cloud Computing - SoCC 19</i>. ACM Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  66. Ana Klimovic, Yawen Wang, Christos Kozyrakis, Patrick Stuedi, Jonas Pfefferle, and Animesh Trivedi. 2018a. Understanding Ephemeral Storage for Serverless Analytics. In <i>2018 USENIX Annual Technical Conference (USENIX ATC 18)</i>. USENIX Association, Boston, MA, 789–794. https://www.usenix.org/conference/atc18/presentation/klimovic-serverlessGoogle ScholarGoogle Scholar
  67. Ana Klimovic, Yawen Wang, Patrick Stuedi, Animesh Trivedi, Jonas Pfefferle, and Christos Kozyrakis. 2018b. Pocket: Elastic Ephemeral Storage for Serverless Analytics. In <i>Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation</i> (Carlsbad, CA, USA) <i>(OSDI'18)</i>. USENIX Association, USA, 427–444.Google ScholarGoogle Scholar
  68. Benjamin D Lee. 2018. Squiggle: a user-friendly two-dimensional DNA sequence visualization tool. <i>Bioinformatics</i> (sep 2018). Google ScholarGoogle ScholarCross RefCross Ref
  69. Benjamin D Lee, Michael A Timony, and Pablo Ruiz. 2019. DNAvisualization.org: a serverless web tool for DNA sequence visualization. <i>Nucleic Acids Research</i> 47, W1 (06 2019), W20–W25. 0305-1048 https://academic.oup.com/nar/article-pdf/47/W1/W20/28879727/gkz404.pdf Google ScholarGoogle ScholarCross RefCross Ref
  70. Pedro García López, Marc Sánchez Artigas, Simon Shillaker, Peter R. Pietzuch, David Breitgand, Gil Vernik, Pierre Sutra, Tristan Tarrant, and Ana Juan Ferrer. 2019. ServerMix: Tradeoffs and Challenges of Serverless Data Analytics. <i>CoRR</i> abs/1907.11465 (2019). [arxiv]1907.11465 http://arxiv.org/abs/1907.11465Google ScholarGoogle Scholar
  71. Andrew Lumsdaine, Douglas Gregor, Bruce Hendrickson, and Jonathan Berry. 2007. Challenges in parallel graph processing. <i>Parallel Processing Letters</i> 17, 01 (2007), 5–20.Google ScholarGoogle Scholar
  72. Pascal Maissen, Pascal Felber, Peter Kropf, and Valerio Schiavoni. 2020. FaaSdom. <i>Proceedings of the 14th ACM International Conference on Distributed and Event-based Systems</i> (Jul 2020). Google ScholarGoogle ScholarDigital LibraryDigital Library
  73. Filipe Manco, Costin Lupu, Florian Schmidt, Jose Mendes, Simon Kuenzer, Sumit Sati, Kenichi Yasukata, Costin Raiciu, and Felipe Huici. 2017. My VM is Lighter (and Safer) than Your Container. In <i>Proceedings of the 26th Symposium on Operating Systems Principles</i> (Shanghai, China) <i>(SOSP ’17)</i>. Association for Computing Machinery, New York, NY, USA, 218–233. Google ScholarGoogle ScholarDigital LibraryDigital Library
  74. Johannes Manner, Martin EndreB, Tobias Heckel, and Guido Wirtz. 2018a. Cold Start Influencing Factors in Function as a Service. <i>2018 IEEE/ACM International Conference on Utility and Cloud Computing Companion (UCC Companion)</i> (2018), 181–188.Google ScholarGoogle Scholar
  75. Johannes Manner, Martin Endreß, Tobias Heckel, and Guido Wirtz. 2018b. Cold start influencing factors in function as a service. In <i>2018 IEEE/ACM International Conference on Utility and Cloud Computing Companion (UCC Companion)</i>. IEEE, 181–188.Google ScholarGoogle Scholar
  76. Ingo Müller, Renato Marroquin, and Gustavo Alonso. 2019. Lambada: Interactive Data Analytics on Cold Data using Serverless Cloud Infrastructure. <i>ArXiv</i> abs/1912.00937 (2019).Google ScholarGoogle Scholar
  77. Richard C Murphy, Kyle B Wheeler, Brian W Barrett, and James A Ang. 2010. Introducing the graph 500. <i>Cray Users Group (CUG)</i> 19 (2010), 45–74.Google ScholarGoogle Scholar
  78. Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. <i>The pagerank citation ranking: Bringing order to the web.</i> Technical Report. Stanford InfoLab.Google ScholarGoogle Scholar
  79. Christos H Papadimitriou and Kenneth Steiglitz. 1998. <i>Combinatorial optimization: algorithms and complexity</i>. Courier Corporation.Google ScholarGoogle Scholar
  80. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In <i>Advances in Neural Information Processing Systems 32</i>, H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 8024–8035. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdfGoogle ScholarGoogle ScholarDigital LibraryDigital Library
  81. Qifan Pu, Shivaram Venkataraman, and Ion Stoica. 2019. Shuffling, Fast and Slow: Scalable Analytics on Serverless Infrastructure. In <i>16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19)</i>. USENIX Association, Boston, MA, 193–206. https://www.usenix.org/conference/nsdi19/presentation/puGoogle ScholarGoogle Scholar
  82. Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, Ramesh Chukka, Cody Coleman, Sam Davis, Pan Deng, Greg Diamos, Jared Duke, Dave Fick, J. Scott Gardner, Itay Hubara, Sachin Idgunji, Thomas B. Jablin, Jeff Jiao, Tom St. John, Pankaj Kanwar, David Lee, Jeffery Liao, Anton Lokhmotov, Francisco Massa, Peng Meng, Paulius Micikevicius, Colin Osborne, Gennady Pekhimenko, Arun Tejusve Raghunath Rajan, Dilip Sequeira, Ashish Sirasao, Fei Sun, Hanlin Tang, Michael Thomson, Frank Wei, Ephrem Wu, Lingjie Xu, Koichi Yamada, Bing Yu, George Yuan, Aaron Zhong, Peizhao Zhang, and Yuchen Zhou. 2019. MLPerf Inference Benchmark. [arxiv]1911.02549 [cs.LG]Google ScholarGoogle Scholar
  83. Sherif Sakr, Angela Bonifati, Hannes Voigt, Alexandru Iosup, Khaled Ammar, Renzo Angles, Walid Aref, Marcelo Arenas, Maciej Besta, Peter A Boncz, et al. 2020. The Future is Big Graphs! A Community View on Graph Processing Systems. <i>arXiv preprint arXiv:2012.06171</i> (2020).Google ScholarGoogle Scholar
  84. Hermann Schweizer, Maciej Besta, and Torsten Hoefler. 2015. Evaluating the cost of atomic operations on modern architectures. In <i>IEEE PACT</i>. 445–456.Google ScholarGoogle Scholar
  85. Mohammad Shahrad, Jonathan Balkind, and David Wentzlaff. 2019. Architectural Implications of Function-as-a-Service Computing. In <i>Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture</i> (Columbus, OH, USA) <i>(MICRO '52)</i>. Association for Computing Machinery, New York, NY, USA, 1063–1075. Google ScholarGoogle ScholarDigital LibraryDigital Library
  86. Vaishaal Shankar, Karl Krauth, Qifan Pu, Eric Jonas, Shivaram Venkataraman, Ion Stoica, Benjamin Recht, and Jonathan Ragan-Kelley. 2018. numpywren: serverless linear algebra. <i>CoRR</i> abs/1810.09679 (2018). [arxiv]1810.09679 http://arxiv.org/abs/1810.09679Google ScholarGoogle Scholar
  87. Edgar Solomonik, Maciej Besta, Flavio Vella, and Torsten Hoefler. 2017. Scaling betweenness centrality using communication-efficient sparse matrix multiplication. In <i>ACM/IEEE Supercomputing</i>. 47.Google ScholarGoogle Scholar
  88. N. Somu, N. Daw, U. Bellur, and P. Kulkarni. 2020. PanOpticon: A Comprehensive Benchmarking Tool for Serverless Applications. In <i>2020 International Conference on COMmunication Systems NETworkS (COMSNETS)</i>. 144–151.Google ScholarGoogle Scholar
  89. Vikram Sreekanti, Chenggang Wu, Xiayue Charles Lin, Johann Schleier-Smith, Joseph E. Gonzalez, Joseph M. Hellerstein, and Alexey Tumanov. 2020. Cloudburst: Stateful Functions-as-a-Service. <i>Proc. VLDB Endow.</i> 13, 12 (July 2020), 2438–2452. 2150-8097 Google ScholarGoogle ScholarDigital LibraryDigital Library
  90. Dan Terpstra, Heike Jagode, Haihang You, and Jack Dongarra. 2010. Collecting Performance Data with PAPI-C. In <i>Tools for High Performance Computing 2009</i>, Matthias S. Müller, Michael M. Resch, Alexander Schulz, and Wolfgang E. Nagel (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 157–173.Google ScholarGoogle Scholar
  91. Jóakim v. Kistowski, Jeremy A. Arnold, Karl Huppler, Klaus-Dieter Lange, John L. Henning, and Paul Cao. 2015. How to Build a Benchmark. In <i>Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering</i> (Austin, Texas, USA) <i>(ICPE '15)</i>. ACM, New York, NY, USA, 333–336. Google ScholarGoogle ScholarDigital LibraryDigital Library
  92. Erwin van Eyk, Alexandru Iosup, Cristina L. Abad, Johannes Grohmann, and Simon Eismann. 2018. A SPEC RG Cloud Groups Vision on the Performance Challenges of FaaS Cloud Architectures. In <i>Companion of the 2018 ACM/SPEC International Conference on Performance Engineering - ICPE 18</i>. ACM Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  93. Jeffrey S Vetter, Ron Brightwell, Maya Gokhale, Pat McCormick, Rob Ross, John Shalf, Katie Antypas, David Donofrio, Travis Humble, Catherine Schuman, et al. 2019. <i>Extreme Heterogeneity 2018-Productive Computational Science in the Era of Extreme Heterogeneity: Report for DOE ASCR Workshop on Extreme Heterogeneity</i>. Technical Report. Lawrence Berkeley National Lab.(LBNL), Berkeley, CA (United States).Google ScholarGoogle Scholar
  94. Liang Wang, Mengyuan Li, Yinqian Zhang, Thomas Ristenpart, and Michael Swift. 2018. Peeking behind the Curtains of Serverless Platforms. In <i>Proceedings of the 2018 USENIX Conference on Usenix Annual Technical Conference</i> (Boston, MA, USA) <i>(USENIX ATC '18)</i>. USENIX Association, USA, 133–145.Google ScholarGoogle Scholar
  95. Tianyi Yu, Qingyuan Liu, Dong Du, Yubin Xia, Binyu Zang, Ziqian Lu, Pingchao Yang, Chenggang Qin, and Haibo Chen. 2020. Characterizing Serverless Platforms with Serverlessbench. In <i>Proceedings of the 11th ACM Symposium on Cloud Computing</i> (Virtual Event, USA) <i>(SoCC '20)</i>. Association for Computing Machinery, New York, NY, USA, 30–44. Google ScholarGoogle ScholarDigital LibraryDigital Library
  96. Miao Zhang, Yifei Zhu, Cong Zhang, and Jiangchuan Liu. 2019. Video Processing with Serverless Computing: A Measurement Study. In <i>Proceedings of the 29th ACM Workshop on Network and Operating Systems Support for Digital Audio and Video</i> (Amherst, Massachusetts) <i>(NOSSDAV ’19)</i>. Association for Computing Machinery, New York, NY, USA, 61–66. Google ScholarGoogle ScholarDigital LibraryDigital Library
  97. Wen Zhang, Vivian Fang, Aurojit Panda, and Scott Shenker. 2020. Kappa: A Programming Framework for Serverless Computing. In <i>Proceedings of the 11th ACM Symposium on Cloud Computing</i> (Virtual Event, USA) <i>(SoCC '20)</i>. Association for Computing Machinery, New York, NY, USA, 328–343. Google ScholarGoogle ScholarDigital LibraryDigital Library
  98. Y. Zhu, D. Richins, M. Halpern, and V. J. Reddi. 2015. Microarchitectural implications of event-driven server-side web applications. In <i>2015 48th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)</i>. 762–774. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. SeBS: a serverless benchmark suite for function-as-a-service computing

              Comments

              Login options

              Check if you have access through your login credentials or your institution to get full access on this article.

              Sign in

              Full Access

              Get this Publication

              • Published in

                cover image ACM Conferences
                Middleware '21: Proceedings of the 22nd International Middleware Conference
                December 2021
                398 pages
                ISBN:9781450385343
                DOI:10.1145/3464298

                Copyright © 2021 ACM

                Publisher

                Association for Computing Machinery

                New York, NY, United States

                Publication History

                • Published: 2 October 2021

                Permissions

                Request permissions about this article.

                Request Permissions

                Check for updates

                Author Tags

                Qualifiers

                • research-article

                Acceptance Rates

                Overall Acceptance Rate 203 of 948 submissions, 21%

              PDF Format

              View or Download as a PDF file.

              PDF

              eReader

              View online with eReader.

              eReader
              View Table of Contents
              About Cookies On This Site

              We use cookies to ensure that we give you the best experience on our website.

              Learn more

              Got it!