Scala By the Bay

We all know technology choices have a profound impact on platform development, especially if that platform needs to scale. Choice of technology at the app server level, DB and specific design choices have an impact on scale. What about the choice of language? What does that have to do with technology choices? A strong team builds stellar software that shines. Does language have impact on the formation of the team? We all know team sets tone for culture. How does language choice impact the team culture, hiring, ramp-up, productivity? We will share the wisdom/learning of developing all Scala based scalable platform here at Verizon, ONcue. During the development of the OnCue service, we will share our experience that language choice is similar to a tip of the iceberg problem, the first area that is easy to observe but the majority of the scaling challenges are hidden and waiting to be discovered as problems arise. As we share our experience on platform development, we will also shed light on the least discussed topic on devops. What impact does a level of automation and the engineering focus on the ops team have to do with the developer productivity and the evolution of the platform.

Apache Spark has been blessed as the replacement for MapReduce in Hadoop environments. It also runs in other deployment modes. Spark provides better performance, better user productivity, and it supports a wider range of application scenarios than MapReduce, including event stream processing, ad hoc queries, graph representations and algorithms, and iterative algorithms, such as those commonly used in machine learning.

This talk discusses Spark from a Data Science perspective, it's strengths and weaknesses, the Scala, as well as Java, Python, and R APIs it offers for common analytics problems, what's missing, and what's planned. We'll look at support for ad hoc queries over large data sets, stream processing, machine learning algorithms, graph processing, and the user experience.

Resilience; most developers understand what the word means, at least superficially, but way too many lack a deeper understanding of what it really means in the context of the system that they are working on now. I find it really sad to see, since understanding and managing failure is more important today than ever. Outages are incredibly costly—for many definitions of cost—and can sometimes take down whole businesses. In this talk we will explore the essence of resilience. What does it really mean? What is its mechanics and characterizing traits? How do other sciences and industries manage it? We will see that everything hints at the same conclusion; there is no "happy path", failure is an option and resilience is by design. In this talk we will explore how. 

If you're here at Scala by the Bay, you're part of the Scala community. Let's take a look at what matters in the Scala community, how you can help, and what matters we still need to address as a community. Any way you can combine the words "Scala", "community" and "matters" we will explore.

Spark has a simple and high-level API, but one thing this enables is making big changes inside the engine without affecting user code. Over the past year, our team has been working on some of the biggest changes to Spark since its creation, improving both its performance and usability and debuggability.

I plan to cover some of the latest updates to Spark, including Project Tungsten for off-heap memory management, I/O layer improvements, and new monitoring tools. I'll also discuss new APIs that allow even richer optimization.

What happens if you take everything that is happening in your company—every click, every database change, every application log—and make it all available as a real-time stream of well structured data?

I will discuss the experience at LinkedIn and elsewhere moving from batch-oriented ETL to real-time streams using Apache Kafka. I’ll talk about how the design and implementation of Kafka was driven by this goal of acting as a real-time platform for event data. I will cover some of the challenges of scaling Kafka to hundreds of billions of events per day at Linkedin, supporting thousands of engineers, applications, and data systems in a self-service fashion.

I will describe how real-time streams can become the source of ETL into Hadoop or a relational data warehouse, and how real-time data can supplement the role of batch-oriented analytics in Hadoop or a traditional data warehouse.

I will also describe how applications and stream processing systems such as Storm, Spark, or Samza can make use of these feeds for sophisticated real-time data processing as events occur.