Loading…
This event has ended. View the official site or create your own event → Check it out
This event has ended. Create your own
View analytic
Monday, August 17 • 5:10pm - 5:50pm
Quantifind's Story: Building Custom Interactive Data Analytics Infrastructure

Sign up or log in to save this to your schedule and see who's attending!

Building interactive data analytics products on top of large volumes of data is challenging. This talk will outline Quantifind's infrastructure story for building analytics software. We will describe the infrastructure that we originally built on top of existing systems such as Spark/Hadoop. However, the bulk of this talk will focus on a new, custom distributed system that we have built in-house for our predictive analytics software. This new system includes a distributed, in-memory, interactive computing platform that supports fast interactive querying against large volumes of compacted raw data that isn't pre-aggregated. This infrastructure can be viewed as an in-memory combination of map/reduce style computation and indexed structures built from bit sets to offsets in the compacted off-heap data. Akka Cluster sits at the core of this system for distributed communication between nodes. We will discuss why we chose to build our own system as well as tips and tricks that we've learned along the way for pushing the JVM for these types of systems.

Speakers
avatar for Ryan LeCompte

Ryan LeCompte

Software Engineer, Quantifind
Ryan LeCompte is the infrastructure tech lead at Quantifind. Ryan has been developing backend systems and infrastructure with Scala for the past two years at Quantifind. Interests include concurrency, distributed programming, and data structures.


Monday August 17, 2015 5:10pm - 5:50pm
Track B

Attendees (11)