Sebastian Brestin

Context:
We wanted to improve the user and developer experience. The Spark application was presenting performance problems, and tech debt became a blocker for adding new functionality.

Solution:
Redesigned Spark application to make it more robust and flexible for data scientists and software engineers:
Partitioned data to avoid shuffles and data skew
Decreased number of partitions while increasing partition size to avoid data skew and driver congestion
Used strategic caching in order avoid recomputation and improve computation by cutting down the query plan
Refactored the application as a pipeline for engineers to add features as plugins to the pipeline
在具有相等比较值的行的情况下生成确定性Spark输出

Results:
减少了50%的执行时间，增加了配置灵活性, 在第二轮中，哪一个提高了数据科学家的生产力
减少了技术债务，从而改善了开发时间和开发体验
Improved functional tests by having a deterministic output

Context:
我们希望提高集群的可用性，因为它被长时间运行的Spark作业阻塞了.

Solution:
Researched to increase Spark S3 parquet write performance. To analyze Spark jobs, I have used Spark History Server and Ganglia to pinpoint where time is spent and compare configuration fixes.

Results:
Increased write performance by a factor of three.

Context:
我们希望提高产品的稳定性和我们的开发过程.

Solution:
为Spark管道实现健壮和通用的功能测试框架;
Allowed multiple instances of a test to run in parallel
允许测试作为套件的一部分以管道方式运行
Simulated production runs by using a representative dataset

Results:
Faster POC development
Reduced development time by 50%
Increased product stability
通过使用更少的资源模拟生产运行，降低了AWS成本

Context:
我们的一些Spark应用程序是长时间运行长达9小时的应用程序. 在此时间范围内，节点可能会内存不足或失去连接. 我们希望我们的应用程序运行得更快，并且在发生故障时更容易恢复.

Solution:
Redesigned Spark application from a monolith into a modular Spark application which writes intermediate results and can be run in parallel.

Results:
No failures due to out of memory
Easier to recover by re-running only the failed module

Context:
We wanted a churn model that we can use to predict the future for our members and also integrate it into other Spark applications.

Solution:
Built a Spark application for ingesting 1TB of data from multiple sources and generate 40k possible features based on which the data science team can perform EDA and test models.

Context:
数据科学团队必须最有效地使用数据和我们的系统来提高生产力.

Solution:
实现新的应用程序api，以帮助其他团队提高生产力;
1. Wrote better API for complex Spark or Cloud functionality
2. Wrote wrappers for complex application configuration
3. Implemented integration with adjacent applications
4. Wrote documentation and Unit Tests
5. Ran data analysis and data quality checks

For a minimum viable product, I designed and implemented a custom Django MapReduce system for matching an object with other million objects. 系统在60个Docker节点上运行，一个匹配任务在60秒内运行. In order to test the system I have extended Django test framework in order to test locally the Django custom MapReduce system without spawning virtual machines only by using multiple processes and cores.

This project aimed to match mutable objects (which users can create/update) with millions of immutable objects in real time (or as close as possible).

We chose Apache Spark because of its Python support, rich analytics toolkit, and streaming capabilities.

The solution was split into three Spark apps:
01. 使用Kafka和Spark流从数据源中检索可变数据, extracting features and saving the result to Cassandra.
02. 使用Kafka和Spark流从数据源中检索不可变数据, 提取特征并将结果保存到磁盘作为Parquet文件.
03. Loads data from Parquet files, computing a match percentage between all immutable objects and a single mutable object from Cassandra.

In order to query data in a reasonable amount of time, 我使用Elasticsearch对PostgreSQL表(数十亿条记录)进行了反规范化. 这将读性能提高了两个数量级，但增加了写损失. 因为只有部分文档频繁更改, the problem was solved using Elasticsearch bulk partial updates with Groovy scripts for complex fields.

I led the implementation of a single-sign-on platform, using OpenID Connect, Django, PostgreSQL, Angular/TypeScript和AWS(它集成了客户的整个产品套件).

While working as a contractor, I used NLTK, Scikit-learn, AWS Transcribe, AWS Lambda运行情绪分析以改善客户支持.

我领导了一个使用Django的高内容流平台的实现, PostgreSQL, Elasticsearch, and AWS. 该系统允许使用RESTful API高效地推送/提取内容.

在使用Spark时，我注意到缺乏关于PySpark的适当文档. This situation motivated me to put together this article aimed at thoroughly explaining the secondary sort design pattern using PySpark.

The article covers two solutions. The first solution uses Spark's groupByKey which does a sort in the reducer phase while the second solution uses Spark's repartitionAndSortWithinPartitions which leverages the shuffle phase and iterator-to-iterator transformation to sort more efficiently and not run out of memory.

I wrote this article about PostgreSQL and data partitioning in general with practical examples using the Django web framework.

The first part of the article goes on about the reasons to partition data and also how partitioning can be implemented in PostgreSQL.

The second part describes a few solutions one can use to take advantage of PostgreSQL partitioning while using a web framework such as Django.

Working with PostgreSQL for many years has inspired me to create an article about indexes that any developer should know.

In the first section of the article, I explain about the fundamentals of the PostgreSQL B-tree index structure with a focus on B-tree data structure and its main components—leaf nodes and internal nodes, what are their roles, 以及在执行查询时如何访问它们(这个过程也称为索引查找). The section ends with the index classification, with a broader view over the index key arity which helps explain the impact on a query’s performance.

In the second section, 下面是对查询计划的快速介绍，以便更好地理解后面的示例.

In the third section, I introduce the concept of predicates, and the explanations cover the process PostgreSQL uses to classify predicates depending on index definition and feasibility.

最后一节将深入讨论扫描的机制以及索引如何定义, data distribution, 甚至谓词的使用也会影响查询的性能.