Repository navigation
Expand file tree
/
Copy pathwhy.qmd
More file actions
370 lines (268 loc) · 11.2 KB
/
Copy pathwhy.qmd
File metadata and controls
370 lines (268 loc) · 11.2 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
---
title: "Why Ibis?"
---
Ibis defines a Python dataframe API that executes on any query engine -- the
frontend for any backend data platform, with nearly 20 backends today. This allows
Ibis to have excellent performance -- as good as the backend it is connected to
-- with a consistent user experience.
## What is Ibis?
Ibis is the portable Python dataframe library.
We can demonstrate this with a simple example on a few local query engines:
```{python}
import ibis
ibis.options.interactive = True
```
::: {.panel-tabset}
```{python}
#| echo: false
t = ibis.examples.penguins.fetch()
t.to_parquet("penguins.parquet")
```
## DuckDB
```{python}
con = ibis.connect("duckdb://") # <1>
t = con.read_parquet("penguins.parquet")
t.limit(3)
```
1. Change only your connection to switch between backends.
```{python}
t.group_by(["species", "island"]).agg(count=t.count()).order_by("count")
```
## Polars
```{python}
con = ibis.connect("polars://") # <1>
t = con.read_parquet("penguins.parquet")
t.limit(3)
```
1. Change only your connection to switch between backends.
```{python}
t.group_by(["species", "island"]).agg(count=t.count()).order_by("count")
```
## DataFusion
```{python}
con = ibis.connect("datafusion://") # <1>
t = con.read_parquet("penguins.parquet")
t.limit(3)
```
1. Change only your connection to switch between backends.
```{python}
t.group_by(["species", "island"]).agg(count=t.count()).order_by("count")
```
## PySpark
```{python}
con = ibis.connect("pyspark://") # <1>
t = con.read_parquet("penguins.parquet")
t.limit(3)
```
1. Change only your connection to switch between backends.
```{python}
t.group_by(["species", "island"]).agg(count=t.count()).order_by("count")
```
:::
## Who is Ibis for?
Ibis is for data engineers, data analysts, and data scientists (or any title
that needs to work with data!) to use directly with their data platform(s) of
choice. It also has benefits for [data platforms](#ibis-for-data-platforms),
[organizations](#ibis-for-organizations), and [library
developers](#ibis-for-library-developers).
### Ibis for practitioners
You can use Ibis at any stage of your data workflow, no matter your role.
**Data engineers** can use Ibis to:
- write and maintain complex ETL/ELT jobs
- replace fragile SQL string pipelines with a robust Python API
- replace PySpark with a more Pythonic API that supports Spark and many other
backends
**Data analysts** can use Ibis to:
- use Ibis interactive mode for rapid exploration
- perform rapid exploratory data analysis using interactive mode
- [create end-to-end analytics workflows](./posts/ibis-analytics/index.qmd)
- work in a general-purpose, yet easy to learn, programming language without the
need for formatting SQL strings
**Data scientists** can use Ibis to:
- extract a sample of data for local iteration with a fast local backend
- prototype with the same API that will be used in production
- preprocess and feature engineer data before training a machine learning model
### Ibis for data platforms
Data platforms can use Ibis to quickly bring a fully-featured Python dataframe
library with minimal effort to their platform. In addition to a great Python
dataframe experience for their users, they also get integrations into the
[broader Python and ML ecosystem](#ecosystem).
Often, data platforms evolve to support Python in some sequence like:
1. Develop a fast query engine with a SQL frontend
2. Gain popularity and need to support Python for data science and ML use cases
3. Develop a bespoke pandas or PySpark-like dataframe library and ML
integrations
This third step is where Ibis comes in. Instead of spending a lot of time and
money developing a bespoke Python dataframe library, you can create an Ibis
backend for your data platform [in as little as four hours for an experienced
Ibis
developer](https://voltrondata.com/resources/new-ibis-backend-shipped-in-four-hours-druid)
or, more typically, on the order of
[one](https://github.com/ibis-project/ibis/pull/7954) or
[two](https://github.com/ibis-project/ibis/pull/7303) months for a new
contributor.
::: {.callout-warning title="Why not the pandas or PySpark APIs?" collapse="true"}
The pandas API inherently does not scale due to its single-threaded design,
ordered index, and a lot of API baggage. The creator of pandas (and Ibis!) has
[talked about the issues with pandas
publicly](https://wesmckinney.com/blog/apache-arrow-pandas-internals/). While
there have been projects attempting to scale the pandas API, they always result
in a dubious support matrix. You can see that with
[Modin](https://modin.readthedocs.io/en/stable/supported_apis/dataframe_supported.html)
or [pandas on Spark (formerly known as
Koalas)](https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/supported_pandas_api.html).
[Google BigQuery
DataFrames](https://github.com/googleapis/python-bigquery-dataframes) is a more
modern attempt to scale the pandas API built on top of Ibis. If you are going
to build a pandas API we recommend you take a look at this project.
PySpark is a great API for Spark, but not very Pythonic and tightly coupled to
the Spark execution engine.
Ibis takes inspiration from pandas and PySpark -- and R and SQL -- but is designed to be
scalable from the start. If offers a neutral,
[self-governed](https://github.com/ibis-project/governance) open source option
for your data platform.
:::
### Ibis for organizations
Organizations can use Ibis to standardize the interface for SQL and Python data
practitioners. It also allows organizations to:
- transfer data between systems
- transform, analyze, and prepare data where it lives
- benchmark your workload(s) across data systems using the same code
- mix SQL and Python code seamlessly, with all the benefits of a general-purpose
programming language, type checking, and expression validation
### Ibis for library developers
Python developers creating libraries can use Ibis to:
- instantly support nearly 20 data backends
- instantly support pandas, PyArrow, and Polars objects
- read and write from all common file formats (depending on the backend)
- trace column-level lineage through Ibis expressions
- compile Ibis expressions to SQL or Substrait
- perform cross-dialect SQL transpilation (powered by
[SQLGlot](https://github.com/tobymao/sqlglot))
## How does Ibis work?
Most Python dataframes are tightly coupled to their execution engine. And many
databases only support SQL, with no Python API. Ibis solves this problem by
providing a common API for data manipulation in Python, and compiling that API
into the backend's native language. This means you can learn a single API and
use it across any supported backend (execution engine).
Ibis broadly supports two types of backend:
1. SQL-generating backends
2. DataFrame-generating backends
```{python}
#| echo: false
from backends_sankey import fig
fig.show()
```
As you can see, most backends generate SQL. Ibis uses
[SQLGlot](https://github.com/tobymao/sqlglot) to transform Ibis expressions into
SQL strings. You can also use the
[`.sql()`](./how-to/extending/sql.qmd#table.sql) methods to mix in SQL strings,
compiling them to Ibis expressions.
While portability with Ibis isn't perfect, commonalities across backends and
SQL dialects combined with years of engineering effort produce a full-featured
and robust framework for data manipulation in Python.
In the long-term, we aim for a standard query plan Intermediate Representation
(IR) like [Substrait](https://substrait.io) to simplify this further.
## Python + SQL: better together
For most backends, Ibis works by compiling Python expressions into SQL:
```{python}
g = t.group_by(["species", "island"]).agg(count=t.count()).order_by("count")
ibis.to_sql(g)
```
You can mix and match Python and SQL code:
```{python}
sql = """
SELECT
species,
island,
COUNT(*) AS count
FROM penguins
GROUP BY species, island
""".strip()
```
::: {.panel-tabset}
## DuckDB
```{python}
con = ibis.connect("duckdb://")
t = con.read_parquet("penguins.parquet")
g = t.alias("penguins").sql(sql)
g
```
```{python}
g.order_by("count")
```
## DataFusion
```{python}
con = ibis.connect("datafusion://")
t = con.read_parquet("penguins.parquet")
g = t.alias("penguins").sql(sql)
g
```
```{python}
g.order_by("count")
```
## PySpark
```{python}
con = ibis.connect("pyspark://")
t = con.read_parquet("penguins.parquet")
g = t.alias("penguins").sql(sql)
g
```
```{python}
g.order_by("count")
```
:::
This allows you to combine the flexibility of Python with the scale and
performance of modern SQL.
## Scaling up and out
Out of the box, Ibis offers a great local experience for working with many file
formats. You can scale up with DuckDB (the default backend) or choose from other
great options like Polars and DataFusion to work locally with large datasets.
Once you hit scaling issues on a local machine, you can continue scaling up with
a larger machine in the cloud using the same backend and same code.
If you hit scaling issues on a large single-node machine, you can switch to a
distributed backend like PySpark, BigQuery, or Trino by simply changing your
connection string.
## Stream-batch unification
As of [Ibis 8.0](./posts/ibis-version-8.0.0-release/index.qmd), the first stream
processing backends have been added. Since these systems tend to support SQL, we
can with minimal changes to Ibis support both batch and streaming workloads with
a single API. We aim to further unify the batch and streaming paradigms going
forward.
## Ecosystem
Ibis is part of a larger ecosystem of Python data tools. It is designed to work
well with other tools in this ecosystem, and we continue to make it easier to
use Ibis with other tools over time.
Ibis already works with other Python dataframes like:
- [pandas](https://github.com/pandas-dev/pandas)
- [Polars](https://github.com/pola-rs/polars)
Ibis already works well with visualization libraries like:
- [matplotlib](https://github.com/matplotlib/matplotlib)
- [seaborn](https://github.com/mwaskom/seaborn)
- [Plotly](https://github.com/plotly/plotly.py)
- [Vega-Altair](https://github.com/altair-viz/altair)
- [plotnine](https://github.com/has2k1/plotnine)
Ibis already works well with dashboarding libraries like:
- [Streamlit](https://github.com/streamlit/streamlit)
- [Dash](https://github.com/plotly/dash)
- [Quarto dashboards](https://github.com/quarto-dev/quarto-cli)
- [Shiny](https://shiny.posit.co/py/)
Ibis already works well with machine learning libraries like:
- [scikit-learn](https://github.com/scikit-learn/scikit-learn)
- [XGBoost](https://github.com/dmlc/xgboost)
- [LightGBM](https://github.com/microsoft/lightgbm)
- [PyTorch](https://github.com/pytorch/pytorch)
## Supported backends
{{< include ./_tabsets/install.qmd >}}
See the [backend support matrix](./backends/support/matrix.qmd) for details on operations
supported. [Open a feature
request](https://github.com/ibis-project/ibis/issues/new?assignees=&labels=feature&projects=&template=feature-request.yml&title=feat)
if you'd like to see support for an operation in a given backend. If the backend
supports it, we'll do our best to add it quickly!
## Community
Community discussions primarily take place on
[GitHub](https://github.com/ibis-project/ibis/discussions) and
[Zulip](https://ibis-project.zulipchat.com).
## Getting started
If you're interested in trying Ibis we recommend the [getting started
tutorial](./tutorials/basics.qmd).