79449236

Date: 2025-02-18 18:14:50
Score: 1
Natty:
Report link

Per Polars Support:

First, you need way more iterations than just 100 for such a small timewindow. With 10,000 iterations I get the following:

avg polars run: 0.0005123567976988852 avg pandas run: 0.00012923809615895151 But we can rewrite the polars query to be more efficient:

df_filtered = (
    df.lazy()
      .with_columns(abs_diff = (pl.col.column_0 - target_value).abs())
      .filter(pl.col.abs_diff == pl.col.abs_diff.min())
      .collect()
)

Then we get:

avg polars run: 0.00018435594723559915 Ultimately Polars isn't optimized for doing many tiny tiny horizontally wide datasets though.

Unfortunately, I didn't experience much of a performance boost when I tried the version above. It does seem the speeds are very machine dependent. I will continue with pandas for this specific use case. Thanks all for looking.

Reasons:
  • Blacklisted phrase (0.5): Thanks
  • Long answer (-0.5):
  • Has code block (-0.5):
  • Self-answer (0.5):
  • Low reputation (1):
Posted by: Raymond Han