Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 27 additions & 28 deletions transcripts/487-building-rust-extensions-for-python.txt
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@

00:03:57 So I started Pydantic back in 2017 as a side project.

00:04:01 And I toodled along.
00:04:01 And I toddled along.

00:04:02 And then 2021, it somehow, something happened.

Expand Down Expand Up @@ -340,7 +340,7 @@

00:09:55 These banks to have enough money to, you know, support open source.

00:09:58 We're just trying to persuade them to use Lockfire right now.
00:09:58 We're just trying to persuade them to use Logfire right now.

00:10:00 So I'll stop being rude about banks.

Expand All @@ -352,41 +352,41 @@

00:10:06 To get those folks to open their checkbooks and pay for it.

00:10:09 Right. They they don't have charity usually in their bylines or whatever they're supposed to be doing, right?
00:10:09 Right. They don't have charity usually in their bylines or whatever they're supposed to be doing, right?

00:10:15 Like they but with something like Lockfire, they can say, well, here's a service that we can use.
00:10:15 Like they but with something like Logfire, they can say, well, here's a service that we can use.

00:10:20 And by using it, we might be able to support Pydantic and the other things that are important to us.

00:10:25 So I think that's great.

00:10:27 You know, tell people just real quickly about Lockfire out there.
00:10:27 You know, tell people just real quickly about Logfire out there.

00:10:30 Yeah, absolutely.

00:10:31 So Lockfire is an observability platform built, obviously, by the team behind Pydantic.
00:10:31 So Logfire is an observability platform built, obviously, by the team behind Pydantic.

00:10:36 And if you go to Lockfire along the along the top, the I suppose the two things that make it different from some of the stuff that's come before is Lockfire is built on open telemetry.
00:10:36 And if you go to Logfire along the along the top, the I suppose the two things that make it different from some of the stuff that's come before is Logfire is built on open telemetry.

00:10:45 So on open standards that mean that, yeah, the rails of that where the data is being transferred are on an open standard.

00:10:50 And if you decide you didn't want to use the Lockfire platform anymore, you can send that data to anything else that supports open telemetry.
00:10:50 And if you decide you didn't want to use the Logfire platform anymore, you can send that data to anything else that supports open telemetry.

00:10:56 But unlike lots of other companies in our space, instead of using open telemetry as an excuse to abandon the SDK space and just say use the horrible open telemetry SDK directly, we have the Lockfire package, which tries to make that super nice and easy to use.
00:10:56 But unlike lots of other companies in our space, instead of using open telemetry as an excuse to abandon the SDK space and just say use the horrible open telemetry SDK directly, we have the Logfire package, which tries to make that super nice and easy to use.

00:11:10 And then the other big change we have, I think there'll be an example maybe further on down the page.

00:11:15 Like we maybe maybe there isn't one right there, but we allow you to write SQL directly against your data.
00:11:15 Like we maybe there isn't one right there, but we allow you to write SQL directly against your data.

00:11:20 So instead of having to use kind of ClickOps to go around and do surveys.

00:11:25 So if you look here, right, you can go and like write SQL to query your data.

00:11:29 And so there are although we're still early, there are things you can do in Lockfire that you can't you cannot and never been able to do in like one of the big incumbents like Datadog because it's just SQL.
00:11:29 And so there are although we're still early, there are things you can do in Logfire that you can't you cannot and never been able to do in like one of the big incumbents like Datadog because it's just SQL.

00:11:38 And it's obviously much easier to learn for you, much easier for LLMs to write.

00:11:43 So we have the like plausible chance in the future that you could basically chat with Lockfire and say, what's wrong with my app?
00:11:43 So we have the like plausible chance in the future that you could basically chat with Logfire and say, what's wrong with my app?

00:11:49 And the agent can go off and run SQL to investigate things in a way that is much harder if you have your own dialect.

Expand Down Expand Up @@ -532,7 +532,7 @@

00:16:52 Is it just a Rust level thing?

00:16:53 If you go up actually, it's a, it's a, this is a monorepa.
00:16:53 If you go up actually, it's a, it's a, this is a monorepo.

00:16:56 So if you go up into source, into crates, which is, so this is, you have a bunch of different crates in here.

Expand Down Expand Up @@ -608,7 +608,7 @@

00:19:32 I've been surprised by how, like, the legs that this has had, which I wasn't expecting when we first started it.

00:19:37 But yeah, the nice bit is it's all the input, the actual JSON parsing is Rust, but then we have the logic to, yeah, basically access that from Python, both in this package and in Pylandic core.
00:19:37 But yeah, the nice bit is it's all the input, the actual JSON parsing is Rust, but then we have the logic to, yeah, basically access that from Python, both in this package and in Pydantic core.

00:19:47 Well, you sent me over to this cargo section, or the crate section, rather.

Expand Down Expand Up @@ -702,7 +702,7 @@

00:23:03 I'm not clever enough to write C.

00:23:04 That's a void pointer pointer.
00:23:04 That's a void pointer.

00:23:08 What are we doing here?

Expand Down Expand Up @@ -818,7 +818,7 @@

00:27:26 We don't really think about compiling much in Python.

00:27:28 We don't think about it, but you can think of import time when you do different stuff or you think about static typing when you go and run PyWrite over your code.
00:27:28 We don't think about it, but you can think of import time when you do different stuff or you think about static typing when you go and run PyWright over your code.

00:27:37 And that is very similar in some ways to compile time.

Expand All @@ -844,7 +844,7 @@

00:28:30 I mean, putting the controversy over Surdi to one side for a minute, you would have to take care of compiling that code yourself every time you wanted to go and go and use it.

00:28:38 Whereas ActionPyPI does an amazing job of distributing compiled Rust for virtually every ecosystem, every architecture, and it just working when you install it.
00:28:38 Whereas Action PyPI does an amazing job of distributing compiled Rust for virtually every ecosystem, every architecture, and it just working when you install it.

00:28:46 Absolutely.

Expand Down Expand Up @@ -1280,7 +1280,7 @@

00:43:32 I don't know how long ago that was you, you were trying, but I think it's, yeah, reduced significantly.

00:43:37 So, I mean, I did this quite a lot of the same work in PyNantic Core to avoid the overhead of calling into and out of it lots of times.
00:43:37 So, I mean, I did this quite a lot of the same work in Pydantic Core to avoid the overhead of calling into and out of it lots of times.

00:43:43 And I think that's less of a thing now than it used to be.

Expand Down Expand Up @@ -1310,11 +1310,11 @@

00:44:46 I don't know if that's in here as well.

00:44:48 I maintain watch files, which is the file watching library used by UV-icorn and some other things.
00:44:48 I maintain watch files, which is the file watching library used by UV-con and some other things.

00:44:55 And that is, again, wrapping a Rust library for getting fast system notifications.

00:45:00 And also R-toml, which is the fastest tomml parser in Python, which is, again, just wrapping the Rust library.
00:45:00 And also R-toml, which is the fastest toml parser in Python, which is, again, just wrapping the Rust library.

00:45:06 So there are a number of places where you can get enormous performance improvements and indeed like fundamentally more reliable libraries because you're building in Rust.

Expand Down Expand Up @@ -1414,7 +1414,7 @@

00:47:55 But RustUp is kind of like one level up.

00:47:58 And it's maybe a bit like PyM if people have used that.
00:47:58 And it's maybe a bit like PyPI if people have used that.

00:48:02 But it sort of does a bit more than that.

Expand All @@ -1426,7 +1426,7 @@

00:48:30 What you'll do, your first step is to go to the Rust website and install RustUp.

00:48:35 And then hopefully at the command line, you'll be able to type RustC and then you can compile a Rust file if you want to.
00:48:35 And then hopefully at the command line, you'll be able to type Rust C and then you can compile a Rust file if you want to.

00:48:40 But you don't actually need to do that because that's what you use Cargo for.

Expand Down Expand Up @@ -1508,7 +1508,7 @@

00:51:57 Maturin effectively gets rid of that whole challenge for you and just works and lets you work on it like you're, it's as easy as writing Python or Rust interacting with the two.

00:52:07 Yeah, because cargo and Rust C will output Rust conventions, but Python wants C extension conventions, naming like .so versus Dynelib and things like that, right?
00:52:07 Yeah, because cargo and Rust C will output Rust conventions, but Python wants C extension conventions, naming like .so versus Dynamo lib and things like that, right?

00:52:18 And getting the inputs right and yeah.

Expand Down Expand Up @@ -1536,7 +1536,7 @@

00:53:26 The problem you then have is you end up with two separate definitions of what your functions have in them.

00:53:31 And so there's a really neat tool, which I didn't know about until quite recently, but we use it in Pydantic Core, which mypy can basically type check that the PYI file stubs match the definitions inside the DLL.
00:53:31 And so there's a really neat tool, which I didn't know about until quite recently, but we use it in Pydantic Core, which MyPy can basically type check that the PYI file stubs match the definitions inside the DLL.

00:53:44 And so you can have like guarantees that where you've written in a PYI file, my function foobar takes A as a string and C as a list of bytes.

Expand All @@ -1548,7 +1548,7 @@

00:53:56 Fixes all together.

00:53:57 Is it a mypy plugin?
00:53:57 Is it a MyPy plugin?

00:53:58 If you have a look at Pydantic Core, we run it in CI or in maybe even pre-commit so you can find it there.

Expand Down Expand Up @@ -1722,7 +1722,7 @@

00:58:24 But I think if you're starting from scratch and you're trying to write something high performance, the experience of doing it in Rust is completely different.

00:58:30 from trying to do it in C or C++ or C Sharp or any of the other or Fortran or Julia or any of those other languages.
00:58:30 from trying to do it in C or C++ or C# or any of the other or Fortran or Julia or any of those other languages.

00:58:35 Rust is awesome.

Expand Down Expand Up @@ -1844,7 +1844,7 @@

01:01:34 We should be right at the top.

01:01:36 You can also find the iTunes feed at /itunes, the Google Play feed at /play, and the direct RSS feed at /rss on talkpython.fm.
01:01:36 You can also find the iTunes feed at /iTunes, the Google Play feed at /play, and the direct RSS feed at /rss on talkpython.fm.

01:01:45 We're live streaming most of our recordings these days.

Expand All @@ -1859,4 +1859,3 @@
01:02:00 Now get out there and write some Python code.

01:02:02 I'll see you next time.