Understanding Ethereum Network Upgrades: Lecture 5

The following transcript was auto-generated, and may content syntactical errors.

00:00:09.000 --> 00:00:17.000

Today is January seventeenth, Wednesday, January seventeenth and, 2024 and we're in the week of testing and implementation.

00:00:17.000 --> 00:00:28.000

So we're going to be talking about both generically, what does testing and implementation look like for, an upgrade and then we will get into some.

00:00:28.000 --> 00:00:42.000

Specifics of what testing and implementation. Looks like for the Denkun upgrade. So. Take ticket off today, you know.

00:00:42.000 --> 00:00:46.000

How do we test what is changing? And I think the short answer. Is we do it, we start incrementally in piecewise.

00:00:46.000 --> 00:01:01.000

So there's always some testing going on. That's incremental and piecewise. It's testing out EIPs that's testing out.

00:01:01.000 --> 00:01:11.000

Potential attack vectors on the network. These are kind of always happening. And they're happening.

00:01:11.000 --> 00:01:20.000

During network upgrade like in the run up to network upgrades, but also in the planning phase it's constantly kind of happening.

00:01:20.000 --> 00:01:31.000

But then as we get closer to actually making a network upgrade. Those. We test everything altogether at the same time.

00:01:31.000 --> 00:01:40.000

So, we start out with this incremental piecewise process and then we move to like doing everything and seeing how it works together.

00:01:40.000 --> 00:01:48.000

So We're gonna touch on. Really. These 4.

00:01:48.000 --> 00:01:55.000

Elements of what gets tested and then we're going to talk about like testing before the upgrade and then testing after an upgrade occurs.

00:01:55.000 --> 00:02:03.000

And there actually was. A test network that happened today. So we're also going to talk about what some high level results of that.

00:02:03.000 --> 00:02:17.000

So just to sort of cover the like our. We're going to talk about the implementation of EIPs by client teams or within in an Ethereum client.

00:02:17.000 --> 00:02:21.000

EIPs by client teams or within an Ethereum client, whether it's execution layer client or the CL client.

00:02:21.000 --> 00:02:37.000

Test cases, who writes them, where do they exist? What are the testing tools that exist we're gonna talk about shadow forks and dev nets which are some of the coolest like Names of stuff that we gave in Ethereum.

00:02:37.000 --> 00:02:44.000

We've done we've done ourselves sometimes a disservice with the naming but not with shadow forks shadow for the naming but not with shadow forks.

00:02:44.000 --> 00:02:55.000

Shadow Forks is that's a sweet name. And then we're gonna talk about test net forks in general and what those do and the overall like upgrade itself, which is the implementation.

00:02:55.000 --> 00:03:11.000

So we'll jump right into it. So the first thing we need to do if we look back at our week that we talked about the like writing EIPs, when we write an EIP, we have to actually implement it in an Ethereum client.

00:03:11.000 --> 00:03:21.000

So whether that's the EIP for an execution layer client or a consensus layer client. We actually have to implement that EIP in order to test it, right?

00:03:21.000 --> 00:03:40.000

You have to have some sort of implementation. So client teams do this. If you join all core Devs calls, There was always a part of the call where we go, they go around to the client teams, particularly on the execution layer and say, have you implemented these changes?

00:03:40.000 --> 00:03:49.000

There's often times folks from the EF who are asking questions around, hey, when is this team going to finish their implementation?

00:03:49.000 --> 00:03:57.000

And there's a lot of back and forth between the various teams on getting that implementation done so that tests can be written.

00:03:57.000 --> 00:04:11.000

But. Because all of these clients are in open source software repositories, I think actually all of them are pretty much hosted on GitHub, maybe if one or 2 on Gitlab.

00:04:11.000 --> 00:04:15.000

You could just fork that repo if you are an author of EIP and if you have the knowledge of that code base, you could implement it yourself.

00:04:15.000 --> 00:04:30.000

So this is a reason why we talked about reference implementations from EIP authors being so helpful. They actually make it faster for client teams for to be able to figure out how they're going to implement that, that EIP in the client.

00:04:30.000 --> 00:04:47.000

So, like I said, anyone can do this. You can just create a fork of a software repository and add those add those changes into it and oftentimes that's what you see is done.

00:04:47.000 --> 00:04:48.000

It's a combination of the client teams, but if someone is properly motivated, they can go ahead and do it.

00:04:48.000 --> 00:05:04.000

With their client of choice. Once that EIP is implemented, it's actually then possible for any user to download that and actually just run.

00:05:04.000 --> 00:05:08.000

From either a Genesis block or from a point, a certain point in the state, a local, a local version of the chain.

00:05:08.000 --> 00:05:23.000

And that could be a fork, but it could also just be there running the chain and seeing how it how it works to see the impact that the changes have, right?

00:05:23.000 --> 00:05:33.000

And when I say see the impact, I'm talking both about like literally like you know, potentially doing some actions on that chain.

00:05:33.000 --> 00:05:36.000

That's sort of just generally probing to see what might be different or it could be actually running like a full test suite.

00:05:36.000 --> 00:05:53.000

That is, is specifically written. So, not making a distinction there. The reason I bring that up is because Once you have something that you're able to run locally, a version of the chain.

00:05:53.000 --> 00:06:05.000

That can be useful for iterating on writing test cases because test cases are written typically towards what are the expected behaviors and what are the what are the behaviors that are out of bounds.

00:06:05.000 --> 00:06:06.000

So having something that you can refer to and actually go ahead and start writing those tests is critically important.

00:06:06.000 --> 00:06:17.000

Anything you want to add here, Matt.

00:06:17.000 --> 00:06:30.000

One thing that is kind of noted is that the client implementations often yield spec changes because we go to build something in we're like the spec actually doesn't work in practice or we realize that the spec is.

00:06:30.000 --> 00:06:41.000

Maybe over overly permissive or overly restrictive in certain cases, it may trigger portions of a client code that is not aware of.

00:06:41.000 --> 00:06:54.000

So for example, if I have a any IP that impacts the way that maybe the state tree works that could have cascading impacts up and down the clients that we are kind of unaware of that trigger new considered security considerations.

00:06:54.000 --> 00:07:07.000

So in reality, the first set of implementations are always fuzzy. Because once we The client implementations exist in isolation until we link them to other clients in these Dev Nets and test nets.

00:07:07.000 --> 00:07:16.000

So we yield a lot of results because We have a spec about a chunk of functionality and then we have all the other company pieces of the client.

00:07:16.000 --> 00:07:30.000

And then we have how your client peers with other clients. So We, you know, we do a lot of iteration based on this because these, initial implementations may pass all of the reference tests and all of these other tests.

00:07:30.000 --> 00:07:44.000

Because the specification is sound but it has upstream impacts in the client code that we don't we're not aware of so these first rounds of iterations are very much so in flux and the spec often changes beyond what was in scope for the EIP.

00:07:44.000 --> 00:07:53.000

But we need the kind of interoperability testing of clients to be able to understand the test cases better and to understand the client code.

00:07:53.000 --> 00:08:03.000

In terms of other clients, the Ethereum protocol itself and things we might not be aware of. So there's a lot of back and forth at this stage, I would say.

00:08:03.000 --> 00:08:09.000

And so while we said that, you know, client implementation, we these slides are sequential as Matt is referencing.

00:08:09.000 --> 00:08:11.000

This is more of a back and forth process. You can almost think of this happening as like concurrent to each other, right?

00:08:11.000 --> 00:08:19.000

So, what's great is that the EF and all core devs do keep a software repository.

00:08:19.000 --> 00:08:33.000

HOST on GitHub, under the Ethereum repo of Ethereum test cases. So I'll just click into that if you want to actually see what that looks like.

00:08:33.000 --> 00:08:41.000

And here we go. It's their name tests. You can see that there is a lot of tests in here, some are not relevant anymore, proof of work tests, right?

00:08:41.000 --> 00:08:49.000

And there is documentation to go along with that. The slides all link that I'll put them in, and afterwards, but we'll have all these these readings you can click out onto.

00:08:49.000 --> 00:09:00.000

There's actually, so like I said, there's general tests and then there's those that are specific to upgrade, right?

00:09:00.000 --> 00:09:13.000

So if you think about it, there's a set of tests that. The, that should be passable by any client pair or just or an execution layer client.

00:09:13.000 --> 00:09:19.000

Or a consensus like your client because that's like just you know, adhering to the overall spec, right?

00:09:19.000 --> 00:09:26.000

And then there are specific tests that, might be things that need to be tested in the upgrade.

00:09:26.000 --> 00:09:33.000

There is a full tutorial. So if you're interested in participating in this, there is the Ethereum test.

00:09:33.000 --> 00:09:51.000

Doc repo that's linked in this software repository. It's like a sub repo and it actually walks you through Testing against your client, it uses GET as the sort of reference implementation client, but you can.

00:09:51.000 --> 00:10:00.000

Try this out with anything else and I think this is a really great place to start if you're interested in being part of testing.

00:10:00.000 --> 00:10:15.000

So These tests. Assume in this case that you're using a tool called retest. And it's a tool that allows for the running of the test and the generation of the stats on tests.

00:10:15.000 --> 00:10:29.000

Now I think Matt. And I in our day jobs. You know, other test suites, right?

00:10:29.000 --> 00:10:43.000

Because we, work. So, Matt is definitely having to work with this test suite, but then there's like other test suites that the base you team and the take-out team take advantage of.

00:10:43.000 --> 00:10:56.000

Me being from like the inferior side and working on decentralized infrastructure network, we also use different test suites in order to test performance of certain, let's, if we just leave it to Ethereum, there's different test suites that we have.

00:10:56.000 --> 00:11:07.000

So what I pointed out because Just because we're only highlighting retest, as sort of this.

00:11:07.000 --> 00:11:16.000

Anchoring tool and it very much is an anchoring tool. It doesn't mean that different, stakeholders in the space.

00:11:16.000 --> 00:11:29.000

Are just limited to using this tool. Oftentimes, there are other testing tools used. The nice thing about retest, is that there's new releases that are pushed out for each fork.

00:11:29.000 --> 00:11:46.000

So We have a release here that allows us to be able to run a suite of tests and it gives us the stats that we need and brings in sort of all the tests that we need for this.

00:11:46.000 --> 00:12:01.000

What you need to know is that there are teams of folks who are like writing these tests, thinking about them, and they are working if you go into the ER and Discord, there's a channel.

00:12:01.000 --> 00:12:09.000

We highlight it later in this deck. It's called hashtag testing. If I Remember correctly, while it's currently, tagged.

00:12:09.000 --> 00:12:16.000

Underneath it's underneath like the heading Den Coon. It's a channel that moves around with different upgrades.

00:12:16.000 --> 00:12:18.000

Like if you go back in the history, it's actually the whole sense that ER. Discord started of testing.

00:12:18.000 --> 00:12:26.000

So it has that the testing history in in conversation that happened for Shanghai and Capella and going back so on and so forth.

00:12:26.000 --> 00:12:41.000

That channel just like gets moved with each upgrade. So that is kind of where things happen. Retest E, there is a web interface.

00:12:41.000 --> 00:12:51.000

It is a, a stark web interface. Let's put it that way. But sometimes, you know, sometimes you like brutalist architecture.

00:12:51.000 --> 00:13:01.000

Sometimes you really like brutalist web design. And so there is a web interface that you can see that can help run the test.

00:13:01.000 --> 00:13:07.000

So very useful that we have this. When you think about it, there's a lot of work that goes into building this shared infrastructure.

00:13:07.000 --> 00:13:20.000

And now I think that helps contextualize the importance of our C, the Ethereum Foundation. If we think back on that graph of overlapping, it's not just the ideation and creation and vetting of ideas.

00:13:20.000 --> 00:13:28.000

It's also building the DevOps infrastructure. So there is an Ethereum Foundation DevOps team that really takes ownership of a lot of this work.

00:13:28.000 --> 00:13:33.000

Anything else you want to add here, Matt?

00:13:33.000 --> 00:13:34.000

Yeah, where we just recently started to migrate a lot of these tests to something called hive testing suite.

00:13:34.000 --> 00:13:44.000

The reason being is that Hive takes a new account, multi-client use cases for testing and it also takes no account the CL to EL separation.

00:13:44.000 --> 00:13:55.000

So I, when I will go back and add hive to this. Slide when we go back to share the materials.

00:13:55.000 --> 00:14:16.000

But hive is a super important one for us to test. A lot of, a lot of the reason being is that it exercises the EVM and state tests but also exercises like the reality of proof of stakes so it it exercises the fork choice rule it exercises the engine API at exercises sync testing.

00:14:16.000 --> 00:14:24.000

It's really a moralistic picture that is slowly becoming this de facto standard. We're also going to, I believe this might be.

00:14:24.000 --> 00:14:42.000

Big news, but I believe we're incorporating into that testing harness. So there'll be kind of a one complete suite going forward because now that we have the merge and proof of stake retest deep is really a way to test execution layer implementations of the EVM and some of the state stuff.

00:14:42.000 --> 00:14:54.000

So we're kind of creating a new picture, but we're rolling all these things together. In the Ethereum Foundation leads these efforts continuously, but Client teams create new test cases all the time.

00:14:54.000 --> 00:15:07.000

So like, you know, For example, the bug that impacted based on main net on January sixth, we're potentially, we're adding back that test, that corner case of what happened in the state dB implementation to the high testing suite so that we can.

00:15:07.000 --> 00:15:17.000

Not have that happen to again or anyone else. So it's kind of like Tom said a collaborative process.

00:15:17.000 --> 00:15:24.000

Alright, so now let's talk Shadow Forks in Dev Nets. So first. These are my definitions of what a shadow fork in Devnet is.

00:15:24.000 --> 00:15:29.000

I'm sure if you Google this, there could be different ones, but I've just from seeing them.

00:15:29.000 --> 00:15:49.000

A shadow fork is basically when you take an existing network that could be the theory of main net that could be a test net like girly, it could be, Sapolia, it could be, and you are literally taking from a point of a set.

00:15:49.000 --> 00:15:57.000

State, you know, so think about it like a certain block. From that point, you're forking the test and applying some changes.

00:15:57.000 --> 00:16:10.000

So we use Shadow Forks because it allows you to implement the upgrade or part of the upgrade and observe the conditions starting from a known specific existing state.

00:16:10.000 --> 00:16:29.000

So there is a, if, you check out in, there's the Ethereum Hack MD, which is essentially a, I would call it a scientific document writing, platform.

00:16:29.000 --> 00:16:35.000

We have these, you could see here's girly, shadow fork 0 analysis.

00:16:35.000 --> 00:16:43.000

And it talks about the test setup, provides some information about. Health of the network. These look like screenshots of.

00:16:43.000 --> 00:16:54.000

Almost certainly Graafanna. Which is an observability tool and it is, you know, sort of running through what happened there.

00:16:54.000 --> 00:17:02.000

So we have a spec sheet for every shadow fork and it lets basically anyone who's participating know and define the parameters.

00:17:02.000 --> 00:17:16.000

It's important for setup. It's also important for analysis, right? So here we have the spec sheet, talks about everything that will will happen, talks about when it will start up, when it will be shut down.

00:17:16.000 --> 00:17:25.000

What is included in it when has that been merged, etc. So that is really important.

00:17:25.000 --> 00:17:39.000

So shadow for think about it. We are taking a Set. A set state and from that state we're applying some change so we're literally forking an existing chain.

00:17:39.000 --> 00:17:51.000

Now, Dev Nets are specific networks that are spun up. To test the implementation of changes. And They are not made from an existing network state.

00:17:51.000 --> 00:18:00.000

That is kind of how I define it. You might hear slightly different definitions, but the idea is that, hey, a Devnet is literally a developer network, right?

00:18:00.000 --> 00:18:20.000

Like Devnet is typically used with other sort of if you're writing solidity, your DAP developer, you might first deploy your, your, depth to a, Devnet that you create from like, a.

00:18:20.000 --> 00:18:24.000

You know, if you're using any of the sort of development frameworks, most of them have something that you can spin up and run a Devnet.

00:18:24.000 --> 00:18:39.000

So you think about this is isn't sort of necessarily taking the set network state. But I think the key and and Rob like you know, showed up on camera.

00:18:39.000 --> 00:18:56.000

So this is a great time to say like a remix being one of those frameworks. Has options to essentially like if you think about it from that depth developer perspective, you can either like start clean like a true Devnet or you could say, hey, I want to start from a certain fork.

00:18:56.000 --> 00:19:07.000

Right? So you could deploy and saying like, Hey, I want to deploy under what I would call the, I think, you know, if you, if you use remix, I was using it the other day, it's gonna be, Shanghai is like your default when, when you load it up in your web browser.

00:19:07.000 --> 00:19:20.000

So, Dev Nets. Once again, same thing, spat like. We're gonna have our.

00:19:20.000 --> 00:19:32.000

Here's the test. Here's the information. Here's you know, what our analysis of it and then we're gonna have a spec sheet, right? Same sort of deal.

00:19:32.000 --> 00:19:46.000

We're gonna have a spec sheet that's gonna help guide us. And in this case, you know, these are, these are really important for actually like running, running these, these tests.

00:19:46.000 --> 00:19:49.000

The results are. Sort of I would say formally reported out from both the shadow forks and the Dev Nets.

00:19:49.000 --> 00:20:11.000

In the, E. Developer operations. Hack MD. So here you could see a bunch of like different published notes and and information about them so you could like go in and just click on it.

00:20:11.000 --> 00:20:18.000

We have the links here gives you an overview of. Of all of these different specs and what happened.

00:20:18.000 --> 00:20:26.000

But what you'll also see is that a lot of times Client teams, other stakeholders.

00:20:26.000 --> 00:20:38.000

Will report out results. I think in the merge, we saw probably the most reporting out of shadow fork results literally by media outlets and other organizations.

00:20:38.000 --> 00:21:00.000

You'll see reporting out via Twitter or X or you know, whatever social media. The key though is that you have when you do Shadow Forks and Dev Nets that it doesn't mean that we're necessarily testing all the changes and it might not be inclusive of all the changes that go into the eventual upgrade.

00:21:00.000 --> 00:21:07.000

It may include changes that are going to go into a next upgrade. It just depends on how that specified, right?

00:21:07.000 --> 00:21:13.000

So. One big thing to take away from Shadow Forks and Dev Nets, they're useful because it's getting closer and closer to the actual conditions that we'd expect.

00:21:13.000 --> 00:21:35.000

But we're able to more piecemeal. Test changes under conditions that are looking more and more like what the actual upgrade will look like, but we're not all the way there.

00:21:35.000 --> 00:21:42.000

So we could be selective. You could, you saw in one of these, there was a Devnet that was really focused on like.

00:21:42.000 --> 00:21:57.000

Blobs right so you might be testing very specific functionality because you need to do that Like I said, this came after talking about the client implementation and talking about writing the test cases, but Shadow Forks and Dev Nets.

00:21:57.000 --> 00:22:06.000

Think about this like a circle where these are kind of always happening and we're working to then. Figure out what's going to be in there.

00:22:06.000 --> 00:22:15.000

So Last week, Matt talked about how ELF was not included. In this upgrade. But EOF was tested in some of the Shadow Forks and Def Nets to be perfectly clear.

00:22:15.000 --> 00:22:29.000

Like, that did happen during this, during the run up to this upgrade. So understanding what goes in and out is really important.

00:22:29.000 --> 00:22:39.000

That's where the Shadow Forks and Dev Nets really help us determine whether the the scope of the upgrade is such that it can include certain changes or not.

00:22:39.000 --> 00:22:48.000

Right? So do not be surprised if you go back to the history and you start to see it's like, Hey, there were implementations even.

00:22:48.000 --> 00:23:06.000

Correct me if I'm wrong, Matt, I think 4 8 4 4 was being tested in elements of 4 8 4 4.

00:23:06.000 --> 00:23:10.000

Yeah, the.

00:23:10.000 --> 00:23:13.000

You'll have to repeat the last part of your question again. Excuse me.

00:23:13.000 --> 00:23:24.000

Oh, basically we even though that 4 8 EIP 4 8 4 4. You know, proto-dank sharding was gonna, is going into this upgrade.

00:23:24.000 --> 00:23:32.000

Testing for that was actually occurring. Back before the Shanghai and Capella upgrade, the Shapella upgrade.

00:23:32.000 --> 00:23:40.000

Yeah, absolutely. So we kind of started developing that at the same time, honestly. So it was.

00:23:40.000 --> 00:23:57.000

Little bit separated logically because they we had to get withdrawals. Ships first. But yeah, we were, we've We kind of are iteratively testing and developing all of this stuff in tandem and they kind of loop over each other and things like that.

00:23:57.000 --> 00:24:02.000

So yeah, I think that's the answer to the question is yes.

00:24:02.000 --> 00:24:07.000

Cool. And here's a great question. Why do we need both Shadow Forks and Dev Nets?

00:24:07.000 --> 00:24:16.000

The simplest explanation is that it allows you to change the scope of your tests.

00:24:16.000 --> 00:24:34.000

And what I mean by that is a shadow fork says, let's take existing. So Devnet is probably less close to What, if you think about like main upgrading main net and upgrading main net as the goal.

00:24:34.000 --> 00:24:46.000

Certain points in time we need to simulate. What a main net upgrade will look like. A shadow for gives us the ability to do that.

00:24:46.000 --> 00:24:59.000

To give a better approximation of what upgrading main net or test net will look like. Whereas a Devnet allows us to start with maybe more idealized conditions, right?

00:24:59.000 --> 00:25:06.000

We can sort of change the parameters of where we're starting. Whereas with a shadow for We are saying we need to take in it.

00:25:06.000 --> 00:25:11.000

We're gonna take an existing state and see how that applies. So basically gives us optionality of what we're testing and how we're testing it, right?

00:25:11.000 --> 00:25:31.000

Think about the difference between the Shadow Fork and the Devnet. State is the variable and with the Devnet, you're saying that we are going to We were able to manipulate state much more easily because we can sort of set state.

00:25:31.000 --> 00:25:46.000

Wherever it should be and by state I literally mean the state try or the state tree right whereas with a shadow fork we're saying Let's hit us like a state so we can actually look at whether it's main net or that test net.

00:25:46.000 --> 00:25:51.000

And then let's manipulate it. And now we have a comparison point of what happens. After that upgrade is applied.

00:25:51.000 --> 00:26:03.000

So it's that optionality. That's why we need to do both, right? It gives us the ability to test different things and in different ways.

00:26:03.000 --> 00:26:07.000

And do different comparisons. Great question.

00:26:07.000 --> 00:26:13.000

Yeah, it's also an approximation of real state. We generate a lot of interestingly fake state.

00:26:13.000 --> 00:26:20.000

So Shadow Forks basically say, oh, this is what would happen if nodes forked right immediately down.

00:26:20.000 --> 00:26:32.000

And are the notes prepared because Not only do we shadow for test random stuff, but those nodes kind of they operate on the same memo sometimes in terms of like if you're shadow working girl you'll still pick up transactions.

00:26:32.000 --> 00:26:39.000

So it's a real approximation of like What would happen if we literally forked a subset of these nodes right now?

00:26:39.000 --> 00:26:45.000

How would they interact with an existing network? Would they talk to each other? Would they live? Would they agree upon the right state?

00:26:45.000 --> 00:26:59.000

So there's 2 approaches. Devonets are more build fast break fast kind of thing. And Shadow Forks are expensive because we have to spend real hardware things and on main net shadow forks we have to spend real E.

00:26:59.000 --> 00:27:13.000

So it's very like those are more legitimate exercises where we feel we're closer to the end goal where we want as Tom said we're testing on a different set of items but we're also testing on a real picture of state and real transactions.

00:27:13.000 --> 00:27:18.000

So that's definitely valuable.

00:27:18.000 --> 00:27:30.000

So. Just hammering 2 points that. We set up, said already what didn't make an an upgrade is often tested in the run up to the upgrade, right?

00:27:30.000 --> 00:27:39.000

So, That is, I think, sometimes a misunderstanding that it's like, hey, the like.

00:27:39.000 --> 00:27:51.000

The upgrade is scoped so well that we only test what goes into it. The reality is that we're constantly scoping the upgrade and Well, how we're scoping the upgrade.

00:27:51.000 --> 00:28:06.000

Allows us, you know, how we're doing doing this testing, we are literally scoping the upgrade through an iterative process and That means that you'll often see certain EIPs.

00:28:06.000 --> 00:28:10.000

Be, tested and maybe not included in upgrade and that might continue on, potentially through multiple upgrades.

00:28:10.000 --> 00:28:21.000

Also testing is A process. We are in the tens of Dev Nets in Shadow Forks.

00:28:21.000 --> 00:28:31.000

Respectively for this current. This current upgrade, I think Devnet 12. Was the last one?

00:28:31.000 --> 00:28:37.000

So, I mean, this gives you a sense of like. This is happening. Iteratively.

00:28:37.000 --> 00:28:47.000

And so we are, really going. Through this process This is a great question. Matt, maybe you know the answers.

00:28:47.000 --> 00:28:54.000

What is the budget of real ether to be spent on an upgrade for testing transactions. Maybe you could just give like a rough estimate.

00:28:54.000 --> 00:28:59.000

I'm sure it's different. Per upgrade, but just trying to get a sense of.

00:28:59.000 --> 00:29:03.000

Of you know the real the real stakes that are involved.

00:29:03.000 --> 00:29:09.000

Yeah, so the EF does not release those numbers because they use real validating nodes to test this stuff.

00:29:09.000 --> 00:29:14.000

And I also don't blame them because it's probably in the tens of thousands of dollars like sustained amount of money.

00:29:14.000 --> 00:29:25.000

If you're thinking about the amount of EF and the validating stakes that they use, It's because again, we have to pay gas.

00:29:25.000 --> 00:29:39.000

Or to keep those nodes. Kind of forks like in their own little worlds. Otherwise, they'll because we'll run out of transactions to sustain that fork and they'll say, oh, there's nothing here, something weird is going on and they'll try to connect to new peers.

00:29:39.000 --> 00:29:52.000

So we basically have to sustain normal main net gas prices from block to block. Let's see what the gas price on Ethereum main net is right now.

00:29:52.000 --> 00:30:05.000

Yeah, it's like 42. It's like $2 per transaction and blocks can fit tons of transactions and we need to sustain the shadow forks for.

00:30:05.000 --> 00:30:12.000

Hundreds and hundreds of blocks. So you can see that it will quickly add up in terms of straight up just gas.

00:30:12.000 --> 00:30:24.000

And since the state that they're using is real and the fork is fake. What we do on the shadow for doesn't impact Ethereum main net state because we are on a fork of the main chain.

00:30:24.000 --> 00:30:35.000

But it does cost like you have to cost money It's a weird thing to say, right, that we're not actually impacting Ethereum, state, but we need to spend money on gas.

00:30:35.000 --> 00:30:40.000

So in theory, you think that that would make any sense, right? Because if I just go off on a fork and spend all the gas in the world.

00:30:40.000 --> 00:30:48.000

It never really makes it back on the main chain. But we have to pay for those. We have to pay to like fork those machines basically and then sustain them with.

00:30:48.000 --> 00:31:01.000

Stuff and they also run hundreds of nodes like client pairs for the shadow forks which cost money in just like cloud infrastructure costs.

00:31:01.000 --> 00:31:02.000

Yeah.

00:31:02.000 --> 00:31:06.000

At the end of the day, it's electricity all the way down and it's and you know, you gotta pay for the electrons one way or the other.

00:31:06.000 --> 00:31:15.000

And that's a good thing. So let me just like to use an analogy, right? It is it's worth spending.

00:31:15.000 --> 00:31:22.000

Let's just think about like. The total value locked on Ethereum, not even considerate.

00:31:22.000 --> 00:31:28.000

Well, and if we consider all the L two's and app chains like L three's.

00:31:28.000 --> 00:31:41.000

It's worth, you know, spending. Tens to hundreds of thousands of dollars to test this because the value that's at stake literally at stake.

00:31:41.000 --> 00:31:57.000

If this is wrong would be, you know, it, You're talking about like 0 point 0 0 0 0 0 like 1% of the value spent testing in order to continue to secure an incredible amount of value.

00:31:57.000 --> 00:32:08.000

So I think that is really key, but what I'll also mention is that like It's not, you know, to test well and to do this and just to do these setups and.

00:32:08.000 --> 00:32:18.000

Just to go back to, Let's look at the specs. And the like a shadow for like, you know.

00:32:18.000 --> 00:32:23.000

Running. This is this is

00:32:23.000 --> 00:32:29.000

A pretty sweet setup, right? Like if we're if we're running. You know, this kind of, like hardware software combination, it has real cost.

00:32:29.000 --> 00:32:56.000

So, When I mentioned this because I would say have it's always good for us to have some grace and perspective when it comes to, you know, I think I think it's after the merge it's maybe become a little bit more like, oh well, you know, the the client teams and different teams doing this.

00:32:56.000 --> 00:32:59.000

They're just, oh, they're so good at this. They just do it, but like.

00:32:59.000 --> 00:33:07.000

There are real costs and If you're running all this infrastructure. There's all the challenges that go along with running that infrastructure just in the day to day as well, right?

00:33:07.000 --> 00:33:20.000

Keeping machines online. Dealing with outages, dealing with, that the cost of that.

00:33:20.000 --> 00:33:21.000

So I point that out because it ends up being, it's not a trivial process, but it's worth it.

00:33:21.000 --> 00:33:34.000

Right? It's very much worth it. So, let's now talk about.

00:33:34.000 --> 00:33:48.000

I put this as, you know, dress rehearsals and the main event. So we've now crossed over and we want to start testing what I would say would be let's test everything at once, right?

00:33:48.000 --> 00:33:56.000

Let's test all the dependencies. All the interdependencies at the same time. And that's test net upgrades, right?

00:33:56.000 --> 00:34:05.000

These are the closest thing to to an upgrade that like can occur because test nets often persist, right?

00:34:05.000 --> 00:34:08.000

If you run at test and upgrade, the test net will continue to exist after the upgrade and be used.

00:34:08.000 --> 00:34:22.000

It has real users. So there are real stakes to this. Like there are real users who are, you know, have a deployment of a DAP.

00:34:22.000 --> 00:34:39.000

Or of a staking setup on a test net that gives them a signal about how their main net setup or depth is gonna operate it is possible to revert these like this isn't as a dire if something goes wrong.

00:34:39.000 --> 00:34:46.000

And The safest way to do this is start with a test that's going to be deprecated.

00:34:46.000 --> 00:34:54.000

There isn't always a test that's going to be deprecated. For the merge it was.

00:34:54.000 --> 00:35:06.000

Rinkby. Thank you for your service, RinkB. There were some other chess sets that were in deprecation process already, Covon.

00:35:06.000 --> 00:35:25.000

In the case of, it's going to be girly, girly hat is technically, I believe, already officially deprecated or at least it's beyond Q 3 of 2023 which was like the planned deprecation or maybe I'm wrong about that.

00:35:25.000 --> 00:35:26.000

Already there. It's done.

00:35:26.000 --> 00:35:32.000

Girlie is going to be deprecated, is already deprecated. So with Den Coon.

00:35:32.000 --> 00:35:35.000

You're gonna start there because It's like the least is at state with that, but there is still something at stake.

00:35:35.000 --> 00:35:44.000

And so similar reporting analysis is going to occur as with the De Nets and Shadow Forks and there's often coordination calls and watch parties as that happens to talk through that happening.

00:35:44.000 --> 00:35:55.000

I'm not saying that there is a coordination of calls with Dev Nets or Shadow Forks, but like.

00:35:55.000 --> 00:36:00.000

You're sort of seeing like these are the dress rehearsals, like people start showing up for the merge like.

00:36:00.000 --> 00:36:13.000

I was blown away by how many people were showing up for the calls that were scheduled for not the merge itself but like for the forks like the Test Net Forks and seeing that play out.

00:36:13.000 --> 00:36:21.000

So Earlier today, we actually had the girly fork. On the thirtieth, we'll have this Apollo one and on the seventh, we'll have the whole ski one.

00:36:21.000 --> 00:36:28.000

So. Map, what happened? Let's talk about the girly fork that happened earlier, today.

00:36:28.000 --> 00:36:38.000

Yeah, so this is a non exhaustive. List of post mortem stuff. We uncovered 3 bugs.

00:36:38.000 --> 00:36:47.000

The biggest issue is that Prism is basically just kind of broken on girly right now. That means that and needs a patch.

00:36:47.000 --> 00:36:53.000

Some of the gossip blobs that they are receiving from their peers have the wrong index.

00:36:53.000 --> 00:37:02.000

And that makes the prism note invalidate them improperly, whereas sometimes the index is not needed because of late.

00:37:02.000 --> 00:37:14.000

Slot late blob stuff. So you don't necessarily know which index of the 0 to 6, well, you know, 0 to 5 because we have 6 max blobs and Things are 0 indexed in computers.

00:37:14.000 --> 00:37:20.000

So we have those 5 in the season. If you go outside of that, normally you're supposed to be able to handle a non sequential thing.

00:37:20.000 --> 00:37:27.000

They believe that the bug is that Prison is just handling this case and properly but it's kind of breaking prison on Gurley.

00:37:27.000 --> 00:37:38.000

Nether mind is not also keeping the max limited blob transactions. In its

00:37:38.000 --> 00:37:56.000

I believe parsing of these transactions in the pool for per block. So for the slot when prison or another mind is examining the number of maximum blobs they don't keep the correct number between transactions transaction, which could lead to an invalidation of the of the block from Netherlands perspective.

00:37:56.000 --> 00:38:10.000

I have a feeling this is not an exhaustive list. The fork happened last night we're still investigating, but it's pretty interesting to see that we have, you know, a few bugs, a client called Grand on that's a consensus layer client is proposing in valid blocks as well.

00:38:10.000 --> 00:38:18.000

This is the first transition that Grand D has been through, so that's probably to be expected. But yeah, we're ongoing investigation.

00:38:18.000 --> 00:38:27.000

The net is we'll expect kind of a full post Mormon client releases tomorrow on the core Devs execution call.

00:38:27.000 --> 00:38:35.000

And we will Hopefully have more information after that.

00:38:35.000 --> 00:38:38.000

Awesome. So follow along. Join the all core devs call if you want to hear more because that's going to be pretty interesting.

00:38:38.000 --> 00:38:51.000

This is so, what's gonna happen after this one is that new releases have to get cut.

00:38:51.000 --> 00:38:59.000

To prepare for the Sapolia fork. And new releases then have to get cut again after that.

00:38:59.000 --> 00:39:10.000

To prepare for the for the whole ski fork. And then after that The main event.

00:39:10.000 --> 00:39:18.000

May not upgrade. So after all that testing. This is the and implementation part. Now we have to implement.

00:39:18.000 --> 00:39:26.000

And so it typically takes a few weeks from the last. Test net fork to make final changes if they're needed to the client repos.

00:39:26.000 --> 00:39:36.000

I said about 2 weeks, that's more based on like just seeing what's happened in the past takes about 2 weeks.

00:39:36.000 --> 00:39:41.000

And then after then an epoch, an epoch, a date of the upgrade is a set.

00:39:41.000 --> 00:39:54.000

You scheduled the coordination call the watch party everything else and then it proceeds you know the upgrade occurs And if we've done our jobs right.

00:39:54.000 --> 00:40:06.000

All the testing will have. Proven to be you know useful and we'll have a custom upgrade.

00:40:06.000 --> 00:40:13.000

But after you upgrade. We're actually still testing it in production, believe it or not. Right?

00:40:13.000 --> 00:40:23.000

So, there are behaviors that cannot be identified until

00:40:23.000 --> 00:40:28.000

We're actually being used, right? Like they're actually being used on, on main net.

00:40:28.000 --> 00:40:35.000

And so because of that, There might be some unanticipated behaviors or vectors of attack that could happen.

00:40:35.000 --> 00:40:46.000

So there is teams of security researchers and ethical hackers that are probing the protocol constantly to identify vulnerabilities to identify potential issues.

00:40:46.000 --> 00:41:01.000

They do receive payment for successfully identifying and appropriately disclosing these. It's always great to have a bug bounty and ethical hacking program because you want to lean into those members of the community who can add value through their, you know, sort of finding behaviors that aren't supposed to occur.

00:41:01.000 --> 00:41:12.000

So I have an example of right here. Here's a fork choice. So I have an example of right here.

00:41:12.000 --> 00:41:14.000

Here's that aren't supposed to occur. So I have an example of right here. Here's the fork choice bug fix disclosure.

00:41:14.000 --> 00:41:28.000

This was, you can see this is also on the Ethereum Hack MD. But basically there was, 4 choice attacks that, were identified that could able to cause long reorgs.

00:41:28.000 --> 00:41:35.000

You can actually go down. There's quite a bit of documentation and proof. Some of it is actually done via formal verification.

00:41:35.000 --> 00:42:04.000

So if you go into this, you could see attached proofs about how this is proven and this helps identify how changes you know can Can occurred that need to be implemented and sometimes these changes don't require like a full update you know full upgrade it can be just done with cutting a new client versions, right?

00:42:04.000 --> 00:42:12.000

Because We're trying to. And to better adhere to the spec, there might be things that need to be changed or removed.

00:42:12.000 --> 00:42:25.000

But basically, this just because a upgrade is done doesn't mean that testing stops, there's just a different set of group.

00:42:25.000 --> 00:42:34.000

Different set of use. Users who are testing and in this case it's at security researchers.

00:42:34.000 --> 00:42:40.000

Ethical hackers, protocol, sort of base people. So What we have then is during this whole process, there's a ton of discussion that's happening.

00:42:40.000 --> 00:42:55.000

So like I said, we have coordination calls, we have, you know, the Alcore Devs calls, the CL layer calls, there's constant talk, sometimes there's specific calls to discuss EIPs and the testing and implementation of those.

00:42:55.000 --> 00:43:07.000

There's a testing channel on E. Discord. There's channels related specifically to each upgrade.

00:43:07.000 --> 00:43:13.000

There's the interop channel that talks between the EL and CL and talks about the engine API.

00:43:13.000 --> 00:43:19.000

The client teams have their own discords and telegrams where there's talking about testing and test conditions.

00:43:19.000 --> 00:43:27.000

There's a hack MD. Documents and we haven't even really broached this yet but I think in the run up to.

00:43:27.000 --> 00:43:39.000

You know, events like Dev Connect or East Denver, you know, this course started after Dev Connect and it's gonna end before East Denver, at least this version of it.

00:43:39.000 --> 00:43:54.000

But those are opportunities, you know, these different. Community get togethers for actually a lot of the teams to get together in person or in a hybrid setting and actually talk and try stuff out.

00:43:54.000 --> 00:44:13.000

Prior to the merge, there was a huge amount of progress made in an August. Meet up that was in I think the greater Athens area where client teams researchers aren't you able to get together, and actually spend, I think 3 to 4 days together working specifically on some of the harder aspects of the merch.

00:44:13.000 --> 00:44:27.000

So as much as we do stuff and sort of this like always a virtual environment. There are times when it does make sense to convene people in person to spend time working to sort of push through and get stuff done.

00:44:27.000 --> 00:44:40.000

So, anywhere else, Matt, that you think the coordination is kind of happening as well.

00:44:40.000 --> 00:44:49.000

This is basically it.

00:44:49.000 --> 00:44:50.000

Yes, the magicians and all of that.

00:44:50.000 --> 00:44:54.000

Yeah, so a lot of I was gonna say a lot of this happens in the GitHub spec repositories.

00:44:54.000 --> 00:45:02.000

So the CL specs and the enterprise execution layers are not the enterprise. The execution. Layer specs so the eels.

00:45:02.000 --> 00:45:06.000

Yeah, the coordination is happening there, but a lot of the actual coordination happens in GitHub discussions.

00:45:06.000 --> 00:45:22.000

On individual spec PRs. So, EIPs might have one or more individual spec PRs and people will ask for comment and opinion from other teams on those PRs.

00:45:22.000 --> 00:45:26.000

So the discussion in GitHub actually becomes pretty important and it stays canonical too, which is nice.

00:45:26.000 --> 00:45:31.000

It doesn't really go anywhere. Yeah, that's the only other thing I would add to this.

00:45:31.000 --> 00:45:36.000

Everything else is very much covered.

00:45:36.000 --> 00:45:41.000

Awesome. Alright, so let's open it. We have some good time for QA. Feel free to type it in or come off.

00:45:41.000 --> 00:45:53.000

Mute, whatever your choice and then I'll try and just wrap it. We'll probably stop with like a minute ago to go over, you know, what's gonna happen.

00:45:53.000 --> 00:46:19.000

At office hours.

00:46:19.000 --> 00:46:27.000

If no questions, that's completely, completely fine.

00:46:27.000 --> 00:46:28.000

I'll give everyone a moment to think and I'll cover this. We go right to the end.

00:46:28.000 --> 00:46:36.000

So obviously office hours will answer any questions that people have as they go over the material. We can go in depth on any specific.

00:46:36.000 --> 00:46:38.000

You

00:46:38.000 --> 00:47:02.000

Questions that you have about any of these processes. So one thing that we won't cover is We will probably we will stay away from the Devnet side because that's like our final week of the class is sort of diving into the Dev Nets and that's specific for when we talk about the Dev Nets, we're going to be specifically talking about how Dev Nets applied to this upgrade.

00:47:02.000 --> 00:47:20.000

So, That's to more sort of dive into to these changes and. Yeah, and we do have a guest speaker coming up to speak about, so, we're gonna get that scheduled.

00:47:20.000 --> 00:47:29.000

We'll announce it on Friday. It will be Justin Florentine from the, Hi.

00:47:29.000 --> 00:47:43.000

And so, we'll put out that information and he will talk about testing probably with, you know, and that's a great person to then ask questions if you have them, not just about the generalized testing process, but like.

00:47:43.000 --> 00:48:00.000

Decisions around testing specific to that execution layer Ethereum find.

00:48:00.000 --> 00:48:07.000

Okay, cool. Well, if there are No questions. I don't wanna hold anyone. For any longer.

00:48:07.000 --> 00:48:28.000

So thank you so much for attending today's. Lecture, we'll see you on Friday if you can attend and just to make a plug, there are some additional Just that forks coming up, so, I will tell you right now.

00:48:28.000 --> 00:48:35.000

We will probably, we will definitely cover what happens in this a polio one at some point.

00:48:35.000 --> 00:48:56.000

And we while the course will be, technically in the sort of final project stage at this point, Matt and I are gonna try and pull something together a little special extra one for, we want to, you know, whether that's a debrief about it or depending on the timing, if it works out, something that we can kinda.

00:48:56.000 --> 00:49:12.000

Do a little bit of a watch. Party. We'll see about that, but we're gonna have there's gonna be at least one or 2 bonus, activities as part of this course to take advantage of the fact that Our timing got lucky, right?

00:49:12.000 --> 00:49:23.000

So better lucky than good sometimes. And so It's very nice that this course is timing up with the test net forks as we're kind of rolling into the finish.

00:49:23.000 --> 00:49:30.000

So, we will take it. We will take it. It went as planned. All right, everyone.

00:49:30.000 --> 00:49:32.000

Have a great rest of your weeks and all will see you, on Friday.

00:49:32.000 --> 00:49:37.000

Thanks everyone

The following transcript was auto-generated, and may content syntactical errors.

00:00:04.000 --> 00:00:18.000

Kick off today is Friday? January nineteenth. And we're gonna have some office hours.

00:00:18.000 --> 00:00:30.000

So just to review. From last

00:00:30.000 --> 00:00:46.000

Lexter. We. Are always going through some sort of testing process. Whether we're testing the current fork or testing.

00:00:46.000 --> 00:00:53.000

Yeah, EIPs for inclusion and another fork. We might also always be testing. We might also be testing.

00:00:53.000 --> 00:01:10.000

Interpretations of the protocol as well. I think that's something that we kind of missed elaborating on, but what I mean by that is just that Sometimes, particularly when there is.

00:01:10.000 --> 00:01:20.000

The implementation of an EIP or of some change, other change to the Ethereum protocol, it may not.

00:01:20.000 --> 00:01:29.000

Be implemented as planned and so there's always testing to make sure that. There's adherence to the specification.

00:01:29.000 --> 00:01:36.000

So. There's always testing going on, but as we think about coming into any sort of upgrade.

00:01:36.000 --> 00:01:55.000

There are these initial test that give us signal about what needs to change. Those are, you know, the shadow forks and dev nets will go next week into more specifics about like sort of the nitty gritty of what happened on the Dev Nets.

00:01:55.000 --> 00:02:07.000

And then. When it's time for like. Prime time, we first start with dress rehearsals of Does, test net forks.

00:02:07.000 --> 00:02:20.000

And then those give a signal about what worked but didn't work. Make changes and that allows us to go and prepare into the, to the main net upgrade.

00:02:20.000 --> 00:02:30.000

So that's just an overview. I just wanted to open it up to any questions that folks had or any topics.

00:02:30.000 --> 00:02:34.000

To that would make sense to just go deeper on.

00:02:34.000 --> 00:02:44.000

So feel free to unmute. Or just drop any questions in the in the chat.

00:02:44.000 --> 00:02:54.000

Something that we pointed out, and this is in the slides is that, you know, there is this end-to-end testing suite for a theory.

00:02:54.000 --> 00:03:01.000

Of Hive and so we put that in there as that will replace retest teeth.

00:03:01.000 --> 00:03:11.000

Okay, so great. Question one. Which networks are accessible? To the general public.

00:03:11.000 --> 00:03:21.000

For trying out their own depths. Would this include an appropriate asset faucets? Would there be a page with all relevant information?

00:03:21.000 --> 00:03:33.000

Great question. So I'll start generally. Let's start outside of an upgrade. The DAP networks right now that folks can test on.

00:03:33.000 --> 00:03:36.000

There is.

00:03:36.000 --> 00:03:48.000

There is by. Coordination of the Ethereum Foundation and those who are running it. Only.

00:03:48.000 --> 00:04:04.000

One network that's really built for testing daps. And that. Would be, at this time.

00:04:04.000 --> 00:04:14.000

Sipolia. For Ethereum may not. So, Ethereum may not Sapolia because Gurley is deprecated.

00:04:14.000 --> 00:04:23.000

That is going to be the network for. Testing daps. There are Sepulia faucets.

00:04:23.000 --> 00:04:33.000

The etherium foundation does a very good job of like keeping up to date on where there are faucets.

00:04:33.000 --> 00:04:41.000

I would be remiss to say, you know, I, I'm wearing an inferior shirt right now.

00:04:41.000 --> 00:04:50.000

I work at consensus. We have a Sapolia faucet as well. So many other, our PC providers also have.

00:04:50.000 --> 00:05:04.000

Faucets as well. So if you want to get any sort of You can debt go to.

00:05:04.000 --> 00:05:15.000

The Ethereum Foundation website, you just go to the theorem.org and you're There is a like a.

00:05:15.000 --> 00:05:19.000

A section that says faucets, if you go to infuri. Dot.

00:05:19.000 --> 00:05:26.000

There's faucets if you go to, if you Google. S, you will find Sipolia faucets.

00:05:26.000 --> 00:05:33.000

There are a lot of faucets. That is for testing. If you just want to test at any given time.

00:05:33.000 --> 00:05:39.000

Adap and you wanna test it in a environment as close as possible to production, that's where you do it.

00:05:39.000 --> 00:05:49.000

Now, if you wanna test your DAP. In a Devnet situation or a shadow fork. That does require joining.

00:05:49.000 --> 00:06:00.000

The, channel and being in communication with the various teams that are testing it because That, Dev Nets and Shadow Forks.

00:06:00.000 --> 00:06:10.000

Tend to be a close validator staking set, right? So. If you have a DAP that has been deployed.

00:06:10.000 --> 00:06:21.000

To that. Network. And there's a shadow fork occurring. You would actually see what happens in that shadow fork.

00:06:21.000 --> 00:06:31.000

If you had the ability to interact with it, but that those shadow forks are, almost always not like open to the public, right?

00:06:31.000 --> 00:06:39.000

And that. There is not a, the coordination to like have that be an open test is very hard.

00:06:39.000 --> 00:06:52.000

So. If there's a reason that it makes sense to like test the DAP and for example, like if the DAP is making use of a certain new opcode that's introduced by EIP.

00:06:52.000 --> 00:07:15.000

That'd be a great reason to have that one be tested. That would require coordination with. All core devs with the The Ethereum like DevOps team is also sometimes called Panda ops just like if you hear us like throughout that name.

00:07:15.000 --> 00:07:29.000

Panda Ops is just a reference to the merge some visualization of the merge of like a like a black bear in a polar bear.

00:07:29.000 --> 00:07:36.000

Doing a little dance together and then becoming a panda bear so that's why that's, what the reference is.

00:07:36.000 --> 00:07:48.000

But if you wanted to test under the conditions, test adapt under the conditions of a. Upgrade, but wanted to get like a smaller set, you'd have to coordinate with them.

00:07:48.000 --> 00:07:57.000

Now, if you have a depth that's deployed to S, on the.

00:07:57.000 --> 00:08:16.000

The Sapolia. Upgrade is gonna happen on thirtieth January and that would be the best sort of like open set of people if you wanted to test out a depth deploy it before then see what happens and then get an opportunity to do it and you could just use Sapolia.

00:08:16.000 --> 00:08:28.000

So the is is now become the DAP testing layer and, Holeski is like the staking layer. So if you're like that, sorry, the staking sort of base test.

00:08:28.000 --> 00:08:34.000

So if you're looking to like test your staking validator setup, this test net will, will be the one.

00:08:34.000 --> 00:08:44.000

This will change over time, because What we're seeing is that, you know, test nets while there's no like actual crude value on them.

00:08:44.000 --> 00:08:58.000

They do like any other sort of computer system require resources to run. And so I would say the general conclusion is.

00:08:58.000 --> 00:09:09.000

That. It's easier to run a test net. That is. For DAP developers.

00:09:09.000 --> 00:09:26.000

At scale, if you have a closed validator set, basically like a closed number of set. Groups that are running the validators for it because it makes a little bit easier for them to ensure that that test that stays online.

00:09:26.000 --> 00:09:43.000

And if you're running a test net that is more like Testing a staking setup. Focused it's that's the one that you can keep open right it's because then it's an open set you're having validators join and the requirements are different if you think about it.

00:09:43.000 --> 00:09:44.000

Like adap doesn't need 30 to to deploy, right? 32 test. That's when you could do like a faucet.

00:09:44.000 --> 00:09:50.000

I's different if you think about it. Like, adap doesn't need 32, to deploy, right?

00:09:50.000 --> 00:09:55.000

32 test. Whereas like if you want to test your staking setup. You need like to have that 32 test, right?

00:09:55.000 --> 00:10:07.000

So it's just like a fundamentally sort of different. Setup. So great question. I went into a little bit more detail there, but just kind of wanted to flush it out.

00:10:07.000 --> 00:10:17.000

And then The page with all the information I can probably pull that up in a second. So let's go on to question 2.

00:10:17.000 --> 00:10:24.000

I would like to fork a client test modifications to the network framework. Would there be a way to do this without having to run a full-fledged network.

00:10:24.000 --> 00:10:45.000

I would say the best way to do that. Is to run it locally. Run it locally and The easiest way to do that is just to Spin up like you can spin up a client and also spin up multiple like other iterations of that client and have it peer with itself.

00:10:45.000 --> 00:10:53.000

And what you would need to decide is like what your starting state would be. So you could either say, I'm going to have my starting state be essentially a snapshot of a certain block number.

00:10:53.000 --> 00:10:59.000

And, and when I say this like, I'm assuming that you know how to set the configuration of the client, right?

00:10:59.000 --> 00:11:15.000

Like basically when you're doing startup, whether it's your ingesting that information via like a GAMMO file, any let's just say any sort of configuration file that you know what the starting state wants to be.

00:11:15.000 --> 00:11:24.000

So if you want to start from like, hey, we're gonna start from. Essentially.

00:11:24.000 --> 00:11:32.000

A coin like a coin based account just, you know, 0 x 0 0 0 and we're gonna start and try network.

00:11:32.000 --> 00:11:45.000

Framework that is essentially like recreating everything from scratch or if you want to say, you know what, I'm really interested in starting from the state, like doing a fork from 5 min ago. You could do that.

00:11:45.000 --> 00:11:57.000

You just need to set up your client and the peers appropriately. And then you can use that information to actually then fork and test modifications within the framework.

00:11:57.000 --> 00:12:04.000

So you're gonna run a network. But you're gonna run it locally. So it isn't as.

00:12:04.000 --> 00:12:25.000

Full, we used to do this all the time in the, one of the exercises in the developer boot camp was taking death and starting it and running basically a for node network back under like proof of work, but just doing it locally on.

00:12:25.000 --> 00:12:32.000

Your machine and you know there was a lot of laughs that would happen as everyone's machines start to you know go the fans spin and it sounds like it's an airplane taking off depending on how powerful their machines are.

00:12:32.000 --> 00:12:46.000

But now with proof of state, you can do this. You could be pretty strategic and like, wait about it.

00:12:46.000 --> 00:13:02.000

Additionally, if you want to. Even be more lightweight, there are different development frameworks that ship with a.

00:13:02.000 --> 00:13:12.000

With the sort of. Forks and you can reach out to those teams and see if that's something that they accommodate.

00:13:12.000 --> 00:13:33.000

Everything, you know, it's, it wouldn't be, for most of it wouldn't be out of the box because normally you're just trying to test the DAP versus test the client, but there could be some possibilities there and I'm talking about everything from remix, open source flavor of Ganesh opens hard hat Foundry all those they have some sort of version of like a

00:13:33.000 --> 00:13:43.000

forked version. So I'm not saying that like it would be easy to bolt on testing a client there, but it's something worth to talk to teams too.

00:13:43.000 --> 00:13:54.000

So those are kind of your options. And then finally during the testing, whether it be any data collection for vital signals of the network, say message is going gossip network, shaped with a blockchain with this data be accessible.

00:13:54.000 --> 00:14:06.000

Yes, there is. This is probably a good time for me to stop share and break out of my

00:14:06.000 --> 00:14:21.000

Pull up a page for you. Because we can talk about The, what is called the, the,

00:14:21.000 --> 00:14:27.000

Let me pull this up. I had this ready and then of course I had it ready and then I clicked off somewhere and I can't find it.

00:14:27.000 --> 00:14:42.000

Which, you know, always happens. This is the.

00:14:42.000 --> 00:14:57.000

First. And now, of

00:14:57.000 --> 00:15:01.000

I'm still sharing my screen. My goodness.

00:15:01.000 --> 00:15:09.000

Thank you. It's my first time using. This computer, but it is. Actually not.

00:15:09.000 --> 00:15:15.000

Okay, where did everything go?

00:15:15.000 --> 00:15:21.000

Come on, Zoom. Help me out. Help me out too. Okay, there we go.

00:15:21.000 --> 00:15:26.000

Goodness. I did not know it's going on there. Apologize to that. Okay.

00:15:26.000 --> 00:15:36.000

Here we go. We're back and share. Okay, so Girlie. This should be the, girly blob analysis.

00:15:36.000 --> 00:15:52.000

So this came out of. Yes. Upgraded. And so, This is the test, this is like the data collection and a write it up in hack MD on the, Panda ops.

00:15:52.000 --> 00:15:59.000

So it's talking about, hey, you know, we, the girly public test, it underwent the, the denkin fork.

00:15:59.000 --> 00:16:09.000

6 30 am universal coordinated time. There was a brief non finality incident. Let me just drop this in here, cause I think this is really interesting.

00:16:09.000 --> 00:16:15.000

This is like an initial report out, like, like we said in lecture, this is kind of what happens.

00:16:15.000 --> 00:16:27.000

There is like an initial report out and then We, we have some additional, reporting that's done, there is an issue, an issue analysis, which is awesome.

00:16:27.000 --> 00:16:45.000

So here is the debugging of that. And talks about What happened? Walks through all of it so We have that issue now.

00:16:45.000 --> 00:17:02.000

Data from precon was collected via centuries running. Eeli think it's like LLL is and you can kinda see this is the.

00:17:02.000 --> 00:17:03.000

Electric, that's what it is. Yeah. This is a multiplexer.

00:17:03.000 --> 00:17:28.000

It's additional. It allows multiple consensus clients to connect to a single execution client, which is Really makes me laugh because like when we were doing a bunch of education, work about the merge we.

00:17:28.000 --> 00:17:32.000

We're like, don't.

00:17:32.000 --> 00:17:41.000

Do this, right? Because it's suitable for monitoring analytics. Which is great. It's good for testing, but you will miss block proposals.

00:17:41.000 --> 00:17:48.000

Basically the idea is that you can't have multiple consensus layer clients driving the same execution that the same engine API, which is associated with that.

00:17:48.000 --> 00:18:02.000

The execution layer. So That's a little bit of a side, but, so we have, you know, the blob analysis.

00:18:02.000 --> 00:18:09.000

And everything is kind of walk through here, there's spanning of blobs.

00:18:09.000 --> 00:18:18.000

There was, you can see some screenshots dropped in here, how data was collected. Everything that was done, I think this is a great overview.

00:18:18.000 --> 00:18:26.000

And so we're collecting those P. 95 statistics. On what happened. There's some.

00:18:26.000 --> 00:18:34.000

Interesting, more mainly illustrative graphics dropped in there, but there is you know a heat map that you can view.

00:18:34.000 --> 00:18:40.000

There's the graph. I think this is all done. Yeah, it is done in and These are like the literal dashboard that you go into.

00:18:40.000 --> 00:18:51.000

So great question, cause this is, this is gonna force me to log in. Okay. I guess you'd have to have a login to do that.

00:18:51.000 --> 00:18:56.000

But I'm not exactly sure how to, to get that. That is probably a question.

00:18:56.000 --> 00:19:00.000

And, and I'm guessing it's a close set of people but This is like all the data about what happened.

00:19:00.000 --> 00:19:09.000

Kind of at a high level. So yes, I guess the short answer to the question is that, all of that data is here.

00:19:09.000 --> 00:19:18.000

It should be in some form accessible. That, all of that data is here. It should be in some form accessible.

00:19:18.000 --> 00:19:24.000

It does seem like it could be exported as well. It should be in some form accessible. It does seem like it could be exported as well. And it is shared around.

00:19:24.000 --> 00:19:28.000

And then there is, and while we don't have the login to see these graphs, it does seem like it could be exported as well, and it is shared around.

00:19:28.000 --> 00:19:30.000

And then there is, and while we don't have the login to see these graphs, it that these graphs do exist.

00:19:30.000 --> 00:19:32.000

So they are something that could be, you know, in talking to like, like someone else from that team that it should be accessible.

00:19:32.000 --> 00:19:43.000

So, yeah, that kind of answer. I hope that answers all 3 questions. Great questions.

00:19:43.000 --> 00:19:52.000

Happy to jump into, more.

00:19:52.000 --> 00:19:53.000

Yeah.

00:19:53.000 --> 00:20:01.000

I got another one. So like the tests are probably run by some script of. What they think are the things they want to test about.

00:20:01.000 --> 00:20:11.000

But because there's seems like So many different ways of getting in. To, there's so many different factors.

00:20:11.000 --> 00:20:16.000

I guess that's why they're limiting the network upgrades to just a couple of PRs.

00:20:16.000 --> 00:20:29.000

But, still there's still so many ways in so. It's amazing they don't miss new ways of of problems.

00:20:29.000 --> 00:20:30.000

Okay.

00:20:30.000 --> 00:20:42.000

You know, like in, I don't know, just like in. In remix where like where there's like our standard tests that we every time we add a new feature we add some tests for it but users always find some new way to break it.

00:20:42.000 --> 00:20:43.000

And, yeah.

00:20:43.000 --> 00:20:53.000

So, yeah. It is it is very impressive. That the number of issues. Our like I would say like it's an impressive.

00:20:53.000 --> 00:21:04.000

An exceedingly impressive rate of, catching issues beforehand. Now, to be clear.

00:21:04.000 --> 00:21:22.000

There are issues that pop up that have talked up. So we go a little bit more into this. Into the for choice bug that was discovered this one really did like this was a deep deep research issue.

00:21:22.000 --> 00:21:39.000

I'm not going to lie. Like I think that this was an issue that would have likely not been caught if not for like a really adept security researchers who have.

00:21:39.000 --> 00:21:51.000

The ability to like write proofs in like formal verification languages. Could do this because they basically found like edge cases that

00:21:51.000 --> 00:22:02.000

Could have been exploitable? But are so deep in there that we require like you know, Quite a bit of talent.

00:22:02.000 --> 00:22:16.000

To be able to discover them, though who knows with like. So the proliferation of AI, it is possible that a lot of these kind of like what we would once think of like, oh, you know, these are pretty deep issues, they might be more.

00:22:16.000 --> 00:22:22.000

I think there is an argument to be made that there's gonna be that sort of like.

00:22:22.000 --> 00:22:36.000

So only gonna accelerate sort of that. That you know finding bugs and fixing them but There actually is a very deep explanation of this issue.

00:22:36.000 --> 00:22:46.000

Roberto. Sultani, who's one of the. The

00:22:46.000 --> 00:22:52.000

The researchers on this, I was fortunate enough to get to spend some really nice time. With him.

00:22:52.000 --> 00:23:03.000

In a small group setting. To really like talk through what, they, what they figured out.

00:23:03.000 --> 00:23:12.000

And. It was. I am not going to necessarily be able to.

00:23:12.000 --> 00:23:19.000

Do you do justice? I've probably, I, he's also shared some like a talk that he gave around this.

00:23:19.000 --> 00:23:32.000

But, what was realized was that There were possibilities of reorging with for a non justifiable reason like.

00:23:32.000 --> 00:23:53.000

So, so the issue was essentially was. The way that the specification was written. And the way it was implemented did not match essentially but on these like sort of edge cases so that there was an attack vector where there could be the possibility of a reorg.

00:23:53.000 --> 00:24:01.000

That could occur that would not be a justified reorg. And it one of the results was that.

00:24:01.000 --> 00:24:16.000

Validators who acted honestly could be slashed. So this was discovered and. There was at you know full

00:24:16.000 --> 00:24:23.000

I mean, I think this is one that was done by Roberto like at a station. Denlocks and then.

00:24:23.000 --> 00:24:36.000

That there was the, this is how it's gonna be. Changed so This was a really so you know it is very and impressive that all of this was like figured out.

00:24:36.000 --> 00:24:50.000

Here is the manual proof list. And this goes like. Super deep. If you want to like start clicking down, I'll share this like again.

00:24:50.000 --> 00:24:58.000

So you can just see. But as you keep on like going down, if you really want to get into some some.

00:24:58.000 --> 00:25:09.000

Interesting. Work, it just. You can see the spec versions in there, but this goes into.

00:25:09.000 --> 00:25:30.000

Sort of the assumptions and then like. How do you prove that this thing but what was supposed to happen was not happening so this whole

00:25:30.000 --> 00:25:39.000

Real, math and justification going on. But to your point, Yes, it is possible. That we can get something.

00:25:39.000 --> 00:25:53.000

Live and there's still be some, gaps in there, but we're very fortunate that For the most part.

00:25:53.000 --> 00:26:02.000

The vast majority of the hit like expected behaviors. Our, are observed. And the upgrade is able to occur.

00:26:02.000 --> 00:26:14.000

Now to to talk about sort of an interesting situation. Let's talk about it. This one is, this is like an open for debate.

00:26:14.000 --> 00:26:29.000

Question, but like at some point, right? We have to say. Is there adjusted like, right, is a good example.

00:26:29.000 --> 00:26:36.000

Was that an intended part of the protocol? Is that an intended part of the protocol is that an intended part of the protocol, is that an intended part of the protocol or not?

00:26:36.000 --> 00:26:47.000

I think we've We've come to say, well, you know, It's something that exists and there are benefits.

00:26:47.000 --> 00:26:51.000

There's positive benefits. There's negative benefit. There's, there's not negative benefits.

00:26:51.000 --> 00:26:57.000

There's, there's negatives to it, but what you can also see is sometimes that.

00:26:57.000 --> 00:27:11.000

Parts of the protocol, there is a inability to always predict what that would. How users will. End up interacting with the protocol.

00:27:11.000 --> 00:27:32.000

And so then there has to be decisions around. You know, is that now just part of it and we're saying it's included a lot of times, you know, that is seen the like code is law sort of subset of of folks that are in this space and that is a group that, I think, you know, it's very important to kinda like take that into consideration.

00:27:32.000 --> 00:27:57.000

On the other hand, there is a like, hey, the intent wasn't for that. So, self-destruct, we're starting to see, you know, all that work around self-destruct and that's gone on and modifying the opcode is basically saying like, hey, this was the original intention and yes, could his law but like because of things that are going on with

00:27:57.000 --> 00:28:03.000

it, can we modify the opcode? So it's always like there isn't like a hard fast rule.

00:28:03.000 --> 00:28:11.000

It does feel like there is always this sort of like tension and pull between various groups within all Cordes within the Ethereum community.

00:28:11.000 --> 00:28:21.000

But what's cool about it is that we end up coming to . A Enough of a consensus that we can move forward to keep on.

00:28:21.000 --> 00:28:32.000

Keep the protocol going. Now, obviously doing upgrades like this, we have said, you know, there is a certain desire to constantly move.

00:28:32.000 --> 00:28:39.000

You know, taking information and react. So as much as that like code is law, there is a desire to say like, hey, you're not going to get everything right.

00:28:39.000 --> 00:28:51.000

You need to be able to. Constantly update and improve. And so that Tension is is a good one because it Makes us respect the fact that like, hey, there are decisions made.

00:28:51.000 --> 00:29:06.000

We're just not gonna try and not take those decisions into account, but at the same time, like a protocol has to evolve to meet used cases and what people want to do with it.

00:29:06.000 --> 00:29:16.000

And in the case of Roberto's. Proof. Like, so you found a problem. And then, then there was probably.

00:29:16.000 --> 00:29:20.000

How long did it take for them to did they address it or did they? Hi, did it have to wait?

00:29:20.000 --> 00:29:34.000

They did address it. I think that the time horizon to address it was ended up being months. So which sounds about right, you know, that seems, it seems like that seems pretty fast.

00:29:34.000 --> 00:29:48.000

Like, like pretty reactive for like such a deep. Proof that they that had to do but it see from my memory it was like identified and and you know

00:29:48.000 --> 00:29:55.000

A solution was come up with and that was all be able to be implemented over the course of months.

00:29:55.000 --> 00:30:02.000

But it does that it was presented. It was like fairly disclosed. And all that. So it was part of like a.

00:30:02.000 --> 00:30:20.000

A an an ethical security researcher work group. I think there that there was a compensation received for identifying that and fairly disclosing it under the terms of a better set out.

00:30:20.000 --> 00:30:31.000

It didn't make a new. Fork network for good. I mean, yeah. Did it or when they up fixed it?

00:30:31.000 --> 00:30:32.000

I think.

00:30:32.000 --> 00:30:35.000

It didn't my understanding is not it was more of a client implementation change that could be made. Yeah, yeah.

00:30:35.000 --> 00:30:46.000

Without going into the deep specifics of it, I think there was like a specific. Field that was able to be some parameters on that were able to be modified.

00:30:46.000 --> 00:30:53.000

And so that's a little bit quicker because it just requires the client teams to cut a new version.

00:30:53.000 --> 00:30:59.000

Yeah.

00:30:59.000 --> 00:31:08.000

So as much as you kind of laid this out linearly, I mean, I would I probably need to get a good make a good diagram.

00:31:08.000 --> 00:31:18.000

Long term that shows like it's really like circles upon circles of testing. And, and usage and so yeah.

00:31:18.000 --> 00:31:33.000

Any other questions? Great questions.

00:31:33.000 --> 00:31:40.000

Okay, well, everyone thinks about if there are any final questions. Do not take the whole time, but did want to.

00:31:40.000 --> 00:31:53.000

Just give some, final, announcement. So on Monday, this is on the course calendar just in Florentine from the basic team is going to explain testing as it relates to him working on an Ethereum execution layer client.

00:31:53.000 --> 00:32:03.000

So we'll go in depth. It will be sort of in his focus. And Matt will be running that session with with Justin.

00:32:03.000 --> 00:32:06.000

Any specific questions you have, if you sort of review anything I shared and you're like, I'm really interested in finding more about this tooling.

00:32:06.000 --> 00:32:15.000

He's great person asked that recording will be posted. It might be a little delayed in getting it posted.

00:32:15.000 --> 00:32:23.000

We have that next week off. Matt and I have some prior obligations that are just gonna make it, you know.

00:32:23.000 --> 00:32:35.000

Tough for us to do the Wednesday in the Friday office hours. But then we come back and we have a, our final week where we talk about, you know, really go in depth on Dev Nets.

00:32:35.000 --> 00:32:38.000

On the

00:32:38.000 --> 00:32:48.000

On the thirtieth and then instead of having office hours on the second what we're going to do is push them to Wednesday, February, the seventh to coincide with the Holski upgrade.

00:32:48.000 --> 00:32:50.000

Hello.

00:32:50.000 --> 00:32:57.000

So that way we can just keep the conversation going. So we won't have office hours that are typical Friday time.

00:32:57.000 --> 00:33:10.000

We'll have us will push the office hours on the whole ski upgrade date just so we because at that point we'll have all the test nets upgrade date just so we because at that point we'll have all the testnuts upgraded and that's like a good way to wrap and summarize.

00:33:10.000 --> 00:33:21.000

Probably it also that one will. Talk about bonus activities. And we are also probably going to at that point.

00:33:21.000 --> 00:33:46.000

Talk about the opportunities that we might have to do a little. if possible, either a hybrid or in person get together as we get closer to, one of the, what I would call like a real cornerstone event of the Ethereum community, alongside the likes of, you know, the, the global series.

00:33:46.000 --> 00:33:55.000

Devconnect, Devcon, East Denver. So, many of us will, be there.

00:33:55.000 --> 00:33:59.000

And so, we're trying to figure out, is there a way to just informally get together, maybe over a coffee.

00:33:59.000 --> 00:34:11.000

And just say hi to anyone who's in the course. Who might be in the area and talk about what we've learned.

00:34:11.000 --> 00:34:19.000

And see if we can gather some of our our guest speakers as well just to come by and say hi so it would be very informal but.

00:34:19.000 --> 00:34:24.000

Yeah, that's right thinking.

00:34:24.000 --> 00:34:25.000

Sounds good. We can meet at Dandy's house.

00:34:25.000 --> 00:34:33.000

Alright. Yeah, that would be nice. I'll bring, I'll bring a bunch of coffee and, I'll just.

00:34:33.000 --> 00:34:34.000

Yeah.

00:34:34.000 --> 00:34:46.000

Get caffeinated. Pull out a whiteboard and and and write some write some mathematical proofs.

00:34:46.000 --> 00:34:58.000

Alright, well if no more questions that was really great to hear everyone guest speaker on Monday. Just to, so you should see that in the calendar and, We're really excited for that.

00:34:58.000 --> 00:35:08.000

We'll have a week off from lecture and then we're back. With our final official lecture of the course on the I said the thirtieth by minute 30 first and then we're pushing back like I said office hours to be the day it's something that Friday I've already moved that that should update it that should have updated in the calendar.

00:35:08.000 --> 00:35:28.000

All right

00:04:37.150 --> 00:04:47.520

Matt Nelson: all right. Thank you for joining folks. Today is week number 4, guest speaker number 3 of our upgrading ethereum network

20

00:04:47.710 --> 00:04:50.759

Matt Nelson: upgrading the ethereum network lecture series.

21

00:04:51.100 --> 00:05:11.770

Matt Nelson: And today's topic is going to be a guest speaker on the week that we discussed testing and implementation of the hardworks. So we have with us today. Justin Florentine is a member of of my team here, consensus on the base. 2 team who was the core developer. It's been a big part of delivering the Cancun upgrade on the execution there.

22

00:05:11.770 --> 00:05:24.069

Matt Nelson: so I won't give too much away. But we'll give let you do a little intro Justin and then we can kind of go into it. I'll start with some initial questions, or if you'd like to start us off in a particular spot.

23

00:05:24.210 --> 00:05:28.840

Matt Nelson: But yeah, again, the context for this this week of lecture was last. We discussed.

24

00:05:29.080 --> 00:05:50.650

Matt Nelson: How? Core devs coordinate around things like Devnett's specification updates testnet forks. We discussed the gorilla fork briefly. So there, there's context already. There there's also context around the eip process. How those are written, how those are coordinated! We taught. Had puja from Eve cat Herters come on and discuss all that stuff.

25

00:05:50.650 --> 00:06:00.260

Matt Nelson: So the the students should have a great kind of understanding of how we get to the point of scope and how we start to put things in practice. But I think your perspective will be super helpful on like.

26

00:06:00.390 --> 00:06:06.380

Matt Nelson: okay, we have kind of a defined list of items. Now we have to get started with.

27

00:06:06.390 --> 00:06:17.980

Matt Nelson: you know, implementation, which is in many cases kind of the step one or the easy step until we start to break stuff with interoperability testing. And then we have, of course, devnets and testnets.

28

00:06:18.060 --> 00:06:27.700

Matt Nelson: so I'll take a pause here. I'll let you intro, and then, if you'd like to start anywhere feel free. If not, I have some questions prepared that we can go through to discuss, but I'll turn over to you.

29

00:06:28.570 --> 00:06:38.200

Justin Florentine (Besu): Cool. Yeah. Hi, Justin Florentine. I'm an engineer on the Baseu team. You'll find me on socials as Robocop's gone mad, or some variant thereof.

30

00:06:38.260 --> 00:06:49.589

Justin Florentine (Besu): I've been working with consensus since 21 right around London. I think London was the first hard fork that I worked on. I worked on pretty much every hard fork since then, except for Shanghai.

31

00:06:49.760 --> 00:06:55.110

Justin Florentine (Besu): So that's you know, London, Paris. And now, cancun

32

00:06:55.680 --> 00:07:11.510

Justin Florentine (Besu):  you will find me hanging out in discords various times when it's time for interop testing ethicians, forums, etc., etc. That's where a lot of the hanging out and finding out what needs to be done where we're at, etc., etc. goes on.

33

00:07:11.540 --> 00:07:13.310

Justin Florentine (Besu): So yeah, happy to be here.

34

00:07:14.910 --> 00:07:25.940

Matt Nelson: Awesome, Justin. So let's think this, let's start these questions kind of in terms of of testing apparatus. Right. So zoom, we've had a

35

00:07:26.030 --> 00:07:34.290

Matt Nelson: rousing set of scoping discussions around the Cancun Hard Fork, and you are a core developer on the basic team who's been given

36

00:07:34.510 --> 00:07:43.560

Matt Nelson: a handful of things to implement as well as a date for a first devnet launch. What does that kind of look like when we're going through?

37

00:07:43.790 --> 00:07:54.269

Matt Nelson: again. You know, there's the broad specs, broad strokes applying the specs. But then we actually have to connect the nodes and get that started. What's kind of those first little steps there. And how do we start to kind of

38

00:07:54.470 --> 00:08:02.489

Matt Nelson: iteratively patch the clients to get them all talking to each other? Get them all working on the same page, and, you know, go from there.

39

00:08:03.490 --> 00:08:15.800

Justin Florentine (Besu): So there's a couple of different aspects that you want to think about when you're when you're when you're discussing how you get from decided specs to clients running and processing blocks on a network.

40

00:08:15.870 --> 00:08:28.660

Justin Florentine (Besu): The first is a traditional software engineering challenge. Right? So this is open source software, which means you don't necessarily have quite the rigorous top down software design and implementation that you would have elsewhere.

41

00:08:28.890 --> 00:08:50.330

Justin Florentine (Besu): So you need to have discussions with your community. You need to decide. You know what are the. You know we. We understand what the actual features to be delivered in a hard fork are, and the granularity you're given is the eips themselves. So there'll be a design document that'll express some sort of feature that we wanna add to ethereum? It's scheduled to be delivered in this hard fork. Cool?

42

00:08:50.330 --> 00:09:08.250

Justin Florentine (Besu): What are the software tasks that need to happen for that? Is there an area of the code that needs to be refactored? Is there a place where this obviously is a quick hit and will be easy to implement? Who is already familiar with that part of the code and wants to work on. So all of that needs to happen. As you're preparing for the hard work, and as you start delivering.

43

00:09:08.710 --> 00:09:13.890

Justin Florentine (Besu): you're going to be discussing with other client teams in discord

44

00:09:14.650 --> 00:09:19.049

Justin Florentine (Besu): how we know the various eips work. so there will be

45

00:09:19.090 --> 00:09:47.319

Justin Florentine (Besu): usually a series of tests that arrive in a number of different ways. We have reference tests. We have state tests. The big one these days is the hive tests, which is a full on systems test. It's actually running clients. It's interacting with them over the engine Apis to drive them from a test Genesis state to a head block that is theoretically past the fork that we're designing. It has all the features that we're trying to build.

46

00:09:47.320 --> 00:10:04.150

Justin Florentine (Besu): and there's a lot of discussion amongst the teams around that. So they will say, Hey, here's a tive test. Is this actually valid? Great? Ef does does a lot of investment on that. They spent a lot of time and effort on testing. We have really good test coverage cover coverage in these different dimensions.

47

00:10:04.170 --> 00:10:15.409

Justin Florentine (Besu): And that's and I think that's really the shelling point for a lot of getting this work done is the tests. The tests really do drive it. And they are granular enough that you can discuss it. And we implement these tests when they're passing.

48

00:10:15.410 --> 00:10:39.240

Justin Florentine (Besu): We discuss with folks when they're not most the time it's on us a lot of times we find things in the test that are either inaccurate or ambiguous, and sometimes it'll go back to the spec. So a lot of times. People have been joking lately like we're still making changes to the spec for Dan Coon, you know. Hey, we kind of are but a lot of these things are very, very small things. They're they're reducing ambiguities that developers may have run into.

49

00:10:39.360 --> 00:10:55.329

Justin Florentine (Besu): So that's you know. That's kind of the I think the the high vision of of how that works. You iterate on that stuff until we're all in a discord room somewhere, saying, Yeah, we're passing all these things, we feel good about it. What do we do next? Great, we're gonna build test network

50

00:10:55.450 --> 00:11:01.869

Justin Florentine (Besu): it'll be a completely fake network. The Ef will spin up a whole bunch of their devops stuff, they'd use it.

51

00:11:02.080 --> 00:11:21.590

Justin Florentine (Besu): Excessive automation on all of this. They can create these new networks. They will socialize that and say, like, Hey, we made a fake network. Here's a bunch of tools that we're actually running on it for people to watch. So when a new network comes out I'll be like, Hey, Matt? E stats is up over here. You can see how things are going, and they'll be an E stats implementation, and they will be running all the different versions of our clients

52

00:11:22.040 --> 00:11:41.400

Justin Florentine (Besu): when they run into an issue they'll contact whoever they've been working with as the point of contact for that for that hard fork. So some. It's nice for us to have one team member kind of quarterback that, and and try to coordinate that information and keep it moving. That could be anybody, any release? And so you know, that happens.

53

00:11:41.730 --> 00:11:48.869

Justin Florentine (Besu): we work the work, the issues that we run into on the debnets, we find the bugs. And again, it's really your standard Sdlc process at that point

54

00:11:49.640 --> 00:11:55.869

Justin Florentine (Besu): find an issue, report an issue. If you can fix it, fix it. Somebody else could help you fix it. Great, awesome

55

00:11:56.220 --> 00:12:03.200

Justin Florentine (Besu): and that's that's usually the first step is like the fully synthetic test nets the completely constructed one.

56

00:12:03.350 --> 00:12:25.389

Justin Florentine (Besu): Once we've done a few of those we feel really good about it, we'll say, like cool. Let's Shadow fork, a test network like a proper test network. So like girly a named test network, polesky sepolia, those are the those are the ones that we have today. Ripe girly. Bye, bye, this is the last. This is the last for off for girly, so say your goodbyes. Raise a glass.

57

00:12:25.890 --> 00:12:28.970

Justin Florentine (Besu): It's served as well. and

58

00:12:29.090 --> 00:12:40.569

Justin Florentine (Besu): we will take a shadow fork, which is a technique that we developed during the merge. Actually, I think, was the first time that we started doing that where we'll intentionally fork the network. So you'll have

59

00:12:41.090 --> 00:12:54.219

Justin Florentine (Besu): a much smaller subset that'll get real transactions, and it'll create its own blocks with the new rules for the Hard Fork while the other network is still running on the old rules. So, for instance, during the merge Mainet would keep

60

00:12:54.260 --> 00:12:56.440

Justin Florentine (Besu): chug it along on London specs.

61

00:12:56.540 --> 00:13:17.960

Justin Florentine (Besu): and we'd set up a whole bunch of networks, a whole bunch of nodes that are on that network, but now have paras specs, and they will intentionally fork off, and they'll continue to run valid transactions against valid main net state for as long as they can like. At some point these things decay. Because you're running contracts that they don't know about the new features. And they're gonna they're gonna start breaking it.

62

00:13:17.960 --> 00:13:27.270

Justin Florentine (Besu): But it's a really really helpful technique that we've developed that we've we're so happy with how it informed the merge that we're just gonna do it all the time. Now.

63

00:13:27.920 --> 00:13:32.120

Justin Florentine (Besu): So there was a lot of that for Shanghai. There's a lot of that going on right now.

64

00:13:32.140 --> 00:13:46.009

Justin Florentine (Besu): We have shadow forked girly a number of times until we were happy with it. And now we've actually fully forked girly. If you're running a note on gorily, and it's keeping up with head congratulations. You're running Dan Coon software.

65

00:13:46.600 --> 00:13:53.869

Justin Florentine (Besu): We then repeat that process for subsequent networks. Next up will be hulesky, and then sepolia

66

00:13:54.380 --> 00:13:57.940

Justin Florentine (Besu): I don't know if I have the more the order, right? It's the other way around. Yeah.

67

00:13:58.510 --> 00:14:03.830

Justin Florentine (Besu): Sopolo and Hillesky, and then we'll do shadow forks for me net and

68

00:14:04.190 --> 00:14:08.550

Justin Florentine (Besu): wash rinse. Repeat until we're super super confident that we're ready to get to main net.

69

00:14:09.190 --> 00:14:22.380

Justin Florentine (Besu):  yeah. So that's that's a big, complicated, very thorough process that we use to deploy new features on a very expensive network without without dropping

70

00:14:22.840 --> 00:14:24.520

Justin Florentine (Besu): customer transactions.

71

00:14:26.870 --> 00:14:37.970

Matt Nelson: Yeah, that's that's awesome overview. Tell me a little bit more about how the kind of collaborative testing process works. Mostly being because, you know, we're all kind of core developers

72

00:14:38.150 --> 00:14:50.030

Matt Nelson: to outside observers. It might seem that we're all we're we're kind of creating the tests we're creating the specs we are doing all of the testing ourselves. It's kind of like, you know.

73

00:14:50.540 --> 00:15:12.110

Matt Nelson: could seem like a single group is doing all this. But in reality it's a lot of different people. How does that kind of collaborative testing environment come about? You know, what are the orgs involved? And how does it kind of look from a perspective where some might say, Okay, no. It's actually just the same groups doing the same things kind of patting themselves on the back when in reality there's a huge diversity of

74

00:15:12.560 --> 00:15:14.849

Matt Nelson: stuff that goes into that testing in reality.

75

00:15:16.120 --> 00:15:17.780

Justin Florentine (Besu): Well,

76

00:15:17.980 --> 00:15:23.589

Justin Florentine (Besu): yeah. So we get contributions from the community. Each person

77

00:15:23.700 --> 00:15:45.899

Justin Florentine (Besu): providing what they're what they're best at. Right? So a lot of folks that are, familiar with driving Json, Rpc. Api's would be contributing to the hive tests right? Folks that are more interested or well versed with lower level Evm tests could be submitting reference tests. And those are those are very different. Kind of opposite ends of the spectrum.

78

00:15:47.100 --> 00:16:15.379

Justin Florentine (Besu): You know. So so you you could get contributions from just about anyone and it they just sort of get socialized and filtered amongst the other teams through, mostly discord. But all this stuff is tracked in version control. You know, there's a project out there for reference tests. There's a project out there for the hive tests, and those are really just traditional software management problems, right? So as folks create new issues or find issues or fix ambiguities.

79

00:16:15.530 --> 00:16:38.930

Justin Florentine (Besu): they create Prs, they get on to main. And then someone will say, Hey, Justin, that test that you were complaining about, that you thought we implemented wrong. Well, you were wrong, and here's how, and they'll have a test that you know clarifies the spec or fixes and ambiguity, etc., etc. And you just pull those tests. You run them against your your client software, and you move forward from there. And so you do that, you you.

80

00:16:39.380 --> 00:16:58.850

Justin Florentine (Besu): if or whoever's running the test, knows that that that was done, they'll pull main branch of your client, run it against their tests to confirm, and they'll continue doing that. So they, whoever whoever's running these tests is always staying very current on the bleeding edge of the corresponding clients that are being tested. So you have a pretty tight feedback loop.

81

00:16:58.850 --> 00:17:10.039

Justin Florentine (Besu): You'll hear something the next day, if like something on nightly went out and broke broke a test, because they're just constantly and automatically usually pulling those new client updates and running existing tests on.

82

00:17:11.920 --> 00:17:23.369

Matt Nelson: yeah, that's great. So I mean, I know that in context of what we've seen on main net, there have been a few bugs and clients, consensus issues that have actually stopped kind of nodes in the network.

83

00:17:23.390 --> 00:17:33.950

Matt Nelson: The reason I'm asking about this is because I think in Devnet's and testnets, it's a little bit different. How do we necessarily kind of look at consensus issues when there may not be

84

00:17:34.010 --> 00:17:42.649

Matt Nelson: a consensus among the clients? Do we always kind of defer to the tests? And in some cases right. The tests are written from perspective of

85

00:17:42.760 --> 00:17:59.230

Matt Nelson: kind of one team, often, or one clients implementation. How do we start from the building block where every block isn't valid, and then kind of slowly progress? These client specs because it's really very much like you said it. It might even be from one commit to the next.

86

00:17:59.250 --> 00:18:04.410

Matt Nelson: as far as, like the tip of a branch is concerned with Github that can cause an issue.

87

00:18:04.570 --> 00:18:26.680

Justin Florentine (Besu): So that happens through the the redundant design of the testing pipeline right? The fact that we're actually running these hive tests like I run high test locally against no network. Sometime it'll just be what the hive test run against a single instance of base, which is a very, very, very simple form of doing that, but it's easy to reproduce, and it's easy to iterate on. It's easy to repeat it over and over and over again. So

88

00:18:26.970 --> 00:18:39.910

Justin Florentine (Besu): once we go through that we go through the next layer and the next layer, and then you're doing. You know your shadow forking things, your shadow, forking testnets, etc., etc. You've done it so many times that it's very rare that

89

00:18:40.120 --> 00:18:48.909

Justin Florentine (Besu): You found a bug that halts the network right? Or you've you found like a really weird edge case or a

90

00:18:49.680 --> 00:18:59.779

Justin Florentine (Besu): spec issue like, those are the things that we really worry about is like it's the spec wrong. And in most of those cases, apart from like a proper test network.

91

00:19:00.270 --> 00:19:08.309

Justin Florentine (Besu): it doesn't matter if it's a shadow fork fine, you can. Just, you know. Let the Shadow Fork die and start over with your adjusted specs.

92

00:19:08.660 --> 00:19:10.520

Justin Florentine (Besu): So

93

00:19:11.040 --> 00:19:25.209

Justin Florentine (Besu): it it's it's a good question, because the stakes are really really high. And I think the answer is redundancy, like the answer is, we're just re really, really redundant. It's really, really important. And we're kind of fortunate that an open source community that we can just do that. I think if we were in a commercial sense.

94

00:19:25.210 --> 00:19:41.199

Justin Florentine (Besu): the cost benefit analysis would not fly past some being calendar and accounting, and they would tell you like, Oh, you're you're this is too redundant. You're you're you're you're wasting money, etc., etc. But you know it's it's live. There's no, there's no better way to do that. Redundancy is important.

95

00:19:44.450 --> 00:19:52.059

Matt Nelson: Yeah, redundancy is key, and we we definitely have redundancy among these test test nets, where we have

96

00:19:52.400 --> 00:19:56.709

Matt Nelson: tens of clients of each combination size, shape.

97

00:19:57.350 --> 00:20:26.729

Matt Nelson:  very cool. I wanted to take 2 s to see if we have any questions that folks on the phone wanted to provide. Again, the context for today's discussion is testing and implementation for the hardworks. And as I mentioned the intro, some of you might not have been on by that point that Justin has been working on the basic team to get cancun developed, tested and shipped. So anything in that context. But you also have a core developer on the phone. Anything else you wanna chat about is is likely fair game here.

98

00:20:26.900 --> 00:20:30.229

Matt Nelson: If not, I will keep probing, probing with questions.

99

00:20:31.790 --> 00:20:41.889

Matt Nelson: also the eips that we discussed where there was potentially some ambiguity. I know we went through the eips themselves in depth. So you have more detailed spec questions that I was either Iffy

100

00:20:41.990 --> 00:20:43.090

Matt Nelson: or

101

00:20:43.340 --> 00:20:49.679

Matt Nelson:  you know we we have Justin here who worked on a lot of those specs themselves, so that'd be great.

102

00:20:50.060 --> 00:20:51.170

Matt Nelson: But hmm.

103

00:21:00.000 --> 00:21:02.929

Matt Nelson: alright! I got another question for you while we wait here, Justin.

104

00:21:03.090 --> 00:21:09.810

Matt Nelson: tell us about the girly fork post mortem. We very briefly walked through what happened.

105

00:21:10.010 --> 00:21:20.889

Matt Nelson: As far as the prism client bug and potential nethermine client bug for the last fork. But we actually didn't have enough time for a proper analysis, because girly was forked, maybe

106

00:21:21.040 --> 00:21:31.449

Matt Nelson: a handful of hours before our call last time. So do you have anything that any lessons learned or anything that would be, you know, interesting in context of this discussion that we learned from that hard fork, how we're going to apply those.

107

00:21:32.010 --> 00:21:33.520

Yeah. So

108

00:21:34.040 --> 00:21:43.039

Justin Florentine (Besu): now you forgive me. I was on vacation this weekend and the day before, so I didn't go too deep into the root causes of what we ran into. But this was

109

00:21:43.390 --> 00:21:57.420

Justin Florentine (Besu): the important question is is like, Oh, how do we miss something right here, because I just went on and on about redundancy, and we're still found something that broke on a test net with, you know, one of our one of our clients. That was interesting, because

110

00:21:59.030 --> 00:22:12.330

Justin Florentine (Besu): it it probably should have showed up in one of the shadow forks. We're a little suspicious as to why it didn't, but the nature of the bug is such that it would not have showed up in hive. It would not have showed up in

111

00:22:12.670 --> 00:22:17.539

Justin Florentine (Besu): The. You know, the ephemeral networks that we're spinning up because my understanding of what

112

00:22:17.800 --> 00:22:27.489

Justin Florentine (Besu): prism ran into had to do with historical receipts that were older than a certain point. So so they had a situation where.

113

00:22:28.100 --> 00:22:53.959

Justin Florentine (Besu): when in our smaller networks, we were creating the 4.4 dancoon, a little too close to origin or to to Genesis. And so we weren't hitting certain paths in the code that are a little bit more relevant when you have a much longer distance there. So that was kind of a a weird edge case that we're gonna have to really consider how we cover with our tests because you don't wanna have like.

114

00:22:54.200 --> 00:23:14.820

Justin Florentine (Besu): you don't wanna have days between your genesis for your test network when you actually hit the thing that you want to test, you know. So we'll have to take a closer look at that, and just a and and realize well, was it a length of the blockchain issue that exacerbated this, or was it a state sized thing, or whatever? It's probably the former, because it had to do the

115

00:23:14.820 --> 00:23:25.940

Justin Florentine (Besu): something with the receipts. I don't. I don't quite recall off the top of my head. But but it would be a new test scenario that we would have to introduce there. To specifically target that.

116

00:23:26.340 --> 00:23:27.949

Justin Florentine (Besu): so yeah, interesting.

117

00:23:28.900 --> 00:23:52.459

Matt Nelson: Yeah, I think that's the the prism. One was especially interesting because this it was sort of an optional spec implementation detail around the way we index blobs, which is part of the reason why we have redundancy, because the spec is intentionally left a little bit ambiguous in certain instances to allow client diversity to solve some of these problems just by virtue of

118

00:23:52.560 --> 00:24:09.070

Matt Nelson: whomever is thinking over the spec in a slightly different way. We do this in many ways by the ways that the Cls will handle Mev blocks or block production in general. Some of them have the. They've implemented different kind of censorship, resistant overrides for these things that have different timeouts.

119

00:24:09.070 --> 00:24:23.010

Matt Nelson: different ways that they kick in and then different present present presentations to the user. But the I remember specifically at a a core Devs call where we said, no, we do not want to codify this in the spec intentionally, to give

120

00:24:23.030 --> 00:24:36.379

Matt Nelson: openness, so that the clients can be a little different than each other, because if we had an extremely rigid spec, they would all look and act identical, which means they would all present potential spec challenges in the same way.

121

00:24:36.630 --> 00:24:49.389

Matt Nelson: so in this instance, it can create kind of issues where clients will have a little bit, maybe too ambiguous or loose interpretation of the spec, and can cause issues. But in in general it's a kind of a good thing for sure.

122

00:24:51.970 --> 00:25:10.409

Matt Nelson: Okay, we got a question in chat from Tj. The question is, what is the process for a new client like ref to be considered legit enough to be an official client. Couldn't any random group of individuals create an execution or consensus client? Obviously we don't want pro poor performing clients to drag down the rest of the teams during the upgrade process.

123

00:25:10.490 --> 00:25:14.150

Matt Nelson: So I'll give a 2 s version. Then I'll kick to Justin

124

00:25:14.750 --> 00:25:16.549

Matt Nelson: In many instances

125

00:25:16.720 --> 00:25:33.780

Matt Nelson: we kind of just say, Hey, if you can keep up with the status quo welcome, we're seeing new consensus and execution clients like ethereum js and ref, and on the consensus side we have a new client called Grandine, which actually produced an invalid block

126

00:25:33.780 --> 00:26:02.259

Matt Nelson: that tripped up a lot of the other clients on on some of these forks. So in reality, I think we we test against a common denominator. We don't really have a criteria. So if people are willing to run your node that can follow the tip of the chain. You basically have a good enough client in many instances. That's an oversimplification. But in the case of things like theium js, and Ref, I think that's absolutely the case. And they continue to push their performance, profile and functionality profile

127

00:26:02.270 --> 00:26:03.460

Matt Nelson: over time.

128

00:26:03.640 --> 00:26:15.049

Matt Nelson: But both of them went through the Fork Transition, and if users want to run that client they can, and if they want to validate with that client they can. I think that a lot of the kind of voice on the core development

129

00:26:15.060 --> 00:26:20.580

Matt Nelson: space comes from expertise, but also representation right? I think the Get team

130

00:26:20.900 --> 00:26:38.890

Matt Nelson: has a big portion of voice on core devs because they represent on the execution layer at least a hugely outsized proportion of users that they want to protect and make sure, have the best functionality available to them, running the client, etc. So that's my take, Justin. Maybe you have a different or more nuanced one there.

131

00:26:39.300 --> 00:26:58.009

Justin Florentine (Besu): II think, might take us fairly similar to that. To add a little nuance, I would say that if you're producing a new client. The good news is that there is a ton of stuff that you can do up to, and prior to actually proposing new blocks on the chain.

132

00:26:58.050 --> 00:27:06.709

Justin Florentine (Besu): and you can do that stuff on all the networks and all the way up to May that and you can measure your performance, and you can measure it against all the other clients

133

00:27:07.040 --> 00:27:09.380

Justin Florentine (Besu): to go further than I think

134

00:27:09.630 --> 00:27:37.869

Justin Florentine (Besu): you know. If I was, if I was endorsing a new client I would wanna see all of that. I would wanna see all of them participating the way Matt suggested, and keeping it up with it. And then I would wanna see all of the lower level testing things that I spoke about before executed and including proposals and including and creating new things for the blockchain. So that includes results from Edm tests running all the state tests and making sure that it was super. You know.

135

00:27:38.540 --> 00:27:52.639

Justin Florentine (Besu): it's making sure that it had been exposed to all the weird edge cases and stuff that we bake into those tests as well as checking its performance to making it to to make sure that you know some of the attacks that we're worried about that would occupy gas

136

00:27:52.940 --> 00:28:00.010

Justin Florentine (Besu): are analogous to all the other clients right like you don't want to find a gas mispricing issue that's specific to one client.

137

00:28:00.110 --> 00:28:06.779

Justin Florentine (Besu): And so there's there's a lot of tests that you would want to do. And then, you know, you've got the proposal stuff, too.

138

00:28:07.070 --> 00:28:15.020

Justin Florentine (Besu): so how do you? How do you create blocks? How long does it take you to create a block all all that stuff. So I think that

139

00:28:15.510 --> 00:28:27.719

Justin Florentine (Besu): it would be. I don't think it would be that difficult to say like, Oh, this client, yeah, this is a legit client like we have all the processes in place we have where we're used to running hive tests against everything. The results are quite comparable.

140

00:28:27.720 --> 00:28:51.289

Justin Florentine (Besu): I know that there's been a discussion ongoing recently about performance. And how do we compare it? And I don't think that's gonna be too hard. We're gonna come to a consensus pretty quickly on that, it'll be documented and reproducible across clients. The hive tests are really good test bed for that. And so once. That's once that exists. New clients. You just gotta you will draw bar somewhere. Says you must be this high to ride on main net and

141

00:28:51.560 --> 00:29:13.619

Justin Florentine (Besu): there's nothing we can do about it. If they don't, you know. We'll we'll there'll be a social consensus. Maybe some social prefer like, hey? Don't run that because it actually, when your turn comes up to propose, might not get a legit block, etc. So that's just common evaluating open source software and waiting for that to shake out through social consensus.

142

00:29:14.920 --> 00:29:24.849

Matt Nelson: There's also the gas limit to consider in this sense that and other clients can shift the gas limit, but they cannot outright change it. So if you're

143

00:29:25.320 --> 00:29:27.469

Matt Nelson: if your client cannot

144

00:29:27.490 --> 00:29:34.789

Matt Nelson: execute roughly 30 million gas and 12 s, you probably aren't going to be able to keep up with the chain.

145

00:29:34.990 --> 00:29:47.210

Matt Nelson: That's why I think it's kind of fun when you look at the python specs to like run those like State tests against the execution layer specifications, which is just like python pseudocode. And you can see, like, you know.

146

00:29:47.380 --> 00:30:01.539

Matt Nelson: the base suit and Geth numbers in the number of milliseconds. And then you have, like, 15 s for some of the python code. So I think that it's kind of like, yeah to Justin's point, what what works, as far as social consensus is concerned, what works in terms of the specs, and how

147

00:30:01.750 --> 00:30:07.930

Matt Nelson: performant is your client to follow the chain, and the rest is kind of social stuff.

148

00:30:08.790 --> 00:30:14.250

Matt Nelson: That was a great question. again, please feel free to post more in the chat.

149

00:30:15.240 --> 00:30:24.919

Matt Nelson:  okay, let me take a look back at my question list here. 1 s.

150

00:30:32.100 --> 00:30:34.250

Matt Nelson: Okay, here's a good one, I think.

151

00:30:35.080 --> 00:30:46.619

Matt Nelson: So you've mentioned kind of late breaking spec changes how, as as kind of developers with these late breaking spec changes.

152

00:30:48.250 --> 00:30:56.559

Matt Nelson: how are we kind of squaring up the modifications that need to be done at the implementation level versus

153

00:30:56.720 --> 00:31:00.599

Matt Nelson: the specification. If if you're following my question.

154

00:31:02.590 --> 00:31:08.070

Matt Nelson: is it all the clients think this is this is silly, let's change it, or

155

00:31:08.240 --> 00:31:18.639

Hmm, because I think that a lot of the problem that ethereum had before the merge was that the specification was primarily driven by developments in the Geth client in general

156

00:31:19.140 --> 00:31:21.529

but now we have this different landscape.

157

00:31:22.040 --> 00:31:27.420

Matt Nelson: How does that kind of how does that new blurred line make sense to to a core? Dev!

158

00:31:29.120 --> 00:31:42.350

Justin Florentine (Besu):  That's one of those questions that's hard to answer. Unless you've got your elbows in the code like, if you're super deep in the code, you can maybe in like 5, 10 min of discussion with your team. You can say like

159

00:31:42.370 --> 00:31:46.010

Justin Florentine (Besu): that would really be hard for us to implement, and

160

00:31:46.020 --> 00:31:54.399

Justin Florentine (Besu): it would be by hard. That generally means we would have to refactor a bunch of stuff to actually implement that change in the spec.

161

00:31:54.650 --> 00:32:01.450

Justin Florentine (Besu): And here's the additional testing and performance impacts. We would want to be clear about

162

00:32:01.650 --> 00:32:06.989

Justin Florentine (Besu): before before submitting that back and merging that in and changing the way that that works.

163

00:32:07.910 --> 00:32:11.060

Justin Florentine (Besu): Does that answer your question, Matt.

164

00:32:12.110 --> 00:32:30.599

Matt Nelson: Yeah, it does. I think that. Yeah, the in my mind, it's like we have a spec that is intended to be somewhat inflexible. like, where and how does our the teams choose to kind of bend or push on that spec versus bend or push on the client? Because I think you're right like

165

00:32:31.400 --> 00:32:49.190

Matt Nelson: us. Let's think of it this way, right? Us saying we need to refactor a bunch of code in order to meet a spec that is actually well designed versus the opposite, you know. We don't wanna say that that's not a fun thing to say. No, of course not, of course not.

166

00:32:49.420 --> 00:32:55.209

Justin Florentine (Besu): Cause. Cause if you don't, if if that's not really a concern, or the answer is almost always sure, go ahead, change it.

167

00:32:55.420 --> 00:33:06.590

Justin Florentine (Besu): It's like there's a there's a a question being kicked around about renaming some fields for block gas use or data gas user. I forget what I forget what it was.

168

00:33:06.860 --> 00:33:15.930

Justin Florentine (Besu): it's it's kind of just a you know, naming things is important, and it's hard. But once that decision is made, it's not gonna be that difficult for anybody anything to get changed, you know.

169

00:33:16.660 --> 00:33:18.420

Matt Nelson: Yeah. Got you there?

170

00:33:19.130 --> 00:33:38.089

Matt Nelson: This is more, a little bit of a more fun question for you. Where do you see, the community kind of coming into play and coming into this testing and implementation process. Because I think we, we have a lot of the initial stuff is done in the core devs kind of sphere a little bit

171

00:33:38.520 --> 00:33:59.579

Matt Nelson: you know. Hush! As we work things out. But then we slowly start to layer in more and more groups, like institutional stakers, communities like East Staker, and even regular users who are like, I'm experiencing some weird client bugs on girly. You know what kind of fun or cooperative or kind of uplifting

172

00:33:59.720 --> 00:34:10.080

Matt Nelson: interactions did you see there? And where do you see? Also incentives to have community members get involved in that process just as individuals or small groups.

173

00:34:11.510 --> 00:34:22.420

Justin Florentine (Besu): I wish I had a better answer for that. But the the real incentive is social. if you're if you're nice enough to be running a test set client.

174

00:34:22.440 --> 00:34:25.179

Justin Florentine (Besu):  Then

175

00:34:25.210 --> 00:34:49.049

Justin Florentine (Besu): you got some interesting people to talk about it with on on the various discords, etc. You're gonna be much further down the stack. You're gonna be, probably in more contact with core developers things like that. It's really really awesome. When our users choose to do that, they have no financial incentives to do that. They're using compute resources to S to sync up with.

176

00:34:49.080 --> 00:35:06.339

Justin Florentine (Besu): you know, girly, which is a very large chain. You know, moving forward. That's gonna get sunset. No, it'll still be. Blobs will be there. They're they're still gonna have the same requirements. Hopefully, that main that's gonna have. The state management will be a lot, you know, a lot smaller. But they're still gonna have to, you know.

177

00:35:06.580 --> 00:35:17.410

Justin Florentine (Besu): They're gonna have to consider their their, their hardware resources. And that's a nice donation. for them to be giving back to the community. And

178

00:35:18.120 --> 00:35:45.129

Justin Florentine (Besu): it's it's it's super super helpful. Right? Because we get we get into these. We get. We get blinders on sometimes, and it's hard to. It's hard to avoid that. And it's very easy for us to automate all of our testing and deployment of our nodes, etc., etc., and they all have the exact same spec out machine, because they're all running in azure aws or whatever. And you don't really have the hey? I tried to run girly on my old laptop the other day.

179

00:35:45.400 --> 00:35:57.809

Justin Florentine (Besu): And this is some weird stuff that happened like, Oh, well, that's cool. Cause that would actually happen on main net. And that's actually kind of hard for us to to to simulate and reproduce. So it's so it's like super helpful.

180

00:35:58.430 --> 00:36:12.190

Justin Florentine (Besu): I would love to see like a stupid girlie tricks award, or a Po app, or something like that, where, like somebody just tried to do something weird and silly on on our test nuts. And you know, build a little bit of a community about around that.

181

00:36:12.430 --> 00:36:23.550

Justin Florentine (Besu): You know, people try to run doom on refrigerators. Why aren't we trying to run, you know, ethereum on on weird stuff? I think that would be that would be fun. That would be that would be kind of

182

00:36:24.220 --> 00:36:29.170

Justin Florentine (Besu): I don't know. I think it would be fun. I think our community would think it would be fun, but that's and that's really what would matter.

183

00:36:29.500 --> 00:36:33.610

Matt Nelson: Zoom in the ebm. Sure.

184

00:36:34.100 --> 00:36:42.690

Justin Florentine (Besu):  so yeah, if you're if you're running, if you're already going through the trouble to to to solo, stake and run things from home. Throw another network on there.

185

00:36:44.810 --> 00:36:59.439

Matt Nelson: Awesome, awesome. Alright. We got another question in the chat here from Tj. This one's a fun one, because I think it's actually very revealing. Declined teams start working on implementing future eips, even if they're not necessarily going to be included in upcoming upgrades.

186

00:36:59.590 --> 00:37:05.740

Matt Nelson: or even sorry parentheses, or even eips that do not have popular support from other teams.

187

00:37:07.300 --> 00:37:23.959

Matt Nelson: As an example, euf could be shipped in Prague, Electra, and the Ap. Specs are mostly written will. Most of the clients have done preliminary work to implement eips such as those needed for ef in advance for do client teams have code written or possible future apips that aren't up for inclusion.

188

00:37:26.230 --> 00:37:28.320

Justin Florentine (Besu): Yeah. Yep.

189

00:37:28.460 --> 00:37:32.080

Justin Florentine (Besu): Yup, eip is very often

190

00:37:32.100 --> 00:37:38.740

Justin Florentine (Besu): coexist alongside implementations that are that are in flight, and that's a good thing, because it kind of reinforms the design.

191

00:37:39.140 --> 00:37:52.980

Justin Florentine (Besu): you know, there there will. Eips are notoriously hard to finalize. In fact, there's you could probably go through the eip database and find tons of them that are, you know, marked still in draft, or whatever that have already been implemented and shipped like

192

00:37:53.410 --> 00:38:02.649

Justin Florentine (Besu): they're probably just marked incorrectly, but I think the spirit of that is not too inaccurate. Like, I said, these things keep changing over over time.

193

00:38:02.960 --> 00:38:15.280

Justin Florentine (Besu):  so yeah, so yeah, we we will campaign for something. I mean, that's a good way to put pressure on eip, you think is good for the good, for the network is to implement it. I mean being able to say.

194

00:38:15.660 --> 00:38:25.860

Justin Florentine (Besu): You know, basic team thinks we should ship eof. By the way, we've already got it implemented is a it's a great way to to to to get what you want out there.

195

00:38:26.520 --> 00:38:32.439

Justin Florentine (Besu): so so yeah, that that definitely happens, I think what's really important, though, is that

196

00:38:33.100 --> 00:38:37.519

Justin Florentine (Besu): prototyping an eip as it's being developed

197

00:38:37.620 --> 00:38:46.610

Justin Florentine (Besu): gives clients first, mover advantages right? So if everybody that's designing eips prototypes them all in Geth.

198

00:38:46.890 --> 00:38:51.369

Justin Florentine (Besu): Geth will always be able to say we already have an implementation for this, we should ship it.

199

00:38:51.810 --> 00:39:13.519

Justin Florentine (Besu): and that's a really subtle way to encourage client diversity is to not just run other clients, but prototype on other clients. Give that first mover advantage to other clients so that they can say, Yeah, we've got this implemented already. Have other clients coming to that team to say like, how did you implement it? What tips do you have.

200

00:39:13.520 --> 00:39:22.640

Justin Florentine (Besu): you know? Did did implementing Verko go better for you because you already have a bonsai implementation in your your data store.

201

00:39:23.060 --> 00:39:25.760

Justin Florentine (Besu): So yeah, it's

202

00:39:26.240 --> 00:39:31.989

Justin Florentine (Besu): it's a, it's a, it's a bit of a weird flywheel feedback loop that happens there.

203

00:39:38.510 --> 00:39:43.150

Matt Nelson: Awesome. Great question. Yeah, I'd have to add as well that I think

204

00:39:44.280 --> 00:39:51.329

Matt Nelson: it's not necessarily an addition. But the punctuation is that like clients will work on pick these Ips that they're passionate about, or that their users are.

205

00:39:51.400 --> 00:40:08.879

Matt Nelson: you know, products are passionate about and like they will just go ahead and build stuff in order to push things through. We've discussed this many times on this this series is that in order to champion stuff, you need a little more than just a good idea. So client teams will absolutely work with external contributors and other groups to

206

00:40:09.390 --> 00:40:15.190

Matt Nelson: oftentimes provide advice so that others can implement the Ips that they're working on.

207

00:40:15.370 --> 00:40:16.390

Matt Nelson: And

208

00:40:16.810 --> 00:40:33.280

Matt Nelson: a lot of the times these reference implementations end up in Geth. But sometimes you've have users proposing Ips that have expertise in Java, or, you know, Dotnet, or go, or different languages. And so we end up with a variety of reference implementations. But those are definitely super interesting here.

209

00:40:33.780 --> 00:40:39.220

Matt Nelson:  Any other questions in the chat? I

210

00:40:39.490 --> 00:40:48.420

Matt Nelson: mostly it will just keep going down my list here. We also have about 20 min left. It could probably duck out a little bit early.

211

00:40:49.670 --> 00:40:53.990

Matt Nelson: give another chance to the peanut gallery.

212

00:41:01.920 --> 00:41:07.220

Matt Nelson: Okay, here's a good question, Justin. What has been the most challenging.

213

00:41:07.430 --> 00:41:16.639

Matt Nelson: frustrating, or rewarding part of your participation in this testing experience. So you can either go good good news, bad news, or

214

00:41:16.850 --> 00:41:17.820

Matt Nelson: we're capable.

215

00:41:22.110 --> 00:41:26.179

Justin Florentine (Besu): Well, testing in particular.

216

00:41:26.320 --> 00:41:30.250

Justin Florentine (Besu): you know, it's weird cause I don't really consider myself a tester like

217

00:41:30.390 --> 00:41:37.249

Justin Florentine (Besu): I consider myself sort of like the customer for the tests. I think that

218

00:41:37.390 --> 00:41:47.899

Justin Florentine (Besu): you know, for as far as like a really rewarding technical experience was seeing, seeing the utility of the hive tests improve and the quality of them improve over time.

219

00:41:48.530 --> 00:42:11.780

Justin Florentine (Besu): There is quickly becoming kind of the de facto way to send somebody a bug report. I think I could see like in a year from now there'll be very, very easy ways to do that, and to say, like, Hey, I think this is an issue in in the bug. Okay, where's the hive test right? And you can run it. You can isolate your clients. It spins up kind of a small little docker network.

220

00:42:11.890 --> 00:42:22.760

Justin Florentine (Besu): It's it's it's a really good piece of infrastructure. I think that has a lot of legs, and it's so easy to automate your testing and so many. so many different ways, but also so many different layers of precision.

221

00:42:23.220 --> 00:42:40.870

Justin Florentine (Besu): And and and that's really the the general purpose. One. II think the hive tests really are the right balance between granularity, repeatability, modularity, being able to kind of point it easily at different clients and different networks, and how they're set up.

222

00:42:40.870 --> 00:43:09.319

Justin Florentine (Besu): and that's not to belittle or say like, Oh, the state tests are not less important. There's those are actually just as important because they do the opposite right. They isolate just the evm, and it's very easy to get a very specific trace back out of it when you run into an issue. And it's very easy to inject very specific pieces of evm coder transactions, or whatever, and point them in a specific piece of state. There's there's other types of testing that have a very different granularity that are equally useful for totally different reasons.

223

00:43:09.320 --> 00:43:17.950

Justin Florentine (Besu): So that I think the diversity of scope of the testing techniques that we have is really really cool to see and really cool, to see how

224

00:43:18.560 --> 00:43:25.899

Justin Florentine (Besu): how, is just making all the developers better. I know. I mean, I should just speak for myself, like I think it's making me better as a developer because

225

00:43:25.900 --> 00:43:49.729

Justin Florentine (Besu): I get frustrated personally when I have to load too big of a model into my head, and I'm always looking for ways to narrow it down. Narrow down, narrow it down just because you know the way that I think I don't have a great memory. I don't have as much experiences of a lot of other people, and it takes time for me to build up those sense memories and intuitions about where to find code or where a problem could be, etc. So, having these different granularities of tests.

226

00:43:49.730 --> 00:44:05.370

Justin Florentine (Besu): speed me up as a developer, and I think speed us up as a team. So I think that's that's been really, really cool to see. I think that the the the shadow forking. I still don't I? Still, sometimes I think I'm like, Oh, man, that's such a good idea like when we started doing that for for main net. I was like.

227

00:44:05.440 --> 00:44:26.370

Justin Florentine (Besu): man. That's that's really a great, great test. Cause it's it's like doing it live. It's like just running it test. It's it's testing in production. And so that's that's really cool. I was really excited when we when we did that it gave me a ton more confidence that the merge was gonna go. Well, and I'm glad to see that it's gotten baked into every hard forks since. And it's just standard operating procedure for us now.

228

00:44:27.260 --> 00:44:35.270

Matt Nelson: So yeah, yeah. I think that the even since the for even since I started working on network upgrades with the merge

229

00:44:35.710 --> 00:44:42.820

Matt Nelson: Shanghai and then cancun have increasingly become more automated and more procedural, which I think is

230

00:44:44.050 --> 00:44:59.020

Matt Nelson: like nice to see for a decentralized community, because there's so many procedures and things like you said, that have to be redundant. That could be missed. That could be gaps in testing that, I think. As we kind of get used to working with 10 client teams and more. And all this other stuff.

231

00:44:59.350 --> 00:45:07.050

Matt Nelson: That it's it's very heartening to see the progress. Okay, let's see, here

232

00:45:08.260 --> 00:45:09.380

Matt Nelson: we have

233

00:45:09.840 --> 00:45:15.570

Matt Nelson: again a little over 15 min left. We don't need to use all that time. I mostly have more

234

00:45:15.960 --> 00:45:21.390

Matt Nelson: basic questions, if we have any, from the group here for Justin.

235

00:45:28.190 --> 00:45:40.209

Matt Nelson:  alright, you gave us a good, a good, a good one there, like a a positive note. What about what do you find the most difficult or frustrating part of

236

00:45:40.250 --> 00:45:43.540

Matt Nelson: working on these network upgrades as a developer?

237

00:45:45.660 --> 00:45:58.190

Justin Florentine (Besu):  I mean, there's always that fear right? The stakes are the stakes are just really high. I don't know that that's specific to the network upgrades or how they're gonna work. It's just working on like a

238

00:45:58.200 --> 00:46:05.930

Justin Florentine (Besu): you know. What are we at? Like a 200, some 1 billion dollar network? I don't even know where we're at as far as Market Cap these days.

239

00:46:05.980 --> 00:46:08.540

Justin Florentine (Besu): you know. So that kind of sucks.

240

00:46:08.930 --> 00:46:10.110

Justin Florentine (Besu): I think.

241

00:46:13.000 --> 00:46:15.200

Justin Florentine (Besu): Gosh! I don't know

242

00:46:16.020 --> 00:46:18.090

Justin Florentine (Besu): what what could be better.

243

00:46:19.580 --> 00:46:30.849

Justin Florentine (Besu): I you know that. So so the the diversity of testing is, you know, probably cuts both ways right? It's like, well, where? Where you think you find an issue. Okay, can you send me a test cool. In what format?

244

00:46:31.020 --> 00:46:39.829

Justin Florentine (Besu): Okay? Well, where do I find the tools to do that? I don't know. So now I need to like walk you through. I got a report the other day that was

245

00:46:40.330 --> 00:47:07.450

Justin Florentine (Besu): it was. I forget exactly the test format, but it was a it was a I think it was a more guest specific test format, and it hadn't been translated into something that bases evm tool could execute. And that was, that's kind of a that's friction, right? That's friction in your testing process, friction in your development process where you need to be like. Oh, now I need to go learn this other tool. So I can convert its output into something that I can run through my tool before I can even start doing my job and analyzing.

246

00:47:07.450 --> 00:47:23.489

Justin Florentine (Besu): What is the problem that you found analyzing the traces? Comparing the traces like comparing traces between Cvm's is still not something that we've really standardized. And II think that's a problem. I think that's a huge area for improvement, because the point of traces is to compare them like

247

00:47:23.890 --> 00:47:46.239

Justin Florentine (Besu): it's it's you. Can. You can. The trace will tell you a lot about your evm in isolation, and that's fine. But so did the code that you were looking at. It's even more useful to be able to say like very quickly, I don't have to read and understand Guest code. I can pull that trace. And literally I should be able to do a diff on it like I should be able to compare them with a tool that says like, Hey, here's where they diverge, and how they diverge, and why?

248

00:47:46.290 --> 00:47:47.640

Justin Florentine (Besu): So

249

00:47:48.560 --> 00:47:58.100

Justin Florentine (Besu): getting that next step? To really make that process better, I think, would be, would be a big improvement. That's not. That's not fun to do when you're when you're actually trying to chase down an Evm bug.

250

00:48:00.260 --> 00:48:10.199

Matt Nelson: And we all know those notorious knowing you. That is as bad as it probably will be on this call. So that's fine.

251

00:48:10.840 --> 00:48:19.710

Matt Nelson:  okay, folks, I'll give one more chance for any last questions. If not, I think we'll give everybody an additional 10 min back to their day.

252

00:48:30.510 --> 00:48:33.990

Matt Nelson: Anything coming in chat? We'll give you just a second.

253

00:48:45.710 --> 00:49:02.920

Matt Nelson: Okay, it looks like that, is it? Thank you again, Justin, for coming and bringing your perspective. It's super valuable. This course, I'm hoping will remain relevant for tons of time to come, because we will continue to upgrade Aetherium, and we will continue to have to test those upgrades.

254

00:49:03.400 --> 00:49:05.589

Matt Nelson: So thank you again, Justin, for your time.

255

00:49:05.760 --> 00:49:07.460

Justin Florentine (Besu): Happy to be here. It was a blast.

256

00:49:07.850 --> 00:49:15.429

Matt Nelson: Alright! Some housekeeping folks, I believe. Our next lecture is the

257

00:49:17.470 --> 00:49:26.949

Matt Nelson: February, January 30 first, so we have nothing this week, and then we will return on the Wednesday, the 30 first at 90'clock, Pacific time, or 12 Eastern.

258

00:49:27.040 --> 00:49:37.369

Matt Nelson: and we will have some guest speakers the following week. There. This will be our last lecture. So make sure to mark your calendars. We'll be discussing

259

00:49:38.270 --> 00:49:51.529

Matt Nelson: maybe some more technical details on how to kind of get literally involved in testing on fork to networks so we might even take some look at early code or girly clients. I might have to sync up some early client for

260

00:49:51.570 --> 00:49:52.770

Matt Nelson: the lecture

261

00:49:53.060 --> 00:50:05.599

Matt Nelson: either way. Thank you again for your time today. And if you have any additional questions that pop in your mind that you want answered later on. I know where to find Justin. So please feel free to shoot me an email, and we can get an answer.

262

00:50:05.910 --> 00:50:07.279

Matt Nelson: But thanks again. Everybody

263

00:50:09.160 --> 00:50:10.000

Justin Florentine (Besu): see folks.