Scorers, Multi‑output Support and Merge/Entry‑Point Review (meeting)

36:503 speakers
36:43
duration
20
chapters
3
speakers
8
Tasks
multi‑outputscorersmodel scoringscikit‑learnentry pointsmerge conflictdataflow input layer

Speakers

3
Speaker A78%63 turns
Speaker B10%24 turns
Speaker C11%38 turns

Key ideas

6
  • Current custom multi‑output scorer implementation is unreliable and differs from scikit‑learn behavior.
  • Prefer using scikit‑learn scorers for multi‑output support and simplify base code.
  • Scores and models are too tightly coupled; scorer should accept an explicit feature config to reduce introspection.
  • Rename the high‑level 'accuracy' function to 'score' (agreed P0 before next release).
  • Remove WIP/non‑scikit multi‑output changes; keep only scikit‑based modifications to merge cleanly.

Key highlights

5
  • Agreed to use scikit scores for multi‑output rather than current custom aggregation.
  • Decision to add a config parameter to scorers specifying which predicted feature to score (to decouple from model internals).
  • Consensus to rename the top‑level accuracy API to 'score'; marked P0 for next release.
  • Contributor asked to remove WIP marker and notify when ready to review/merge.
  • Troubleshooting session concluded that the operation is registered but import/load may fail silently—recommend inspecting exception origin and entry‑point load.

Decisions

4
  • Use scikit‑learn scorer implementations for multi‑output support instead of current custom aggregation.
  • Add a config parameter to scorers to specify which predicted feature to score (decoupling scorer from model internals).
  • Rename the high‑level 'accuracy' function to 'score' (priority P0 for next release).
  • Keep multi‑output changes only in the scikit‑based scorer files and back out other custom multi‑output modifications.

Action items

8
  • Remove non‑scikit multi‑output changes and keep only scikit‑based scorer modifications (cleanup).
  • Remove WIP marker and comment/status when ready for review so reviewers know it's mergeable.
  • Finish cleanup of the patch and ping the reviewer on Gitter with branch details for review.author (Hashim implied)
  • Ensure scorer instantiation uses explicit feature config (change mctx.parent.config to self.parent.config in implementations).
  • Send Gitter message with the branch/patch for troubleshooting the input layer entry‑point.Sudhanshu

Terms

6
multi‑output — Models that predict multiple outputs/features for each input (e.g., multiple labels or regressands).
scikit (scikit‑learn) — A widely used Python machine learning library providing standard scorer functions and metrics.
scorer / scoring — Functions or objects that compute a numeric quality metric (accuracy, error, etc.) for model predictions.
introspection (model introspection) — Code inspecting model configuration or predict signature at runtime to determine behavior.
entry points (Python packaging) — A mechanism in Python packages (setup.py/entry_points) to register plugin components that can be discovered at runtime.

Main content

20 chapters · 36:43
0:01

1. Opening and meeting agenda

Speaker A0:01

Okay. Okay, I think I understand what you're saying. Yeah. All right, well, I'm excited to see. Let's see. So, all right, so what do we need to review that today?

Speaker B0:19

Uh, no, uh, there are some issues with that, so I want to like get some inputs on that.

Speaker A0:27

Okay. Is it in a branch somewhere? Or is it in a pull request? Or where can we find it?

Speaker B0:45

It's actually in a branch, like in a random branch, which I haven't pushed.

Speaker A0:50

Okay, yeah, let's make sure I have access to that while I'm doing this here.

And then, um, we can, uh, and you could maybe get ready to present it too.

Speaker B1:01

Yep.

1:02

2. Code review and merge conflicts

Speaker A1:02

Great. Oh my gosh, what the hell? This is like the world's most ridiculous looking merge conflict.

Okay, come on now. So, what is happening here? Wow. All right, let's see what the hell happened here. That's like such a weird way for it to format this.

Okay, and this is totally wrong. All right, so what the hell was in here before? I don't know. All right, well, good enough.

All right. Oh, God. What is this? OK, so— oh, yeah. We have location stuff here. So what was this? All we needed to do here was just say directory to location.

OK. Okay, let's check this out, see if everything is sane. So, and was this pretty much done? Like, did we need anything else on this, or were we just— why hadn't we merged this? Do you remember? I know we did it all in the meeting, so.

Speaker C4:33

Um, I don't know. Uh, you gave me the patch to try, and I tried the patch, and there was some error. Then you gave it to Saptar.

Speaker A4:43

Yeah, that's right. We needed to finish the— oh yeah, okay. So we needed to finish the thing where— okay, this is not going to get done in this meeting then.

So because that was— okay, I'm going to have to do this afterwards because this was the— oh man, yeah, okay. I think you tried it, Saksham tried it.

So now we're back to me trying it for the second time. Okay. All right. Yeah, this thing was— it was the git adder instead of adder thing, right?

Speaker C5:22

Yeah.

Speaker A5:22

Yeah. Okay. Damn. All right. Okay. Damn. Not as easy as we thought.

Okay. So I was hoping we were about to just merge this.

Speaker C5:30

Okay. What?

Speaker A5:32

I

Speaker C5:35

said, I thought you had tried it, so I didn't say anything.

Speaker A5:38

Oh, okay. Oops. Sorry. Yeah, no, I haven't had a chance.

Okay, I was thinking maybe we just needed to rebase the location stuff. I forgot about this. Um, okay, so let's table this. Um, and then, because yeah, so let's see.

6:02

3. Timeline and action items (GSOC wrap-up)

Speaker A6:02

Yeah, so let's do—let's meet Friday again because I think, what, we're wrapping up GSOC next week, is it?

Speaker C6:13

I'm sorry?

Speaker A6:15

I'm sorry, is GSOC wrapping up next week? Let's see, August 16th.

Speaker B6:24

I think it's next to next week, 16th to 23rd.

Speaker A6:33

Okay. Yeah, let's make sure we get enough input on your guys's stuff if you have any. So we got basically 3 weeks.

Um, okay, um, you guys are all on track, um, so I don't think we'll have any major hiccups here. But, you know, I want to make sure that we get everybody, you know, anything that they need. So I'll make sure to get that done today, Hashim.

Um, and if I don't get it done, ping me in the morning. Um, I will, and we'll make sure to get it done ASAP within the next 2 days. I think this, once I get this demo, I should be pretty more free up here.

So things have been, things have gotten extremely hectic. So let's see, we're doing some—

Speaker C7:19

Sure, no problem.

Speaker A7:20

Modernization thing. Okay, so, all right, let me put in a reminder on my calendar. So this is

7:30

4. Introduction and initial questions

Speaker C7:30

Uh,

Speaker A7:31

It will config setters and getters.

Speaker C7:33

I also wanted to ask something.

Speaker A7:35

Yeah,

7:36

5. Problems with multi-output scorers

Speaker C7:36

Regarding multi-output.

Speaker A7:38

Yeah,

Speaker C7:40

So, uh, the implementation of multi-output support on the scorers that we implemented ourselves in BFFML isn't really convincing, and I don't think, uh, that's the way to go.

Like, we are using means of accuracies of different predictions to, you know, calculate the multi-output accuracy. And if we are using the same scorer from scikit implementations, they give out different scores than ours do.

And I can't really follow their multi-output implementation. They're using different scorers inside scorers, and it was getting messy.

I was, uh, going to ask if, uh, you know, we could consider just using the scikit scores for—I think that's a good idea.

Speaker A8:37

I think that's a good idea. Um, yeah, I wanted to talk about that too because I'm not convinced. I'm not, I'm not convinced on the way that we have this right now. I'm concerned—or the scores in general—I'm concerned that we don't specify what the feature is to score. I realize, and we're just inspecting the model's predict.

9:10

6. Proposal to decouple scorer from model (feature parameter)

Speaker A9:10

I think we maybe need to specify the feature to the scorer itself because I don't know, it seems like this is going to be prone to issues, especially if now we have multi-output and now features is all a list. Like, what if a model wants to do something else with features, right? Like, we should really—we should probably separate the connection between the model and the scorer a little more by specifying to the scorer what we want it to score, right? What predicted feature we want it to score, right?

Does that sound reasonable to you guys? Or do you think—I mean, Sutanshu, you have more experience with this than the rest of us, so do you think that we should leave it with the introspection for any strong reason, or are you—do you think it—are you open to the idea of adding a config parameter to the model scores? You know, just—do you guys know what I'm saying? I think we did it with—

Speaker B10:20

I'm not sure how things would work out.

Speaker A10:25

Okay. So, Vim examples, quick start.

Speaker C10:29

I'm not sure what's the idea behind, you know, specifying the feature.

Speaker A10:45

So, you guys can see this, right? Yeah. Okay. All right, so I'm concerned that we're grabbing— I'm concerned that we're grabbing predict. Like, I'm concerned that we're grabbing from the parent model, from the model config. For, you know, for what essentially amounts to something that I think is— that would change, right? Basically.

So what happens with your patch, Hashim, is, um, if we— where was that? The multi-output, right? So your changes sort of make it apparent that this— that the scores and the models are too intertwined right now. Right? Because you now begin to inspect the model can predict to see if it's features or if it's a singular feature. And now all this extra code just explodes.

So basically, the code in each scorer is too tied into the models is what I'm saying. Right. I believe, right? Because right now you, you— we, we, this, this, the change that you made resulted in a big change, not a big change, but like it resulted in quite a few lines added in the score, right? And it showed that the functionality of the models is now too deeply tied to the functionality of the scores or the scores of functionality of the scores is the implementation of the scores is too deeply tied to the implementation of the model. Since we're introspecting the predict config of the model.

So what I'm suggesting is that we do something like this, essentially, right? Where we just make this self.parent. And so now the only dependency on the model becomes the prediction. And so we can ask, you know, we had the scorer do, um, let's see, like, this is where it's like, okay, well, maybe we need to say label equals, uh, model.predict. And yeah, so I don't know, predict or feature, we should probably just say feature.

So I don't know, this is where it's like, that's not— now I'm not sure what terminology to use here, but you see how now we end up with— so now it's like the functionality isn't so tied together, right? So if you had— so now if you end up with a multi-output model, you could verify, right? Like feature only works like— let's see.

So we're also gonna— there's also gonna be some— we need to do some more input verification on the config dictionaries and stuff, but we can— so safe— let's safely assume that, you know, if we say feature then and somebody provides features, right, a list, then it would kick them out and not let them instantiate that classification accuracy. Right? So they would have to use a multi-output-specific classification accuracy score, right?

And that, you know, that minimizes the amount of, you know, the possibilities for issues with this scorer here, right? Because it's no longer tied into every single model and having to deal with, oh, well, what if this model does this, right? Well, it doesn't matter because we're just calling predict and then we're looking at the feature data.

Does that— do you guys see what I'm saying? Is this making any sense?

Speaker B14:56

I think it's a good idea.

Speaker A14:58

Okay.

Speaker B14:58

Yeah.

Speaker A14:59

Okay. So we just need to decide, and then this also sort of, while we're talking about the scores.

Speaker C15:03

Yeah, that makes sense.

15:04

7. Renaming accuracy to score

Speaker A15:04

Okay, cool. Great. So while we're talking about the scores, you wanted to talk about renaming the accuracy function as something else. So what we— and you'd suggested score. And it suggested score. Oh, and then the theme thing, but that's not a big deal.

So basically, we'd looked at a few themes, and I like that one that you'd posted, Hashim. So yeah, go check that out, Suchandra, if you get a chance.

So yeah, so score or evaluate. I'm not, I'm not, I'm not in favor of evaluate personally because I think we already have the record evaluation function, and I'm not sure if I love that name for that either. Um, but score seems fine to me.

Um, what do you— is there a better name than score? Should we change it to score? Should we not change it to score? Um, what do you think, Sutanshu? I'd like to hear your input on this because you've—

Speaker B16:09

Uh, I think we can keep it score as well.

Speaker A16:17

Okay. Because, yeah, the high level, and I mean, we can always, we can, yeah, let's see. So, high level scores with regard to different scores selected. Let's see. I didn't understand that last sentence. These could also be a resultant value of error. What is that? Like, there's like if it throws an error or if there's some kind of actual error returned?

Speaker C16:45

Basically, there are algorithms that calculate the error, right? The error of your model, and they give out the value of the error that's present in the prediction.

Speaker A16:57

I see what you're saying. Yeah, I think score is probably a better name here then. Yeah, I mean, score is what we've implemented the method to be already, so that I don't think it strays too far from that. And that's the term that Scikit uses. So let's, let's go ahead and change this.

17:18

8. Implementation plan, reviews, and priorities

Speaker A17:18

Do you want to edit the issue so that it says that we're going to change this to score? Thank you. Let's see. I can, I can just edit it. So let's see. Change, rename high level. Let's just rename the high-level function unless we should rename the whole thing to, I guess, scorer. We could, we could, let's rename the high-level function right now, and then we can focus on everything else later because we could move the accuracy directory to scorer, and that would keep the command line flags consistent.

So, but it doesn't really matter too much right now. I think the main interface to this is the high-level function. So rename high-level accuracy to score. Does that sound good? Yeah. Cool. All right. Okay. So let's make sure this— yeah, let's get this done here.

So P0, we definitely want that before the next release. So, okay, great. Um, oh, and it's already pinned. All right, um, so let's try to get that done. Um, anything else? So, so review— reviewing this I can't do right now, um, not enough time, but I'll try to get to it and I'll try to do the, uh, the— I think this— yeah, let, let me know. I think there's some comments and stuff in here. I thought I saw some stuff that looked like it wasn't cleaned up. Like, how— how—

Speaker C19:11

Yeah, the comments were just in the, uh, the native implementations, and, uh, I thought I'd ask before making—

Speaker A19:20

Yeah, okay, that makes sense.

Speaker C19:22

I'll remove it. It's the last comment. It's the last commit.

Speaker A19:26

That sounds great. Okay. So then, what is your path forward on this then?

19:32

9. Scoring multi-output and scikit-scorer changes

Speaker A19:32

So, are you just going to score multiple— are you going to score— are you going to show multiple scores for multiple features that were predicted? Or are you going to do some sort of aggregate still?

Speaker C19:48

Or— I'm sorry, can you repeat that?

Speaker A19:53

So, is your plan still to— is your plan to show the score now that we've decided that— and correct me if I'm wrong, but it sounded like you're saying let's not try to implement, or let's either implement separately or not implement right now. Scores for multi-output, right?

Speaker C20:25

I said that we can just use the scikit-scores and not have this maybe implemented into this one.

Speaker A20:36

Okay. And with the scikit-scores, do those— are you going to score— are you going to use different scikit-scores to score different features? Are you going to use the same scorer to score different features? Are you going to use one scorer to score multiple features? Or how is that going to work?

Speaker C20:57

Yeah, we're using one scorer for multiple features.

Speaker A21:02

Okay. And is that—does that require changes to the scikit-scores?

Speaker C21:10

In just a second, let me move—your voice is breaking up. I think it's my internet. All right.

Speaker A21:20

Does that require multiple—

Speaker C21:22

Can you repeat that?

Speaker A21:23

Does that require changes to multiple or to the scorers to make them support multi-output?

Speaker C21:32

No, that doesn't require changes to multiple scorers. I just changed the base scikit-scorer file.

Speaker A21:42

Okay, great. Okay, cool. Um, and then we did—we decided that we need to add, uh, some kind of config parameter to the scores. Um, we don't—

Speaker C21:54

Okay, uh, just a second. I'm a little bit confused.

Speaker A21:57

Yeah.

Speaker C21:58

Uh, do you mean, uh, that if we want to use multiple scores in the same scoring session or something?

Speaker A22:10

Uh, I mean, I was just trying to figure out what you were going to do, but it doesn't really matter because I'll see it once you do it.

So, uh, it's, it's, uh, I'll just review it when it— when we get time for it. I was just trying to understand what your plan was, but it sounds like you have a plan, and I'll see it when it's ready here.

So, and, and I— it sounds right. So I was just trying to, trying to ask if you were gonna— if you were planning on using multiple scorers. But it sounds like you're going to use the scikit-scores, and you're going to use one score to do the scoring on all of the predicted features, right?

Speaker C22:50

Yeah, that's what I'm doing so far.

Speaker A22:53

Okay, and that sounds good. So, okay, so yeah, sorry, continue.

Speaker C23:03

I meant that's, uh, what I've done already.

Speaker A23:06

Okay. Oh, that's what you've done in here. So, or—

Speaker C23:10

Yeah.

Speaker A23:11

Okay. Oh, and here's the changes.

Okay. No, this is the— okay, so wait, so where's the score changes?

Okay, there we go. All right, so we're already good to go.

So what— okay, so you're gonna remove that comment. Are you going to leave the changes to the— are you going to leave the multi-output changes in the base scores, the ones that aren't a part of the scikit-scores?

Speaker C23:46

No, I'm going to remove the ones other than scikit-scores.

Speaker A23:50

That sounds probably like a good idea there.

23:50

10. Tests, cleanup, and merge readiness

Speaker A23:50

Okay, so model type in Regressor tests.

Okay. So, okay, you've got the test cases. Let's see.

Okay, great. Fantastic.

Okay. So I'll give you feedback on this.

So this is no longer a work in progress. This is basically you're ready to merge this if it's good.

Speaker C24:14

Yeah. Except for backing out those changes. Yeah. The ones you just mentioned earlier.

Speaker A24:21

Okay, great. So could you—

Speaker C24:23

I think we are also using—sorry.

Speaker A24:28

Just remove the WIP and comment that when you're ready, so that I know exactly. All right. Because that way I'm sure.

Speaker C24:43

Cool, cool. I think we are also— we were also using, yeah, the config predict other than multi-output as well.

Speaker A24:55

Config predict. Okay. You mean in the scores?

Speaker C25:09

I think so. Because that's why—or maybe in the models. Okay, I'm not sure. I'll just do it.

Speaker A25:19

Yeah, just let me know.

Yeah, why don't you do— finish the cleanup and then let me know the current status, you know, and, and, you know, comment and ping me on Gitter too so that I see. Because I kind of tried to clean up my GitHub notifications, and then we're doing this thing where we're trying to move to GitHub, and so they have automation that's not entirely perfect in Intel, and it keeps adding me to everybody's repos.

And so then I get a million notifications. So sorry if I'm not seeing things, but

Speaker C25:56

all

25:56

11. Request for feedback / Introduction

Speaker A25:56

Right. So, Sudhanshu, did you have something that you wanted feedback on right now?

Speaker B26:04

Yes, I want to share my screen.

Speaker A26:06

Okay, great.

26:21

12. Input layer design (matrix conversion)

Speaker B26:21

Yeah. So I have actually created this operation input layer, which will, uh, take all the data points and, uh, the length of the source, and it will actually create it into a matrix.

So I have created a matrix here. So it will take all the inputs and add it into the list.

Speaker A26:46

Okay.

Speaker B26:46

And the way I'm doing here is in the seed. I'm actually providing all the values to the input layer data point.

Speaker A26:57

So, why are we doing this? I don't quite understand.

Speaker B27:05

Uh, so, basically, we are— so, the operation, the cleanup operations actually work on matrix data, right? So for that, I have created this input layer which will take all the features, feature data points, and it will convert it into a matrix so that we can perform operations on them.

27:25

13. Operation instantiation error & entry point registration

Speaker B27:25

So, for example, like, if you have this operation, it actually takes this input data in the form of a list of lists. And the error I'm getting here is that it is showing that it is not instantiable.

Speaker A27:47

Uh, okay, so, okay, so I would say if it's showing the current error, I would say it looks like an entry point registration issue.

So, you need to make sure that the operation has a name associated with it, and then you need to make sure that the entry point is registered and you've installed the package.

Speaker B28:15

So, I have actually done both of them.

So, in the entry_points.txt, so this was actually what—

Speaker A28:22

Perfect.

Speaker B28:22

Running the scale gave us. So, I have added the input layer here. And also, the package was installed.

28:34

14. Entry point registry checks and dfml commands

Speaker A28:34

Okay, great. So, okay, impressions data. Okay, and it's still not showing up. So the next thing I usually do is I do dfml— I do, uh, I would do dfml space service space dev space entry points space list and then dfml.operation.

So dfml service dev entry points list. Okay. dfml.operation. And this will just show you all the registry operations so that you can verify.

Okay, entry points list. Oh, no S at the end. Ah. Operations, just on the end of operations, no S. Leave it on entry points. No, it's entry points with an S and operations without an S. It's, it's dfml.operation.

Okay, so input layer does exist and it is registered. So, all right, let's see. So let's see the data flow that's causing this.

29:53

15. Data flow, logs and traceback inspection

Speaker B29:53

So this was the create data flow, and this was the data flow.

Speaker A30:11

Put layer. Okay, put layer. So we have a one-operation data flow, and the flow is— okay, so the flow says get everything from seed. Okay. Um, and it says operation not instantiable. So, uh, can you run it again? With the,

Speaker B30:55

Uh, the exception, uh, merge Pinot Flow to actually see what is the output. So this was the merge thing.

Speaker A31:07

Okay, so you're running this. Okay.

Speaker B31:14

Yes.

Speaker A31:21

All right, can we see the full traceback? Input layer was not found in blank. Can we scroll up on the logs?

Okay. So we have a data flow source, and we have a data flow. Player is not instantiable. Player was not found in— okay, um, and okay, uh, I don't know what to tell you right now. If it's showing up in— I mean, I would say what I would start doing right now is going into the code that throws that exception and say, well, why is it throwing that exception?

But I think that— yeah, let's open that.

32:59

16. Operations import issue and code debugging

Speaker B32:59

So actually, I looked into it. This part here, this operation implementation is actually an empty dictionary. Like, I tried to print it.

Speaker A33:19

Dataflow operations, I know.

Speaker B33:19

And it was empty, I guess.

Speaker A33:27

Okay, so it attempts to instantiate it. So I think it's having a problem instantiating that operation. Oh, I think I ran into this recently. So let's see.

OK, it's not instantiable. So what does your file that has the operations in it look like? Oh, wait. So yeah, so that's none, but it's going to try to load it from the entry point. So— or let's see, not instantiable was not found in— Blank.

Okay, so let's see. OK, so I would go to your operations.py.

Speaker B34:15

So this is the operation.

Speaker A34:19

Okay.

Speaker B34:21

And this is the context.

Speaker A34:25

Okay, okay. So, I think, I think this is due to an error in the load call of the entry point. I think I ran into this the other day. It doesn't throw the right exception, or like it silently throws some exception if you're importing— if, if something happens when it tries to import this file. Um, can you run other operations from this file?

35:17

17. Tests, GitHub patch and next steps

Speaker B35:17

Yes, like the tests which I have written actually run fine.

Speaker A35:25

Okay. I wonder, can you send me this patch and stuff?

Speaker B35:30

It's already up in my GitHub repo in a random branch.

Speaker A35:36

Okay, great. Can you send me a Gitter message with that, and I'll take a look at it? Great, great. All right, and then I'm gonna— do we have any final things, or else I'm gonna drop off now?

Speaker B35:52

I know that's it from my side.

Speaker A35:54

All right, so let's meet Friday. Yeah.

Speaker C35:58

I'd like to make sure of something.

Speaker A36:02

Yeah.

36:04

18. Scorer instantiation and feature passing

Speaker C36:04

So basically, what I'm going to do with the scorers is just make it so that we are, you know, while we are instantiating the scorers, we just send in the features like we instantiate models, the predict features.

Speaker A36:20

Yep.

Speaker C36:21

Right?

Speaker A36:21

Yep. And it should just be a change from— yeah, it should just be a change from mctx.parent.config to self.parent.config when you're in the implementation. Yeah, yeah.

Speaker C36:33

Yeah.

Speaker A36:33

Sweet.

36:34

19. Clarification and apology

Speaker C36:34

Okay. I misunderstood earlier. Sorry about that.

Speaker A36:37

No worries. No worries. Yeah, confusing. We have a lot of stuff going on. So, all right.

36:43

20. Meeting wrap-up and goodbye

Speaker C36:43

All right.

Speaker A36:43

Thanks, guys. Have a good one. Bye.