QuillHub
QuillHub
EN

Concourse: release process, CVEs, Vault credential-manager metrics

19:443 спикера
19:38
длительность
10
главы
3
спикеры
7
Термины
Concourse releasessecurity CVEsrelease pipelinescredential metricsVault integrationcontainerd/Garden

Спикеры

3
Спикер A46%30 реплик
Спикер B39%33 реплики
Спикер C15%12 реплик

Основные идеи

5
  • Recent CVEs were handled; security releases require reliable, possibly faster backup release mechanisms.
  • It would be beneficial to allow trusted contributors to trigger or assist with patch/security releases when maintainers are unavailable.
  • Credential-manager integrations (Vault, CredHub) are hitting request-rate limits; metrics for successful/failed requests would help troubleshoot and tune behavior.
  • Automating or clarifying parts of the release pipelines (e.g., separating security vs major releases) would reduce friction.
  • Containerd/Garden CVEs should be tracked for inclusion in the 8.4 release milestone.

Важные моменты

5
  • Proposal to enable a fail-safe/redundant process for triggering security/patch releases when maintainers are unavailable.
  • Suggestion to add metrics for credential managers: count of successful requests, failures, and total requests.
  • Agreement that metrics are a reasonable starting point; health endpoint could surface credential-manager failures.
  • Discussion to possibly rejigger pipelines to make release types (security vs major) clearer and easier to trigger.
  • Action item to create/open issues for remaining container-related CVEs and consider adding them to 8.4 milestone.

Решения

3
  • Start with adding metrics for credential managers (successful/failed/total requests) as a first step.
  • Treat metrics as the initial, generic approach; tune per credential manager as needed and consider piping metrics to health endpoints.
  • Approach to pipeline changes: rejigger/section pipelines to clarify security vs major releases rather than large automation work.

Задачи

6
  • Tag Taylor to write the low-priority blog post for the 8.3 releaseTaylor
  • Open an issue to the community proposing metrics for credential managers (successful, failed, total requests)
  • Consider adding remaining container-related CVEs (containerd/Garden) to the 8.4 milestone
  • Rejigger/consolidate pipelines to make release types (security vs major/ minor) clearer and easier to trigger
  • Discuss and align a backup strategy allowing the core group to trigger security/patch releases when maintainers are unavailable

Термины

7
CVE — Common Vulnerabilities and Exposures — publicly disclosed cybersecurity vulnerabilities tracked by ID.
Concourse — continuous integration/continuous delivery (CI/CD) system discussed throughout the meeting.
Pipeline — automated CI/CD workflow in Concourse that builds, tests, and releases software.
Credential manager — component that provides credentials/secrets to Concourse (examples: Vault, CredHub).
Vault / CredHub — external secret stores/credential managers used by Concourse integrations.

Основное содержание

10 глав · 19:38
1:29

1. Introductions and Small Talk

Спикер A1:29

Hello, Derek.

Спикер B1:30

Hey folks, how you doing? Hi, Derek.

Спикер A1:34

Great. Middle of the week, so.

Спикер B1:39

So that's the good thing. It's always kind of a question for me, is like, is that the good thing or is that the—

Спикер A1:44

Always the good thing. We're over the hill of the week, so.

Спикер B1:49

Okay. Oh yeah, you're at the end of your day. That's fair. That's fair. You're over the hill. I'm like, I don't know. I'm at the beginning of that hill, right? Like I'm reaching for that precipice.

Спикер A1:59

Just before the apex. So, the last climb.

Спикер B2:03

Yeah.

Спикер A2:03

We are falling from the hill.

Спикер B2:07

Yeah, just don't fall off on the other side. Yeah, yeah, that's a good one.

2:12

2. Attendance, Agenda, and CVE Updates

Спикер B2:12

So for what it's worth, Taylor's not gonna be able to join us this week. I think he's traveling.

So as a result, is there any— do you expect anyone else from your side today? First question.

Спикер A2:36

I'm— I don't know. Me too.

Спикер C2:39

I don't think anyone's going to join.

Спикер B2:43

Okay, awesome. Is there anything you all want to discuss today? I know that we had a few CVE releases, right, for the wave of things that flowed through the last few weeks here.

But other than that, I think it's been kind of a business as usual moment in general. So, nothing major that Taylor wanted to leave me with to bring up before he went on vacation.

Спикер A3:14

Okay. Yeah, thanks for fixing that.

Спикер B3:21

Of course. Nobody likes CVEs. Nobody likes getting yelled at about CVEs. So, across the board, make them go away.

3:30

3. Release Process, CI Pipelines, and Release Automation

Спикер A3:30

From my side, something that comes to mind. So there are two things. One of them I saw in the docs that we forgot about the blog post that was for July or something like that planned for the 8.3 release. But it's not that problematic since we're still on 8.3, I guess. So I just tagged Taylor, but it's low priority. I guess it's not something at the moment that needs any attention. It's just an article anyway, so whenever it comes out, it could be beneficial for other people. That was his main reason for why he wanted that article written.

So that's one thing. The other thing, while you were doing the 8.3.3, something I noticed—so I'm—we're still not quite sure with Ivan around the process of releasing, and I more or less looked into the CI repository of Concourse and played around locally with some stuff to try and understand how the pipeline works with forks of the repositories.

But anyway, I think if you guys need any assistance for such chaotic moments, so for example, you're doing the 8.3.3, you were just before shipping it. And for example, you wanted something to pass and then just ship it. Yeah. I think that we can more or less go through a process together at some point and maybe show us what, so how the things happen so we can assist you with such activities.

Because it seemed, for me, it seemed like I just needed to trigger a single build then and everything would have been released for the community. And that was it. I'm more or less aware that it depends on the release, right? So if it's a bug fix, it's different. If it's a minor or major, different. But yeah, more or less for me, I think it would be beneficial to understand that part, and if needed, we can assist as well in such activities, or at least I can do it from my side whenever you need that.

It again, it's not something urgent, but for such security vulnerabilities, it might be beneficial at some point to, yeah.

Спикер B5:48

Yeah. Containers, this likely means two things we'll need to spend time on. At least two things come to mind, right? Which is access to manage the pipelines.

And the first part is automating pieces of the release process that are not currently automatic. I think that's a very small piece right now. Like, the things that aren't automatic are like, um, there's some bits related to releasing an updated Helm chart after cutting a release. But, like broadly speaking, cutting the Concourse release, like on the Concourse and BOSH releases side, plumbs through pretty automatically.

So there may not be much to do there, but I wanna put that in like kind of that order. That sounds reasonable. I'll talk with Taylor about that. I agree that for the core group that's kind of been developing here, it makes sense to make that possible, right, for us to trigger, especially for security-based releases and sensitivity.

Спикер A7:14

Yeah, I think we can even align on some process. So it might not be, let's say, it's sort of a backup. Let's say, let's do it like a backup strategy.

So if the, for example, the maintainers can do the official releases, but if something is required as assistance, it would be beneficial. Like, for example, when there is a 9.8 vulnerability for TCP and we want a fix merged, it could happen that you guys are, let's say, it happens to be late at night for you or something like that until the pipeline gets through.

So it might be just a single job trigger or something like that before we deliver such a release to everyone to consume. And at some point, that might be beneficial. It's not something that's urgent in my opinion, but it's something that we can think on how to improve.

So let's say for the general part, the process might still continue to be the same. So you and Teodor can align and release based on the discussions in these steering meetings.

But other than that, for such patch releases or security releases, we can have sort of a fail-safe scenario. For example, Teodor is vacationing now, and if you're also vacationing, and we need to patch a security fix in Concourse, it's, yeah, it could be worth it to investigate as an opportunity.

Yeah, that's the main idea.

Спикер B8:41

Makes sense. I don't think Taylor will have any fundamental concerns there. We might just want to, like, rejigger the pipelines a little bit to make it so it's more obvious and clear, and maybe even sectioned off about which, you know, parts cut, like security releases versus like a major release, right? Or things of that nature, which is not particularly a large bit of work, but just a thoughtful piece of work.

Okay, that sounds great. Anything else y'all wanted to talk about?

9:15

4. Vault/Credential Manager Performance & Metrics Proposal

Спикер C9:15

Yeah, maybe one question from my side. Since in the past months, we were seeing some, let's say, performance problems with Vault integration. We are hitting max requests per second with our integrations, meaning we have one namespace with a lot of pipelines and teams connected to that namespace. Nice.

We work around it with extending the request per second limit. However, I was thinking about, would you benefit from extending OJIC for credential managers to give us information like metrics, how many successful requests are done to the credential manager and how many fails? I'm asking about that because, in the whole process of collecting the data, we tightly work with both colleagues, and it's pretty hard to collect all the information and, yeah, create a good plan for improvement.

Спикер B10:25

So just so I understand, the way that failure is manifesting, I'm assuming, is in failed pipeline, like orange builds essentially, like failing to contact or rejected requests.

Спикер C10:42

Our stakeholders contact us, and we investigate the failing job. So, yeah.

Спикер B10:48

So, while metrics would be interesting in identifying that problem faster, right, I'm not necessarily opposed to that, by any means.

Is there also a concern that you're trying to address about how many requests Concourse is making to, like, achieve these things? Like, making that tunable, or do you mostly just want the signal side of this a bit faster?

Спикер C11:15

Both of them. Both of them. It gives us, yeah, uh, perspective to know how many requests can we make to the vault and also how many of them are failing. Yeah.

Спикер B11:28

Sure.

Спикер C11:29

Uh, maybe just an idea. It's not a requirement here. Yeah. Just throwing.

Спикер B11:34

Yeah, that doesn't sound unreasonable, particularly the second thing, right? Of like, how can we tune it to or allow operators to tune depending on what makes sense there, right? Because you also don't wanna, you know, the things that come to mind is like, well, you can cache or things like that, but at the same time, you maybe don't wanna change that behavior outright, right, for folks at non-scale of like, okay, how long do these things cache for, and then I change and change a secret, and you don't want it to necessarily have a delay of something.

So, some things to think about there that directionally sounds reasonable to me on both sides. Yeah. And I think on the, you'll have to treat it differently per credential manager, at least with respect to the tuning piece, the metric side, I mean. Yeah, that should be generic enough, right? Failed to do X and all that jazz. Makes sense.

Спикер C12:39

Yeah, it only addressed the metrics part, not—we are not going for any cache functionality. Yeah, we cannot benefit from caching here. Our stakeholders want credentials to be.

Спикер B12:56

Instantaneous.

Спикер C12:56

Right?

Спикер B12:56

It's like, okay, well, yep. Yep.

Спикер A13:01

I think just for the metric side, then maybe, so it would be applicable for every credential manager. So it might be more of an epic and have dedicated issues for the different.

13:10

5. Implementation Considerations: Tuning, Metrics, and Health Integration

Спикер A13:10

And also the thing to consider, I'm not sure, I think that the load var abstraction, so the additional credential manager abstraction, it doesn't conflict with the interface of Vault. For example, having CredHub and Vault both configured, but also the metrics would be—they cannot be credential manager requests or something. They need to be for the specific abstraction, right?

Спикер B13:35

You got to type it. Yeah, you got to type it or something like that. I don't—you got to figure out. Um,

Спикер A13:40

Yeah, based on the enablement, but I think since there are different interfaces in Concourse, that shouldn't also be that much of a problem. Just each of them should have separate metrics, I guess.

Спикер B13:52

Yep, how about that? No, super reasonable.

Спикер A13:59

And, uh, so just for the metrics, though, you're talking about, uh, successful and failed requests, and, uh, more or less on the request side, so nothing, uh, else?

Спикер C14:10

Uh, yeah, this is my top of mind, but I guess during the implementation, it may appear that some other events might be beneficial to be logged.

But for the current investigation, we pretty much rely on the failing and the successful ones and the total count of the requests.

So,

Спикер A14:42

Okay.

Спикер C14:42

I guess I can open an issue to the community to see. Yeah.

Спикер A14:52

Yep, sure.

Спикер C15:02

Thanks.

Спикер A15:07

Based on your idea, I would say that if we even have that information, we might at least configure a metric for the health endpoint as well.

So, if you see end times failed to request the credential manager, it should— it could also be visible in the health.

So, you can also even, for example, your CredHub isn't working or request CredHub piled up or failed. That might also be beneficial for operators then.

But yeah, the metrics is a good starting point. Then maybe just pipe it towards that as well would be—

Спикер B15:42

Yeah, you gotta start there, and then, yeah, yeah.

Спикер A15:47

Yeah, just thinking out loud to some possibilities that might come with it.

Спикер B15:54

Of course, of course.

Спикер A15:56

Because I would assume you wouldn't want to concur with the constantly failing request to the credential manager, and that being, that would be a reasonable hot event to figure out. Yeah, cool.

Спикер B16:08

Yes.

16:15

6. Vulnerabilities and 8.4 Release Planning

Спикер A16:15

Awesome. I just want one thing. So, the two vulnerabilities that were left for container, the end, what was the other one for?

Спикер B16:24

It was Garden. No, was it Garden?

Спикер A16:27

Maybe Garden. So those were not because they need a minor release, right? Because they wouldn't change the file system.

So the code for Concourse or, yeah. Okay, yeah, yeah, I was just asking because, uh, so then they would be targeted for 8.4 release, right?

Спикер B16:51

I think so. I have to look at it again. I've admittedly looked at way too many CVEs, uh, recently across too many different software projects, so I'm not gonna pretend that I remember exactly.

Спикер A17:05

Um, I think that they actually planned 8.4 for the beginning of November, but now that with the patch releases and a lot of PRs that piled up, I think he decided to postpone it a bit.

So I guess maybe that's something, if we don't have an issue for, it might be worth adding to the milestone for 8.4 from my perspective, because even though the other more critical vulnerabilities were fixed, I would still— Yeah, I think that those would need to be handled here for 8.4.

Спикер B17:44

Yeah, that makes sense.

17:47

7. Issue Tracking and Concourse CVE Job

Спикер A17:47

I'm not sure if you have an issue for them or if they're in the security and quality section here. I saw that there's a job in Concourse that you track CVs in.

Спикер B18:01

Yes.

Спикер A18:02

ScanRC or something like that.

Спикер B18:05

Yeah.

Спикер A18:07

Yeah. So that doesn't open issues. So we need to make open issues for them. Okay.

18:18

8. Dependency Updates and Release Bumps (containerd)

Спикер B18:18

But so, containerd, I don't think it had a— so did they stop releasing? Is your point they didn't? They cut a new minor on their side?

Спикер A18:33

No, no, my point was, if we see those CVEs and there's a containerd version that fixes them, maybe we need to update to it in 8.4. Uh, in 8.4. Yeah, sure.

Спикер B18:48

Yep, yep. No, we'll certainly take whatever the newest is. I mean, the reality is we take the newest of each of those things, right? Unless, I don't know, there's—we do auto-bump those, so we'll take the latest of what's available for them right now.

The next release we cut, the question just becomes how many CVEs do they contain, right, at the end of the day.

Спикер A19:13

Yeah, I guess when we bump them, this job will let us know if there's still—

Спикер B19:18

It will.

Спикер A19:19

Yeah,

Спикер B19:19

It will. Yep.

Спикер A19:21

Okay,

Спикер B19:22

Awesome.

19:23

9. Meeting Wrap-up

Спикер A19:23

Thanks, thanks for the info.

Спикер B19:26

Of course. Awesome. Well, anything else?

Спикер A19:30

No, not from my side.

Спикер C19:31

From my side too. Thanks, Eric.

19:34

10. Farewells

Спикер B19:34

Yeah, thank you both. Have a great rest of your week.

Спикер A19:37

Bye-bye.

Спикер C19:37

You too.

Спикер B19:38

Bye.