Move auth to Grafana instead of puppet

This ticket tracks moving authentication out of puppet for Grafana. In #40102 (closed) we created a shared account for people accessing Grafana. This doesn't really scale and it would be nice if we could create accounts in Grafana directly.

added Backlog label

added Icebox label and removed Backlog label

added Grafana label

marked this issue as related to #40102 (closed)

marked this issue as related to #40948 (closed)

so we have another such case that happened today which shows the current system doesn't work at all.

but instead of adding "yet another authentication system", i'd like to consider the possibility of using LDAP as a source of truth for this. if my memory is correct, we have a webPassword field in there that's currently unused.. we could just use that as a ... well... web password, that could get reused elsewhere.

it requires a bit more investigation, but it's the way i would go forward here instead of delegating authentication to grafana.

that doesn't mean, of course, that grafana would be down when ldap goes down, quite the contrary: we would push a "htpasswd" style file through the regular ud-ldap sync mechanisms so we'd be resilient against that type of failure.

i have just checked and there is a web-passwords file with (md5-salted-hashed passwords, meeeh) that we could use. i think i could manage to fix the hash and ship it to solve this problem; that way people could just set that password in their LDAP account to get access.

this, however, would mean that anyone with an LDAP account would get access to the thing.

@cohosh @dgoulet @hiro @meskio how confidential is the stuff in Grafana2/Prometheus2? who should get access to this or not?

is tor-internal (aka "LDAP" + contractors + some other outsiders right now, more or less) an acceptable compromise?

assigned to @anarcat

added Backlog label and removed Icebox label

We could probably still bridge it with LDAP, that said, it just might be a little less resilient. Something to be researched as well, I guess.

moving this in a new thread... i looked at Grafana's support for LDAP, and it's pretty basic, in the opensource version:

https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-authentication/ldap/

configure a bind, bind user, etc. not great. you can however, configure multiple servers, so we could have HA on the LDAP server side. we don't have that right now, so that's kind of a bigger undertaking.

there's this cool feature where Grafana automatically and perdiodically syncs LDAP users, presumably to survive LDAP outages ... but of course that's in the Entreprise version! arghl.

moving this to a new thread:

@meskio said

Actually I would love to have public access to grafana, so people can see our dashboards. Is there any possible future where we can expose some grafana dashboards publicly?

@sready replied: You can then move to grafana.tpo, which has guest access, no?

So yeah, grafana.tpo has a guest account. But I think what I would like to have here is a single server to make it easier to manage redundancy, retention, and deployments. Having two servers right now is pretty confusing, exactly for that kind of situation.

I think what this all tells me is that we need some dashboards (and data!) to be public and some dashboards (and data!) to be private. We need all of the prometheus HTTP server to be completely private, and generally should be shielded by a strong authentication system (possibly basic HTTP auth with a strong password accessible only to TPA, if not blocked at the IP level as well).

Then inside Grafana, I'm not sure how to protect the metrics, other than give tor-internal or a subset access to Grafana, and trust they will do the right thing. To let some dashboard be made public, that would imply not having HTTP-level authentication, and instead delegate authentication to Grafana, which goes back to the question of how to keep high availability in that scenario.

added Icebox label and removed Backlog label

added Next label and removed Icebox label

marked this issue as related to #40755 (closed)

added Icebox label and removed Next label

marked this issue as related to #41280

changed the description

we touched this ticket in #41280 and there was a clear tension in that ticket between some metrics that should absolutely stay private and some that we would like to have public. @meskio, for example, said that all their exporters are public, and @dgoulet and @hiro strongly opposed any move to make some of their metrics public at all.

it does seem like "tor-internal" is sufficient access control there, however... @meskio, is that what you had in mind by "public", or did you really mean "no account necessary"?

oh and in that ticket @micah also pointed out it might be possible to expose some dashboards securely, see #41280 (comment 2929757)

it does seem like "tor-internal" is sufficient access control there, however... @meskio, is that what you had in mind by "public", or did you really mean "no account necessary"?

My idea was "no account necessary". I can live with "tor-internal" for now, and keep telling people to use our exporters directly. Is nice to be able to point people to our dashboards, but we've lived without it up to now.

Another thing I'll try to do is to collect the most useful bits of our exporters into the metrics team to be included somehow in the metrics website. But that requires more work and time than changing dashboards in grafana.

meskio commented on a discussion: #40124 (comment 2955815)

it does seem like "tor-internal" is sufficient access control there, however... @meskio, is that what you had in mind by "public", or did you really mean "no account necessary"?

My idea was "no account necessary". I can live with "tor-internal" for now, and keep telling people to use our exporters directly. Is nice to be able to point people to our dashboards, but we've lived without it up to now.

Another thing I'll try to do is to collect the most useful bits of our exporters into the metrics team to be included somehow in the metrics website. But that requires more work and time than changing dashboards in grafana.

One thing we could do would be to have a secondary Prom/Grafana server that would federate from the private one, and extract only a specific allow-list of metrics that are considered sane. This one would be fully public, could have a lower scrape interval and much longer retention policy.

@micah would that resolve the concerns you have outline about disclosing a subset of metrics in #41280 (comment 2927299)? You comment was:

...

On 2023-10-17 17:40:26, meskio wrote:

For private data, I wouldn't try to 'redact' them in some way, because that will be difficult to get right and too dangerous to get wrong.

We have two different desires for Grafana:

1. internal, private metrics that we need to have access to for our internal development, debugging and monitoring of the network -- the data the exporters put out there, and the graphs that are generated should never be available to the public, they are sensitive and contain information that would be A+ bad if it leaked out. This is the data that @hiro and @dgoulet are talking about.
2. a set of metrics that we want to use to build public facing graphs, based on data that is not private, and can be accessed by anyone. We want visibility into the graphs, and we want people to be able to build their own representations of that data and consume that data for their own research purposes. this is what @meskio is talking about

Is it impossible to satisfy both of those criteria in one grafana? Can you say more about what "fully public" would mean for the public grafana? Would that mean that anyone on the internet could go and construct their own queries on the data? Because as I said in #41280 (comment 2927299) if the general public is able to construct arbitrary queries, that will render that service useless. I'm also not sure what you are thinking about how to go about extracting an allow-list of metrics that are sane, is there some mechanism for doing that?

I think it might be easier to look at how to extract public dashboards, that we've curated, and make those available either outside of grafana itself, or via grafana's public dashboard mechanism.

Until that time exists, having the private grafana available to tor-internal seems fine, but it seems like the existing http basic auth could be shared there without having to setup ldap authentication.

changed title from Move auth to Grafan instead of puppet to Move auth to Grafana instead of puppet

mentioned in issue #41312

So I'm working on #40755 (closed) to review how we do Prometheus monitoring (and retire Nagios), and I have been thinking a bit more about authentication. I reviewed this:

we have a webPassword field in there that's currently unused.. we could just use that as a ... well... web password, that could get reused elsewhere.

The field does exist, it's just hidden from the web interface, and would be trivial to recover.

And given that:

there's this cool feature where Grafana automatically and perdiodically syncs LDAP users, presumably to survive LDAP outages ... but of course that's in the Entreprise version! arghl.

This seems like the best option right now.

So, in short, I will propose, as part of TPA-RFC-33, to move Grafana/Prometheus authentication to LDAP, but with static file copies so the monitoring server keeps operating in the case of an LDAP outage, of course.

This would grant everyone with LDAP access (which is, by policy, tor-internal, I believe, with some exceptions) access to the dashboards.

I'll factor this into the TPA-RFC-33 proposal and close this issue, as we won't be moving auth to Grafana per se... A new ticket will be created after the proposal is adopted to make the relevant changes.

closed

mentioned in issue userdir-ldap-cgi#1

mentioned in issue #41636