I would love to see us move to GitLab Ultimate for several reasons related to how we plan software engineering in the Network Team and I would assume some of the other roadmap heavy teams that need to plan for long term feature additions, far into the future, would benefit from this move too (particularly the Anti-censorship Team and Network Health Team comes to my mind here).
Today, our planning happens in various phases: idea, proposal, grant team involvement, capacity planning, and execution by the team. During this, we use many different tools, including GitLab, Google Docs, Google Sheets, NextCloud's docs/sheets, and many emails and doodles. But there are also tools we currently do not have available that GL Ultimate provides. Currently, the network team's long-term roadmaps are stored in a mixture of pads and notes that Mike and I hold on to. There is no good way for us to keep these upcoming project ideas somewhere sensible where both the Network Team and the Grant team have access to these with all the metadata that both need: estimations, relevance, dependencies from other projects (past and present), and so on.
We currently utilize a mixture of several free and non-free+non-self-hosted solutions to store data about upcoming project work. Not because we have such a strong emotional attachment to these tools but because it's currently less of a pain there than trying to implement reasonable solutions within the Gitlab limitations we have today.
I believe having access to the following features from Gitlab Ultimate would be a massive win for how we do long-term and short-term planning in the Network Team:
Stuff I care deeply about that would significantly make my life at Tor more joyful:
Epics ( https://docs.gitlab.com/ee/user/group/epics/ ) - Being able to organize our life into user stories and checking internal dependencies both from the idea/grant phase over to the engineering execution phase would significantly reduce the overhead from the Project Management and Team Lead side of this work. Visualizing and storing roadmaps encoded as epics would substantially reduce my time reviewing spreadsheets with Gaba. This would also benefit the team as we have one source of truth: one friction point we have seen historically in the network team is when data is stored in multiple places, they tend to get out of sync, so eliminating that opportunity of failure would be helpful. This feature alone is big enough for me to want to move to it today.
Custom roles ( https://docs.gitlab.com/ee/user/permissions.html#custom-roles ): Permissions have ALWAYS been an annoying issue in GitLab compared to the overly flexible permission system we had in Trac. I believe custom roles will help us somewhat without being a 100% solution to our issues.
Label constraints where we can say "maximum one Operating System (OS) label per issue, and make OS::macOS, OS::Windows, OS::FreeBSD, etc. valid labels.
Stuff I don't care so deeply about that we may benefit from:
Suggested reviewers ( https://docs.gitlab.com/ee/user/project/merge_requests/reviews/#suggested-reviewers ) may make some of our Triage Bot usage smarter than just its random reviewer assignee feature for ourselves, but may help more active contributors quickly find a reviewer for the specific piece of a change. Less critical in the bigger picture though.
(I have not looked into whether the above features are only in GL Ultimate or if a minor level can do; I know they are not in our CE version).
With regards to the non-free software nature of GitLab Ultimate, I believe that it's much more essential to reduce potential non-self-hosted content on external proprietary platforms here. I think the negatives of not having these features in our Gitlab today, impacts our planning work and project management abilities and forces us to be overly creative, build hacks, and causes us to just move some of these conversations away from the sensible place for storing project management/planning discussions which, IMO, is on Gitlab. There is no doubt that GitLab understands their split of free/non-free features very well here: Access to epics is a very valuable tool for the PM role in any organization :-)
I hope to see this move happen! I believe this would make Gaba's, Mike's, and my life a lot easier for organizing Network Team tasks and roadmap work. At the same time, it would add some transparency to upcoming work for internal teams and external parties interested (for network team I am particularly thinking of the research community here) in tracking upcoming roadmap work. Exciting!
Additionally for the longterm roadmap, it would be ideal to organize it as a board, and to capture when long term epics are blocked on an item, or otherwise require another item to be done first. That is the following features:
The multi-level epic feature of Gitlab Ultimate sounds useful too.
The problem we repeatedly run into with the proposal process is that when a proposal is rejected or cut down, all of that rejected/cut material still needs to be preserved for a future proposal, especially if other items we had planned on proposing (or had in other active proposals) depend on that rejected/cut item. Right now, we have zero tooling to represent this information, leading to huge amounts of scrambling and confusion whenever this happens, and continually afterwards. It is a huge drag.
With regards to the non-free software nature of GitLab Ultimate, I believe that it's much more essential to reduce potential non-self-hosted content on external proprietary platforms here.
I think that's an important point to make in the free software/non-free software trade-off. I bet if we spelled out which non-self-hosted external proprietary platforms we would avoid by switching to Gitlab Ultimate it would be much easier to convince folks who have qualms about us starting to use yet another proprietary tool.
Yup! The big one for me here is being the Google suite of collaborative editing tools. Some have been replaced somewhat by NC, but we still use it a lot in the planning of our projects.
This is my main objection to the current proposal, or lack thereof. I
have basically been asked to "do gitlab ultimate" without being given
anything back in terms of reduction in the variety of tools we currently
use.
I would be much more enthusiastic about this project if I knew the
plethora of google docs and nextcloud spreadsheets would be replaced
with the GitLab features, or at least significantly.
I have also expressed, in phone calls, serious objections to the use of
non-free software to build free software tools, but I'm not sure it's
worth rehashing that argument here. I do have a post I sent to -vegas
(remember that?) years ago I could copy-paste here though. :p
With regards to the non-free software nature of GitLab Ultimate, I believe that it's much more essential to reduce potential non-self-hosted content on external proprietary platforms here.
I think that's an important point to make in the free software/non-free software trade-off. I bet if we spelled out which non-self-hosted external proprietary platforms we would avoid by switching to Gitlab Ultimate it would be much easier to convince folks who have qualms about us starting to use yet another proprietary tool.
This is my main objection to the current proposal, or lack thereof. I have basically been asked to "do gitlab ultimate" without being given anything back in terms of reduction in the variety of tools we currently use.
I would be much more enthusiastic about this project if I knew the plethora of google docs and nextcloud spreadsheets would be replaced with the GitLab features, or at least significantly.
You're given a set of arguments above from at least two people who spend a significant amount of their paid time dealing with grants and both long- and short-term roadmap planning whom are already dealing with tons of non-free, non-self-hosted, SaaS solutions.
I have also expressed, in phone calls, serious objections to the use of non-free software to build free software tools, but I'm not sure it's worth rehashing that argument here. I do have a post I sent to -vegas (remember that?) years ago I could copy-paste here though.
That's mighty fine -- that ship sailed a long time ago, though. The paid engineers in Tor who deal with our product portfolio deal with Windows, macOS, iOS, and all kinds of proprietary crap every day to do the work our users expect from us. The UX team, I assume, is largely on macOS and uses the tools they need to get their work done efficiently, and it's up to them to make the call for the best tools for their trade.
While I agree that I think it's vital that both our end-users and developers must be able to build and use our tools and products using "purely" free software, I think it's a poor idea to firmly attach yourself to that as an argument against a self-hosted solution that solves a problem space that team lead's and project management spends a significant amount of time on.
I feel like such an argument quickly becomes, "I don't want to suffer from proprietary software in my work life, but the rest of you can just live with it."
This is my main objection to the current proposal, or lack thereof. I have basically been asked to "do gitlab ultimate" without being given anything back in terms of reduction in the variety of tools we currently use.
I would be much more enthusiastic about this project if I knew the plethora of google docs and nextcloud spreadsheets would be replaced with the GitLab features, or at least significantly.
You're given a set of arguments above from at least two people who spend a significant amount of their paid time dealing with grants and both long- and short-term roadmap planning whom are already dealing with tons of non-free, non-self-hosted, SaaS solutions.
Yes, I read and heard those arguments multiple times, and I think they
are valid. I'm sensitive to those, and I think it's a good idea to go
with GitLab Ultimate.
I would like a stronger commit on this, and so far I haven't heard
anything like that. The best I've heard is that we'd get some reduction
in the amount of out-of-band stuff but no serious commitment to stopping
that at all.
[...]
I feel like such an argument quickly becomes, "I don't want to suffer from proprietary software in my work life, but the rest of you can just live with it."
That is certainly an argument I have made, and will make again, although
I will not say "you will just have to live with it". I think if the Tor
Project wants TPA to start managing non-free software, that's fine, but
it doesn't mean I personally have to deal with it in my professionnal
life.
I have spent decades of my work life managing and advocating free
software, it's what I do. I'm not saying I won't do this at all, but I
do want to point out it's a big ask to walk over that principle I've
held on for decades.
It's certainly not what I was told I would do when I was hired here, in
any case.
...
On 2023-09-06 13:42:02, Alexander Færøy (@ahf) wrote:
Finally, I should point out that there's a certain lack of process going
on here. I understand I might look at a total process and documentation
nerd from the outside, but I think there's certainly a middle ground
between "everything should be an RFC" and "everyone let's pile up on
that GitLab issue".
Right now, I think the next step in this ticket was for @gaba to lay out
requirements, and specify more clearly what we would gain by going with
GitLab ultimate, and what we would stop using if we do so.
If that's been delegated to you and mike, maybe that's fine, but I
should point out the best space for this might not be GitLab issues
comments but possibly a wiki page or design document of some
sort... Typically, I write an RFC for this, but whatever flies your boat
is fine with me.
a.
--
Antoine Beaupré
torproject.org system administration
I would like a stronger commit on this, and so far I haven't heard anything like that. The best I've heard is that we'd get some reduction in the amount of out-of-band stuff but no serious commitment to stopping that at all.
With all due respect, this is an absurd request, and you will not get that from me. TPA doesn't hold a position within the organization where they can request commitments from others on how others do work.
What tools individual staff in TPI choose to use isn't a plenary session within the organization. I will use the best tool that seems most handy at the given time to complete my tasks, and I hope everybody else in the organization does the same. We are a grants and donation-funded product organization, so making our products the best possible under our various constraints is our key priority here IMO. If there exist readily available tools that make our products better, I think we should investigate using those tools by looking at the pros and cons of them and whether they will have a positive impact on our collaboration abilities and products :-)
One example of a proprietary tool that has significantly upped our products' overall quality (and security) is the Network Team's use of Coverity's hosted solution for C Tor. I will not say we ditch that as some arbitrary commitment to "reduce proprietary tooling."
Also, if I need to collaborate with an external party on a security insensitive document, I will highly likely use Google Docs again, and that doesn't depend at all upon which version of Gitlab we run today or in the future.
I would like a stronger commit on this, and so far I haven't heard anything like that. The best I've heard is that we'd get some reduction in the amount of out-of-band stuff but no serious commitment to stopping that at all.
With all due respect, this is an absurd request, and you will not get that from me. TPA doesn't hold a position within the organization where they can request commitments from others on how others do work.
I guess not, but I can certainly request that I don't get thrown a random google docs every other month shuffling my priorities from the ground up.
I cannot give a guarantee about that, but this does sound like the same issue we are talking about above about how we manage priorities and which tools we use for that?
But yeah, I do start to get a feeling that "TPA" cannot request commitment on pretty much anything...
This isn't specific to TPA here to be fair -- if the apps team or UX team told me (or other members of the Network Team) how to do work, I'd give the same kind of response :-)
I would like a stronger commit on this, and so far I haven't heard anything like that. The best I've heard is that we'd get some reduction in the amount of out-of-band stuff but no serious commitment to stopping that at all.
With all due respect, this is an absurd request, and you will not get that from me. TPA doesn't hold a position within the organization where they can request commitments from others on how others do work.
Fair enough. However, as I understood your point, likely as @anarcat did, it seemed to imply some kind of commitment at an organizational level. Because otherwise it easily boils down to adding up yet another proprietary tool we use: you might replace the Google suite of collaborative editing tools for your daily usage with Gitlab Ultimate tools but for all the other folks at Tor this might just lead to Google suite of collaborative editing tools + Gitlab Ultimate.
What tools individual staff in TPI choose to use isn't a plenary session within the organization. I will use the best tool that seems most handy at the given time to complete my tasks, and I hope everybody else in the organization does the same.
I think this reasoning is way too simple and does not reflect the complexity in the current context. If your team has set on Coverty or the UX folks use a Mac and whatever tools that implies that's a whole different story than TPO thinking about its organization-wide bug tracker and switching it to Gitlab Ultimate. What you do by bringing up those examples feels a bit like comparing apples to oranges to me. The former just affects single teams while the latter affects the whole organization instead. So, in general and in our context it could very well make perfect sense to not use the best tool for individual staff if there are overriding organizational concerns or (a lot of) other teams affected. To give you just three examples what I feel we could think about in the context of this ticket apart from single teams saying "yes, please" (and I bet there are a bunch of more things to ponder):
I'd find it reasonable to discuss questions about lock-in and how concerning those are. Are we willing to take the risk that features we want to depend on lock us in to whatever Gitlab Inc. deems reasonable? Maybe they want at some point unreasonable prices for licenses? Or maybe they move the features we want to the next even more expensive and restricted level? Etc. Do we have contingency plans, in case things go south in that regard? Should we?
Moreover, what additional security risk poses it to us to just run some proprietary blob we don't even have a remote chance to inspect or verify so we can be reasonably sure we run the same code as intended? Does switching to Gitlab Ultimate make it easier to target us specifically with malware? I recall we had quite some security-related discussions when switching from git.tpo to gitlab.tpo given Gitlab's (free version) huge attack surface and concerns about our repository integrity.
What costs does it involve on other teams to switch to Gitlab Ultimate, in particular TPA? I suspect maintaining Gitlab Ultimate on our infrastructure is even more challenging assuming (as I do) we have at least additional security risks we want to take into account. Do we dump yet another task on TPA then? What about teams that do not need Gitlab Ultimate features but suddenly feel that they need to switch their workflow (e.g. because the PMs at Tor aligned their workflow to Gitlab Ultimate features and they, reasonably enough, don't want to keep in mind several different workflows per team or because cross-team collaboration means we apply a single workflow to all teams)?
Now, as you mentioned the Network Health Team in your first comment let me just add some points from that angle here: I can do my planning for network health team work very well without switching to Gitlab Ultimate both with the Gitlab tooling we have right now and other tools like pads, local notes etc. Obviously, I am not saying this holds for other teams as well. It's just meant to be a data point. If it were just for the Network Health Team then, given the above concerns, I'd say switching to Gitlab Ultimate would not be worth it.
Finally, given that the important free software argument is missing so far in my comment, just as a personal note: I do think that free software needs free tools. :)
Fair enough. However, as I understood your point, likely as @anarcat did, it seemed to imply some kind of commitment at an organizational level. Because otherwise it easily boils down to adding up yet another proprietary tool we use: you might replace the Google suite of collaborative editing tools for your daily usage with Gitlab Ultimate tools but for all the other folks at Tor this might just lead to Google suite of collaborative editing tools + Gitlab Ultimate.
Perhaps that could have something to do with the collaborative tools Google provides for them is making their job easier? I have a bit of a hard time thinking this is a form of a trade we make inside the org where we get one thing, but then we look each other in the eyes and says "less of X now."
I think the notion of some organizational commitment would block us from making most decisions across the team boundaries then. Suppose the grants team actually prefers using Google Docs because its collaborative editing is the only thing that works well with several people editing and doesn't randomly lose edits. In that case, I think they should continue to use that until an alternative is provided to them.
Interestingly enough, the recent email issues we all are impacted by are a bit relevant here: the sporadic loss of sent emails negatively impacts everybody, but it affects people with more external (outside of the TPO domain, often the grant/admin folks) email communication worse than, say, the average engineer we have. Here several things play in: resource additions to the TPA team, of course, but also that we have historically not been good at making decisions in this space because some people have (IMO) unreasonable expectations, such as being able to send emails using some local, self-hosted, SMTP server they have used forever and believe the internet should continue to accept that. Fortunately, TPA is doing pretty nifty stuff in this space now, but it has taken what? 5+ years of conversations, at least before they have been able to execute on it.
I think this reasoning is way too simple and does not reflect the complexity in the current context. If your team has set on Coverty or the UX folks use a Mac and whatever tools that implies that's a whole different story than TPO thinking about its organization-wide bug tracker and switching it to GitLab Ultimate.
I agree that it has a broader implication to discuss shared tooling for the organization vs. local tooling of individual team members, which is also reflected in our conversation here. My (strongly) simplified reasoning is definitely tied to the idea of some commitment from others on what tool they use to carry out their work and @anarcat's reflection on his own beliefs of choices of software and what its consequences potentially have to other people in the organization who are significantly more negatively impacted by poor PM tooling when it comes to grants and roadmap organization than TPA is.
What you do by bringing up those examples feels a bit like comparing apples to oranges to me. The former just affects single teams while the latter affects the whole organization instead. So, in general and in our context it could very well make perfect sense to not use the best tool for individual staff if there are overriding organizational concerns or (a lot of) other teams affected. To give you just three examples what I feel we could think about in the context of this ticket apart from single teams saying "yes, please" (and I bet there are a bunch of more things to ponder):
Agreed -- I think this ticket would benefit from finding out the pros and cons on an organizational level.
I'd find it reasonable to discuss questions about lock-in and how concerning those are. Are we willing to take the risk that features we want to depend on lock us in to whatever Gitlab Inc. deems reasonable? Maybe they want at some point unreasonable prices for licenses? Or maybe they move the features we want to the next even more expensive and restricted level? Etc. Do we have contingency plans, in case things go south in that regard? Should we?
We are already locked into that today -- GitLab Inc. can make whatever decisions they want about both the free software and the non-free version of GitLab, and we wouldn't be able to do much about if if they make poor decisions. It's a large enough project that we wouldn't be able to maintain it ourselves, and I doubt the free software movement could pick it up quickly. This was discussed during our move from Trac to GitLab as well in some of the All Hands during the summer (June/July) of 2020.
If they start making poor decisions that impact GitLab EE, then GitLab CE is potentially affected, too, and we would have to look for an alternative to GitLab entirely.
The pricing argument is a big one here. There is a dependency before we even able to do any of this on getting a license for GitLab EE from GitLab's open source program. When we did the migration from Trac to GitLab we were offered such license (unlimited users license, self-hosted) at one point during a brief conversation we had with some of the GitLab staff at that time, but things may have changed.
Moreover, what additional security risk poses it to us to just run some proprietary blob we don't even have a remote chance to inspect or verify so we can be reasonably sure we run the same code as intended? Does switching to Gitlab Ultimate make it easier to target us specifically with malware? I recall we had quite some security-related discussions when switching from git.tpo to gitlab.tpo given Gitlab's (free version) huge attack surface and concerns about our repository integrity.
It's not a "blob" as such. The code is there for us to look at if we deem it necessary. @micah is probably the person who has spent the most time within the GitLab source code, though as he has gotten some of the issues we had with email headers fixed in the past.
Regarding the safety of running this, it's "just" another package from the same GitLab Inc. provided Debian mirror.
What costs does it involve on other teams to switch to Gitlab Ultimate, in particular TPA?
Migration is the obvious big one. Maintenance cost is similar to Gitlab CE from the feature set we have described so far in this ticket (e.g. we don't need to run additional external services for the feature set listed so far).
I suspect maintaining Gitlab Ultimate on our infrastructure is even more challenging assuming (as I do) we have at least additional security risks we want to take into account. Do we dump yet another task on TPA then?
What makes you think maintaining Gitlab EE is more challenging than maintaining Gitlab CE?
The debuggability of GitLab EE should be precisely the same as Gitlab CE in that you have access to all the source code, but parts of it are not under a libre license. It's essentially the same code tree the products are built from, but GitLab CE is a stripped version of EE.
What about teams that do not need Gitlab Ultimate features but suddenly feel that they need to switch their workflow (e.g. because the PMs at Tor aligned their workflow to Gitlab Ultimate features and they, reasonably enough, don't want to keep in mind several different workflows per team or because cross-team collaboration means we apply a single workflow to all teams)?
They do not have to change anything. They can continue with their workflow with GitLab CE if they want to. But if this is a question for the PM, then @gaba can probably give some thought to what tooling she prefers that would make her job easier -- probably worthwhile to have in mind a future where we may have multiple PM's as well.
Now, as you mentioned the Network Health Team in your first comment let me just add some points from that angle here: I can do my planning for network health team work very well without switching to Gitlab Ultimate both with the Gitlab tooling we have right now and other tools like pads, local notes etc. Obviously, I am not saying this holds for other teams as well. It's just meant to be a data point. If it were just for the Network Health Team then, given the above concerns, I'd say switching to Gitlab Ultimate would not be worth it.
I can also continue to do my work without Gitlab Ultimate because it's not a blocker for me, but it's a significant cause of friction due to data loss, data desync when going in and out of spreadsheets, etc. I also sitting each week and looking at a spreadsheet with roadmap items from NHT tasks the NT needs to deliver on and our capacity estimations on those items -- in that sense we have some overlap there on our teams too.
It's interesting to see which teams think they would benefit from some of the features it provides, so the data point is good! :-)
Finally, given that the important free software argument is missing so far in my comment, just as a personal note: I do think that free software needs free tools. :)
It's generally essential for us to ensure our software can work/build/etc. for people relying entirely on free software on their machines, but now, 13 years after that post was written, Github (not a typo) has just gotten even more prominent, and a significant amount of the things many of our teams depend upon is happily hosted on a source code platform owned and hosted by Microsoft. I don't think Tor is particularly capable of handling the failures of the free software movement when it comes to the rapid inflight of hosted software solutions that provide greater developer satisfaction than the systems we thought we would be using 15 years ago
Interestingly enough, the recent email issues we all are impacted by are a bit relevant here: the sporadic loss of sent emails negatively impacts everybody, but it affects people with more external (outside of the TPO domain, often the grant/admin folks) email communication worse than, say, the average engineer we have. [...] Fortunately, TPA is doing pretty nifty stuff in this space now, but it has taken what? 5+ years of conversations, at least before they have been able to execute on it.
Well that's certainly an interesting thing to bring up considering I've
been explicitly asked to de-prioritize work on the email proposals in
favor of GitLab Ultimate...
I, for one, would be really happy to work on fixing our email services,
more than debating the merits of proprietary software...
[...]
If they start making poor decisions that impact GitLab EE, then GitLab CE is potentially affected, too, and we would have to look for an alternative to GitLab entirely.
I feel that's throwing the baby out with the bathwater here... It's a
huge difference between GitLab CE and EE. If GitLab shuts down CE,
people are free to fork it, not so with EE. That's a huge difference.
Again, here, this is something that must be investigated. @gaba was
supposed to see if the offer still stood and we're waiting on GitLab to
confirm they can give us EE for free... I would love to also know "for
how long", but I suspect they won't give us a promise on that, which is
the whole problem with this "free but not really free" approach.
[...]
It's not a "blob" as such. The code is there for us to look at if we deem it necessary. @micah is probably the person who has spent the most time within the GitLab source code, though as he has gotten some of the issues we had with email headers fixed in the past.
Regarding the safety of running this, it's "just" another package from the same GitLab Inc. provided Debian mirror.
There are possible legal concerns about looking at proprietary source
code. You could be accused of copying proprietary software, for
example.. Even if the source is available doesn't mean it's actually
legal to look at it.
This is one of the problems with proprietary software, it's another
license to examine, and we're not lawyers so we don't really grok the
implications. GitLab's proprietary license also has huge escape hatches
that point at online versions that can change at any moment, so it's
hard to interpret.
What costs does it involve on other teams to switch to Gitlab Ultimate, in particular TPA?
Migration is the obvious big one. Maintenance cost is similar to Gitlab CE from the feature set we have described so far in this ticket (e.g. we don't need to run additional external services for the feature set listed so far).
From what @micah told me, the migration path is pretty simple. I suspect
the migration path back to the free version might be more painful, if
only because of the feature loss.
I suspect maintaining Gitlab Ultimate on our infrastructure is even more challenging assuming (as I do) we have at least additional security risks we want to take into account. Do we dump yet another task on TPA then?
[...]
The additional burden will be minor, but it is a thing. There are more
features to support, so more things that can break, and ultimately it's
issues that are raised with TPA that we are sometimes expected to fix.
[...]
I can also continue to do my work without Gitlab Ultimate because it's not a blocker for me, but it's a significant cause of friction due to data loss, data desync when going in and out of spreadsheets, etc. I also sitting each week and looking at a spreadsheet with roadmap items from NHT tasks the NT needs to deliver on and our capacity estimations on those items -- in that sense we have some overlap there on our teams too.
See, this here is what I mean by a commitment to reduce the use of other
spreadsheets. Wouldn't you like that overlap to go away somehow with the
coming of GitLab Ultimate? Isn't that the goal? Why can't we state that
more explicitly?
Finally, given that the important free software argument is missing so far in my comment, just as a personal note: I do think that free software needs free tools. :)
It's generally essential for us to ensure our software can work/build/etc. for people relying entirely on free software on their machines, but now, 13 years after that post was written, Github (not a typo) has just gotten even more prominent, and a significant amount of the things many of our teams depend upon is happily hosted on a source code platform owned and hosted by Microsoft. I don't think Tor is particularly capable of handling the failures of the free software movement when it comes to the rapid inflight of hosted software solutions that provide greater developer satisfaction than the systems we thought we would be using 15 years ago
I don't think it's solely the burden of the Tor project to handle this
particular problem, but we have clearly chosen to not contribute to it
too much by hosting our own GitLab instance instead of just moving to
GitHub. Going Ultimate is certainly a step in the wrong direction, in
that sense, and I don't think it should be taken lightly.
...
On 2023-09-07 11:46:03, Alexander Færøy (@ahf) wrote:
The gitlab opensource program requires a yearly membership renewal. It requires that the person who applied initially do the renewal, or transfer ownership to the customer portal to the new person.
Members of the GitLab Open Source Partners program agree to: Engage in co-marketing efforts with GitLab Complete a public case study about their innovative use of GitLab Plan and participate in joint initiatives and events
It doesn't detail what these are explicitly.
It also requires abiding by the Open Source Program Agreement which I've looked over and it appears fairly innocuous.
Requirements also are:
Use OSI-approved licenses for their projects: Every project in the applying namespace must be published under an OSI-approved open source license.
Not seek profit: An organization can accept donations to sustain its work, but it can’t seek to make a profit by selling services, by charging for enhancements or add-ons, or by other means.
Be publicly visible: Both the applicant's GitLab.com group or self-managed instance and source code must be publicly visible and publicly available.
Note there are Private Project Exceptions: In some cases, we allow program members to host a small number of private projects if those projects contain sensitive data. Members should send an email to opensource@gitlab.com in order to discuss this exemption. Program members must obtain written permission from the GitLab Open Source Program team in order to use their licenses outside of program requirements.
What about teams that do not need Gitlab Ultimate features but suddenly feel that they need to switch their workflow (e.g. because the PMs at Tor aligned their workflow to Gitlab Ultimate features and they, reasonably enough, don't want to keep in mind several different workflows per team or because cross-team collaboration means we apply a single workflow to all teams)?
They do not have to change anything. They can continue with their workflow with GitLab CE if they want to. But if this is a question for the PM, then @gaba can probably give some thought to what tooling she prefers that would make her job easier -- probably worthwhile to have in mind a future where we may have multiple PM's as well.
Let me just pick this one up here as I think it deserves a broader discussion. Yes, my comment was not meant for how particular teams file tickets, organize tickets, do their CI etc. That is: do their day-to-day work. I was more concerned about the PM side of things and how plans on that side would affect individual team work. The ticket description is a bit thin but I happened to learn yesterday that one (main) driver for looking at Gitlab Ultimate is actually trying to figure out whether that would help with our broken process for tracking work, people's time and allocation and capacity etc. Now, I know essentially all team leads are more or less frustrated with that problem and are routing around that broken process differently, trying, together with @gaba, to keep workload manageable and our commitments to sponsors on track.
However, I am very wary that just pouring over shiny new tooling we'd get with Gitlab Ultimate won't fix our processes automagically (nor would disseminating PM work to an additionally hired PM accomplish that). I think we should have a necessary step 0 here that is involving all the team leads where we get on the same page as to what a good process would look like and then we can decide whether Gitlab Ultimate would actually provide the missing tooling for that. For instance: recall the OKR experiment we did? Is that still a thing we wanna do? If so, how? And, once we know the how, then in a second step: does Gitlab Ultimate actually provide a means for helping with that which the CE does not?
How would we get there? I imagine two parts here, one involving @gaba and probably @isabela who could prepare some text or email outlining what process we tried so far, where we failed and what the new process could look like. That would help providing necessary context for everyone involved (both in figuring out what worked and did not work in the past) and having something written to think about, explaining the imagined future process. I strongly believe that juat a meeting like an All Hands or so is not working for that part due to language barriers, potentially insufficient notetaking and other problems inherent to synchronous communications. The other part would involve some homework for team leads to think about what process they would like to see given organizational needs XYZ so that they do not need to route around it anymore and do not feel the frustration they currently encounter.
After that we could have a meeting hashing the details out, getting together on the same page and nailing the process down. And then we could look which currently lacking tooling would fit to this new process. Maybe Gitlab Ultimate could be helpful in that case, maybe not. Not sure yet. :)
+1 to all of the above. This is actually very connected to the Support Work related to the Flexible Fridays Policy.
What I am thinking is to have one All Hands to present how these processes are related to important decisions inside of the organization. To illustrate that, I am thinking of picking 3 'study cases'. Based on this 3 cases presentations, we will identify steps of our internal processes like 'estimation', 'roadmap management', 'burnout calculation' and so on. The tools and 'human labor' behind doing those things.
All of it will be done at an All Hand because we need to have everyone in the organization on the same page, with the same knowledge, around this.
Then we will move to a more private/dedicated workshop/meetings sessions where team leads and PMs will be working together to review these tools and the human labor part to see how we can improve it.
I honest think that on this second part of the process, more people will for instance, understand why many believe that Gitlab Ultimate will help us. I am part of this group and I would like to say that I believe those who are thinking like this are not necessary 'just believing that a new shiny tool will solve our problems'. For the other hand, we have tried to do it without such tools and we have seeing that is not possible (especially from the 'human labor' part of maintaining this) and we do need certain features to alleviate the 'human labor' part.
Anw, I will work on this first All Hands to help people understand what I am talking about here. And then kick off this process
I am part of this group and I would like to say that I believe those who are thinking like this are not necessary 'just believing that a new shiny tool will solve our problems'.
Great and that's what I actually hoped for. But given that I am not part of that group nor the ticket is description providing any context I thought I'd point out that potential pitfall explicitly. :)
Anw, I will work on this first All Hands to help people understand what I am talking about here. And then kick off this process
I don't feel like a all hands is what we need right now. What we need is for someone to sit down, study this problem, come up with personas and proposed solutions, and write that down in a document so we know what we're actually proposing here.
Questions I'd like to see answered include:
what's the work / cost for us in going with Ultimate (@micah did some research on that, so that's a good start, but there's a lot of gray areas in there)
what are the actual problems we're trying to solve here (@ahf and @mikeperry mentioned some of their problems, we should collate this somewhere and see if we missed anything)
who are the people impacted by this (again, some voices were heard here, but we don't have a good personas story yet)
what is the proposed solution (right now this is "GitLab Ultimate", and no clear view of how that's going to impact the current workflow or change the use of Google Docs / Nextcloud OnlyOffice for planning, there's even been clear pushback from @ahf that we might not even want to commit to any sort of change there)
I'm happy to help with that process, but right now I get the feeling people are tired of me writing RFCs and documentation, so I'm not pushing that way. I should also note that I'm generally very demotivated by this entire process, so I'm taking a back seat...
Let me just pick this one up here as I think it deserves a broader discussion. Yes, my comment was not meant for how particular teams file tickets, organize tickets, do their CI etc. That is: do their day-to-day work. I was more concerned about the PM side of things and how plans on that side would affect individual team work. The ticket description is a bit thin but I happened to learn yesterday that one (main) driver for looking at Gitlab Ultimate is actually trying to figure out whether that would help with our broken process for tracking work, people's time and allocation and capacity etc. Now, I know essentially all team leads are more or less frustrated with that problem and are routing around that broken process differently, trying, together with @gaba, to keep workload manageable and our commitments to sponsors on track.
One big issue here (IMO) with time management and GitLab is that we quickly end up having multiple sources of truth with how we handle this today: if GitLab's time tracking estimates is changed by an engineer, it's very opportunistic to expect the same engineer to update them in a spreadsheet too. We need to eliminate one of the tools, and I doubt the engineers want to do time management for issues in spreadsheets.
When you start looking at the above problem and think of ways to solve it in GitLab today, one additional issue arises, which is that when we go from grant objectives and activities, these reflect poorly in tickets: you have no way of chopping up a ticket into many smaller tickets and keeping track of the initial estimates. If you do the workflow where you have a parent ticket that "hosts" the activity/objective goal, then every time you create a child ticket with an estimation, you have to go and decrease the estimate with the same amount on the primary ticket for the objective/activity. We are not good at doing that today and it's unlikely this will get any better soon. People have tried to experiment with using milestones as the solution for this, which works great, except for people who are doing software projects that uses milestones for release tracking, then you have to pick whether you want to have a great milestone tracking tool or a spreadsheet or remembering to do the split/decrement action together, which we know by now doesn't happen. And no, suggesting to use labels here is not useful.
The desync Gitlab <-> Spreadsheet issues have been an incredible source of frustration for everybody (at least in the NT) as it completely ruins planning meetings. I'd personally also doubt tracking these things in a spreadsheet would be a successful story, knowing one of our engineering teams well). Generally, people in Tor seem to be very good at reacting to events that are visible to them in GitLab (via its todo list, review queue, assignment queue, etc.). I believe looking at the survey TPA did at one point is likely also worthwhile here: people enjoy GL in general and spends a significant amount of time in it, so for things to be visible, having it in GL is likely smart. The average TPI engineer needs to spend very little time in NC, Harvest, db.torproject.org, etc. but a whole lot of their day is spend with GitLab.
Another thing that I mentioned earlier in this ticket is the other end of our "grant funnel": right now our process is largely idea -> grant writing -> getting $$$ -> execute the grant. If we split this up and add ownership, we get the following:
idea (usually origins from a ticket, engineering knowledge, experimentation, etc. but usually comes from people doing R&D in Tor. The owner here is often the ideamaker + team lead + team members interested in this)
grant writing (mixture of grant team and engineering teams works on this together, usually the grant team leads this and have ownership of the tooling here - this part works very smoothly for me at least)
getting $$$ (mixture of grant team + engineering team, usually a lot of meetings. Most of our engineers avoid these meetings as they are grant technicalities or negotiation strategies)
execute the grant (everything is extracted from the signed grant and moved into gitlab and ownership is entirely transferred to teams + team leads + PM).
Right now the idea phase lacks long-term storage and this have lead to a number of issues, including:
loss of data (pad expiry because these things take forever or simply "forgotten")
the social issue of "who has the most recent copy of document X (where X is some test, estimates, etc.)"
The workflow also has poor transparency for newer members of our engineering teams. If they had an obvious place they could see what projects we want to propose for grants in the future, comment on them, add data points, even add novel ideas, I think that would make it easier for newer people to understand the entire grant funnel. Especially today where the grant funnel is, to be very fair, working extremely smoothly and ideas goes from an email thread to writing very quickly (thanks to Al and the team) this means that often these things have even less time for knowledge about these potential grant ideas to be shared within the team.
If I could get to a point where I could tell both external, and internal people, that they can go look at $somewhere to see what things we have in mind for future network team roadmaps, including the estimates we have, the rationale for doing it, the pros/cons for doing it, etc. then I think that would be useful for our ability to organize ourselves well.
A very important component that we lack today in the tooling around these is being able to track dependency amongst grant ideas: for us to request a grant, we need to be sure that all dependencies of the grant is either included or is requested elsewhere and solved in order.
However, I am very wary that just pouring over shiny new tooling we'd get with Gitlab Ultimate won't fix our processes automagically (nor would disseminating PM work to an additionally hired PM accomplish that). I think we should have a necessary step 0 here that is involving all the team leads where we get on the same page as to what a good process would look like and then we can decide whether Gitlab Ultimate would actually provide the missing tooling for that.
A historical data point here: I think this notion was playing in to why we were unable to make a decision to move to GitLab in 2018 to 2020. We wanted to solve all the "if's" and "what's" involved in the process. Some people expected that a central group of people would come up with solutions for their concerns with regard to leaving Trac and entering Gitlab. Had we aimed for consensus here, even amongst team leads back then, we would still be on Trac today.
Instead, we ended up mapping data over, 1-to-1, so people could entirely continue with the workflows they had from Trac with their projects (with a 1:1 mapping from labels, but without multiple assignee's for ownership and review, and Trac component was mapped to a project name in the GL namespace under tpo/) if they wanted to, but other teams could embrace and experiment with the features GitLab provided for them (where interestingly enough, a big driver here was getting rid of Jenkins and getting access to something similar to appveyor/travis: Gitlab CI).
I see this situation very similar: I don't think it is possible to make a universal workflow that all of Tor will embrace. As an example, Richard and I have completely different ways of managing our projects on GitLab, but both team manages to produce what they need without having to compromise on what we think is best in a given situation. Similarly, I am thrilled that the sysadmin team uses the @triage-bot in the way they do with automatic stale management of issues and MR's, but it's not a workflow I want to adopt for projects under tpo/core/ (at least right now).
I think workflows are generally highly organic, and the teams need very high amounts of autonomy on how they want to accomplish their work, and doing so should happen amongst the team, its team lead, and its PM (of course in a manner that is in respect to the values of our organization -- I don't expect people to say "users are super annoying, so let's disable non-membership ticket intake" and other absurd things here :-).
However, I am very wary that just pouring over shiny new tooling we'd get with Gitlab Ultimate won't fix our processes automagically (nor would disseminating PM work to an additionally hired PM accomplish that). I think we should have a necessary step 0 here that is involving all the team leads where we get on the same page as to what a good process would look like and then we can decide whether Gitlab Ultimate would actually provide the missing tooling for that.
A historical data point here: I think this notion was playing in to why we were unable to make a decision to move to GitLab in 2018 to 2020. We wanted to solve all the "if's" and "what's" involved in the process. Some people expected that a central group of people would come up with solutions for their concerns with regard to leaving Trac and entering Gitlab. Had we aimed for consensus here, even amongst team leads back then, we would still be on Trac today.
Yes, but that is kind of orthogonal to what I was aiming at with my comment. There is no need to have the same workflow for every team in their day-to-day work, I said that above ("Yes, my comment was not meant for how particular teams file tickets, organize tickets, do their CI etc. That is: do their day-to-day work."). I am concerned with the team/PM/organization boundary and maintain that we should have a working process in that regard, which is essentially the same for all the teams. That's not about MR worfklow, or how you do 1:1s, or using triage-bot or whatnot, but more about how we account our team time, how this is tracked across sponsors and projects and how do we plan capacity etc. I think it is nuts if every team did that differently. :) And, yes, there might not be consensus about that, sure. But that's not the point either. Team leads need to have a say here and need to get on the same page together with the PM, ED etc. otherwise this is not going to fly.
Spend a few minutes before my 1:1 with Micah today on toying around with GitLab Ultimate (as a 30 day trial on gitlab.com). This is first time I try it (not just reads fancy feature lists here).
Here we see two epics representing two parts of a grant: The Arti Onion Service Client part and the Arti Onion Service Service API side (yeah, horrible name here - it is for creating onion services in Arti).
In addition to the two concrete epics for Arti, there is a general one that contains the two other epics within it. It's duration is set from the first start time of its sub-epics to the latest end time of its sub-epics (basically, its time frame is inherited). You likely need to expand this meta grant to see the two concrete ones in the roadmap view.
Here one can clearly see that that the two roadmap items for UDP support and Super encryption is blocked by the epic containing the "Misc protocol upgrades". The "Misc protocol upgrades" epic itself contains two smaller items that are probably too small to get a grant for in itself, but could be bulked together with other grants on protocol enhancements.
For grant ideas/process I'm not sure epics do bring much on top of using issues and an issue board, which we kind of already do at https://gitlab.torproject.org/tpo/operations/proposals/ but maybe needs improvements. But I see how Project Management might improve with epics having visibility on the project roadmap.
I don't have a strong opinion here. In the Anti-Censorship I don't think this is high priority in our needs, but maybe is me ignoring many of the PM/grants issues and not being so much involved on those processes.
I don't see a way to directly respond to @anarcat comments about All Hands above.. so I will do it here. I think people are confusing what I tried to explain. The All Hands workshop is a process that we will do independently of this gitlab ultimate discussion. It is part of the Flexible Fridays support work that we need to do for the organization. The gitlab ultimate might be 'a part of the solution' for the problems we will discuss at the All Hands. The goal of the All Hands is to level the knowledge, get everyone on the same page regarding the need of certain processes within Tor, like capacity calculation and estimations. It is not to discuss gitlab ultimate, just to be clear.
GitLab Free customers with a self-managed instance running GitLab Enterprise Edition can receive paid features by registering with GitLab and sending us activity data through Service Ping.
In 16.6, Advanced search is now part of this program, which is quite interesting for wikis, in particular, which could become searchable.
Ouch. A large OpenStreetMap group has been using a proprietary chat platform as a community space for ~10 yrs. Now they gotta pay a $80k/yr (or $10k??) for usage. 🤯
Slack (now Salesforce) now wants to charge @OpenStreetMapUS for all ~6k users on their server. Ouch.
This sort of bait & switch is why open, community owned platforms (like this!) are vital!
If we somehow manage to trim our seat list down to 100, it's still 9900
/mth or 120k
/year.
Obviously, we probably wouldn't want to pay that much, but we need to keep in mind that, if GitLab turns around and discontinues the program we're relying on to get this for free, it means a major organizational change to recover from this sudden cost increase. And this is a situation that has happened to other organizations, this year.
with @gaba on IRC today, we looked at the actual users we have in GitLab, and that scales down that number quite a bit. i tried to figure out the "MAU" (Monthly Active Users) and, from the looks of it by browsing the user list sorted by "last activity", we have:
count
last activity
220
< 30 days
400
6 months
1000
~2 years
1500
~3 years
2000
never
So we probably have closer to 200-400 actual active users: we have 400 "owners" as mentioned above, so that's a solid number. Then we probably have somewhere around 1000 to 1500 actual users who did something in the last 3 years, but it's kind of a long tail.
I think we should disable all users that have not sign-in Gitlab for more than 2 years. Right now all the users that their last sign-in was 2021 or before.
Custom roles ( https://docs.gitlab.com/ee/user/permissions.html#custom-roles ): Permissions have ALWAYS been an annoying issue in GitLab compared to the overly flexible permission system we had in Trac. I believe custom roles will help us somewhat without being a 100% solution to our issues.
while reviewing @gaba's proposal about gitlab ultimate, i found a mention of custom roles, but it's not clear to me what exactly they would bring to help with our current situation. if anything, i worry it would make things more obscure and complex than what they already are...
do you have any specific example of how this could improve our situation?
We need to write a policy on how to setup roles and permissions in Gitlab. Right now there is no explicit agreement on how we are doing it. I think it would help more than the custom roles. The custom roles will give us more flexibility. For example roles that can only edit the wiki without seeing confidential issues.
@lavamind : was there a case for custom role that we discussed before? I remember you adding people to developer role when was something else needed?
In any case, I think that particular issue has a satisfactory conclusion, even if we had access to custom roles I wouldn't consider them in this particular scenario.
We should definitely mention that, explicitly, in the proposal then.
...
On 2024-09-04 15:18:38, Gaba wrote:
We need to write a policy on how to setup roles and permissions in Gitlab. Right now there is no explicit agreement on how we are doing it. I think it would help more than the custom roles. The custom roles will give us more flexibility. For example roles that can only edit the wiki without seeing confidential issues.
We need to write a policy on how to setup roles and permissions in Gitlab. Right now there is no explicit agreement on how we are doing it. I think it would help more than the custom roles. The custom roles will give us more flexibility. For example roles that can only edit the wiki without seeing confidential issues.
We need to write a policy on how to setup roles and permissions in Gitlab. Right now there is no explicit agreement on how we are doing it. I think it would help more than the custom roles. The custom roles will give us more flexibility. For example roles that can only edit the wiki without seeing confidential issues.
Its unclear to me the discussion process here. There are a bunch of TODOs on the wiki page, but which of those must be resolved to move it forwards, and who is working on them? Can we establish some kind of process to evaluate which of these are necessary to resolve, and who is working on resolving them?
For example, it feels like many of these are not necessary to resolve, like "TODO: explain why burndown charts are useful and how we'll use them" The concept of burndown charts is very Project Management 101, and seems unnecessary to have to explain this to move things forward.
Here is a definition of burndown charts and how they will be used, maybe have read of this and point out what about this is necessary to define before continuing:
A burndown chart is a graphical representation used in project management to track the progress of a project over time. It visually shows how much work remains to be completed (the "burn") against a timeline, allowing teams to monitor if they are on track to meet deadlines.
How will it be used? How every project manager uses a burn-down chart:
Visual Progress Tracking: Teams and stakeholders can see at a glance how much work is left and whether the team is on pace to meet the deadline.
Identifying Bottlenecks Early: If the actual progress line stays flat (no work being completed), it signals issues early, like roadblocks or an underestimated workload.
Motivation for Teams: The visual feedback from the chart can motivate teams by showing tangible progress as work is completed.
Facilitating Stand-ups: Planning meetings often reference the burndown chart to adjust priorities or strategies as needed.
So I'm the one who added a bunch of TODO items in there, so perhaps i
can clarify.
I think those are items that @gaba intends to work on. She asked for my
review of the document, and the TODOs are a result of that review. The
next step is for gaba to enhance the document in response to said
review.
Regarding burn down charts specifically: I am familiar with those! I
didn't mean to get told what burn down charts are, in general. What I
meant is that the proposal should explain how we intend to use them
going forward. There's a lot of information in the document, and
there's various degrees of clarity in how each feature is described and
whether or not we'd use them.
I think it's important that we clarify what changes are expected from
users, team leads, and project managers here, as it will impact how
people will evaluate the proposal.
For example, perhaps a lot of people would be extremely excited by the
idea of having burn down charts to replace the current "everything
everywhere all the time" spreadsheet, but we're not explicitly saying
that here. I think the proposal would have more impact if it clearly
outlined which parts of Ultimate we intend on using, why, and how.
Its unclear to me the discussion process here. There are a bunch of TODOs on the wiki page, but which of those must be resolved to move it forwards, and who is working on them? Can we establish some kind of process to evaluate which of these are necessary to resolve, and who is working on resolving them?
For example, it feels like many of these are not necessary to resolve, like "TODO: explain why burndown charts are useful and how we'll use them" The concept of burndown charts is very Project Management 101, and seems unnecessary to have to explain this to move things forward.