GitLab is used only for code review, issue tracking and project management. Canonical locations for source code are still https://gitweb.torproject.org/ https://git.torproject.org/ and git-rw.torproject.org.

2020.mdwn 7.92 KB
Newer Older
1
This document is manually synchronized regularly between <https://help.torproject.org/tsa/roadmap/2020/> and <https://pad.riseup.net/p/CYOUx21kpxLL_5Eui61J-tpa-roadmap-2020>. Keep the format in markdown for now. It should eventually be moved into Trac keywords.
Antoine Beaupré's avatar
Antoine Beaupré committed
2

Antoine Beaupré's avatar
Antoine Beaupré committed
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
[[!toc]]

Introduction
============

This page documents a possible roadmap for the TPA team for the year 2020.

Items should be "SMART" <https://en.wikipedia.org/wiki/SMART_criteria>, that is:

 * specific
 * measurable
 * achievable
 * relevant
 * time-bound

18 19 20 21 22 23
DEPRECATED (only to be able to have gaba's notes on the roadmap)

TODO
----

 * nextcloud roadmap
Antoine Beaupré's avatar
Antoine Beaupré committed
24
 * identify critical services and realistic improvements [#31243][] (done)
Antoine Beaupré's avatar
Antoine Beaupré committed
25
 * (anarcat & gaba) sort out each month by priority (mostly done for feb/march)
Antoine Beaupré's avatar
Antoine Beaupré committed
26
 * (gaba) add keywords #tpa-roadmap- for each month (doing for february and march to test how this would work) (done)
Antoine Beaupré's avatar
Antoine Beaupré committed
27
 * (anarcat) create missing tickets for february/march (partially done, missing some from hiro)
28 29 30
 * (at tpa meeting) estimate tickets! (1pt = 1 day)
 * (gaba) reorganize budget file per month https://nc.torproject.net/apps/onlyoffice/7374?filePath=%2FTeams%2FSysadmin%2FBudget%20Sysadmin.xlsx
 * (gaba) create a roadmap for gitlab migration
Antoine Beaupré's avatar
Antoine Beaupré committed
31
 * (gaba) find service admins for gitlab (nobody for trac in https://trac.torproject.org/projects/tor/wiki/org/operations/services ) - gaba to talk with isa and alex and look for service admins (sent a mail to las vegas but nobody replied... I will talk with each team lead)
32 33 34 35
   * have a shell account in the server
   * restart/stop service
   * upgrade services
   * problems with the service
Antoine Beaupré's avatar
Antoine Beaupré committed
36

Antoine Beaupré's avatar
Antoine Beaupré committed
37 38
[#31243]: https://bugs.torproject.org/31243

Antoine Beaupré's avatar
Antoine Beaupré committed
39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66
Main objectives (need to have):

 * decommissining of old machines (moly in particular)
 * move critical services in ganeti
 * buster upgrades before LTS
 * within budget

Secondary objectives (nice to have):

 * new mail service
 * conversion of the kvm* fleet to ganeti for higher reliability and availability
 * buster upgrade completion before anarcat vacation

Non-objective:

 * service admin roadmapping?
 * kubernetes cluster deployment?

Assertions:

 * new gnt-fsn nodes with current hardware (PX62-NVMe, 118EUR/mth), cost savings possible with the AX line (-20EUR/mth) or by reducing disk space requirements (-39EUR/mth) per node
 * cymru actually delivers hardware and is used for moly decom
 * gitlab hardware requirements covered by another budget
 * we absorb the extra bandwidth costs from the new hardware design (currently 38EUR per month but could rise when new bandwidth usage comes in) - could be shifted to TBB team or at least labeled as such

January
=======

67 68
 * [x] catchup after holidays
 * [x] agree internally on a roadmap for 2020
Antoine Beaupré's avatar
Antoine Beaupré committed
69 70 71
 * [x] first phase of installer automation (setup-storage and friends) [#31239][]
 * [x] new FSN node in the Ganeti cluster (fsn-node-03) [#32937][]
 * [ ] textile shutdown and VM relocation, 2 VMs to migrate [#31686][] (+86EUR) IN PROGRESS
72 73
 * [ ] enable needrestart fleet-wide ([#31957][]) 
 * [ ] review website build errors ([#32996][]) 
Antoine Beaupré's avatar
Antoine Beaupré committed
74 75
 * [ ] evaluate if discourse can be used as comments platform for the blog ([#33105][]) <-- can we move this further down the road (not february) until gitlab is migrated? -->
 * [x] communicate buster upgrade timeline to service admins DONE
76
 * [x] buster upgrade 63% done: 48 buster, 28 stretch machines
Antoine Beaupré's avatar
Antoine Beaupré committed
77

Antoine Beaupré's avatar
Antoine Beaupré committed
78 79 80 81
[#31957]: https://bugs.torproject.org/31957
[#31686]: https://bugs.torproject.org/31686
[#32937]: https://bugs.torproject.org/32937
[#31239]: https://bugs.torproject.org/31239
82 83
[#32996]: https://bugs.torproject.org/32996
[#33105]: https://bugs.torproject.org/33105
Antoine Beaupré's avatar
Antoine Beaupré committed
84

Antoine Beaupré's avatar
Antoine Beaupré committed
85 86 87
February
========

88 89
<https://trac.torproject.org/projects/tor/query?status=!closed&keywords=~tpa-roadmap-february>

Antoine Beaupré's avatar
Antoine Beaupré committed
90
 * 2020 roadmap officially adopted
Antoine Beaupré's avatar
Antoine Beaupré committed
91 92 93 94
 * second phase of installer automation [#31239][] (esp. puppet automation, e.g. [#32901][], [#32914][])
 * new gnt-fsn node (fsn-node-04) -118EUR=+40EUR ([#33081][])
 * storm shutdown [#32390][]
 * unifolium decom (after storm), 5 VMs to migrate, [#33085][] +72EUR=+158EUR
Antoine Beaupré's avatar
Antoine Beaupré committed
95
 * buster upgrade 70% done: 53 buster (+5), 23 stretch (-5)
Antoine Beaupré's avatar
Antoine Beaupré committed
96 97 98 99 100 101 102 103 104 105 106 107 108
 * migrate gitlab-01 to a new VM (gitlab-02) and use the omnibus package instead of ansible ([#32949][])
 * migrate CRM machines to gnt and test with Giant Rabbit [#32198][] (priority)
 * automate upgrades: enable unattended-upgrades fleet-wide ([#31957][] )
 * anti-censorship monitoring (external prometheus setup assistance) [#31159][]

[#31159]: https://bugs.torproject.org/31159
[#32198]: https://bugs.torproject.org/32198
[#32949]: https://bugs.torproject.org/32949
[#33085]: https://bugs.torproject.org/33085
[#32390]: https://bugs.torproject.org/32390
[#33081]: https://bugs.torproject.org/33081
[#32914]: https://bugs.torproject.org/32914
[#32901]: https://bugs.torproject.org/32901
Antoine Beaupré's avatar
Antoine Beaupré committed
109 110 111 112

March
=====

113 114
<https://trac.torproject.org/projects/tor/query?status=!closed&keywords=~tpa-roadmap-march>

Antoine Beaupré's avatar
Antoine Beaupré committed
115 116 117
High possibility of overload here (two major decoms and many machines setup). Possible to push moly/cymru work to april?

 * 2021 budget proposal?
Antoine Beaupré's avatar
Antoine Beaupré committed
118 119 120 121 122
 * possible gnt-cymru cluster setup (~6 machines) [#29397][]
 * moly decom [#29974][], 5 VMs to migrate
 * kvm3 decom, 7 VMs to migrate (inc. crm-int and crm-ext), [#33082][] +72EUR=+112EUR
 * new gnt-fsn node (fsn-node-05) [#33083][] -118EUR=-6EUR
 * eugeni VM migration to gnt-fsn [#32803][]
Antoine Beaupré's avatar
Antoine Beaupré committed
123
 * buster upgrade 80% done: 61 buster (+8), 15 stretch (-8)
124
 * solr deployment ([#33106][])
Antoine Beaupré's avatar
Antoine Beaupré committed
125 126 127 128 129 130 131 132 133 134 135
 * anti-censorship monitorining (external prometheus setup assistance) [#31159][]
 * nc.riseup.net cleanup [#32391][]
 * SVN shutdown? [#17202][]

[#17202]: https://bugs.torproject.org/17202
[#32391]: https://bugs.torproject.org/32391
[#32803]: https://bugs.torproject.org/32803
[#33083]: https://bugs.torproject.org/33083
[#33082]: https://bugs.torproject.org/33082
[#29974]: https://bugs.torproject.org/29974
[#29397]: https://bugs.torproject.org/29397
136
[#33106]: https://bugs.torproject.org/33106
Antoine Beaupré's avatar
Antoine Beaupré committed
137 138 139
April
=====

Antoine Beaupré's avatar
Antoine Beaupré committed
140
 * kvm4 decom, 9 VMs to migrate [#32802][] (w/o eugeni), +121EUR=+115EUR
141
 * new gnt-fsn node (fsn-node-06) -118EUR=-3EUR
Antoine Beaupré's avatar
Antoine Beaupré committed
142
 * buster upgrade 90% done: 68 buster (+7), 8 stretch (-7)
143
 * solr configuration
Antoine Beaupré's avatar
Antoine Beaupré committed
144

Antoine Beaupré's avatar
Antoine Beaupré committed
145 146
[#32802]: https://bugs.torproject.org/32802

Antoine Beaupré's avatar
Antoine Beaupré committed
147 148 149
May
===

Antoine Beaupré's avatar
Antoine Beaupré committed
150
 * kvm5 decom, 9 VMs to migrate [#33084][], +111EUR=+108EUR
151
 * new gnt-fsn node (fsn-node-07) -118EUR=-10EUR
Antoine Beaupré's avatar
Antoine Beaupré committed
152 153 154
 * buster upgrade 100% done: 76 buster (+8), 0 stretch (-8)
 * current planned completion date of Buster upgrades
 * start ramping down work, training and documentation
155
 * solr text updates and maintenance
Antoine Beaupré's avatar
Antoine Beaupré committed
156

Antoine Beaupré's avatar
Antoine Beaupré committed
157 158
[#33084]: https://bugs.torproject.org/33084

Antoine Beaupré's avatar
Antoine Beaupré committed
159 160 161
June
====

Antoine Beaupré's avatar
Antoine Beaupré committed
162
 * Debian jessie LTS EOL, chiwui forcibly shutdown [#29399][]
Antoine Beaupré's avatar
Antoine Beaupré committed
163
 * finish ramp-down, final bugfixing and training before vacation
164
 * search.tp.o soft launch
Antoine Beaupré's avatar
Antoine Beaupré committed
165

Antoine Beaupré's avatar
Antoine Beaupré committed
166 167
[#29399]: https://bugs.torproject.org/29399

Antoine Beaupré's avatar
Antoine Beaupré committed
168 169 170
July
====

171
 * Debian stretch EOL, final deadline for buster upgrades
Antoine Beaupré's avatar
Antoine Beaupré committed
172 173
 * anarcat vacation
 * tor meeting?
174
 * hiro tentative vacations
Antoine Beaupré's avatar
Antoine Beaupré committed
175 176 177 178 179

August
======

 * anarcat vacation
180 181 182
 * web metrics R&D (investigate a platform for web metrics) ([#32996])
 
[#32996]: https://bugs.torproject.org/32996
Antoine Beaupré's avatar
Antoine Beaupré committed
183 184 185 186 187 188

September
=========

 * plan contingencies for christmas holidays
 * catchup following vacation
189
 * web metrics deployment
Antoine Beaupré's avatar
Antoine Beaupré committed
190 191 192 193

October
=======

Antoine Beaupré's avatar
Antoine Beaupré committed
194 195
 * puppet work (finish prometheus module development, puppet environments, trocla, Hiera, publish code [#29387][])
 * varnish to nginx conversion [#32462][]
196
 * web metrics soft launch (in time for eoy campaign)
Antoine Beaupré's avatar
Antoine Beaupré committed
197
 * submit service R&D [#30608][]
198

Antoine Beaupré's avatar
Antoine Beaupré committed
199 200 201
[#30608]: https://bugs.torproject.org/30608
[#32462]: https://bugs.torproject.org/32462
[#29387]: https://bugs.torproject.org/29387
Antoine Beaupré's avatar
Antoine Beaupré committed
202 203 204 205

November
========

Antoine Beaupré's avatar
Antoine Beaupré committed
206
 * first submit service prototype? [#30608][]
Antoine Beaupré's avatar
Antoine Beaupré committed
207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223

December
========

 * stabilisation & bugfixing
 * 2021 roadmapping
 * one or two week xmas holiday
 * CCC?

2021 preview
============

Objectives:

 * complete puppetization
 * experiment with containers/kubernetes?
 * close and merge more services
Antoine Beaupré's avatar
Antoine Beaupré committed
224
 * replace nagios with prometheus? [#29864][]
Antoine Beaupré's avatar
Antoine Beaupré committed
225 226
 * new hire?

Antoine Beaupré's avatar
Antoine Beaupré committed
227 228
[#29864]: https://bugs.torproject.org/29864

Antoine Beaupré's avatar
Antoine Beaupré committed
229 230 231 232
Monhtly goals:

 * january: roadmap approval
 * march/april: anarcat vacation
Antoine Beaupré's avatar
Antoine Beaupré committed
233

Antoine Beaupré's avatar
Antoine Beaupré committed
234