Skip to content

GitLab

  • Menu
Projects Groups Snippets
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
  • Trac Trac
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Issues 246
    • Issues 246
    • List
    • Boards
    • Service Desk
    • Milestones
  • Monitor
    • Monitor
    • Metrics
    • Incidents
  • Analytics
    • Analytics
    • Value stream
  • Wiki
    • Wiki
  • Activity
  • Create a new issue
  • Issue Boards
Collapse sidebar
  • Legacy
  • TracTrac
  • Issues
  • #22514

Closed (moved)
(moved)
Open
Created Jun 06, 2017 by Karsten Loesing@karsten

Replace DescriptorReader's setMaxDescriptorFilesInQueue() with setMaxMemory()

We currently let applications define a limit for parsed descriptor files in DescriptorReader#setMaxDescriptorFilesInQueue(). If the background can parse descriptor files faster than the application can process them, it pauses when the configured number of descriptor files is contained in the queue and only resumes as soon as the application has dequeued a descriptor file.

As we say in the docs, "the default is 100, but if descriptor files contain hundreds or even thousands of descriptors, that default may be too high". But honestly, it's impossible to define a useful number there that works for all descriptor types, from microdescriptors to votes or even files containing tens or hundreds of concatenated votes.

The real purpose of having a limit is to avoid running out of memory when parsing descriptors. So, we should just replace setMaxDescriptorFilesInQueue() with setMaxMemory(). This can't be a hard limit, because we cannot put partial descriptors in the queue (or not even partial descriptor files right now). But this limit can define a magnitute of available memory, which could be anything between, say, 50M and 1G.

This change is useful on its own, but it also prepares #22141 (moved) (after which we won't add DescriptorFile instances to the queue anymore, but Descriptor instances) and #21751 (moved) (when we'll likely want to define a setMaxThreads() method).

I'll attach a branch as soon as I have a ticket number. The code there is for discussion, not meant to be merged just yet. It provides another method to define an upper limit for descriptors (not descriptor files) which I briefly considered but deemed less useful than a limit for descriptor bytes.

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Assignee
Assign to
Time tracking