Issues with grub install after update

have been some issues with machines not being able to come back after a reboot because of grub being updated with unattended upgrades and not being properly installed.

Related issues:

#40040 (closed)

#40041 (closed)

@weasel was able to get Alberti back by installing grub on the mounted disk.

root@fsn-node-01:~# gnt-instance activate-disks alberti.torproject.org
fsn-node-07.torproject.org:disk/0:/dev/drbd6
fsn-node-07.torproject.org:disk/1:/dev/drbd7

root@fsn-node-07:/# kpartx -a -v -p '-p' -v /dev/drbd6
add map drbd6-p1 (253:71): 0 41940992 linear 147:6 2048

root@fsn-node-07:/# mkdir /TARGET
root@fsn-node-07:/# mount /dev/mapper/drbd6-p1 /TARGET

root@fsn-node-07:/TARGET# cp -av `readlink -f /dev/drbd6` dev/new-root
'/dev/drbd6' -> 'dev/new-root'

root@fsn-node-07:/TARGET# cp -av `readlink -f /dev/mapper/drbd6-p1` dev/new-root1
'/dev/dm-71' -> 'dev/new-root1'




root@fsn-node-07:/TARGET# chroot . grub-install --modules=part_msdos /dev/new-root
Installing for i386-pc platform.
/proc/devices: fopen failed: No such file or directory
/proc/devices: fopen failed: No such file or directory
/proc/devices: fopen failed: No such file or directory
/proc/devices: fopen failed: No such file or directory
/proc/devices: fopen failed: No such file or directory
Unknown device "/dev/new-root1": No such device
Installation finished. No error reported.


cd /
umount TARGET
kpartx -d -v -p '-p' -v /dev/drbd6

Grub status after upgrade:

root@alberti:~# DEBIAN_FRONTEND=readline apt install --reinstall grub-pc
Reading package lists... Done
Building dependency tree       
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 131 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 https://mirror.hetzner.de/debian/packages buster/main amd64 grub-pc amd64 2.02+dfsg1-20+deb10u2 [131 kB]
Fetched 131 kB in 0s (790 kB/s)   
[master ebf3bc14] saving uncommitted changes in /etc prior to apt run
 Author: Peter Palfrader <peter@palfrader.org>
 1 file changed, 0 insertions(+), 0 deletions(-)
Preconfiguring packages ...
(Reading database ... 57271 files and directories currently installed.)
Preparing to unpack .../grub-pc_2.02+dfsg1-20+deb10u2_amd64.deb ...
Unpacking grub-pc (2.02+dfsg1-20+deb10u2) over (2.02+dfsg1-20+deb10u2) ...
Setting up grub-pc (2.02+dfsg1-20+deb10u2) ...
Configuring grub-pc
-------------------

The GRUB boot loader was previously installed to a disk that is no longer present, or whose unique identifier has changed for 
some reason. It is important to make sure that the installed GRUB core image stays in sync with GRUB modules and grub.cfg. Please
check again to make sure that GRUB is written to the appropriate boot devices.

If you're unsure which drive is designated as boot drive by your BIOS, it is often a good idea to install GRUB to all of them.

Note: it is possible to install GRUB to partition boot records as well, and some appropriate partitions are offered here. 
However, this forces GRUB to use the blocklist mechanism, which makes it less reliable, and therefore is not recommended.

  1. /dev/sda (21474 MB; QEMU_HARDDISK)  2. - /dev/sda1 (21473 MB; /)  3. /dev/sdb (2147 MB; QEMU_HARDDISK)

(Enter the items you want to select, separated by spaces.)

GRUB install devices: 1


Installing for i386-pc platform.
Installation finished. No error reported.
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.19.0-10-amd64
Found initrd image: /boot/initrd.img-4.19.0-10-amd64
Found linux image: /boot/vmlinuz-4.19.0-9-amd64
Found initrd image: /boot/initrd.img-4.19.0-9-amd64
done
Processing triggers for man-db (2.8.5-2) ...
Scanning processes...                                                                                                            
Scanning linux images...                                                                                                         

Running kernel seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.
root@alberti:~# 

We should take care of grub before rebooting the remaining machines.

• reboot all machines with grub reinstalled (in progress)
• keep grub away from automated upgrades

On cache-02 I tried to update grub by running:

apt install --reinstall grub-pc

The system failed to boot.

I rescued the system with the rescue image from Hetzner. Mounted the LUKS drive with:

# cryptsetup luksOpen /dev/sda2 croot
# mkdir /TARGET
# mount /dev/mapper/croot /TARGET
# fsck /dev/sda1
# mount /dev/sda1 boot
# mount -o bind /proc proc
# mount -o bind /dev dev
# mount -o bind /sys sys
# mount -o bind /run run
# chroot .
# update-initramfs -u
# grub-install /dev/sda
# update-grub
# exit
# umount ./*
# cd /
# umount TARGET
# cryptsetup luksClose croot
# reboot

added Backlog label

added Next label and removed Backlog label

added Doing label and removed Next label

assigned to @hiro

changed the description

marked the checklist item reboot all machines with grub reinstalled (in progress) as completed

marked the checklist item keep grub away from automated upgrades as completed

closed

i wonder if this issue might have actually been fixed in Debian's grub scripts... I am not sure what error we were getting on the console here exactly, but if i remember correctly, it looked something like bug 966575 in the Debian BTS, which is:

GRUB loading.
Welcome to GRUB!

error: symbol 'grub_calloc' not found.
grub rescue>

does that sound familiar to anyone?

mentioned in issue #40185 (closed)

i opened #40185 (closed) to reconsider the grub block from unattended-upgrades, since it seems the bug was fixed... followup there.

uhm. So last time I did update grub I had to select the partition to reinstall grub on. I am not sure that happens with unattended-upgrades which I think was part of the problem?

replying in #40185 (closed)

marked this issue as related to #40185 (closed)