SCALE Nightlies - Failed to start kubernetes cluster for Applications

Description

Testing 22.02nightlies following the code-freeze announcement
Version: TrueNAS-SCALE-22.02-MASTER-20220205-112923

Same issue that I've reported in the past is still not resolved ahead of the 22.02 Release.

Apps remain stuck in deploying and there is a alert saying failed to start kubernetes (even though it has started)

Previous issue: https://jira.ixsystems.com/browse/NAS-114203

This really needs to be resolved before release.

Debug attached while Apps are still shown as in "Deploying" but after catalog sync

I will also attach the output of `journalctl --no-pager -u k3s > k3s-scale.log`

Problem/Justification

None

Impact

None

Linked work items

is duplicated by

NAS-114677

[SCALE] Apps does not start cleanly after first boot on update

Activity

Show:

Bug Clerk
February 7, 2022 at 11:30 AM

22.02 PR: https://github.com/truenas/middleware/pull/8216

Bug Clerk
February 6, 2022 at 10:42 PM

22.02.1 PR: https://github.com/truenas/middleware/pull/8215

ksimm1
February 6, 2022 at 6:41 PM
(edited)

I'm attaching additional logs here that are trimmed to specific boot events, two where the cluster failed to start upon reboot on 05-Feb and 06-Feb, and one where the cluster successfully started after a reboot on 05-Feb. Nothing changed on my system at all between these reboots.

The difference between failed and success seems to start around here:

FAILED:

SUCCESS:

Resize issue view side panel

Complete

Details

Assignee

Muhammad

Reporter

ksimm1

Labels

Time remaining

Components

Fix versions

SCALE-22.02.1

SCALE-22.02.0

Affects versions

Master - TrueNAS SCALE

Priority

Low

Katalon Platform

Created February 5, 2022 at 7:49 PM

Updated July 1, 2022 at 2:38 PM

Resolved February 7, 2022 at 3:15 PM