Scale 22 RC1-1 Kubernetes service is not running

Description

Kubernetes service is not running. No changes were made to the system. 2 apps stopped working, when trying to reinstall the following message is displayed.

Code:
Error: Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 382, in run
await self.future
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 418, in __run_body
rv = await self.method(*([self] + args))
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1131, in nf
res = await f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1263, in nf
return await func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/chart_release.py", line 466, in do_create
await self.middleware.call('kubernetes.validate_k8s_setup')
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1310, in call
return await self._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1267, in _call
return await methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/update.py", line 336, in validate_k8s_setup
raise CallError('Kubernetes service is not running.')
middlewared.service_exception.CallError: [EFAULT] Kubernetes service is not running.

Also reported in Forum here

Kubernetes service is not running. | TrueNAS Community

Problem/Justification

None

Impact

None

Activity

Show:
Trevor Cannard
November 22, 2021 at 8:31 PM
(edited)

Hi,

I'm a truenas nooby so with the GUI. BTW apps have been running fine since deleting ix-applications from msata, recreating it on ssd and then reinstallling them. I feel like perhaps some configs were not pointing to the correct drive after the move as per chens situation. 

Waqar
November 22, 2021 at 2:03 PM

 can you please confirm if you changed your pool to "msata-storage" from "ssd-storage" using zfs replication or the UI ? Thanks!

Chen
November 16, 2021 at 7:04 AM

Problem solved. In "pool-name"/ix-applications/k3s/server/cred,there are servel *.kubeconfig files containing *.crt and *.key file path, which is wrong,(still the old pool path), I changed them to new pool in every *.kubeconfig file, and all apps are back.

Chen
November 16, 2021 at 6:10 AM

I have the same problem. First I migrated all apps to another pool(all apps are stopped), then I restarted all apps and they worked fine. But today, after I rebooted the system, all apps are gone, the Installed Applications page shows "Applications are not running", when I tried to reinstall the app, this problem shows up.

Trevor Cannard
November 15, 2021 at 10:04 PM
(edited)

Thanks, here is the Debug File

Duplicate

Details

Assignee

Reporter

Labels

Impact

Time remaining

0m

Fix versions

Affects versions

Priority

Katalon Platform

Created November 15, 2021 at 9:42 PM
Updated July 6, 2022 at 8:57 PM
Resolved November 23, 2021 at 6:53 AM