As @reder9 suggested I checked if java gets automatically updated on agent and it is.

Before job starts failing Java version on agent is 17.0.12. After the job failed the first time agent has 17.0.13 Java version.

I disabled autoupdated as described here: https://linuxconfig.org/disable-automatic-updates-on-ubuntu-22-04-jammy-jellyfish-linux

Disabling auto updates worked!