You need to upload your .jar files to an S3 bucket and then reference them in your Glue job.

Until you migrate them into jobs, Glue notebooks run in an interactive development environment where you have a dedicated Spark session running on a single instance. When you use magic, you're working within this interactive session that has direct internet access (as a default) and can dynamically load dependencies. The notebook environment is more flexible because it's not distributed across multiple nodes and maintains its state throughout the session. This allows for real-time library installations and direct loading of .jar files using magic functions.

You still need to put your .jar files into S3 bucket and then reference it once you convert these interactive sessions into jobs. Otherwise you will get the same error. .jar files loaded using magic in interactive session are only available during the current interactive session's lifecycle.

When you create a Glue job using scripts (or from interactive sessions), the code needs to be accessible to multiple workers that might be running in different locations. S3 serves as the central storage location from which all these workers can access the same script.