Let me start with the Dataflow question first. It appears that the pipeline is using a custom source/sink. Only supported source and sinks are supporting lineage out of box: https://cloud.google.com/dataflow/docs/guides/lineage#limitations . Emitting lineage from the custom source is possible, but would require code changes to the source/sink. Relevant API documentation: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/metrics/Lineage.html.

Per Dataplex documentation, https://cloud.google.com/dataplex/docs/about-data-lineage#lineage_sources lineage is also supported automatically from integrated Google Cloud services and manually, by using the Data Lineage API for custom sources.