Dataflow job is failing - google-cloud-platform

Question

I'm trying to trigger a dataflow job to process a csv file then save the data into postgresql.
the pipeline is written in Java
this is my pipeline code:
public class DataMappingService {
DataflowPipelineOptions options;
Pipeline pipeline;
private String jdbcUrl;
private String DB_NAME;
private String DB_PRIVATE_IP;
private String DB_USERNAME;
private String DB_PASSWORD;
private String PROJECT_ID;
private String SERVICE_ACCOUNT;
private static final String SQL_INSERT = "INSERT INTO upfit(upfitter_id, model_number, upfit_name, upfit_description, manufacturer, length, width, height, dimension_unit, weight, weight_unit, color, price, stock_number) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?)";
public DataMappingService() {
DB_NAME = System.getenv("DB_NAME");
DB_PRIVATE_IP = System.getenv("DB_PRIVATEIP");
DB_USERNAME = System.getenv("DB_USERNAME");
DB_PASSWORD = System.getenv("DB_PASSWORD");
PROJECT_ID = System.getenv("GOOGLE_PROJECT_ID");
SERVICE_ACCOUNT = System.getenv("SERVICE_ACCOUNT_EMAIL");
jdbcUrl = "jdbc:postgresql://" + DB_PRIVATE_IP + ":5432/" + DB_NAME;
System.out.println("jdbcUrl: " + jdbcUrl);
System.out.println("dbUsername: " + DB_USERNAME);
System.out.println("dbPassword: " + DB_PASSWORD);
System.out.println("dbName: " + DB_NAME);
System.out.println("projectId: " + PROJECT_ID);
System.out.println("service account: " + SERVICE_ACCOUNT);
options = PipelineOptionsFactory.as(DataflowPipelineOptions.class);
options.setRunner(DataflowRunner.class);
options.setProject(PROJECT_ID);
options.setServiceAccount(SERVICE_ACCOUNT);
options.setWorkerRegion("us-east4");
options.setTempLocation("gs://upfit_data_flow_bucket/temp");
options.setStagingLocation("gs://upfit_data_flow_bucket/binaries");
options.setRegion("us-east4");
options.setSubnetwork("regions/us-east-4/subnetworks/us-east4-public");
options.setMaxNumWorkers(3);
}
public void processData(String gcsFilePath) {
try {
pipeline = Pipeline.create(options);
System.out.println("pipelineOptions: " +pipeline.getOptions());
pipeline.apply("ReadLines", TextIO.read().from(gcsFilePath))
.apply("SplitLines", new SplitLines())
.apply("SplitRecords", new SplitRecord())
.apply("ReadUpfits", new ReadUpfits());
.apply("write upfits", JdbcIO.<Upfit>write()
.withDataSourceConfiguration(JdbcIO.DataSourceConfiguration.create(
"org.postgresql.Driver", jdbcUrl)
.withUsername(DB_USERNAME)
.withPassword(DB_PASSWORD))
.withStatement(SQL_INSERT)
.withPreparedStatementSetter(new JdbcIO.PreparedStatementSetter<Upfit>() {
private static final long serialVersionUID = 1L;
#Override
public void setParameters(Upfit element, PreparedStatement query) throws SQLException {
query.setInt(1, element.getUpfitterId());
query.setString(2, element.getModelNumber());
query.setString(3, element.getUpfitName());
query.setString(4, element.getUpfitDescription());
query.setString(5, element.getManufacturer());
query.setString(6, element.getLength());
query.setString(7, element.getWidth());
query.setString(8, element.getHeight());
query.setString(9, element.getDimensionsUnit());
query.setString(10, element.getWeight());
query.setString(11, element.getWeightUnit());
query.setString(12, element.getColor());
query.setString(13, element.getPrice());
query.setInt(14, element.getStockAmmount());
}
}
));
pipeline.run();
} catch (Exception e) {
e.printStackTrace();
}
}
}
I have created a user managed service account as the documentation says: https://cloud.google.com/dataflow/docs/concepts/security-and-permissions#specifying_a_user-managed_worker_service_account and I'm providing the service account email in the pipeline options.
The service account has the following roles:
roles/dataflow.worker
roles/storage.admin
iam.serviceAccounts.actAs
roles/dataflow.admin
Service Account Token Creator
when I upload the csv file, the pipeline is triggered but I'm getting the following error:
Workflow failed. Causes: Subnetwork https://www.googleapis.com/compute/v1/projects/project-name/regions/us-east-4/subnetworks/us-east4-public is not accessible to Dataflow Service account or does not exist
I know that the subnetwork exists so I'm assuming its a permission error. The vpc network is created by my organization as we're not allowed to create our own.